For me, it's about knowing the mnemonic for the various vim commands and what the keys do, not where they are placed on the Qwerty layout. I've been touch-typing Dvorak for over 20 years and think about what I am trying to do in vim/vi (or evil-mode) rather than where on the keyboard I need to hit a key.

Like others said, you can adapt to using most keybinds on colemak. I had a tough adapting to positional keybinds such as the home arrows. It just was in very weird places on colemak.

Eventually I started using Extend layer from BigBagTrix which provides many useful keys near home. It uses caps as a modifier but you could possibly add dual behavior for tap and hold to have backspace function if you want.

Now I am used to a mix of extend and vim bindings.

I’ve been using vi/vim with Dvorak for over 30 years. Why would you have to re-bind keys?