Wondering what the "vim-pager" use flag on app-editors/vim did, I gave it a shot. Turns out it gives you 2 extra commands: vimpager and vimmanpager. So adding
to your login script of choice (.bashrc works just fine), man will use vim as a pager, complete with systax highlighting and ability to use the mouse scroll wheel and other such fun things to navigate man pages.
You could also set PAGER (and/or alias less) to vimpager, but I found that vimpager waits for EOF before displaying anything, meaning it won't work for commands that run until interupted, and commands that run for a long time wait until completion to show anything at all.
UPDATE to ask: If anyone knows a way to use this but to use the colors from a program with color output, eg. "ls --color=always | vimpager", please let me know.