Wondering what the “vim-pager” use flag on app-editors/vim did, I gave it a shot. Turns out it gives you 2 extra commands: vimpager and vimmanpager. So adding
export MANPAGER=/usr/bin/vimmanpager
to your login script of choice (.bashrc works just fine), man will use vim as a pager, complete with systax highlighting and ability to use the [...]