Starts in insert mode, and that seems to be the default. Press escape, though, and suddenly the bash prompt responds to vi keys. hjkl, dw, 0, w... it all works. It is such a relief after using the arrow keys to delete that one package name that causes a huge apt command to break.
If you are using Bash 4.3, you can add the following to ~/.inputrc as well:
This will draw a + at the very start of the PS1 when in insert mode, and a : when in command mode.