Thanks! Changing the ownership on these devices has cleared up the problem. All of the googling I did for the problem talked about doing work in the guest (such as making sure 'vboxvideo' was in /etc/modules) which I did. However, didnt seem to matter what I did in the guest.
Appreciate the pointer, was not even aware vboxdrv was a device