So, since +Greg Kroah-Hartman
has hinted with that picture there's going to be an influx of people with new +Yubico
Yubikeys that will be doing PGP-for-SSH, here's the bit of magic that I hadn't exactly found via googling. When you unplug your key, and then re-plug it, gpg-agent (scdaemon really, as I understand it) won't handle the event gracefully until you do a gpg2 --card-status. So, whack the udev rule you'll be adding anyhow (or may have from your distro) to do gpg2 --card-status after in the RUN line.
Or I've gone off the rails and someone can point out what I need to do instead to have things work cleanly with a current Yubikey NEO (I've been told an older key+debian old-stable doesn't have this problem).