You're exactly right in that "visibility: hidden" hides the element, but still reserves the space on the page while "display: none" renders the page as if the element never existed. That is really the main difference.
I find that, unless for very specific occasions, "display:none" to be preferable. Take the example in this blog post in which the "suborders" don't show unless the main order is selected. If you were to use "display:hidden", the table rows would still take up space and while you might remove some "clutter," you wouldn't save any space on your page.
Also (and this is a minor thing most of the time), since "visibility: hidden" reserves the space for the element(s), it has to essentially render the element to determine how much space to reserve. In doing so, the page takes a minor performance hit vs. "display: none"