Normally I'd agree with you that I don't care who a politician is having sex with and it has no bearing on their job. However, things are SLIGHTLY different when you have a Top Secret security clearance and are Director of the CIA.
The Code of Federal Regulations (Title 32, Chapter 1, Part 147) to which the CIA Director is subject, makes an affair a security concern. A person may lose a security clearance for “personal conduct or concealment of information that may increase an individual’s vulnerability to coercion, exploitation, or duress, such as engaging in activities which, if known, may affect the person’s personal, professional, or community standing or render the person susceptible to blackmail.”
In 1995, Clinton expanded that with Executive Order 12968. It stated that individuals eligible for access to classified material must have a record of “strength of character, truthfulness, honesty, reliability, discretion and sound judgment, as well as freedom from conflicting allegiances and potential for coercion.”
Basically, being caught having an affair means you're not a very good spy.