Well, things work differently in the US but from an England/Wales perspective there's a small (but only small) point of sense there.
The concern with former clients is that if I have acted for A and, in so doing, gained access to lots of privileged information, then if I later act for B suing A, I might potentially make use of that secret information, which I have a duty to keep confidential on A's behalf as a former client. Potential conflict.
There ought to be ways of avoiding that conflict in a firm of any decent size - but small outfits like me can't put a "Chinese wall" up in our heads. I've no idea to what extent that principle is applied in the US.
In fact its possible - but not advised in the usual run of things - to represent opposing clients in different cases where there's no conflict between those cases. I realised - but only after the fact - that I was once double-booked to appear for and against someone in two separate hearings in the same court (they held one hearing so I could do both). But that is rather unusual and probably not a good idea in general (in that case the "someone" was a large public body that didn't care, I was making legal argument in both cases, neither overlapped, and the decision in one case had no relationship with the one in the other - still I won't do it again).
I'm rather more concerned about the idea that former clients who might be called as witnesses might be an issue. There's no property in a witness and it seems to me the possible conflicts may still be there but are much less sharp. It ought to be possible to deal with them without drastically reducing a firm's possibility of representation.
NB: It used to be SOP in London to instruct all the QC's in a particular area of practice on retainer so that one's opponents couldn't instruct them. That practice is now frowned upon and doesn't (as far as I know) happen.