That it's based on the browser or OS setting is in fact the ENTIRE reason the WCAG says to use EM for font sizes...
I'm particularly sensitive to it as I come to places that use px for font sizes or play goofy tricks, and I end up forced to either dive for the zoom or go find some other site that isn't telling me to piss off.
But to be fair, I've been using the 20px base size OS-wide since Windows 3.0 called it "8514" since it was meant to be usable on the IBM 8514 display with it's whopping 800x600 resolution being shown at the same size as a 640x480 VGA display. Back in the day before we had font smoothing, upping the resolution one notch higher than what was "normal" for that size display and then telling Windows to just up-scale everything was a cheap and easy way to improve legibility and decrease the jaggies.
... and the browsers USED to inherit it by default, though you can now manually set it in most browsers, or force it with plugins in others. (Naturally webkit/blink browsers give a flying purple fish about accessibility so you have to go to extensions for that now)
Which is why I laugh when Apple starts bragging about their display scaling -- hey look they've added the bleeding edge of 1990's Windows technology!!!
It's kind of sad that with Windows 10 they've pretty well shtupped a good deal of that functionality in favor of trying to actually anisotropic filter a zoomed in version of programs. It's ugly, it's less useful, and much like the rest of the OS it's a giant middle finger to users with accessibility needs. Much like how the interface changes are a giant middle finger to notebook and desktop users!
As with anything else, you could do far worse than to follow the WCAG's recommendations... and that means fonts whenever possible in EM's, and layouts whenever possible in EM's.
They say use EM, so use 'em!