A student writing an article for an on-campus magazine about whether higher math (ie, algebra on) should be required or optional in high school contacted me for my thoughts. I thought my off-the-cuff response was worth sharing:
I agree that making math optional would be harmful, for the reasons you described. Without learning high school math, many future paths are blocked off, so if math is optional, the stereotypes will end up getting reinforced. There's may in the the computer world who believe that this is part of why programing and software engineering has stayed so overwhelming male: computer programming is treated as an elective, and only the nerdy/geeky take it. Most students -- particularly girls -- never have the opportunity to see what it's like, and if they would actually enjoy it. (This is tangential at best to your original question, but there is a movement to incorporate computer programing and algorithmic though into the standard math curriculum, which I absolutely support.)
Back to you question: I think that, at its best, algebra teaches important patterns of thought. Let's think about high school English class for a moment. You'll probably never analyze a piece of literature after college, but learning to write an essay involves learning to organize your evidence and use it to and make a coherent argument -- valuable ways of approaching the world in situations far outside the English classroom. Similarly, algebra should be teaching people to learn to recognize and describe patterns, identify and isolate an unknown quantity, and model a real world situation with symbols and graphs are valuable skills, and are ideally applicable far outside the confines of a math or science classroom. Unfortunately, at its worst, algebra becomes a series of tricks to memorize and perform, taught with little attention to concept or understanding. In that form, math serves as little more than a gatekeeper, baring the door to college and further advancement to people who failed to make it through a series of arbitrary hoops. In other words, it's not math, it's the way it can be taught that is the problem.
I'd love to see a deep and broad re-think of the order and content of the high school math curriculum. The basic structure of college-prep math was set long in a world where calculations were done with pencil and paper, and laborious use of tables for things like logs and square roots. I don't think this makes sense any more. Here's one example:
Every time someone writes about how useless high school math is, factoring or the quadratic equation come up as examples. Why do we make polynomials such an important part of algebra? It's actually really hard to motivate a reason to care about quadratics, cubics, etc. (Well, quadratics do come up naturally in physics, but only once you're using calculus modes of thought -- a 12th grade topic, not a 9th grade one.) On the other hand, exponential functions and logarithms (think interest, or population growth) are actually really easy to make relevant and meaningful. But they're shunted off to 10th and 11th grade, and taught as a series of arcane steps. Why? I believe it's because polynomials are far more accessible to pencil and paper calculations. But we're not no longer living in a world where ease of calculating should be driving the curriculum.
Let's not throw away math; let's refocus on making sure that we're teaching math in a way that's appropriate and connected to the 21st century world we live in. Give students the skills that they would need to go forward in any field, and don't cut off pathways in high school, but also make sure that we're teaching relevant and meaningful content.