The reason is fairly simple: if a market were truly "free" (with all the subtleties that that phrase entails), people would be doing what they're best at, in the way that rewards them the most. Discrimination is, by its nature, something that keeps people from doing what they want to. And general principles of economics tell us that, when resources are allocated inefficiently, the economy as a whole slows down.
More concretely, if there's someone who would have been a great doctor, but is instead forced by circumstance -- be it being the wrong race or growing up poor -- to be a janitor instead, then not only does she lose out on everything she would have gained as a doctor, but everyone else loses out on everything she would have done as a doctor.
This brings us to this rather interesting paper from four researchers at the National Bureau of Economic Research, who wanted to put some numbers to this. (http://klenow.com/HHJK.pdf) For example, in 1960, 94% of doctors and lawyers were white men; in 2008, 62% were. Since there's no reason to believe that white men are intrinsically better at being doctors than anyone else, that gives us a way to estimate how many people would have been doctors who weren't. By building a mathematical model out of this, they estimate the real economic impact of discrimination -- and it turns out that somewhere between 15-20% of our total economic growth during that period comes from that.
For a sense of scale, during this time the inflation-adjusted GDP grew from $3.95T (in 2008 dollars) to $14.7T. That means that at the lower end of the estimate, the discrimination that went away between 1960 and 2008 was costing the US about $33 billion (in 2008 dollars, again) per year.
Note that this is just the aggregate cost to society as a whole, summed up between rich and poor. Obviously some people gained from this as well -- e.g., the people who became doctors who wouldn't have been able to, had the full pool of people who could have been doctors been allowed to participate. (And there's your next unsettling thought for the day: if you take the people who want to be doctors and line them up in order of how good a doctor they would be, and cut it off after you have enough doctors, you've got the best possible pool of doctors. If you take any one of them out of eligibility for some reason, then his replacement is mathematically guaranteed to be a worse doctor. You have just promoted some undeserving schmuck to perform surgery on you. Congratulations.)
But leaving aside the question of how different people fared under this, consider as well: this difference accounts for all of the discrimination that went away between 1960 and 2008.
We have not, by any stretch of the imagination, gotten rid of all the discrimination. The girl growing up in the Appalachian back country, the boy growing up in Baltimore, the child of migrant farm workers, these people are not likely to be able to go to college, get a BA or MD, work in the job of their choice.
When we talk about the economic costs of inequality, this is the sort of thing that really matters: not just the costs to those at the bottom, but the fact that inequality of opportunity has huge costs for society as a whole. Since in our society in particular, opportunity is greatly tied to existing resources -- consider anything from access to out-of-school enrichment, to having a good suit to wear to an interview, to knowing how to interview for a job in the first place (you learned that; it wasn't innate. You learned it from other people, and access to those people is a resource) -- resource inequality leads in turn to opportunity inequality, and that drags everyone down, even as it enriches the incompetent few.
Macroeconomics says: Trade makes everyone wealthier. You can't impoverish some people without that screwing everyone else over, as well. Trying to flout those laws tends to work about as well as trying to flout gravity: it might work really well, briefly. There's just that sudden stop at the end.
(Illustration via Paul Townsend: https://flic.kr/p/dVva6h. What goes up tends to come down somewhat rapidly, at times.)