There are problems the market can't solve. Discrimination is one of them. Slavery wasn't a drag on the southern economy. Neither was segregation. Those institutions ended because of government intervention reflecting evolving societal attitudes; even though some still clung to not just economic, but also religious justifications for slavery.
The constitution prohibits the establishment of state religion and allows the free exercise of religion (or no religion, if you prefer). It also, and conservatives tend to dispute this point, creates a separation between church and state. In the final analysis, though, religious "law" isn't allowed to simply trump secular law. We aren't a theocracy. Mormons aren't allowed to engage in polygamy. So-called "honor killings" are illegal. Etc.