What you're saying (quite accurately) is that freedom allows you to do what you will, so long as you harm no innocent person, and that you're responsible for your misuse or abuse of your freedom when its exercise causes harm.
I would add that yelling "fire" In a crowded theater cannot be illegal, because it is the right thing to do under certain circumstances. It's important to realize that any a priori restraint on the exercise of a right is "prior restraint," a restriction not permitted in our system of juris prudence. We don't, or at least shouldn't, prevent someone from performing an act because there are circumstances under which the act MAY be criminal. For instance, we generally frown on the intentional killing of human beings, but don't have a law prohibiting all such killing. We recognize that under certain circumstances, killing a human is appropriate. So we have laws that define the circumstances under which such a killing is considered lawful and the circumstances under which it is considered unlawful, and the law sets the punishment for the criminal killing of a human, but there is no law to prevent you from killing another human in the first place. The law merely reviews the actions of the killer and can hold the killer responsible for his act if it was not the exercise of a right (legitimate self-defense), that is to say, it was a crime. If it WAS the exercise of a right, by definition it didn't harm an innocent person and there was, therefore, no criminal act. (I understand that there are circumstances under which the killing of a malefactor is considered a crime. But just because causing intentional harm to a malefactor is sometimes considered a crime, this does not refute the conclusion that the exercise of a right intentionally harms no innocent person.) (BTW, I am aware that our government does prohibit certain innocent acts, such as the possession of certain types of firearms that makes them, for practical purposes, illegal to possess. This is a perversion of our system of laws, because it criminalizes intrinsically innocent acts - indeed, in some instances criminalizes the exercise of constitutionally-guaranteed rights - on the presumption that the innocent act is a prelude to the commission of a criminal act. As mentioned above, this is prior restraint - persons are restrained prior to actually harming another person, their property, or their interests.)
The exercise of no right intentionally harms any innocent person. If an act intentionally harms an innocent person, it is by definition a crime, not a right. This is another way of saying nobody has a right to commit crimes.
"It will be found an unjust and unwise jealousy to deprive a man of his natural liberty upon the supposition he may abuse it."