I think that it's rather unproductive to try to draw such a sharp line between "science fiction" and "science fantasy". Isaac Asimov's Foundation, for example, is broadly recognized as science fiction, but it involves elements which are clearly not a part of our scientific reality (telepathy) and has no real basis in any science. Even the mathematical basis of its central conceit is more fantasy than science.
My definition of science fiction is any storytelling that is inspired by scientific or technological progress. As such, Star Wars and Foundation clearly have a right to claim membership in the genre as do more science-centric stories such as Diamond Age, Ringworld or The Moon Is a Harsh Mistress. Simply having fantastic elements does not cause me to reject a story as science fiction, and the line is infinitely broad and grey (e.g. with much of the "modern fantasy" subgenre also qualifying as science fiction).
There are terms that are less widely used but narrow the scope the way you suggest. For example, "hard science fiction" or "speculative fiction".