Here's how I understand it as a GM:
The difficulty of the role should be measured by how difficult the task at hand is in a perfect world with no distractions. When you start introducing distractions (weather, a firefight, a time-crunch, one too many drinks, etc.) is when you start introducing setbacks to the roll, because now its no longer a perfect world for the character to complete the task in.
One other point you should think about, which hasn't been mentioned so far, is that by upgrading dice from Difficulty to Challenge, you're creating the possibility that your PCs roll a Despair (Despair only show on Challenge dice).