AlphaGo lost today after the human player made an unexpected move, throwing off AlphaGo expectations. This reminded me of our  "Beat the Machine" system, where we incentivized humans to find examples that the machine misses, and makes incorrect decisions with confidence. Typically these examples were cases that were  unlike any examples in the (existing) training data. These are the "unknown unknowns" for the machine learning systems, and humans can help discover such cases quickly.