[Another repost from my former Google Buzz feed, which I first posted on Aug 16, 2010.]
Formally, a mathematical proof consists of a sequence of mathematical statements and deductions (e.g. "If A, then B"), strung together in a logical fashion to create a conclusion. A simple example of this is a linear chain of deductions, such as "A -> B -> C -> D -> E", to create the conclusion "A -> E". In practice, though, proofs tend to be more complicated than a linear chain, often acquiring a tree-like structure (or more generally, the structure of a directed acyclic graph), due to the need to branch into cases, or to reuse a hypothesis multiple times. Proof methods such as proof by contradiction, or proof by induction, can lead to even more intricate loops and reversals in a mathematical argument.
Unfortunately, not all proposed proofs of a statement in mathematics are actually correct, and so some effort needs to be put into verification of such a proposed proof. Broadly speaking, there are two ways that one can show that a proof can fail. Firstly, one can find a "local", "low-level" or "direct" objection to the proof, by showing that one of the steps (or perhaps a cluster of steps, see below) in the proof is invalid. For instance, if the implication C -> D is false, then the above proposed proof "A -> B -> C -> D -> E" of "A -> E" is invalid (though it is of course still conceivable that A -> E could be proven by some other route).
Sometimes, a low-level error cannot be localised to a single step, but rather to a cluster of steps. For instance, if one has a circular argument, in which a statement A is claimed using B as justification, and B is then claimed using A as justification, then it is possible for both implications A -> B and B -> A to be true, while the deduction that A and B are then both true remains invalid. (Note though that there are important and valid examples of near-circular arguments, such as proofs by induction, but this is not the topic of my discussion today.)
Another example of a low-level error that is not localisable to a single step arises from ambiguity. Suppose that one is claiming that A->B and B->C, and thus that A->C. If all terms are unambiguously well-defined, this is a valid deduction. But suppose that the expression B is ambiguous, and actually has at least two distinct interpretations, say B1 and B2. Suppose further that the A->B implication presumes the former interpretation B=B1, while the B->C implication presumes the latter interpretation B=B2, thus we actually have A->B1 and B2->C. In such a case we can no longer validly deduce that A->C (unless of course we can show in addition that B1->B2). In such a case, one cannot localise the error to either "A->B" or "B->C" until B is defined more unambiguously. This simple example illustrates the importance of getting key terms defined precisely in a mathematical argument.
The other way to find an error in a proof is to obtain a "high level" or "global" objection, showing that the proof, if valid, would necessarily imply a further consequence that is either known or strongly suspected to be false. The most well-known (and strongest) example of this is the counterexample. If one possesses a counterexample to the claim A->E, then one instantly knows that the chain of deduction "A->B->C->D->E" must be invalid, even if one cannot immediately pinpoint where the precise error is at the local level. Thus we see that global errors can be viewed as "non-constructive" guarantees that a local error must exist somewhere.
A bit more subtly, one can argue using the structure of the proof itself. If a claim such as A->E could be proven by a chain A->B->C->D->E, then this might mean that a parallel claim A'->E' could then also be proven by a parallel chain A'->B'->C'->D'->E' of logical reasoning. But if one also possesses a counterexample to A'->E', then this implies that there is a flaw somewhere in this parallel chain, and hence (presumably) also in the original chain. Other examples of this type include proofs of some conclusion that mysteriously never use in any essential way a crucial hypothesis (e.g. proofs of the non-existence of non-trivial integer solutions to a^n+b^n=c^n that mysteriously never use the hypothesis that n is strictly greater than 2, or which could be trivially adapted to cover the n=2 case).
While global errors are less constructive than local errors, and thus less satisfying as a "smoking gun", they tend to be significantly more robust. A local error can often be patched or worked around, especially if the proof is designed in a fault-tolerant fashion (e.g. if the proof proceeds by factoring a difficult problem into several strictly easier pieces, which are in turn factored into even simpler pieces, and so forth). But a global error tends to invalidate not only the proposed proof as it stands, but also all reasonable perturbations of that proof. For instance, a counterexample to A->E will automatically defeat any attempts to patch the invalid argument A->B->C->D->E, whereas the more local objection that C does not imply D could conceivably be worked around.
(There is a mathematical joke in which a mathematician is giving a lecture expounding on a recent difficult result that he has just claimed to prove. At the end of the lecture, another mathematician stands up and asserts that she has found a counterexample to the claimed result. The speaker then rebuts, "This does not matter; I have two proofs of this result!". Here one sees quite clearly the distinction of impact between a global error and a local one.)
It is also a lot quicker to find a global error than a local error, at least if the paper adheres to established standards of mathematical writing.
To find a local error in an N-page paper, one basically has to read a significant fraction of that paper line-by-line, whereas to find a global error it is often sufficient to skim the paper to extract the large--scale structure. This can sometimes lead to an awkward stage in the verification process when a global error has been found, but the local error predicted by the global error has not yet been located. Nevertheless, global errors are often the most serious errors of all.
It is generally good practice to try to structure a proof to be fault tolerant with respect to local errors, so that if, say, a key step in the proof of Lemma 17 fails, then the paper does not collapse completely, but contains at least some salvageable results of independent interest, or shows a reduction of the main problem to a simpler one. Global errors, by contrast, cannot really be defended against by a good choice of proof structure; instead, they require a good choice of proof strategy that anticipates global pitfalls and confronts them directly.
One last closing remark: as error-testing is the complementary exercise to proof-building, it is not surprising that the standards of rigour for the two activities are dual to each other. When one is building a proof, one is expected to adhere to the highest standards of rigour that are practical, since a single error could well collapse the entire effort. But when one is testing an argument for errors or other objections, then it is perfectly acceptable to use heuristics, hand-waving, intuition, or other non-rigorous means to locate and describe errors. This may mean that some objections to proofs are not watertight, but instead indicate that either the proof is invalid, or some accepted piece of mathematical intuition is in fact inaccurate. In some cases, it is the latter possibility that is the truth, in which case the result is deemed "paradoxical", yet true. Such objections, even if they do not invalidate the paper, are often very important for improving one's intuition about the subject.