Nate Silver 2012, The Signal and the Noise:
# ch11
In 2009, a year after a financial crisis had wrecked the global economy, American investors traded $8 million in stocks every second that the New York Stock Exchange was open for business. Over the course of the typical trading day, the volume grew to $185 billion, roughly as much as the economies of Nigeria, the Philippines or Ireland produce in an entire year. Over the course of the whole of 2009, more than $46 trillion1 in stocks were traded: four times more than the revenues of all the companies in the Fortune 500 put together.2
This furious velocity of trading is something fairly new. In the 1950s, the average share of common stock in an American company was held for about six years before being traded-consistent with the idea that stocks are a long-term investment. By the 2000s, the velocity of trading had increased roughly twelvefold. Instead of being held for six years, the same share of stock was traded after just six months.3
If there really were a Bayesland, then Justin Wolfers, a fast-talking, ponytailed polymath who is among America's best young economists, would be its chief of police, writing a ticket anytime he observed someone refusing to bet on their forecasts. Wolfers challenged me to a dinner bet after I wrote on my blog that I thought Rick Santorum would win the Iowa caucus, bucking the prediction market Intrade (as well as my own predictive model), which still showed Mitt Romney ahead. In that case, I was willing to commit to the bet, which turned out well for me after Santorum won by literally just a few dozen votes after a weeks-long recount.* But there have been other times when I have been less willing to accept one of Wolfers' challenges. Presuming you are a betting man as I am, what good is a prediction if you aren't willing to put money on it?
Nevertheless, there is strong empirical and theoretical evidence that there is a benefit in aggregating different forecasts. Across a number of disciplines, from macroeconomic forecasting to political polling, simply taking an average of everyone's forecast rather than relying on just one has been found to reduce forecast error,14 often by about 15 or 20 percent.
But before you start averaging everything together, you should understand three things. First, while the aggregate forecast will essentially always be better than the typical individual's forecast, that doesn't necessarily mean it will be good. For instance, aggregate macroeconomic forecasts are much too crude to predict recessions more than a few months in advance. They are somewhat better than individual economists' forecasts, however.
Second, the most robust evidence indicates that this wisdom-of-crowds principle holds when forecasts are made independently before being averaged together. In a true betting market (including the stock market), people can and do react to one another's behavior. Under these conditions, where the crowd begins to behave more dynamically, group behavior becomes more complex.
Third, although the aggregate forecast is better than the typical individual's forecast, it does not necessarily hold that it is better than the best individual's forecast. Perhaps there is some polling firm, for instance, whose surveys are so accurate that it is better to use their polls and their polls alone rather than dilute them with numbers from their less-accurate peers.
When this property has been studied over the long run, however, the aggregate forecast has often beaten even the very best individual forecast. A study of the Blue Chip Economic Indicators survey, for instance, found that the aggregate forecast was better over a multiyear period than the forecasts issued by any one of the seventy economists that made up the panel.15 Another study by Wolfers, looking at predictions of NFL football games, found that the consensus forecasts produced by betting markets were better than about 99.5 percent of those from individual handicappers.16 And this is certainly true of political polling; models that treat any one poll as the Holy Grail are more prone to embarrassing failures.17 Reducing error by 15 or 20 percent by combining forecasts may not sound all that impressive, but it's awfully hard to beat in a competitive market.
Also, while I accept the theoretical benefits of prediction markets, I don't know that political betting markets like Intrade are all that good right now-the standard of competition is fairly low. Intrade is becoming more popular, but it is still small potatoes compared with the stock market or Las Vegas. In the weeks leading up to the Super Tuesday primaries in March 2012, for instance, about $1.6 million in shares were traded there;18 by contrast, $8 million is traded in the New York Stock Exchange in a single second. The biggest profit made by any one trader from his Super Tuesday bets was about $9,000, which is not enough to make a living, let alone to get rich. Meanwhile, Intrade is in a legal gray area and most of the people betting on American politics are from Europe or from other countries. There have also been some cases of market manipulation*19 or blatant irrational pricing20 there. And these markets haven't done very well at aggregating information in instances where there isn't much information worth aggregating, like in trying to guess the outcome of Supreme Court cases from the nebulous clues the justices provide to the public.
Could FiveThirtyEight and other good political forecasters beat Intrade if it were fully legal in the United States and its trading volumes were an order of magnitude or two higher? I'd think it would be difficult. Can they do so right now? My educated guess21 is that some of us still can, if we select our bets carefully.22
Efficient-market hypothesis is sometimes mistaken for an excuse for the excesses of Wall Street; whatever else those guys are doing, it seems to assert, at least they're behaving rationally. A few proponents of the efficient-market hypothesis might interpret it in that way. But as the theory was originally drafted, it really makes just the opposite case: the stock market is fundamentally and profoundly unpredictable. When something is truly unpredictable, nobody from your hairdresser to the investment banker making $2 million per year is able to beat it consistently.
Suppose that we looked at the daily closing price of the Dow Jones Industrial Average in the 10 years between 1966 and 1975-the decade just after Fama had published his thesis. Over this period, the Dow moved in the same direction from day to day-a gain was followed by a gain or a loss by a loss-58 percent of the time. It switched directions just 42 percent of the time. That seems nonrandom and it is: a standard statistical test38 would have claimed that there was only about a 1-in-7 quintillion possibility (1 chance in 7,000,000,000,000,000) that this resulted from chance alone.
But statistical significance does not always equate to practical significance. An investor could not have profited from this trend.
Suppose that an investor had observed this pattern for ten years-gains tended to be followed by gains and losses by losses. On the morning of January 2, 1976, he decided to invest $10,000 in an index fund39 which tracked the Dow Jones Industrial Average. But he wasn't going to be a passive investor. Instead he'd pursue what he called a Manic Momentum strategy to exploit the pattern. Every time the stock market declined over the day, he would pull all his money out, avoiding what he anticipated would be another decline the next day. He'd hold his money out of the market until he observed a day that the market rose, and then he would put it all back in. He would pursue this strategy for ten years, until the last trading day of 1985, at which point he would cash out his holdings for good, surely assured of massive profits.
How much money would this investor have at the end of the ten-year period? If you ignore dividends, inflation, and transaction costs, his $10,000 investment in 1976 would have been worth about $25,000 ten years later using the Manic Momentum strategy. By contrast, an investor who had adopted a simple buy-and-hold strategy during the same decade-buy $10,000 in stocks on January 2, 1976, and hold them for ten years, making no changes in the interim-would have only about $18,000 at the end of the period. Manic Momentum seems to have worked! Our investor, using a very basic strategy that exploited a simple statistical relationship in past market prices, substantially beat the market average, seeming to disprove the efficient-market hypothesis in the process.
But there is a catch. We ignored this investor's transaction costs. This makes an enormous difference. Suppose that the investor had pursued the Manic Momentum strategy as before but that each time he cashes into or out of the market, he paid his broker a commission of 0.25 percent. Since this investor's strategy requires buying or selling shares hundreds of times during this period, these small costs will nickel-and-dime him to death. If you account for his transaction costs, in fact, the $10,000 investment in the Manic Momentum strategy would have been worth only about $1,100 ten years later, eliminating not only his profit but also almost all the money he put in originally. In this case, there is just a little bit of predictability in stock-market returns-but not nearly enough to make a profit from them, and so efficient-market hypothesis is not violated.
The other catch is that the pattern has since reversed itself. During the 2000s, the stock market changed direction from day to day about 54 percent of the time, just the opposite of the pattern from earlier decades. Had the investor pursued his Manic Momentum strategy for ten years beginning in January 2000, his $10,000 investment would have been whittled down to $4,000 by the end of the decade even before considering transaction costs.40 If you do consider transaction costs, the investor would have had just $141 left over by the end of the decade, having lost almost 99 percent of his capital.
Some of the prices listed on the NADSAQ seemed to be plainly irrational. At one point during the dot-com boom, the market value of technology companies accounted for about 35 percent of the value of all stocks in the United States,41 implying they would soon come to represent more than a third of private-sector profits. What's interesting is that the technology itself has in some ways exceeded our expectations. Can you imagine what an investor in 2000 would have done if you had shown her an iPad? And told her that, within ten years, she could use it to browse the Internet on an airplane flying 35,000 feet over Missouri and make a Skype call* to her family in Hong Kong? She would have bid Apple stock up to infinity.
Nevertheless, ten years later, in 2010, technology companies accounted for only about 7 percent of economic activity.42 For every Apple, there were dozens of companies like Pets.com that went broke.
Identifying a bubble is of course much easier with the benefit of hindsight-but frankly, it does not seem all that challenging to do so in advance, as many economists did while the housing bubble was underway. Simply looking at periods when the stock market has increased at a rate much faster than its historical average can give you some inkling of a bubble. Of the eight times in which the S&P 500 increased in value by twice its long-term average over a five-year period,43 five cases were followed by a severe and notorious crash, such as the Great Depression, the dot-com bust, or the Black Monday crash of 1987.44
At various times, the P/E ratio for all companies in the S&P 500 ranged everywhere from about 5 (in 1921) to 44 (when Shiller published his book in 2000). Shiller found that these anomalies had predictable-seeming consequences for investors. When the P/E ratio is 10, meaning that stocks are cheap compared with earnings, they have historically produced a real return46 of about 9 percent per year, meaning that a $10,000 investment would be worth $22,000 ten years later. When the P/E ratio is 25, on the other hand, a $10,000 investment in the stock market has historically been worth just $12,000 ten years later. And when they are very high, above about 30-as they were in 1929 or 2000-the expected return has been negative.
However, these pricing patterns would not have been very easy to profit from unless you were very patient. They've become meaningful only in the long term, telling you almost nothing about what the market will be worth one month or one year later. Even looking several years in advance, they have only limited predictive power. Alan Greenspan first used the phrase "irrational exuberance" to describe technology stocks in December 1996,47 at which point the P/E ratio of the S&P 500 was 28-not far from the previous record of 33 in 1929 in advance of Black Tuesday and the Great Depression. The NASDAQ was more richly valued still. But the peak of the bubble was still more than three years away. An investor with perfect foresight, who had bought the NASDAQ on the day that Greenspan made his speech, could have nearly quadrupled his money if he sold out at exactly the right time. Instead, it's really only at time horizons ten or twenty years out that these P/E ratios have allowed investors to make reliable predictions.
But now consider what happens when the investor gets his bet wrong. This choice is much clearer.
• The trader buys but the market crashes. This is no fun: he's lost his firm a lot of money and there will be no big bonus and no new Lexus. But since he's stayed with the herd, most of his colleagues will have made the same mistake. Following the last three big crashes on Wall Street, employment at securities firms decreased by about 20 percent.63 That means there is an 80 percent chance the trader keeps his job and comes out okay; the Lexus can wait until the next bull market.
A common experiment in economics classrooms, usually employed when the professor needs some extra lunch money, is to hold an auction wherein students submit bids on the number of pennies in a jar.77 The student with the highest bid pays the professor and wins the pennies (or an equivalent amount in paper money if he doesn't like loose change). Almost invariably, the winning student will find that he has paid too much. Although some of the students' bids are too low and some are about right, it's the student who most overestimates the value of the coins in the jar who is obligated to pay for them; the worst forecaster takes the "prize." This is known as the "winner's curse."
There is reason to suspect that of the various cognitive biases that investors suffer from, overconfidence is the most pernicious. Perhaps the central finding of behavioral economics is that most of us are overconfident when we make predictions. The stock market is no exception; a Duke University survey of corporate CFOs,78 whom you might expect to be fairly sophisticated investors, found that they radically overestimated their ability to forecast the price of the S&P 500. They were constantly surprised by large movements in stock prices, despite the stock market's long history of behaving erratically over short time periods.
The economist Terrance Odean of the University of California at Berkeley constructed a model in which traders had this flaw and this flaw only: they were overconfident in estimating the value of their information. Otherwise, they were perfectly rational.79 What Odean found was that overconfidence alone was enough to upset an otherwise rational market. Markets with overconfident traders will produce extremely high trading volumes, increased volatility, strange correlations in stock prices from day to day, and below-average returns for active traders-all the things that we observe in the real world.
Say, for instance, that you had borrowed five hundred shares of the company InfoSpace on March 2, 1999, when they cost $27, promising to return them one year later. Borrowing these shares would have cost you about $13,400. One year later, however, InfoSpace was trading at $482 per share, meaning that you would be obligated to return about $240,000-almost twenty times the initial value of your investment. Although this bet would have turned out to be brilliant in the end-InfoSpace later traded for as little as $1.40 per share-you would have taken a bath and your ability to make future investments would be crippled. In fact, the losses from shorting a stock are theoretically unlimited.
In practice, the investor loaning you the shares can demand them back anytime she wants, as she assuredly will if she thinks you are a credit risk. But this also means she can quit anytime she's ahead, an enormous problem since overvalued stocks often become even more overvalued before reverting back to fairer prices. Moreover, since the investor loaning you the stocks knows that you may have to dig into your savings to pay her back, she will charge you a steep interest rate for the privilege. Bubbles can take months or years to deflate. As John Maynard Keynes said, "The market can stay irrational longer than you can stay solvent."
...Few holders of Palm stock were willing to loan their shares out, and they had come to expect quite a premium for doing so: an interest rate of well over 100 percent per year.82 This pattern was common during the dot-com bubble:83 shorting dot-com stocks was prohibitively expensive when it wasn't literally impossible.
In practice, most everyday investors do not do even that well. Gallup and other polling organizations periodically survey Americans94 on whether they think it is a good time to buy stocks. Historically, there has been a strong relationship between these numbers and stock market performance-but the relationship runs in the exact opposite direction of what sound investment strategy would dictate. Americans tend to think it's a good time to buy when P/E ratios are inflated and stocks are overpriced. The highest figure that Gallup ever recorded in their survey was in January 2000, when a record high of 67 percent of Americans thought it was a good time to invest. Just two months later, the NASDAQ and other stock indices began to crash. Conversely, only 26 percent of Americans thought it was a good time to buy stocks in February 1990-but the S&P 500 almost quadrupled in value over the next ten years (figure 11-10).
Daniel Kahneman likens the problem to the Müller-Lyer illusion, a famous optical illusion involving two sets of arrows (figure 11-11). The arrows are exactly the same length. But in one case, the ends of the arrows outward, seem to signify expansion and boundless potential. In the other case, they point inward, making them seem self-contained and limited. The first case is analogous to how investors see the stock market when returns have been increasing; the second case is how they see it after a crash.
FIGURE 11-11: MÜLLER-LYER ILLUSION
"There's no way that you can control yourself not to have that illusion," Kahneman told me. "You look at them, and one of the arrows is going to look longer than the other. But you can train yourself to recognize that this is a pattern that causes an illusion, and in that situation, I can't trust my impressions; I've got to use a ruler."
# ch11
In 2009, a year after a financial crisis had wrecked the global economy, American investors traded $8 million in stocks every second that the New York Stock Exchange was open for business. Over the course of the typical trading day, the volume grew to $185 billion, roughly as much as the economies of Nigeria, the Philippines or Ireland produce in an entire year. Over the course of the whole of 2009, more than $46 trillion1 in stocks were traded: four times more than the revenues of all the companies in the Fortune 500 put together.2
This furious velocity of trading is something fairly new. In the 1950s, the average share of common stock in an American company was held for about six years before being traded-consistent with the idea that stocks are a long-term investment. By the 2000s, the velocity of trading had increased roughly twelvefold. Instead of being held for six years, the same share of stock was traded after just six months.3
If there really were a Bayesland, then Justin Wolfers, a fast-talking, ponytailed polymath who is among America's best young economists, would be its chief of police, writing a ticket anytime he observed someone refusing to bet on their forecasts. Wolfers challenged me to a dinner bet after I wrote on my blog that I thought Rick Santorum would win the Iowa caucus, bucking the prediction market Intrade (as well as my own predictive model), which still showed Mitt Romney ahead. In that case, I was willing to commit to the bet, which turned out well for me after Santorum won by literally just a few dozen votes after a weeks-long recount.* But there have been other times when I have been less willing to accept one of Wolfers' challenges. Presuming you are a betting man as I am, what good is a prediction if you aren't willing to put money on it?
Nevertheless, there is strong empirical and theoretical evidence that there is a benefit in aggregating different forecasts. Across a number of disciplines, from macroeconomic forecasting to political polling, simply taking an average of everyone's forecast rather than relying on just one has been found to reduce forecast error,14 often by about 15 or 20 percent.
But before you start averaging everything together, you should understand three things. First, while the aggregate forecast will essentially always be better than the typical individual's forecast, that doesn't necessarily mean it will be good. For instance, aggregate macroeconomic forecasts are much too crude to predict recessions more than a few months in advance. They are somewhat better than individual economists' forecasts, however.
Second, the most robust evidence indicates that this wisdom-of-crowds principle holds when forecasts are made independently before being averaged together. In a true betting market (including the stock market), people can and do react to one another's behavior. Under these conditions, where the crowd begins to behave more dynamically, group behavior becomes more complex.
Third, although the aggregate forecast is better than the typical individual's forecast, it does not necessarily hold that it is better than the best individual's forecast. Perhaps there is some polling firm, for instance, whose surveys are so accurate that it is better to use their polls and their polls alone rather than dilute them with numbers from their less-accurate peers.
When this property has been studied over the long run, however, the aggregate forecast has often beaten even the very best individual forecast. A study of the Blue Chip Economic Indicators survey, for instance, found that the aggregate forecast was better over a multiyear period than the forecasts issued by any one of the seventy economists that made up the panel.15 Another study by Wolfers, looking at predictions of NFL football games, found that the consensus forecasts produced by betting markets were better than about 99.5 percent of those from individual handicappers.16 And this is certainly true of political polling; models that treat any one poll as the Holy Grail are more prone to embarrassing failures.17 Reducing error by 15 or 20 percent by combining forecasts may not sound all that impressive, but it's awfully hard to beat in a competitive market.
Also, while I accept the theoretical benefits of prediction markets, I don't know that political betting markets like Intrade are all that good right now-the standard of competition is fairly low. Intrade is becoming more popular, but it is still small potatoes compared with the stock market or Las Vegas. In the weeks leading up to the Super Tuesday primaries in March 2012, for instance, about $1.6 million in shares were traded there;18 by contrast, $8 million is traded in the New York Stock Exchange in a single second. The biggest profit made by any one trader from his Super Tuesday bets was about $9,000, which is not enough to make a living, let alone to get rich. Meanwhile, Intrade is in a legal gray area and most of the people betting on American politics are from Europe or from other countries. There have also been some cases of market manipulation*19 or blatant irrational pricing20 there. And these markets haven't done very well at aggregating information in instances where there isn't much information worth aggregating, like in trying to guess the outcome of Supreme Court cases from the nebulous clues the justices provide to the public.
Could FiveThirtyEight and other good political forecasters beat Intrade if it were fully legal in the United States and its trading volumes were an order of magnitude or two higher? I'd think it would be difficult. Can they do so right now? My educated guess21 is that some of us still can, if we select our bets carefully.22
Efficient-market hypothesis is sometimes mistaken for an excuse for the excesses of Wall Street; whatever else those guys are doing, it seems to assert, at least they're behaving rationally. A few proponents of the efficient-market hypothesis might interpret it in that way. But as the theory was originally drafted, it really makes just the opposite case: the stock market is fundamentally and profoundly unpredictable. When something is truly unpredictable, nobody from your hairdresser to the investment banker making $2 million per year is able to beat it consistently.
Suppose that we looked at the daily closing price of the Dow Jones Industrial Average in the 10 years between 1966 and 1975-the decade just after Fama had published his thesis. Over this period, the Dow moved in the same direction from day to day-a gain was followed by a gain or a loss by a loss-58 percent of the time. It switched directions just 42 percent of the time. That seems nonrandom and it is: a standard statistical test38 would have claimed that there was only about a 1-in-7 quintillion possibility (1 chance in 7,000,000,000,000,000) that this resulted from chance alone.
But statistical significance does not always equate to practical significance. An investor could not have profited from this trend.
Suppose that an investor had observed this pattern for ten years-gains tended to be followed by gains and losses by losses. On the morning of January 2, 1976, he decided to invest $10,000 in an index fund39 which tracked the Dow Jones Industrial Average. But he wasn't going to be a passive investor. Instead he'd pursue what he called a Manic Momentum strategy to exploit the pattern. Every time the stock market declined over the day, he would pull all his money out, avoiding what he anticipated would be another decline the next day. He'd hold his money out of the market until he observed a day that the market rose, and then he would put it all back in. He would pursue this strategy for ten years, until the last trading day of 1985, at which point he would cash out his holdings for good, surely assured of massive profits.
How much money would this investor have at the end of the ten-year period? If you ignore dividends, inflation, and transaction costs, his $10,000 investment in 1976 would have been worth about $25,000 ten years later using the Manic Momentum strategy. By contrast, an investor who had adopted a simple buy-and-hold strategy during the same decade-buy $10,000 in stocks on January 2, 1976, and hold them for ten years, making no changes in the interim-would have only about $18,000 at the end of the period. Manic Momentum seems to have worked! Our investor, using a very basic strategy that exploited a simple statistical relationship in past market prices, substantially beat the market average, seeming to disprove the efficient-market hypothesis in the process.
But there is a catch. We ignored this investor's transaction costs. This makes an enormous difference. Suppose that the investor had pursued the Manic Momentum strategy as before but that each time he cashes into or out of the market, he paid his broker a commission of 0.25 percent. Since this investor's strategy requires buying or selling shares hundreds of times during this period, these small costs will nickel-and-dime him to death. If you account for his transaction costs, in fact, the $10,000 investment in the Manic Momentum strategy would have been worth only about $1,100 ten years later, eliminating not only his profit but also almost all the money he put in originally. In this case, there is just a little bit of predictability in stock-market returns-but not nearly enough to make a profit from them, and so efficient-market hypothesis is not violated.
The other catch is that the pattern has since reversed itself. During the 2000s, the stock market changed direction from day to day about 54 percent of the time, just the opposite of the pattern from earlier decades. Had the investor pursued his Manic Momentum strategy for ten years beginning in January 2000, his $10,000 investment would have been whittled down to $4,000 by the end of the decade even before considering transaction costs.40 If you do consider transaction costs, the investor would have had just $141 left over by the end of the decade, having lost almost 99 percent of his capital.
Some of the prices listed on the NADSAQ seemed to be plainly irrational. At one point during the dot-com boom, the market value of technology companies accounted for about 35 percent of the value of all stocks in the United States,41 implying they would soon come to represent more than a third of private-sector profits. What's interesting is that the technology itself has in some ways exceeded our expectations. Can you imagine what an investor in 2000 would have done if you had shown her an iPad? And told her that, within ten years, she could use it to browse the Internet on an airplane flying 35,000 feet over Missouri and make a Skype call* to her family in Hong Kong? She would have bid Apple stock up to infinity.
Nevertheless, ten years later, in 2010, technology companies accounted for only about 7 percent of economic activity.42 For every Apple, there were dozens of companies like Pets.com that went broke.
Identifying a bubble is of course much easier with the benefit of hindsight-but frankly, it does not seem all that challenging to do so in advance, as many economists did while the housing bubble was underway. Simply looking at periods when the stock market has increased at a rate much faster than its historical average can give you some inkling of a bubble. Of the eight times in which the S&P 500 increased in value by twice its long-term average over a five-year period,43 five cases were followed by a severe and notorious crash, such as the Great Depression, the dot-com bust, or the Black Monday crash of 1987.44
At various times, the P/E ratio for all companies in the S&P 500 ranged everywhere from about 5 (in 1921) to 44 (when Shiller published his book in 2000). Shiller found that these anomalies had predictable-seeming consequences for investors. When the P/E ratio is 10, meaning that stocks are cheap compared with earnings, they have historically produced a real return46 of about 9 percent per year, meaning that a $10,000 investment would be worth $22,000 ten years later. When the P/E ratio is 25, on the other hand, a $10,000 investment in the stock market has historically been worth just $12,000 ten years later. And when they are very high, above about 30-as they were in 1929 or 2000-the expected return has been negative.
However, these pricing patterns would not have been very easy to profit from unless you were very patient. They've become meaningful only in the long term, telling you almost nothing about what the market will be worth one month or one year later. Even looking several years in advance, they have only limited predictive power. Alan Greenspan first used the phrase "irrational exuberance" to describe technology stocks in December 1996,47 at which point the P/E ratio of the S&P 500 was 28-not far from the previous record of 33 in 1929 in advance of Black Tuesday and the Great Depression. The NASDAQ was more richly valued still. But the peak of the bubble was still more than three years away. An investor with perfect foresight, who had bought the NASDAQ on the day that Greenspan made his speech, could have nearly quadrupled his money if he sold out at exactly the right time. Instead, it's really only at time horizons ten or twenty years out that these P/E ratios have allowed investors to make reliable predictions.
But now consider what happens when the investor gets his bet wrong. This choice is much clearer.
• The trader buys but the market crashes. This is no fun: he's lost his firm a lot of money and there will be no big bonus and no new Lexus. But since he's stayed with the herd, most of his colleagues will have made the same mistake. Following the last three big crashes on Wall Street, employment at securities firms decreased by about 20 percent.63 That means there is an 80 percent chance the trader keeps his job and comes out okay; the Lexus can wait until the next bull market.
A common experiment in economics classrooms, usually employed when the professor needs some extra lunch money, is to hold an auction wherein students submit bids on the number of pennies in a jar.77 The student with the highest bid pays the professor and wins the pennies (or an equivalent amount in paper money if he doesn't like loose change). Almost invariably, the winning student will find that he has paid too much. Although some of the students' bids are too low and some are about right, it's the student who most overestimates the value of the coins in the jar who is obligated to pay for them; the worst forecaster takes the "prize." This is known as the "winner's curse."
There is reason to suspect that of the various cognitive biases that investors suffer from, overconfidence is the most pernicious. Perhaps the central finding of behavioral economics is that most of us are overconfident when we make predictions. The stock market is no exception; a Duke University survey of corporate CFOs,78 whom you might expect to be fairly sophisticated investors, found that they radically overestimated their ability to forecast the price of the S&P 500. They were constantly surprised by large movements in stock prices, despite the stock market's long history of behaving erratically over short time periods.
The economist Terrance Odean of the University of California at Berkeley constructed a model in which traders had this flaw and this flaw only: they were overconfident in estimating the value of their information. Otherwise, they were perfectly rational.79 What Odean found was that overconfidence alone was enough to upset an otherwise rational market. Markets with overconfident traders will produce extremely high trading volumes, increased volatility, strange correlations in stock prices from day to day, and below-average returns for active traders-all the things that we observe in the real world.
Say, for instance, that you had borrowed five hundred shares of the company InfoSpace on March 2, 1999, when they cost $27, promising to return them one year later. Borrowing these shares would have cost you about $13,400. One year later, however, InfoSpace was trading at $482 per share, meaning that you would be obligated to return about $240,000-almost twenty times the initial value of your investment. Although this bet would have turned out to be brilliant in the end-InfoSpace later traded for as little as $1.40 per share-you would have taken a bath and your ability to make future investments would be crippled. In fact, the losses from shorting a stock are theoretically unlimited.
In practice, the investor loaning you the shares can demand them back anytime she wants, as she assuredly will if she thinks you are a credit risk. But this also means she can quit anytime she's ahead, an enormous problem since overvalued stocks often become even more overvalued before reverting back to fairer prices. Moreover, since the investor loaning you the stocks knows that you may have to dig into your savings to pay her back, she will charge you a steep interest rate for the privilege. Bubbles can take months or years to deflate. As John Maynard Keynes said, "The market can stay irrational longer than you can stay solvent."
...Few holders of Palm stock were willing to loan their shares out, and they had come to expect quite a premium for doing so: an interest rate of well over 100 percent per year.82 This pattern was common during the dot-com bubble:83 shorting dot-com stocks was prohibitively expensive when it wasn't literally impossible.
In practice, most everyday investors do not do even that well. Gallup and other polling organizations periodically survey Americans94 on whether they think it is a good time to buy stocks. Historically, there has been a strong relationship between these numbers and stock market performance-but the relationship runs in the exact opposite direction of what sound investment strategy would dictate. Americans tend to think it's a good time to buy when P/E ratios are inflated and stocks are overpriced. The highest figure that Gallup ever recorded in their survey was in January 2000, when a record high of 67 percent of Americans thought it was a good time to invest. Just two months later, the NASDAQ and other stock indices began to crash. Conversely, only 26 percent of Americans thought it was a good time to buy stocks in February 1990-but the S&P 500 almost quadrupled in value over the next ten years (figure 11-10).
Daniel Kahneman likens the problem to the Müller-Lyer illusion, a famous optical illusion involving two sets of arrows (figure 11-11). The arrows are exactly the same length. But in one case, the ends of the arrows outward, seem to signify expansion and boundless potential. In the other case, they point inward, making them seem self-contained and limited. The first case is analogous to how investors see the stock market when returns have been increasing; the second case is how they see it after a crash.
FIGURE 11-11: MÜLLER-LYER ILLUSION
"There's no way that you can control yourself not to have that illusion," Kahneman told me. "You look at them, and one of the arrows is going to look longer than the other. But you can train yourself to recognize that this is a pattern that causes an illusion, and in that situation, I can't trust my impressions; I've got to use a ruler."