Musings on Assessing "Irrational Exuberance"

Back in 1996 Yale economist Robert Shiller wrote:

Price Earnings Ratios as Forecasters of Returns: The theory that the stock market is approximately a random walk does not look right at all: Figure 1... show[s]... the ratio of the real Standard and Poor Index ten years later to the real index today (on the y axis) versus... the ratio of the real Standard and Poor Composite Index for the first year of the ten year interval, divided by a lagged thirty year moving average of real earnings.... If real stock prices were a random walk, they should be unforecastable, and there should really be no relation here between y and x. There certainly appears to be a distinct negative relation here. The January 1996 value for the ratio shown on the horizontal axis is 29.72, shown on the figure with a vertical line. Looking at the diagram, it is hard to come away without a feeling that the market is quite likely to decline substantially in value over the succeeding ten years; it appears that long run investors should stay out of the market for the next decade...

In 1996 Yale economist Robert Shiller looked around, considered the historical record on the performance of the stock market, and concluded that the American stock market was overvalued. Prices on the broad index of the S&P 500 stood at 29 times the average of the past three decades' earnings. In the past, whenever price-earnings ratios had been high future long-run stock returns had turned out to be low. On the basis of econometric regression studies carried out by him and by Harvard's John Campbell, Shiller predicted in 1996 that the S&P 500 would be a bad investment over the next decade. In the decade up to January 2006, he predicted, the real value of the S&P 500 would fall, and even including dividends his estimate of the likely real inflation-adjusted returns to be earned by investors holding the S&P 500 was zero--a far cry below the 6% per year or so real return that we have come to think typical of the American stock market.

Robert Shiller's arguments were convincing. They convinced Alan Greenspan enough so that in December of 1996 he gave his "irrational exuberance" speech to the American Enterprise Institute. They certainly convinced me.

But Robert Shiller's arguments were wrong--at least, wrong ex post. Unless the American stock market collapses before the end of January, the past decade has seen the stock market offer returns a little bit higher than the historical averages--much, much greater than zero. Those who invested and reinvested their money in America's stock market over the past decade have nearly doubled it, even after taking account of inflation.

Why was Shiller wrong? In an arithmetic sense, we can point to three factors, each of which can take roughly one-third the credit for real American stock returns of 6% per year over the past decade rather than zero:

• 2% per year because the acceleration of productivity growth produced by the high-tech revolutions behind the very real "new economy" has made American companies much more productive.
• 2% per year because of shifts in the distribution of income away from labor and toward capital that have boosted corporate profits as a share of production.
• 2% per year because the argument of Glasman and Hassett in Dow 36000 turned out to be only nineteen-twentieths wrong: they argued that increasing risk tolerance on the part of stock market investors would raise long-run price-earnings ratios by 400%; it actually appears that increasing risk tolerance has raised long-run price-earnings ratios by 20% or so.

None of these three factors were obvious as of 1996 (although there were signs of the first and inklings of the third for those smart or lucky enough to read them). As of 1996, betting on Shiller's regression studies was a reasonable thing to do, perhaps an intelligent thing to do--but it was also an overhelmingly risky thing to do, as anybody who followed the portfolio strategy implicit in Shiller's analysis now painfully feels in his wallet or her purse.

Economists muse about just why it is that stock markets around the world are subject to fits of "irrational exuberance" and "excessive pessimism." Why don't rational and informed investors take more steps to bet heavily on fundamentals and against the enthusiasms of the uninformed crowd? The past decade gives us two reasons. First--if we grant that Shiller's regression analyses had correctly identified long-run fundamentals a decade ago--betting on fundamentals for the long term is overwhelmingly risky: lots of good news can happen over a decade, enough to bankrupt an even slightly leveraged bear when stocks look high; and lots of bad news can happen over a decade enough to bankrupt an even slightly leveraged bull when stocks look low. Thus even in extreme situations--like the peak of the dot-com bubble in late 1999 and early 2000--it is very difficult for even those who believe they know what fundamentals are to make large long-run bets on them. And it is even more difficult for those who claim they know what long-run fundamental values are and want to make large long-run contrarian bets to convince others to trust them with their money. As J.P. Morgan said when asked to predict what stocks would do: "They will fluctuate."

Perhaps this is how it should be: if it were easy to pierce the veils of time and ignorance and to assess long-run fundamental values with a high degree of confidence, it would be easy and safe to make large contrarian long-run bets on fundamentals. In this case the smart money would smooth out the enthusiasms--positive and negative--of the overenthusiastic crowd. And stocks would fluctuate less. And there wouldn't be teasing evidence at the edge of statistical significance of large-scale deviations of stock market prices from fundamental values.

Comments

[Cal]

Isn't that how Soros made all his money. Big contrarian bets.

[dsquared]

what has happened to the 1996 S&P500? This would certainly have included Enron and probably quite a few other companies which aren't in the SP500 any more.

[Brad DeLong]

IIRC, Enron wasn't dropped from the S&P until after its stock-market collapse was nearly complete. The "return on the S&P" does require that you rebalance your portfolio when the S&P changes, yes...

[Robert Waldmann]

Very diplomatic of you not to say to Shiller "I told you so" but you did and it's on your web page.

[Howard]

3 relevant quotes.

old wall street saw: "i can tell you what's going to happen, but not when, or when something's going to happen, but not what."

Ben Graham: "in the short term, the market is a voting machine; in the long term, the market is a weighing machine."

Lord Keynes: "in the long run, we're all dead."

me? i still think time/warner is undervalued, no matter what the market seems to have concluded....

[anne]

Nicely analyzed, and I rather agree that in the long run stocks can sell at valuations that are higher than historical norms before 1990. Still, there is the matter of time. The S&P has averaged gains of 9.2% a year after Vanguard costs for the last 10 years through November 30. However the 5 year gain is only 0.52% and the gain from January 2000 is about zero.

The S&P has not gained for 6 years now, while there has been one of the greatest bull markets in long term bonds. The difference between the gain in the Vanguard long term bond index and the S&P since January 2000 seems to be the largest since the Depression.

[Brad DeLong]

Re: 'Very diplomatic of you not to say to Shiller "I told you so" but you did and it's on your web page.'

I don't think Barsky and DeLong is "I told you so"... I wasn't that smart... Is it?

[anne]

http://flagship2.vanguard.com/VGApp/hnw/FundsByName

The Vanguard long term investment-grade bond fund is up 8.7% a year for the last 5 years, while the S&P index is up 0.52%. The price earning ratio for the S&P index is 17.3, return on equity is 18.5%, earnings growth rate over the last 3 years is 12.9%.

[anne]

The price earning ratio for the Vanguard S&P went from about 8 in 1979 to 15 in 1989 to 30 in 1999. The historical level was about 15, so it was reasonable to assume the 1990s would be less robust in stock returns than the 1980s. This did not turn out to be so. This decade however the return for the S&P has been about zero.

[anne]

The lesson that might have been drawn during the dramatic climb in the price earning ratio of the S&P during the 1990s, was value value value. Of course, information technology growth might well have been hedged if pockets were deep and patience enough to get to this decade :) But value was the key to really fine gains for the last 6 or 10 years, while relative value was the key to gains for the last 16 or 26 years.

[Aaron]

Brad,

If I told you, "Don't buy lottery tickets; the expected payoff is less than the price," but you bought one anyway, and you won, did I give you bad advice? There can be negative surprises as well as positive ones. Americans -- who went from April 1865 through August 2001 without a war being fought on their home soil -- tend to forget that.

[Cranky Observer]

Some observations:

1) From 1950-2000 the western world's economy generated an enormous surplus of wealth; probably greater than the sum total of all human wealth prior to 1950
2) Until 1970 there was some distribution of this wealth that was not uniform but didn't have a huge bias either (translation: the wealth was shared reasonably fairly with some differential rewards to ability/effort; worker bees made 40,000, CEOs made 400,000).
3) Starting in 1970, and accelerating rapidly after 1990, the distribution changed to greatly concentrate the wealth at the high end
4) However, 401(k)s were also introduced around that same time.

So what exactly did you expect to happen? Stocks to go down? Where would the now ultra-wealthy put all that wealth that they can't eat? Where do the worker bees put their 401(k) deferred wealth in the desperate hope of getting some return?

In both cases stocks are the only answer. Meaning they can only go up. Until of course they go down - just a bit.

Cranky

[Dave Iverson]

Here is a link to Russ Randall's 19 Reasons to expect a 30% drop in stock prices in 2006 (before the traditional year-end rally). Even though I find his "19" to be interesting, I disagree with Russ's optimistic forecast for a 2007 rally in the wake of the 2006 downturn. But he might be right. Who knows? Certainly I don't claim to know.
http://austrianenginomics.com/id9.html

And here is a link to my list of 24 Reasons not to put all your eggs in one basket based just on productivity gains and other supply-side indicators; titled "Economic Nightmares: Potential Triggers" http://forestpolicy.typepad.com/economics/2005/06/economic_nightm.html

Although I agree with Brad that we ought not to be too quick to think we can predict the future, I am hesitant to think that 2006 is going to be a gleefully "up year" even though productivity gains seem very real. Brad didn't say that 2006 would be an "up year," by the by, but I do think this post can be read as "optimistic" about future prospects.

My guess is that the secular bull is over (or if not then it is likely set up to be kited upward one more time before it runs out of steam—-with a worse reckoning if that be the case). But I probably read too much from Doug Noland and Stephen Roach who have both been quite bearish for some time.

Remember that secular bear markets can be ugly for optimistic investors for a very long time. So beware of unbridled optimism, and beware of those who put all their eggs in the "productivity gains" basket, or baskets filled only with "supply-side" indicators like Brad's threesome: "acceleration of productivity growth … behind the 'very real' new economy," "shifts of income away from labor and toward capital," and "increased risk tolerance on the part of stock market investors." Those three things may well explain why Shiller was wrong, but may not help us discern what comes next.

[anne]

There is a problem that should be addressed. An investor, any investor, who routinely bought shares in the Vanguard S&P index from 1980 or 1990 has reason to be content. But, this is quite deceptive for relatively investors bought the index. As David Swensen and John Bogle have pointed out the actual returns to mutual fund investors have been strikingly below the S&P returns. Generally high costs and portfolio turnover and weak management timing have been a decided drain on returns.

[anne]

There is an interesting puzzle now with commercial real estate. The price earning ration for the Vanguard REIT index is 40.5, while the return on equity is 8.4% and the earnings growth rate is negative -4.8 over the last 3 years. The earnings growth rate for the REIT index has been negative for almost 5 years. Now I know about alternate valuations for real estate investment trusts, but we surely have not seen such index valuations in the last 30 years. What this means however, I do not know.

[dragan]

Ah, but I wonder who will be "I-told-you-so-ing"
in, say, March of 2010. Will the SPX returns have averaged 6%+ in this decade as well? Should I hold
my breath for SPX 3000 in 2010?

[Christopher Ball]

On 31 Dec. 1996, the S&P 500 was at 740.737
On 9 Oct. 2002, it was 776.763.

What we really need to explain is why it rose back from that Oct. 2002 low. Just over three years ago, Schiller looked pretty smart.

There is a lot of volatility over the past decade that neither Schiller's regression nor Delong's tripartite theory explains.

[Barkley Rosser]

There is another reason why the risk premium may have declined long term for stocks since 1990 or thereabouts. It is the collapse of world socialism. The threat of nationalization in many countries around the world severely declined. Indeed, what we have been seeing has been widespread privatizations of state-owned enterprises, but almost no nationalizations.
BTW, I threw this argument at Shiller himself last spring, and he accepted it.

OTOH, with some of the political changes in Latin America, the risk of that may be reappearing...

[anne]

Robert Shiller, by the way, considers the S&P significantly overvalued at present. He uses a 10 year moving average for the price earning ratio, and includes corporations with negative earnings, where the Vanguard p/e ratio is a 1 year ratio that does not include losses. The p/e for Shiller then is about 25.

[anne]

There is an interesting international bull stock market this year, a broad and deep bull market that is somewhat hidden from us because of the strength of the dollar. Every developed market is positive and every emerging market save for Venezuela. Europe index is up about 23% in domestic currency, Pacific index is up about 33%. Emerging markets index is up about 29% in dollars. It is as though the values of international stocks are adjusting to the strength of the dollar.

[anne]

http://www.msci.com/equity/index2.html

National Index Returns [Domestic Currency]
12/31/04 - 12/13/05

Australia 20.6
Canada 24.2
Denmark 38.9
France 26.8
Germany 24.9
Hong Kong 7.2
Japan 43.1
Netherlands 30.6
Norway 42.2
Sweden 30.8
Switzerland 35.6
UK 17.7

http://www.msci.com/equity/index2.html

National Index Returns [Dollars]
12/31/04 - 12/13/05

Australia 16.0
Canada 29.3
Denmark 21.8
France 11.3
Germany 9.6
Hong Kong 7.5
Japan 22.1
Netherlands 14.7
Norway 29.4
Sweden 9.8
Switzerland 19.1
UK 8.4

[anne]

Suppose we use the Vanguard price earning ratio of 17.3. Given a return on equity of 18.5% and an earnings growth rates of 12.3%, and a tax favored status for capital gains and dividends, why should we be surprised? Long term interest rates are low, and interest income is tax disadvantaged, so bonds now are really not competitive with stocks.

We are also going through a time when even with 18 months of short term interest rate increases by the Federal Reserve, and significantly rising oil and gas prices, economic growth has been sustained. Indeed economic growth has been sustained in every developed economy, even when housing markets have slowed. Economic flexibility appears to be far greater than in 1980 or even 1990.

[anne]

"Looking at the diagram, it is hard to come away without a feeling that the market is quite likely to decline substantially in value over the succeeding ten years; it appears that long run investors should stay out of the market for the next decade..."

Quite a problem :) The market appeared to be over-valued, and would appear to be much more so in 1997 and 1998 and 1999, and investors are advised to stay away for a decade. As it turned out the S&P has returned 9.2% for the last decade, but what if it had not? The market after all has been flat for the last 6 years. How are investors to possibly time a market in such a vein? What was suggested makes no sense unless there are alternatives, but there were always alternatives.

[p.lukasiak]

I'd like to suggest that there is another reason why Schiller was "wrong" -- the increasing use of stock purchases through payroll deductions.

The existence of a huge source of funds that is programmed to be invested each week in the market regardless of market conditions significantly affects the way stocks are bought and sold. The constant stream of new money coming into the market through IRAs and 401ks means that there will always be a far greater demand for stocks than a "rational" market would suggest.

[anne]

Taking account of more retirement funds flowing to stocks, should allow stocks to trade at higher valuations but not increasingly higher. So, the market might be able to trade a few percent higher in price earning terms as suggested but the adjustment will not take the price earning ratio continually higher.

[Arun Khanna]

1. Maybe the smart money bets on volatility rather than mean S&P 500 moves.
2. Your argument reminds me (to some extent) of Shleifer and Vishny's limits to arbitrage paper.

[perfectlyGoodInk]

Seems to me that the lesson here is to dollar-cost-average instead of trying to time the market.

[realist]

over the last three years, shiller has been saying the same thing about the real estate market, that he said about the stockmarket in 1996. i agree that there has to be some pressure on prices to fall for the next few years, but that will probably change, again, a few years after that.

i don't know if anyone on this blog has been following the muscle car market, but prices of these restored classic cars have risen dramatically. because the restoration work is domestic, the chinese can't compete. in a few years the chinese will be exporting their own cheaper cars, but that will not affect the prices of restored u.s.muscle cars.

this same analogy may be applicable to new housing. in the building of new houses, the materials prices, energy prices and labor prices keep going up. the land is totally subject to domestic pricing. consequently, the final product can't be imported. the more our dollar depreciates, the more material costs, labor costs and land costs should appreciate. higher new house prices should also push the prices of existing houses, higher.

schiller could be wrong, again.

[anne]

And remember how easily an investor can diversify using indexes :) A selected broad American stock index, large cap to small, European index, REIT index, health care, energy, bond index.... From 1998, REIT, health care, energy and bond indexes were so nicely priced.

[Robert Waldmann]

OK Brad this is definitely the silliest debate yet on a comment thread to your blog but you sure did tell him so (with Barsky). The point of your paper IIRC was that variance bounds tests (as in for example Shiller) are only valid if there is not a second unit root in the dividend process as in persistent productivity slow downs and speed ups. You note that the evidence against is not overwhelming that the US might just happen to have stayed near apparantly constant drift.

Since then there has been a speed up. You didn't say it would happen, but you said it could. Running title IIRC "Why Shiller Can Not be Convincing."

[dk]

how about the obvious answer: STOCKHOLDERS ARE RIPPING OFF WORKERS. that also explains why "productivity" is up, but wages remain flat.

i live in hope that economists will notice the obvious someday.

[Brad DeLong]

Re: 'Running title IIRC "Why Shiller Can Not be Convincing."' Ah. I had forgotten about that running draft title...

:-)

[jm]

In Northern Virginia the wheels on the housing market appear to be getting kind of loose and wobbly.

http://www.nvar.com/market/mktreports.lasso

Reports for other regions on the various housing bubble blogs indicate this is a phenomenon not confined to NVa.

From '94 through '00, new home construction in the range of 1.1-1.2 million annually was enough to consistently raise the home ownership rate about 0.5 percentage points a year. Construction over the last four years has been well above that pace.

Comparing Census housing vacancy survey data for the last three years ...
http://www.census.gov/hhes/www/housing/hvs/qtr305/q305tab7.html
http://www.census.gov/hhes/www/housing/hvs/qtr304/q304ind.html
http://www.census.gov/hhes/www/housing/hvs/qtr303/q303ind.html
reveals:
- near-record 68.8% overall home ownership rate (historically 64%);
- record-high home ownership rates among under-30s;
- rental vacancy rate now over 10%;
- reversal of previous downward trend in households renting;
- marked slowing of home ownership growth;
- actual slight decline in home ownership percentage.

Might the housing market be running out of greater fools? Indeed, why shouldn't there be a "Hubbert's Peak" in America's greater fool supply? If there is, and we're past it, it's quite possible that even a drastic reduction in interest rates will not prevent a plunge in housing.

If that plunge has already started, and gathers momentum next year (perhaps fueled by tightening lending standards and upwardly adjusting ARMs?), it's likely to take the stock market with it. Shiller may have come within just months of being right.

[ilsm]

Fewer fools:

I went on a personal journey in 2000. Sold out, went into renting in lower cost area, way outside commuting distance of my origin.

In 2003, came back to where I sold, appraised housing price rises vis a vis renting and decided to rent.

No sense in chaining my networth to a house which will decline in price and be depressing for 10 to 12 years.

Also, this fool is not sure the house would be a long term enjoyment.

[Rob Bennett]

I don't agree that Robert Shiller has been proven "wrong" about the most important points he made in his 1996 testimony.

It is fair to say that he has been proven wrong on the question of how long it would take for valuations to return to the mean. But there has been lots of evidence put forward in recent years showing that he was right about the more important question--whether it is possible to make long-term predictions re stock prices by making reference to the historical stock-return data.

We have gone into uncharted waters in recent years. Stocks have in recent years remained at extreme levels of overvaluation for a longer period of time than they ever have in the past. The statistical likelihood of that happening was low. So I don't think that Shiller was off base in offering the prediction he did (he did include caveats, of course).

My guess is that when stocks do go down (they always do), there will be lots of people looking to Shiller as a hero for having said what he said at a time when it took a lot of courage to say it. Shiller was one of the first to tell it straight about the risks of the extreme levels of overvaluation we have experienced in recent years. I give him a lot of points for sticking his neck out in the way he did.

There's an article at my web site that explores the famous Robert Shiller prediction in a bit more depth -- http://www.passionsaving.com/Robert-Shiller.html

Rob

[keivn quinn]

Brad: In your comments you seem to rule out the possible existence of "rational bubbles." Why?

[RT]

I would have said the second of the three factors - shifts in the distribution of income away from labor and toward capital - was the most obvious of the three a decade ago. By then, I'd already been using a simple phrase - "there's a lot of money sloshing around at the top" - for a couple of years to explain seeming anomalies like the failure of minimum wage hikes to cause inflation.

Of course, I was looking at the economy from the lower end of the distribution, which may account for why that factor stood out, from my POV.

[Barkley Rosser]

Shiller may have been wrong in 1996. But he was on the money at the beginning of 2000 when his book came out. He is also on the money for his second edition as of this past spring, in which he added a chapter on the housing bubble. It appears that in the bubbliest markets, housing prices peaked around July.

[dk]

come on, guys, just give me even a HINT of a counterargument. i come here to avail myself of your genius, you know.

[anne]

Housing appears to have generally slowed, though according to localities. Commercial real estate appears generally robust.

[bronxite]

One effect that's been left out of the argument is the tendency for stocks in the S&P to be "permenantly" overvalued just due to their membership. Since S&P 500 index funds are so popular (and getting moreso), the inclusion metric creates a demand bias.

The value weighing procedure (market cap) is also a source of permenant mispricing. There is now a PIMCO fund that creates an index with weighting based on earnings, book value, and other factors. This is probably a wiser investment than the S&P 500.

[hack]

Note from the future: Shiller was right!

[hack]

Note from the future: Shiller was right!

[hack]

Note from the future: Shiller was right!