[quote=Arraya]The big question is why did academics fail at forecasting this? THAT is the question and how that applies to society overall?[/quote]
Shiller, Roubini, Roach, and a few other luminaries, academic or otherwise, did in fact diagnose the bubble and forecast a crash.
But they were a tiny minority, and were marginalized by the mainstream media. The reasons are several:
1. Inside academia, most economists spend their entire careers fine-tuning theories that predict a few economic variables for the next quarter. As a profession, economists spend very little time developing a more rigorous understanding of economics or finance over the long term.
2. In the finance world, those who develop a theory saying that the current bonuses and profits of their firm might be largely bogus, simply not sustainable over the long run, or an example of egregious economic rent extraction, will either decide they don’t have enough evidence and keep collecting their paychecks, or they will speak up and promptly be cast out of the community. It’s a tough choice. Most take the easy path.
3. Most mainstream media organizations are supported or controlled by the same people and firms that do well as long as the bubble continues. They will be made to feel disapproval if they suggest we have a bubble. Even now, how many people are suggesting that the Federal Reserve should let asset prices slip back to their levels in the mind-1990’s, before the huge expansion beyond historical trendlines since then? No one in the mainstream media wants to call that.