Silver’s The Signal and the Noise

I’d recommend Nate Silver’s The Signal and the Noise: Why So Many Predictions Fail but Some Don’t to anyone looking for a layman’s tour of applied statistics. It is not a “how to” book, although there are plenty of principles (and suggestions to be humble) worth following. It’s also not a book that gets too deep into any subject area, so for those areas I was familiar with, there were no surprises. But where I wasn’t, I generally enjoyed it.

The section of the book I enjoyed most was the section on weather forecasting. As a start, weather forecasting is getting a lot better. In the early 1970s, a temperature forecast by the US National Weather Service three days in advance tended to miss the actual temperature by around 6 degrees F. Today, they miss by around 3.5 degrees F. For hurricane forecasts 25 years ago, the prediction of where a hurricane would make landfall three days in advance had an average miss of 350 miles. Today it is marginally over 100 miles. For Hurricane Katrina, the prediction of landfall at New Orleans was made five days in advance.

It’s cool to see that there is progress in forecasting in complex systems such as the weather. But the current limits to predictive ability are also interesting. Silver presents a chart comparing the accuracy of temperature forecasts from three sources: climate averages, persistence (the weather tomorrow will be the same as today) and commercial forecasts. Predicting the climate average for any number of days in advance of today results in an error of around 7 degrees F. Persistence outperforms the climate average for the next day, but for more than two days ahead, you are better off assuming the climate average. Finally, commercial forecasts start out as much superior to assuming the climate average, with an average error of around 3 degrees F for the next day. But by day 8, the commercial forecasts barely beat the climate average. After day 10, the commercial forecasts are worse.

What is interesting is that the commercial forecasts rapidly deteriorate to a point where they display negative skill after 10 days. You could simply look to historical averages for a better indication. This is caused by feedbacks in the computer program, which start to build on themselves. With the programs highly sensitive to the initial conditions, the noise ends up dominating (Silver gives a useful simple synopsis of chaos theory).

The weather chapter provides an interesting contrast to the chapter on economics. Economic forecasting might be considered similar to weather - a dynamic system sensitive to initial conditions - but with a markedly different record of improvement. And not only is economic forecasting demonstrating little improvement, but the confidence with which economic forecasts are made is far in excess of what they deserve.

A couple of issues underlie this. First, economics does not have the same fundamental laws underlying it (and as I often argue in posts in this blog, those that exist are too often ignored). But also, the incentives are different. Weather forecasting is tested day after day, so people note errors. There is a small degree of bias where there is something at stake, such as some forecasters exaggerating the probability of rain when it is unlikely to occur, because people remember rain ruining an event when the forecast was for sun. And local TV weather presenters push this wet bias even further. But in general, performance is not too bad.

For economics, predictions are rarely tested and often made years in advance. There are strong incentives to herd where being wrong and the only one who is wrong is costly. There are also incentives to make outlandish claims when people will forget all the other times you are wrong (keep predicting the financial crisis, and you will be right sooner or later). It’s hardly an environment where accuracy is the best bet.

Otherwise, the chapters on disease and chess were also excellent. I wasn’t a big fan of the climate change chapter (neither was Michael Mann) and I still don’t understand why Americans like baseball.