Disclaimer: Both Robert MacCallum (e.g., 2003) and George Box (e.g., 1979) have written extensively about the value of models and I will basically steal parrot their ideas in this post. Moreover, I did not sleep much last night…
Let the postmortem on the 2012 election begin! One story will likely involve the accuracy of well-conducted polls and the success of Nate Silver’s methods over “gut-based” methods favored by pundits and campaign workers. Not surprisingly, I like much of this story as is nicely summed up by this cartoon (thanks to Skip G. for posting this one on “the” Facebook).
But what if Nate Silver was wrong? What if we woke up today and Romney won 303 electoral votes and Obama lost the election? I think Mr. Silver would have been in a much better position than his “gut-based” critics are in today. The reason boils down to the advantages of models. It is really useful to have a formalized recipe for prediction. To quote Box (1979): “The great advantage of the model-based over the ad hoc approach, it seems to me, is that at any given time we know what we are doing.”
I happen to like the model for science depicted by George Box (see Figure A1 in his 1976 Science and Statistics paper). The basic idea is that errors drive the accumulation of knowledge in an iterative cycle. Learning is produced when there is “a discrepancy between what tentative theory suggests should be so and what practices says is so” (p. 791). In other words, there is something to be gained when predictions from models and empirical facts disagree. These errors lead to better models, at least in the ideal case.
So if Mr. Silver’s model was gravely wrong, he could have spent the next days and weeks figuring out where his model went wrong. He has his predicted values and he has the actual values. He can test alternative models to find ones that outperformed his original model. He is in a good position to learn something. Compare his plight with that of the gut-based pundit. How are they going to figure out why their predictions failed? What are they going to learn?
Moral of my story: Models rule. I think there might be a bigger lesson in here for “soft” psychology but I am too tired to express it properly.