Annals of Derp: Upshot or Upstoned?

Image via Kumar Deepak.

Just caught a remarkable piece of fluff or perhaps navel lint signed by Brendan Nyhan at the Times's new Upshot site: a Wisdom-of-Crowds piece on how while we all know that coin tossing does a better job of making predictions than media pundits, if you put them together to make one big parallel-processing device you might get something worth knowing.

Thus, for example, a Wall Street Journal exercise (see table below) put together a list of 17 Republican and Democratic holders-forth and their predictions as to which party would win the Senate in November. Their numbers turned out to be strictly partisan [jump]
(Republicans said the GOP would take over, Democrats said they wouldn't), and altogether off the wall in some cases, but Nyhan discovered that if you looked at them as a group, you found that the median number for the likelihood of the Republican takeover was 55%, same as the odds offered by the Betfair Futures Market, and just a point ahead of the 54% proposed coincidentally by the Upshot its magnificent self (this morning, since the piece was written, they've gone up to 55%, no doubt also coincidentally).

Well, I was going to complain about the utter nullity of the sampling in this study—how was this specific list selected, and why should I regard it as in any way representative of anything, supposing that there is something representable there?—when I noticed some pretty classic derp:

Despite the public nature of the predictions, which might seem to increase the incentives for accuracy, the results break down perfectly along partisan lines. The eight Republicans all see a takeover as more likely than not (median forecast: 65 percent chance of G.O.P. control) while the Democrats see it as a coin flip at best (median forecast: 44 percent).

If Nyhan had been able to count correctly to nine, he might have noticed that there was something wrong with his picture. See, there weren't eight Republicans, as you can verify for yourself. There were eight Democrats, but nine Republicans, meaning that assuming all were to give partisan answers, as all in fact did, the median number (not a calculation like a mean, but a position, at the exact midpoint in the list of all the answers) would of necessity be a Republican number. If you balanced it out by taking one away, there would be two numbers at the midpoint, so you'd determine the median by splitting the difference between the two, and get a median of 52.5% instead. This would be the case no matter which Republican you deleted, as it happens.

Moreover, that's just for the likelihood numbers. If you look at the predictions of how many Republicans are going to be elected, the median prediction (Kellyanne Conway's) is that the Republicans will take over with 51 seats total, but if you balance Republicans vs. Democrats you get a median of 50 seats (unless you delete Conway, yielding 50.5, which I call a rounding error). Thus the Nyhan parallel pundit-processing model predicts (a) a 52.5% chance that the Republicans will control the Senate by (b) a majority of zero. I.e., given our Constitution and the vice president's party, they predicted something that could not possibly happen.

Name	Likelihood of GOP Takeover	*Democrat Seats	Republican Seats
Katie Packer Gage, former Mitt Romney deputy campaign manager	80%	48	52
John Brabender, former Rick Santorum campaign manager	75%	48	52
**Chip Saltsman, former Mike Huckabee and Bill Frist adviser	70%	47	52
Danny Diaz, GOP strategist	70%	47	53
Kevin Madden, former Mitt Romney and George W. Bush aide	65%	47	53
Rick Wilson, Florida-based GOP strategist	65%	46	54
Former Sen. Judd Gregg (R., N.H.)	60%	48	52
Brian Walsh, former NRSC communications director	55%	48	52
Kellyanne Conway, GOP pollster	55%	49	51
Donna Brazile, former Al Gore campaign manager	50%	53	47
Jim Manley, former Harry Reid adviser	50%	51	49
Chris Lehane, former Clinton/Gore adviser	45%	51	49
Robert Shrum, former John Kerry campaign manager	45%	53	47
Michael Simon, former Obama analytics director	42%	51	49
Celinda Lake, Democratic pollster	40%	53	47
Bob Creamer, Democratic strategist	33%	53	47
Hilary Rosen, Democratic strategist	25%	51	49

*Democrat seats include independents Bernie Sanders and Angus King
**Saltsman predicts Louisiana will be undecided following the November election.
Source: Compiled by Reid J. Epstein/The Wall Street Journal

Oh, one other fun fact: Dr. Google informs me (I swear I wasn't looking for this) that Nyhan's experiment has been tried before, by David Silbey at the Chronicle of Higher Education, on November 4 2012. Working with what looks to me like a much more respectably chosen sample ("I gathered as many predictions as I could") and using the more appropriate means rather than medians, Silbey derived a pundit-crowd prediction that Obama would defeat Romney by about (there are couple of variant models but the differences are minor) 275 electoral votes to 263. Since the actual results of the election were 332 to 206, I don't just rest my case, I put it to bed with a great big kiss. The wisdom of pundit-crowds, sadly, isn't.

Via Jimmy Fungus.