Annals of derp: Comparing apples and Oregons

Paul Cézanne, ca. 1900, via WikiPaintings.

Hahahaha Megan McArdle Department:

Last May, researchers studying a randomized controlled trial of Oregon’s Medicaid expansion released a bombshell report that found no significant improvements in mortality rates or key health markers for blood pressure, hypertension and diabetes control. I did three major posts on it (one of which was a guest post from Jim Manzi...).

We'll decide what's major and what isn't, and how many of them you get to take credit for, thanks. Also too, they weren't studying the trial, they were performing the trial, or studying the expansion. Anyway,

A different team of researchers has looked at mortality rates in Massachusetts after Romneycare. And they found what is, for mortality rates, a huge decrease: 8.2 per 100,000 per year for adults 20 to 64. Doesn’t sound like a lot, but your risk of dying in any year between 20 and 64 isn’t that high. So a big, fat decline like 8.2 per 100,000 makes you sit up and take notice.... It is hard to reconcile these results with Oregon. Good Bayesians will update their beliefs accordingly.

(I guess a "good Bayesian" is somebody who believes in using statistical inference for beneficent purposes, as opposed to bad Bayesians, who use it for evil?)

McArdle is somewhat puzzled as to how to show that the "conservative" results for Oregon are more reliable than the "liberal" results for Massachusetts, but what she comes up with is this: that the Oregon study was a better one because it was a genuine randomized control experiment (the state used a lottery to create one population receiving Medicaid benefits and another comparable control-group population of people without health insurance) while the Massachusetts study compared each of the Commonwealth's 14 counties to a sample of  513 counties in other states

that are supposed to be similar but might not actually have been in practice.

Because even though the comparison counties were specifically selected for similarity in terms of race, age, health, and income, you never know. Like, Massachusetts had considerably higher unemployment levels during the last couple of years of the study (that's higher than the US in general, not necessarily higher than the 513 counties in the study), and

if the financial crisis affected Massachusetts differently from the rest of the country -- and it did -- then that may well show up in mortality in ways that are hard to measure.

Which would be what, exactly? The jobless have a greater will to live because they're on permanent vacation?

But the thing is that

1. The Oregon study was in fact not a genuine randomized control experiment but used a "quasi-experimental design"  whose results are thus not demonstrably more reliable than those of the Massachusetts study, also quasi-experimental;
2. The Oregon study was about Medicaid benefits, considering only Medicaid-qualified subjects at 100% of poverty, and people who were self-selected (volunteering for the Medicaid lottery), while the Massachusetts study was about statewide insurance reform and considered the entire population, so the two are not comparable in any way; and
3. The Oregon study did not look at mortality rates at all, since there weren't enough deaths in their sample (of a total of 35,000, compared to the 40-odd million of the Massachusetts study) to draw any significant (or "good Bayesian") conclusions (in spite of McArdle's casual assertion), so there is nothing to compare the Massachusetts study to, even if it were, as it is not, legitimate to compare them.

In this way McArdle is writing about a purely imaginary relationship; her comments are entirely empty. An unceasing astonishment, isn't it, how this (morally questionable Bayesian) person still has a job?

Tatsuya Ishida/MuseWorks.

It also needs to be noted, as McArdle fails to note, that while the Oregon study did not find that Medicaid significantly improved patients' results for high blood pressure, high cholesterol, and high glycated hemoglobin, they did find that it

increased individuals' health care access and use of services, lowered out-of-pocket costs, reduced medical debt, and improved self-reported health and well-being.... raised rates of diabetes detection and management, lowered rates of depression, and reduced financial strain. (California Health Care Foundation)

And that the Massachusetts reform, passed by the legislature over Governor Romney's vetoes of eight separate provisions, including the crucial employer mandate, really shouldn't be called "Romneycare", however much the Mittster may have enjoyed being its stepfather in later life.