Can Measuring Quality Actually Reduce It?

The claim that measuring and publishing quality indicators will always improve health care quality has become an article of faith in health policy circles. The passage of ObamaCare made it dogma. Contrary to widespread belief, evidence-based research suggests that devotion to poorly designed quality measures reduces quality. The literature on this was reviewed by David Dranove and Ginger Zhe Jin in the December, 2010, issue of The Journal of Economic Literature.

Imprecise measurements are a major obstacle. For example, hospital report cards are generally based on easily observable outcome measures like mortality. But mortality is a relatively rare event, so quality measures based on it may have large errors associated with them. In one Medicare study, only 3 percent of hospitals could be identified as having either high or low quality. Another source of imprecision is that in measuring observable outcomes we may be indirectly measuring the qualities of a particular patient population. Some insurers include immunization rates in physician report cards, for example, even though it is well known that immunization rates are strongly affected by parental education and income.

Attempts to adjust even for the known qualities of the patient population with measures of its underlying riskiness (“risk adjustment”) are often unsatisfactory. The range of potential risk adjusters is “vast,” and their “predictive power” is “low.” The problem is orders of magnitude worse if quality is measured along more than one dimension. While it might be straightforward to measure the relative quality of drugs to treat hypertension based on the number of points they lower blood pressure, developing a ranking with any meaning is far more difficult if it must also be based on the rate at which patients on the drugs develop diabetes, suffer from myocardial infarctions, experience drug side effects affecting their quality of life, or die from some other cause while taking the drug.

Quality measures also degrade quality by distorting behavior. Just as teachers who are rewarded or punished based on student test scores will often “teach to the test,” doctors and hospitals pressured to improve their performance on reported measures may allow their performance to deteriorate in ways that are not measured. For example, overall nursing home responses to federal quality indicators suggest that there were few net benefits for patients. The quality of care along reported dimensions showed insignificant improvement, the quality of care along unreported dimensions declined, and there was no evidence that nursing homes increased quality related inputs. Another example is provided by the mandatory cardiovascular mortality report cards developed in New York and Pennsylvania. They increased resource use while degrading patient care simply because surgeons stopped operating on sicker patients who were more likely to die in an effort to protect their mortality rankings.

There is also bad news for those who believe that government quality certifiers act in the best interests of consumers. Contrary to the articles of faith in some policy circles, government inspectors often rely on subjective measures in making their judgments, and their personal preferences may differ from those that they are supposed to protect. The article cites the wide variability in FDA and Nuclear Regulatory Commission inspections as examples of this.

Comments (9)

Trackback URL | Comments RSS Feed

  1. devon herrick says:

    This is the problem of hospitals treating observable metrics as proxies for quality rather than a byproduct of quality. For instance, a check list of procedures may be associated with better outcomes but not the cause of better outcomes. Yet, the incentive is to follow the check list and document adherence to the protocols even if the list does not lead to better outcomes.

  2. Bruce says:

    Excellent post. This is what John Goodman talked about at Brookings yesterday.

  3. Ken says:

    The JEL article completely undercuts Obama Care’s entire approach to health reform.

  4. Erik says:

    Contrary to widespread belief, evidence-based research suggests that devotion to poorly designed quality measures reduces quality.

    Okay remove the qualifier “Poorly Designed” quality measures and start over.

    Straw-man much?

  5. Ken says:

    Erik, not a strawman. Read the JEL paper. They are describing real cases. Have you never heard of teachers teaching to the test? Do you want your doctor proacticing medicine to the test?

    No matter how good the test, it’s no substitute for judgment and experience. If it were, we wouldn’t need doctors. We could suffice with clerics following a cookbook.

  6. Neil H. says:

    This is fascinating. Thanks for posting it.

  7. Erik says:

    I somewhat agree with you post. I am saying what would this look like with a thoroughly agreed upon protocol? I bet it would help those doctors who “studied abroad.”

    I like the “clerics following a cookbook” line. I tend to believe that is exactly the case. That is why it is called “Practicing Medicine.” Eventually a good doctor will stumble across the right recipe.

  8. Ken says:

    Erik, the probem is, you can’t right down on a piece of paper what high quality medicine looks like. There have been several posts at this site showing that high performance practices and regions have very few objective characteristics in common.

  9. Bruce says:

    John Goodman said it best at Brookings: Low cost/high quality care is like pornography. We know it when we see it, but be can’t define it.