Monday, 23 August 2010

Quality measures and the individual physician: A UK perspective

A few weeks ago, Dr. Danielle Ofri, published her personal view in the NEJM of receiving individual feedback on how patients attain certain performance targets. On his personal blog, Dr. Kent Bottles wrote a rebuff, where he suggested that Dr Ofri was implying that because she and other doctors cared about their patients, these scorecards were irrelevant. My reply  stated that Dr. Ofri was not against feedback per se, but that she believed it should be at the level of the institution. Kent's thoughts are now reposted on the Health Care Blog where it has created much debate. I’m sure that Dr. Ofri doesn’t really need defending but here is my take on what she has written.

She starts by pointing out the silliness of treating success in reaching targets as a binary outcome and particularly mentions blood pressure control in diabetes. This is a good point. Treating hypertension in diabetes is about treating a risk factor in a population;  one can never know the benefit for any individual patient. The original UKPDS study  which looked at the impact of ‘tight BP control in diabetes’ compared targets of 150/85 (tight control) vs 180/105 (less tight control). The tight control group managed a mean BP of 144/82 whilst the less tight control group averaged 154/87. The tight control group had reduced all cause mortality and also lower rates of nephropathy and stroke. However, the tight control group were given a regimen including an ACE-inhibitor (a treatment that we now know to decrease all cause mortality in diabetic patients, and to be protective of kidneys) while the less tight control group were to be deliberately not given this. So we don't really know how much of the benefit was down to the actual blood pressure attained, or the treatment used.

But back to Dr. Ofri's point; looking at how many patients achieve a target might, but doesn't necessarily, tell you about how the overall BP in the population has changed. And that is what matters. An aggregate measure of how much change has been produced in the BP of all patients might be a better way of describing how well BP is managed in any practice. She could spend all her time trying to get the patients with a BP of 145/85 down to 130/80 to meet the target, but completely ignore the patients with much higher blood pressures because they will be so much harder to get to the target. I doubt that Dr. Ofri would ever be so cynical as to take this approach because as she says most doctors have the good of their patients at heart, and are not just trying to make a fast buck as quickly as they can. I work in the UK, so I can't say if Dr. Ofri is seeing her colleagues through rose-tinted glasses. She afterall has chosen to work in Bellevue Hospital, the oldest public hospital in the US, where 80% of patients come from under-served poulations.

Some of the discussion has been around what are meaningful endpoints for quality measures. We might presume that reducing blood pressure is always good, but it seems to be more complicated than that. Atenolol, which was one of the agents used in the tight control group in the UKPDS trial above, does decrease blood pressure but not overall mortality so it isn't a sensible choice for first-line treatment. And in diabetes achieving even lower blood pressures through aiming for a target of 120/80, is associated with more side-effects from medication but no benefits for patients. Dr. Ofri's failure to get her patients' blood pressures below 130/80may be a good thing for some of them.

But her main complaint about these scorecards for individual doctors is not about the choice of targets, although I hope I have helped you to understand what she meant when she said it is easy to pick fault with them. No her main issue is that the scorecards place responsibility with individual doctors for the outcomes of  their patients. She doesn't have a problem with tracking outcomes, and says she would be keen to see how her institution compared to others. She cites a systematic review which shows that this kind of feedback may work at the level of the organisation but not at the level of the individual. She also cites an opinion piece in JAMA  where the authors suggest there may be unintended consequences to publicly sharing feedback on the performance of individual doctors, and organisations. These consequences may include "causing physicians to avoid sick patients in an attempt to improve their quality ranking, encouraging physicians to achieve "target rates" for health care interventions even when it may be inappropriate among some patients, and discounting patient preferences and clinical judgment."

Could that be true? Only this evening I came across the following tweet

Pay-for-performance quality measures will result in docs firing noncompliant patients. I know I don't want a dipshit diabetic to sink me.less than a minute ago via HootSuite

In the UK, we have already introduced pay-for-performance in primary care. When this was introduced in a new GP contract in 2003, it was lauded by Paul Shekelle, as "the boldest such proposal on this scale ever attempted anywhere in the world". But he was also worried about unintended consequences. One was that areas which were not assessed in the performance measures would suffer. This is hard to assess, and new areas have been added in each annual review in any case, but it is thought that there has been no impact.  The other was that the relationship between doctor and patient would change, with loss of some of the holism thought to define UK general practice. We can gain some insights in to how that might have been realised through the ethnographic work of Checkland et al. who documented the changes that the contract brought to two UK general practices. There was an increased focus on recording 'hard' biomedical data over 'soft' patient-centred data. But the staff involved did not see any change in their practice. It is well worth reading this and their other work to gain insights in to some of the impacts that pay for performance may be having in the UK.

Primary care doctors in the UK can not choose their patients. If their 'list is open' (they think that they have spare capacity) then they must take any patient that wants to join. So there is no risk that an  individual patient may not receive care because of pay-for-performance. However, unlike in the US, patients may be excluded from denominators if they are having 'maximally tolerated treatment'. This may reduce some of the frustrations that doctors in the US feel about such performance measures.

Secondary care in the UK does not have pay-for-perfomance, or even publicly accessible feedback on performance, but evidence exists that there is a tension between protocol-driven care and tailoring care to the needs and preferences of patients. Sanders et al. have done some excellent work describing how this plays out in specialist heart failure clinics.

Getting back to Dr Ofri's concerns,  it is worth noting that the feedback on achievement of targets in UK general practice is at the level of the practice, not the individual doctor (although there is still a size-able number of single-handed GP practices). How does this feedback play out in real life? Here is the prevalence data of disease areas covered by the contract for my own practice in South Wales. The practice is in a deprived area, so unsurprisingly the prevalence of diabetes is 34% higher than the UK average, and the prevalence of hypertension is 32% higher. In our patients with chronic disease the prevalence of smoking is 26% compared to a UK average of 22%. This next link shows how well we compare to other practices in the area in meeting the target of having BP readings of less than 145/85 in our patients with diabetes. Despite the higher than average burden of disease in our practice, we have managed this in 75% of our patients. However, this puts us only on the 25th centile for performance within the area. Through a process of internal peer review we try to figure out how we can improve our success in these targets. We are continually reviewing our recall systems for patients, and how we can share work within the practice team.

Dr. Ofri does not say  that doctors should not be subject to performance measures just because they are good people. Instead she points out that the measures should be sensible, and that they should probably be applied at the level of the institution and take in to account wider systemic issues, for example availability of cheap medications. She has drawn attention to the complexity of such an apparently simple process.

Finally, I would ask you to watch this short video of Dr. Julian Tudor Hart, a doctor who inspires many in primary care with his research and  work in South Wales. He has demonstrated  what can be achieved when caring for underserved populations, and the mindset and caring attitude required.

EDIT: 24/8/10 I came across this YouTube "Can we tell physicians apart without better scorecards?" I find it interesting because it starts with talking about feedback from patients about empowerment. shared decision making etc. Next, the comment is made that if this could be done by email it would significantly reduce costs, however no-one has the email addresses of patients. Lastly, the point is made that although institutional/system measures 'should' be the way to address quality improvement in real life it doesn't work that way, and individual physicians seems to be a key determinant themselves. However, this fits with the notion of process measures- not the outcome measures described by Ofri. The debate continues!


  1. Good commentary... publishing report cards in the US has certainly led to cherry picking or choosing to operate on the low hanging fruit. In NYS the results of cardiothoracic surgeons and institutions is available online. During my training I remember many cases where surgeons would chose not to operate on someone because they were such high risk and they were worried what affect that would have on their scores. There was one surgeon who didn't care and would try and help everyone... balloon pump, pressors, alert, awake and willing to assume the risks too. That surgeon only lasted 1 or 2 years at the hospital I trained at. So, while scoring is *important*... without determining what to score, how to score it, what realistic endpoints or variables are and how to control for those variables a number of doctors and institutions may be singled out as *poor* unnecessarily.

  2. Thank you. That is a very helpful.

  3. Okay, Doc. I hear you, and I've said (on The Health Care Blog) that I agree with Dr. Ofri with respect to analog comparison of quality records. Knowing that you're moving in the right direction and recording the positive efforts is just as important as recording the milestones.

    Beyond that, I have to disagree with a few points. In response to Dr. Bottles's post, I suggested Atul Gawande's article, "The Bell Curve," from his 2007 book Better, as a model for quality improvement. The efforts of the Cincinnati Children's Hospital to control CF life-expectancy demonstrate that being smart and well-intentioned don't always equate to success. You still need a model for growth. The performance pay model from the UK, as you've reported, demonstrates this again. Can we doubt that the UK physicians wanted to get that performance pay?

    Back in January Kent Bottles posted an article about "Kent Bottles: Why Smart People Don’t Learn from Failures" ( ). In that article he notes that the problem isn't the learning, it's the admitting they're wrong that offers a real stumbling block for smart people.

    You say, "Dr. Ofri does not say that doctors should not be subject to performance measures just because they are good people." I think she does something just as bad. She submits that, "Who are the people who choose
    to enter medicine, and what are
    their motivations and character?
    I have yet to meet a medical student,
    intern, nurse, or doctor who
    doesn’t feel a powerful sense of
    professional responsibility." Clearly, Ofri priveleges intention over actual performance. Dr. Ofri also lashes out at the folks who created her hospital's quality standards, implying that they don't actually practice medicine. If they're finding her scores inadequate, she implies, then they don't understand the "real world" where diabetics are unhealthy people with a lot of other problems. Ofri clearly doesn't want to understand the application of any kind of quality standard that finds her quality lacking. Instead of seeing an inability to meet a quality standard as an indicator that she's incompetent, she needs to see it as an indicator that she needs to change the way she works.

    Ofri occasionally sounds like she almost gets it, but does she? She says, "Are most doctors
    doing a reasonable job? If so, then our analytics should aim to weed out the few who are inept. Or are most doctors mediocre, with shoddy clinical skills that put patients at risk? If so, then our data-driven system must prod doctors as a group to up their game." This is not the statement of someone who understands quality. This is someone who understands only shame and blame. If the scores are low, then either they're wrong or she has "shoddy clinical skills."


    No, no, no. Read the Gawande article. The clinicians at Cincinnati Children's weren't "mediocre." They just lacked a little insight.

    Everyone favors putting quality metrics to work until they have to see a report card. Everyone wants to be seen as inherently great. Quality metrics, if they work, always piss people off. They tell you where you're not doing the best job possible. The trick is not to take it personally. A low quality score doesn't mean you're a low quality clinician. It does mean you need to change the way you do something.

    As I've said again and again, quality is not about being good, smart, or well-intentioned. It's not a personality critique. Quality is about finding a way to get the results you want.

  4. @MedicalBillDog Yes! Yes! Yes! Ofri champions the approach to quality taken by organisations such as the Cincinnati Children's Hospital.
    She says: "I certainly want to know how my hospital is doing with an overwhelming disease like diabetes", and that the data could "highlight fixable systemic impediments to good care." She cites a review showing that institutional feedback works, and she says that this is the kind of information that she wants for her own hospital.
    But nowhere can I see that Gawande has been championing scorecards on individual physicians. He recognises that improving outcomes depends on improving systems of care, and that includes getting feedback from the families involved and sharing with them.
    I don't know how to make clearer to you than Ofri is not in anyway disagreeing with your concept (which is absolutely correct) of of how to improve quality.
    I hope you will read her piece again and take this on board.

  5. Okay, Anne Marie, I see your point. I can see shifting the scoring to the institution overall. I can also understand Dr. Ofri's exasperation with her colleagues refusing to address her concerns.

    Ultimately, however, every time I go back to her writing, three points stand out as slaps in the face to anyone who is not a physician.

    First, and in my mind most egregious is the idea that all doctors are smart, capable, and well-intentioned. Even Gawande gets on my nerves with this. In his articles, all the doctors appear to intend only the best for their patients. Are the rest of us to ignore our perceptions that some doctors are incompetent, crooked, lazy, megalomaniacal, or just burnt out? I think she's wrong. I think he's wrong. I've met bad doctors. They're in the news all the time. Doctors complain about malpractice insurance costs and blame the lawyers, but they cover for bad doctors.

    The second slap in the face is the complaint that her quality metrics were created by non-practicing physicians. Is this true? Is this a problem in her institution alone? What kind of idiot would allow quality assessments to be drawn up by people with no practical experience? If this problem is in her hospital alone, all I can say is thanks for warning us to avoid that institution. If this problem is rampant in US hospitals, then yes, she's probably right. The metrics are probably crap. Ultimately, though, she doesn't say that the metrics are crap. She even admits that

    "The data do offer a snapshot
    of the clinical complexities
    of the disease, the challenges
    posed by our patients’ cases, and
    the limits of how much we can
    alter a disease that is affected by
    so many variables. And they could
    highlight fixable systemic impediments
    to good care."

    So, finally, she comes to the point you've made. She says the data should be pointed at the institution, that she can see changes she could make to improve her patients' outcomes but the changes are all impossible under the current CPT and ICD-9/10 billing practices. She also says that the only positive effect the data can have is to weed out the occasional incompetent, which really is not the point of the data. She finds it demoralizing, she says. She doesn't find the evidence compelling (yet, the two studies she cites disagree with her).

    Maybe she's right. Ultimately, I'd need to see a comparison with her colleagues. If any of them show an improvement under the current metrics, maybe she's wrong. Maybe one doctor can make a difference. If you've read the Gawande article I cited, you'd know that the methods he reports were the results of one man's change in standards of practice.

    So, even if she didn't, as you contest, intend for her article to read as just another, "trust me; I'm a doctor." Her straw man argument about smart, well-intentioned medical professionals seems to say that, but it's not the main point. As for the main point, I'm not convinced by that either. Yes, perhaps the hospitals shoud be publishing overall metrics, but I don't see any reason to stop measuring individuals unless they have absolutely no autonomy. I know doctors in the US work within pay-based constraints, but I wouldn't agree that they have no autonomy.

  6. Thanks @medicalbilldog. I think we are reaching some understanding here and are not as far apart as it initially seemed.
    Sadly not all doctors are good. We have bad ones here in the UK too. But I am a little more optimistic overall than you. Not everyone is like Julian Tudor Hart who could have had a much more luxuriant lifestyle working in leafy suburbs with a healthy income from private practice, rather than working in a poor "valleys" community. Maybe he is looking at his colleagues with the same rose-tinted glasses that Ofri wears when he says that healthcare still operates essentially as a gift economy in the UK. (It was partly for that reason that I included the @burbdoc quote).

    Next, you ask many questions which only Danielle Ofri can answer- and I'd like to see those answers too. What is the nature of that data? Is the paragraph you quote what does happen now or what might happen? I read it as being aspirational.

    Next, of the two studies she cites, one is in fact an opinion piece so it adds no evidence. The other is the systematic review (Shekelle is one of the authors). I should have made clear that the evidence isn't just unclear about whether individual performance measures improve quality, no studies have been done at all!
    "We identified no published studies of the effect of publicly reporting performance data on quality improvement activity among physicians or physician groups."
    So we don't have one iota of evidence that this measure might work.

    And lastly, you make the appoint about autonomy. You probably were posting this comment as I was adding the edit with the AEI symposium youtube. They make the point that doctors do have autonomy and team-based care is just a myth. But interestingly they also mention individual measures which are not about outcome but about process... eg shared decision making. I think that is a really interesting point and gets to the heart of Ofri's point about what are the right things to measure in an individual doctor.

    Thanks very much for replying. I appreciate it!

  7. As ever Ann-Marie you have raised a number of points here about both measuring healthcare provision and health of the individual. Clearly there are many models of care emerging and I'm sure many e-patients (empowered in this context) see their medical practitioners more of a coach/leader rather than just a fixer. I was struck (and I can see how many people are inspired by) Dr Julian Tudor Hart, the sentence that resonated was that the doctor cared that they are human beings.
    There are many stakeholders currently trying to appraise the value of components of the health service. With the emergence of GP commissioning I do worry that further pressure will be borne on GPs to perform. Whilst its right that GPs are accountable it is clear to me that having the right measures, measuring in the right way is paramount. Furthermore we've seen this in many industries that just watching the speedo to find out whether you are on course usually crashes! The design of these services is complex. Asking/persuading people to change habits, empowering people with information to make better micro decisions may seem easy to the Latte Guardian community, however buying 5 donuts rather than 1 apple may give more overall short term utility for a single parent stuck indoors with a young children whilst surviving an illness! The ability then of the medical practitioner to lead a community will be key.
    My further worry is how these systems will be gamed in order to show outcomes from the organisation, which you show convincingly in your blog.
    I would really like to see how communities could cocreate a measurement system of health where they show how healthy their community is and the health improvement that they have made.

  8. Sorry for the delay in publishing this Mike- my holiday got in the way, I think!

    I have been very interested in community participatory research in the past- have you came across much work like this?

  9. I appreciated your perspective very much. There are two articles which I have found very helpful in my thinking. One is Dr. Atul Gawande's January 2011 New Yorker piece - The Hot Spotters - Can we lower medical costs by giving the neediest patients better care. It may be possible, even among those who have little. It requires thinking differently about how to solve problems. Our health care system, whether US or NHS, as it currently exists may not be able to address the poverty issue which impacts health outcomes which is likely more prevalent in Dr. Ofri's practice.

    The other is from the book Switch - How to Make Change When Change is Hard - and looking for bright spots. Though I understand that all physicians don't wish to have report cards and that outcomes should be at the level of the institution, isn't the institution outcome the aggregate of the individual doctors? Even within Bellevue / NYU there may be doctors, despite the system and societal challenges they and their patients face, who have better clinical outcomes - the bright spots. Like the excerpt in the book, even among impoverished villagers in Vietnam, a few families were able to raise healthy children despite the challenges they all faced. The key is learning from those positive deviants and insuring everyone knows.

    This only happens if one believes that it is possible. Yes, great clinical outcomes and bedside manner are not mutually exclusive. If one does not believe solutions exist or that it is possible, then you simply see what you want to see.

    Great post.

  10. Dear Dr Liu,
    Thank you very much for your comment and I will read these pieces with interest. I guess UK general practice is more analgous to the 'medical home' in the US. I don't have a personal list of patients who I help with managing their diabetes. They see the practice nurses often and possibly a diabetologist in a community hospital, as well as me (or my colleagues). So it would be difficult to even say who were 'my patients'.
    But I agree we need to look for the bright spots. A few years ago I considered doing a PhD on quality improvement in diabetes and wanted to explore what one could learn from the practices that had the best care in the area, that could be used to help those with the poorest outcomes.

    So I think we are on the same page! Many thanks again.


I am reintroducing word verification to cut back on spam posts. I'm sorry if you find it frustrating,