Investigating Andy: the MMR scandal
The title of my talk for the HealthWatch 2011 annual general meeting was, “Regrets? I have a few—Inside the MMR investigation”. Being honoured with an award seemed like a moment to go off-script, and after more than a year of banging back and forth to give lectures in the United States, and dashing around Britain to “Skeptics in the Pub” meetings, I’d gotten a little jaded with my standard “Fixing the Link” PowerPoints. So I took a chance and said something more personal.
My first regret was a failure, years ago, which had nothing to do with vaccines. In 1990, which Nick Ross reminded everyone at the meeting was the year that HealthWatch was launched, I was ensconced on Potrero Hill as The Sunday Times man in San Francisco (don’t ask how I pulled off that stunt). And on a sunny day, across the Bay at UC Berkeley, I interviewed Dr Peter Duesberg. This was (and still is) the man who said (and still says) that HIV is not the cause of Aids.
At the start of the 1990s, there were big gaps in science’s grasp of how the retrovirus could do its mischief. Duesberg, at the time a distinguished retrovirologist, was onto them like flies on sheep. His analysis was fascinating, and I took a pile of notes, while we chatted at a campus picnic table. But when I solicited his views on the “real” cause of the pandemic, he seemed unconvincing, even irrational. So, to my lifelong regret, I never wrote the piece. This is among my more painful hindsights. Agree with him or not—and that wasn’t really my business—what he said, back then, was worth a thousand words in The Sunday Times. “Boffin thinks you can’t catch Aids.”
But, as I say, I didn’t write it. Couldn’t be bothered, to be honest. Which left a modest gap to be filled by someone else. Newspapers, like nature, abhor a vacuum. And in the office from which I’d departed to write dispatches from America’s roadsides, was a medical correspondent under pressure. He’d been advised by our managers to leaven the usual diet of “new treatment offers hope” reports to “get stories like Deer’s” that cause a fuss.
Within months of my failure, he waded in to the HIV issue with a notorious crusade. It was full-blooded denialism, but was allocated pages and pages by the then editor, Andrew Neil. And, bestowing on Duesberg the public credibility of a world class newspaper, it may well have fed into what was to occur in South Africa after 1999: president Thabo Mbeki’s denialist health strategy, which brought death and suffering on a horrifying scale.
If only I’d written a few well-picked words, conceivably things might have turned out differently. My colleague’s first pitch wouldn’t have survived the editor’s Tuesday conference. “Deer’s done that already,” Neil would have snapped.
Why is this relevant? Well, because journalists at the sharp end are bound to consider what plays out from their work. At the top, national editors are often supremely ideological, constantly thinking about the upshot of their coverage. But the most successful set this aside for the imperative to sell: to tell stories that people actually want to read. It’s an anarchic part of our culture, but is near the root of democracy. You tell stories, like climbing mountains, because they’re there.
My mistake with Duesberg was to evaluate his theories, as if I was the arbiter of his views. And when Andrew Wakefield came along as a topic of inquiry1 I made sure not to make the same error. I was determined primarily to be a servant of the story, which, for all my hard work, told itself. “Excuse me, ma’am, did you see what happened?” Like a train crash, or a robbery. Get the facts.
Although I didn’t have time for all this in my talk, if I’d had the choice, I’d sooner have revealed proof that MMR indeed causes autism. I’d have got more space, won more journalism awards, have been accorded more spots on TV, and earnt more money, than I did revealing the Wakefield scandal. Overshadowing these, moreover, and worth more than them all together, I’d have accomplished something remarkable for children. If vaccines caused autism, that’s important information to head off another kind of pandemic.
The truth I found was a lesser story, although I think one of consequence. People have told me it’s a justification for a life. But here’s another regret: the (perhaps understandable) spin which my editors sometimes sought to introduce. They were always going on about measles outbreaks and whether MMR causes autism or not. I don’t belittle those questions—which I’ve researched a great deal— but they were never what drove my inquiries. I simply wanted to set out what it was that happened: in short, what Wakefield did.
Recently, I was invited to lunch by the legendary reporter Bruce Page, who led The Sunday Times campaign over Thalidomide. He made me wince with envy at how he said he completed the investigative part by pulling research on the drug from the library. And he reminded me of an old adage, passed to him by someone else, that there are really only two stories in our tradition. The first, he said, was, “We name the guilty men.” The other: “Arrow indicates defective part.”
Mine was the first kind (the second would have been the science): nailing an all-singing-and-dancing charlatan. For my HealthWatch talk, I showed a few of my slides about how my investigation moved forward over the years. I put up tables which compared what the BMJ has calledWakefield’s “elaborate fraud” with the Piltdown Man hoax of 1912. And I presented a series of slides concerning others in the saga, which cause me worry about the integrity of medical science.
These included the graduate student who knew something was amiss, but had nowhere to go with his concerns; co-authors taking credit from a high-impact publication; and a reviewer who was disposed to believe the MMR data because of an old-boy connection.
Back to measles: of course I don’t minimise this disease, which is sometimes fatal and, more often, brain-damaging. Nor do I underestimate the importance of the science that lies behind vaccine safety. But alongside these issues, theWakefield scandal says much about the potential for wrongdoing in research. It reveals the ease with which misconduct can be effected in academia, and the difficulty, or reluctance, to tackle it. Bill Clinton once called the 21st century the “century of biology”. And after what I’ve seen can happen, just looking at MMR, I fear another Thalidomide-style catastrophe.
HealthWatch, to its credit, often touches on such issues, but I must admit to a small regret about some emphases. As with the amazing Skeptics in the Pub2 phenomenon, which has sprung up in the last few years, there seems to me to be a sporadic obsession with, say, rubbishing homeopaths, or brandishing the Daily Mail’s contradictions. By all means promote critical thinking, and call out absurdity, but, from my point of view, I’m not sure that it’s here that health requires watching most intently.
As I said in my talk, I think that—for young people especially— trying “complementary” medicines can be a rite of passage. They may be a way in which some explore their route to rationality, rather than a gullibility from which we must rescue them. To borrow a related thought from the former politician Denis Healey: “If you’re not a communist before you’re 30, then you’ve got no heart. But if you’re still a communist after you’re 30, you’ve no brain.”
People often ask about the abuse I get, having had the temerity to take on Wakefield. Here’s an example I quoted from my hate mail inbox, which produced laughter enough to delight any speaker. It was written by a lady. She wrote in bold capitals which, to avoid shouting, I’ve dropped to lower case:
“F... you. You are the scum of the earth. You are just trashing people that are totally dedicated to find cures for a horrible disease. Protecting your selfcentered ass, trashing others for selfgain, or protecting your buddies at a corrupted pharmaceutical industry. Well, now I’m trashing you right back. You are trash. We’re even.”
Upset, me? Nope. Understandable? Entirely. Wakefield has been encouraging people to be afraid of vaccines for 14 years. He was doing it at autism conferences long before I got involved. At one of these, I should point out, he gave his infamous “callous disregard” speech,3 about buying blood from children at a birthday party. In another speech, you won’t have heard, he betrayed an apparently sincere whistleblower,4 who voiced his vaccine safety concerns. If you watch the second video and listen to the audience, you’ll understand how Wakefield works the crowd.
And this brings me to the final regret which I raised at the meeting. It concerns people like the lady who wrote that email. That woman was suffering. She was clearly in pain—and a pain not inflicted by me. For years, some parents of children with developmental disorders have been led to believe that it was their own fault for not listening to Wakefield that a son or daughter is autistic or disabled. That can deepen a sense of guilt, from which springs a vulnerability which in some cases festers into hatred.
But my regret isn’t vicarious. It’s not mere bystander sympathy. It’s that I sometimes find my compassion feels strained. There was a day, for example, during Wakefield’s GMC hearing when a strange collection of people gathered outside. One was an overwrought woman bearing a placard with a picture of her young son who had undergone radical bowel surgery. But he wasn’t even involved in the Wakefield case and had been diagnosed at another hospital with a food intolerance.
Wakefield’s disciples, however, pretended that this boy was part of the case. It was a disgusting deception, for which those who knew should be ashamed. But I wondered, most uncharitably, about that mother. Why did she allow herself to be used in this way? And what about those who used her?
I don’t fear the abuse of those victimised by Wakefield. But I sometimes feel my reactions are misplaced. It’s easy to dismiss these suffering people as cranks or idiots. It used to happen in the controversy around Duesberg. But it’s a greater accomplishment to stay focussed in compassion, and on the needs which should be brought to the fore. There’s no right response to hatred in hatred. I need to try harder next time.
Brian Deer, Journalist
In addition to receiving the 2011 HealthWatch award, Brian Deer was named specialist journalist of the year in the 2011 British Press Awards. Brian Deer can be contacted through his website, http://briandeer.com/
Further reading from Brian Deer’s website on topics mentioned
Prof Raymond Tallis: for his thoughtful promotion of evidence-based medicine
Anecdotes, Data and the Curse of the Media Case Study
Let me say straight away how honoured I feel to here tonight to be talking to you and to be receiving your annual HealthWatch Award. This came totally out of the blue – which has very positively altered my attitude to the blue as a result. The blue is more usually a source of nasty surprises – the frozen block of urine dropped from a 747 – than nice ones. I feel not only honoured but distinctly undeserving, particularly when I think of the previous winners of your award. I am thinking of writers whom I admire enormously such as Geoff Watts and Polly Toynbee and the witty and courageous John Diamond; and great clinical scientists such as David Sackett and Richard Doll and Michael Baum. And as for Ben Goldacre – the recipient of last year’s award – what can I say, except that every time I read one of his columns, I punch the air with delight until it is bruised. When get my weekly fix of Bad Science all my exasperation evaporates. So thank you Ben, and thank you all my predecessors, for strewing a little broken glass in the path of charlatans, mountebanks, loblolly men, and all those alternative practitioners whose loud noises have threatened at times to drown the softer-voice of hard-won scientific truth. And thank you for inviting me to join their select company. I would like to add my congratulations to the students for their awards. ‘O brave new world that hath such people in it…’ Thank you also Nick and David and you other HealthWatch stalwarts who over the last 19 years have kept HealthWatch as a major force for busting quacks and standing more broadly for reason and evidence against the forces of unreason that flourish best in an evidence-free zone. I know you do this purely because you are passionate about the things David talked about in his address. As Nick, said, you don’t get anything back – though 50,000 – 70,000 hits is very impressive indeed.
I don’t know whether my frontal lobes are starting to lose their grip – or perhaps it’s my latest detox at the health farm – but I am starting to feel a little more optimistic and have the feeling that the quacks, at least in the UK, are perhaps on the retreat; that this has been a pretty good year for quack-busting. Perhaps it’s a matter of contrast-enhancement because we did start from a pretty low point when in September 2006 the MRHA (Medicines and Healthcare Products Regulatory Agency) gave homoeopathy a boost by licensing its products on the basis not of medical evidence but something called provings. I must admit, I had never heard of provings up till that time and this is what they are.
They’re conducted on volunteers in a reasonable state of health (provers) who generally have no knowledge of what substance they are taking. Doses are repeated until provers start to experience symptoms of a change in state. The provers record everything they experience, whether physical, emotional, mental, or even spiritual, as long as the change in state persists.
At the end of the proving, all the records are collated and compared to find the symptoms, states of mind, feelings and experiences the provers have in common and which can be reasonably attributed to the emerging signature resonance of the substance. Wendy Howard Homoeopathic Practitioner quoted in The Skeptic Express, 2006
It would be an interesting exercise for 6th Formers or even 12 year-olds to be challenged to say why this methodology is flawed. Anyway, the outrageous decision by the MRHA – which seems in this instance to have been influenced by populism rather than by science – galvanised opposition and the annus fairly mirabilis has resulted, with homoeopaths, and herbalists in particular, having a fairly rough time of it.
Many of you will be aware of the campaign fronted by Michael Baum, David Colquhoun, myself and others in partnership with Sense About Science that has had scored some palpable hits against homoeopathy. The campaign to persuade PCTS not to fund homoeopathic treatments on the NHS has attracted huge publicity and, what is more, prompted many PCTs to remove such treatments from their bill of fare. The latest scalp is the withdrawal of funding by the Tunbridge Wells PCT from their local homoeopathic hospital which consequently is likely to be mothballed. There are five Homoeopathic Hospitals in the UK: one down 4 to go.
And then there has been increasing awareness not only of the uselessness of most herbal remedies but, indeed, of their power to do harm. While it is possible that there are some benefits from some herbal remedies – after all, if you fire enough buckshot in the dark, you will sooner or later hit a mosquito’s legs and then say that that was what you were aiming at all the time – there is, as we all know, no evidence that the kind of mixtures dished out by herbalists in their shops do any good. There was huge publicity a couple of weeks ago arising from the study by Edzard Ernst – another person who I am very proud to have as a predecessor in the receipt of this award and who has, I believe, done more than anyone to force CAM to engage with the scientific method – recently published by the Postgraduate Medical Journal. This has provoked widespread demand for properly regulation of these medicines and of those who flog them. Let us hope that this time the MRHA doesn’t fumble the pass.
In short, much grounds for rejoicing. It is as if the nation’s serum Carol Caplan levels are falling and the parts per million of bullshit in the collective consciousness has dropped a notch or two. Now, if this speech was written by a computer, rather than written on a computer, and it had predictive texting, it would next print out the words – ‘There are, however, no grounds for complacency’. Or, as your Chairman, David Bender, no computer he, observed in his report last year – ‘We’, that is to say you, ‘are still needed’. And that’s certainly true. Unreason is always ready to break out, or to return, for reasons that go to the heart of what I shall be talking about this evening and, indeed, what it is about quack medicine that makes it more significant than might appear at first sight. Even as we speak, rejoicing in a year in which in the UK at least quackolysis has definitely had the upper hand over the quacks, herds of bulls are massing, ready to evacuate the contents of their ample colons.
Colonic material of a Taurine provenance is in infinite supply
One of the reasons for this is, of course, the way science and non-science are presented to the general public by the media. And the problems are epitomised by what is referred to in the title of my talk: the curse of the media anecdote. Much discussion of medical care – both orthodox and useless – tends to be revolve around, or even to be tethered to, anecdotes.
The media love anecdotes because those who work in them believe (usually correctly) that their customers do. Anecdotes – about a new miracle cure, a drug that is not being made available on the NHS, or the side effects of treatment, or some environmental hazard - sell product: they help to boost circulation, increase audience rating figures, and so on. Data, on the other hand, which take us towards the truth about these things, are less popular.
Anecdotes, however multiplied, do not point the way to reliable knowledge. As the aphorism says, ‘The plural of anecdote is not data’. For an anecdote does not even count as a datum. It may prompt a search for data, as when a chance clinical observation makes someone wonder whether A causes B, or C may have therapeutic benefits in condition D.
These deficiencies count less than they should. For anecdotes, unlike data, have a human face: they are closer to gossip, which is a universal human pastime, and to the things that shape belief formation in everyday life. This is something to which I will return presently. Scientific data - for all that science is one of the most human, as well as the greatest, of human achievements – lack this human face. Their very strength – that they are unattached to persons and personalities – is a PR weakness. At best, data belong to ‘the science community’; worse, to ‘the scientific establishment’; or, worst of all, when they are adopted by, and guide, those who make policy on our behalf, ‘the authorities’.
Let me illustrate the accursed power of the anecdote with one of the most potent examples of recent times: the story told by Jackie Fletcher, the founder of Justice, Awareness and Basic Support or JABS, a group devoted to campaigning on behalf of parents who believe that their children have been damaged by immunisation, most notably by the MMR triple vaccine. Her story is a tragic one. Nearly a decade after the MMR furore broke, it is still news and still being used to influence opinion.
In 2007, The Daily Mail devoted yet another article to her terrible experience, accompanied by a poignant picture of her and her son, ‘who developed autism after being given the MMR jab’:
The change in their bright, loving toddler was sudden. Ten days before his first fit, Robert had been vaccinated against measles, mumps and rubella (MMR). Unlike his older brothers, who had been given the jabs in single doses, Robert had been injected with three viruses at once.
To the Fletchers, the connection seemed obvious..
The Fletchers’ experience is unimaginably terrible: the transformation of a perfectly normal toddler into a child with autism, severe epilepsy, and incontinence, confined to a wheelchair, speaking only the words he knew as a baby, and requiring to be washed and fed. In their case, grief is envenomed with (unjustified) guilt. Mrs Fletcher says, ‘Deep down, I feel that I have caused all Robert’s problems by taking him for the MMR jab and I’m trying to fix it for him and I can’t’.
One would have to have a heart of stone to challenge Mrs Fletcher’s interpretation of what happened; her attempt to make sense of a senseless catastrophe; and her endeavour, through campaigning to protect others’ children against what she sees as a threat, to come to terms with a guilt she feels which, though unwarranted, is real. An anecdote such as this, set among data, is like a queen ant in a colony of drones. Stories have a power that bears no relation to their number or validity. The very fact that those who invoke anecdotes in support of their theories are so convinced by them, adds to their cognitive lustre.
Notwithstanding Nietzsche’s assertion that ‘Convictions are greater enemies of truth than lies’ , certainty, repeatedly asserted, and the anger and grief with which it is charged, increases the impact of the claim embedded in the story, on the reader or listener. When we are invited to assent to an empirical truth in an area in which we have no expertise, we are more inclined to do so if we identify with the person who is purveying it. It is a way of showing solidarity.
The contrast between the media anecdote and the way data are presented in scientific meetings or papers – where there is no reference to the feelings (anger, grief, conviction) of the authors and much concern about sources of error and uncertainty - underlines how disadvantaged the latter are. Of course there are passions behind science – who otherwise could stand the utter tedium of data gathering and learning the techniques of doing so reliably? – but the passions of themselves cut no ice. Indeed, they would be grounds for suspicion from those who are used to what Paul Valéry described as ‘the elementary courtesy of proof’.
Anecdotes also have the ability to attract other anecdotes: when the Fletchers took their stricken child to his many hospital appointments, ‘they encountered other parents who said things like “our child was fine until MMR”’.
This ‘multiplier effect’ is vastly scaled up when the anecdote comes from a celebrity in the media, or is endorsed by such a celebrity. A face that is everywhere is the vehicle for an opinion that is everywhere. The same story is replicated in a million places and has the impact of a million data. In a society where achievement in one area (for example, combining being a moderately competent actor with having an eye-popping figure) makes one an authority in many others, the views of celebrities are the equivalent of the union block vote.
From the point of view of the media, there are secondary benefits of the human story. When data falsifying the Anecdote Based Theory (ABT) start to look overwhelming, other, secondary, anecdotes emerge that nurture the primary one. There is talk of conspiracies, cover-ups by the establishment, and we are reminded of persecution of people (of genius and courageous) who have heretical views.5 The campaigners on behalf of ABT and their supporters look like plucky loners facing a mighty monolithic opposition. Powerful data are discredited simply because they originate from sources that are considered to be on the side of the Big Boys: the scientific establishment, the politicians (who have their own reasons for hiding the truth), and others, such as Big Pharma (who have vested interests). Even the sharing of these data with the public can be presented as an act of oppression. At the height of the MMR controversy, the Department of Health was accused of ‘bullying parents with facts’6.
The power of the anecdote is enhanced by the fact that, once it is in circulation, it looks like an entire half of the overall picture. The media claim that they are required, by their commitment to balance, to give equal hearing to both ABT and science. So Jackie Fletcher talking about her son Robert and giving her untrained critique of the epidemiological data is allowed as much airtime as a scientist bearing news of millions of data obtained under carefully controlled conditions. Given that controversy sells product, maintaining the myth that there are two equal sides to the story is desirable in itself.
The notion of balance is one that should be treated with care, the kind of care that it does not always, or even usually, receive in the media. If applied uncritically, it can result in giving data-free believers in the flatness of the earth the same amount of air-time and column inches as those who support the well-attested notion that the surface of the earth is curved. The Royal Society, quoted in an email by that doughty fighter against junk science, David Colquhoun, has made this point:
When obtaining specialist advice from experts, it is not diversity of opinion that should be balanced, but the weight of opinion accorded to various strands of scientific advice within that diversity. In the name of ‘balance’, the media invariably present opposing views on each side of an argument, regardless of the relative scientific merit of those opinions.
The price of so-called balance that does not take account of the ‘relative scientific merits’ of the opinions being given equal air-time, may be very high indeed. Most people in this room will be aware of Peter Duesberg the maverick molecular biologist who denied the link between AIDs and the human immunovirus. His views were largely discounted in the West and have, thank God have had no influence on science policy, public health policy and those scientific research programmes which have led to the discovery of highly active antiretroviral drugs which have transformed AIDS from a fatal to a chronic disease. Elsewhere, mankind has not been so lucky. Thabo Mbeki’s catastrophic AIDs policies have been hugely influenced by Duesberg’s views and the result is a pyramid of corpses sufficient to block out the sun.
Fortunately, the MMR controversy has a less calamitous outcome. When it was obvious that the link between MMR and autism was unsustainable, and that it never had any basis, some of the media showed signs of contrition. Not all of the press, of course. Melanie Philips still continues her one woman campaign against MMR and combination vaccines that she considers lethal. If her views were listened to, of course, and had any influence on public health policy, the result would be a catastrophe on the scale of what has happened in South Africa. The habit, however, of giving appealing individuals with their moving stories at least as much credence and coverage as unappealing data, of preferring faces to graphs, and vox pops to statistics, does not seem to have been kicked. The recent judgement by the National Institute of Clinical Excellence that certain drugs would not be cost-effective in the treatment of early Alzheimer’s disease on the basis of the evidence so far was dominated by human interest stories.
I found this particularly exasperating because (to declare an interest) I was a member of one of the NICE appraisal committees for three years. This, quite labour-intensive, unremunerated role brought with it two rewards: the knowledge that one was contributing in some small way to ensuring that the finite resources of the NHS would be spent in such a way as to maximise health gain; and the satisfaction, indeed pleasure, in participating in a very high-level, supremely well-informed discussion about the costs and benefits of medicines that took into account basic science, clinical evidence, health economics and social value judgements. Very little of this appeared in the media coverage. The World at One gave extensive air-time to a woman whose father had been given the drugs early in his disease and was ‘convinced’ that he would not have done so well had he been denied them.7
This example illustrates something that is common to much anecdote-based discussion of matters of public interest such as the benefits or harms of medical treatments or the impact of some environmental factor on health: anecdotes privilege the visible over the invisible. In the case of MMR, the millions of children who had been saved from harm by the vaccine were given little foreground; in the case of Alzheimer’s drugs, the impact on all other patients that would result from inappropriate allocation of resources to less cost-effective treatments, was hardly discussed.
I want to step back a little and look beyond their immediate consequences to another reason why the media anecdote is accursed. To think about this I want to rehearse some arguments I have had with people about the use of alternative medicine in the UK. I was recently discussing reflexology and it was pointed out to me that it is pretty harmless and that it doesn’t after all cost much. There was very little evidence that those who used it denied themselves the benefits of orthodox medicine. So why are you getting in such a pet? Well it seems to me that alternative remedies and the theories that are used to support them are a major contribution to the moronisation of the collective consciousness and, at some level, a betrayal of one of the greatest achievements of humanity, and so of humanity itself.
The journey humankind has taken to robust scientific knowledge of the world in which it finds itself has been a long and arduous one. For science has required not only that we somehow penetrate the opacity of nature but also that we overcome the prejudices, the intuitions, the assumptions, the convictions, the dogmas that we ourselves place in the path to reliable, usable knowledge. These obstacles are especially stubborn in medicine, not only because its object is the human body, that from which knowledge takes its rise, but also because its concern is with disease, where emotions, particularly those of hope and despair, tend to have the most powerful influence over our beliefs. The endeavour to cure illness is thus bound to be dogged by deception and self-deception. Hence medicine’s status as a parvenu among the sciences: it is, as Lewis Thomas said, ‘the youngest science’. The greatest miracle of scientific medicine is that medicine became scientific.
The contrast with traditional, or pre-scientific, medicine – the medicine that was all that was available for all but the last 100 years of the 300,000 years of H. sapiens sapiens - is extraordinary in many ways. One of the most important is the fact that scientific medicine, unlike traditional medicine, which is deeply self-satisfied with its knowledge and what it believes to be its effectiveness, is driven by an active uncertainty that is sceptical of received ideas and of authority and is continually seeking to improve on the status quo. It is an irony that scientific medicine is called ‘orthodox’ medicine because, unlike acupuncture, or homeopathy, is constantly scrutinising, challenging, overturning its own beliefs. Like the rest of science, it is in a permanent state of self-critique; and the authority of authorities goes no deeper than the facts that they can invoke. By contrast, acupuncture claims to derive its authority from individuals who laid down the law thousands of years ago. One of the most barefaced examples of proud intellectual stagnation must be the fact that homeopathy has not changed since Samuel Hahnemann offered it as an alternative to the poisoning and butchery that was prevalent at the time. Homeopath still rely on his book, first published in the 1790s, to guide their practice. What he spake, they shall do. There could be no more striking example of frozen orthodoxy and authoritarianism.
Alternative medicine has not progressed to anything approximating clinical science because it has not been prepared to challenge its own authority. So-called orthodox medicine, in an act of collective humility, has cultivated a routine distrust of its own practices. This humility has been almost as important in the development of effective therapies as the discoveries in basic biomedical science. The rejection of anecdote-based medicine in favour of treatments tested in randomised controlled trials and re-tested in meta-analyses etc has accounted for the remarkable advances particularly over the last 50 years, and the quickening pace of increases in life and health expectancy. The humility of shaping clinical practice in accordance with the unmanipulated outcomes of therapeutic trials is also connected with another layer of scepticism: that what looks good in theory (irrespective of how good the theory is) may not benefit patients in the real world.
Perhaps the most remarkable facet of this humility is the willingness of doctors to enter patients (with the latter's consent) into clinical trials run by other clinicians. Submission to the authority of such trials means subordinating one's own personal authority to that of other professionals, many of whom (such as statisticians and biomedical scientists) do not even belong to the medical profession. As Marc Daniels pointed out:[for clinicians] to be willing to merge their individuality sufficiently to take part in group investigations, to accept only patients approved by an independent team, and to submit results for analysis by an outside investigator involves considerable sacrifice.
The committment to minimising the play of chance, of bias, or of wishful thinking, is the supreme expression of the self- overcoming that scientific medicine requires. Avoiding beliefs guided by delusive hope, unfounded authority, superstition and plain stupidity, it cultivates an attitude of healthy scepticism towards itself to prevent its practitioners from misleading themselves or their patients. Its permanent strategy of active uncertainty, and the humility this implies, is the distinctive virtue of scientific medicine. In the world outside of scientific medicine, however, humanity has had little time to adjust to this almost inhuman scrupulousness.
Hence the fact that alternative medicine – and more broadly anecdote-based treatments - can still live, and even flourish, at least in terms of the number of customers, and the amount of cash that changes hands, side by side with medicine that actually makes patients better for reasons that are rooted in a scientific understanding of the human body and its diseases. Otherwise, it would seem unaccountable – like an ox-cart at stand in an airport next to a jumbo jet or an Asclepian temple fitted out with WiFi. Indeed, the constant self-critique of medicine and the biomedical sciences upon which it is based is often misunderstood. The self- dissatisfaction and self-transformation - resulting in ever more effective and, for the most part, less unpleasant, treatments - is seen as a kind of weakness. A discipline which is marked by a carefully nurtured scepticism towards itself is sometimes seen as arrogant or in disarray – unlike traditional healers who are proud never to have had a second thought in 5,000 years.
So this is perhaps the true curse of the media-propagated anecdote and anecdote-based beliefs about medical care: they represent a sickening regression to the cognitively primitive state from which medical science, science, and humanity has fought its way out.
So if we are going to free ourselves from the curse of the media-base anecdote, we need to have a major initiative explaining why anecdotes are misleading, why double-blind randomised controlled trials are necessary – how they work and how, over the last 50 years, they have played such a crucial role in the objectively observed advances in medical care.
A good place to start would be cardiovascular disease, in which randomised controlled trials have had almost as great a role in bringing about improvements in mortality as advances in basic biomedical science. A couple of statistics could introduce the topic. Firstly that cardiovascular disease mortality in men of all ages has fallen in 2005 to 30% of the age specific rate in 1968. Secondly, that deaths from coronary heart disease of people under 65 has fallen by 46% - by nearly a half – in the last ten years. People should be asked to celebrate these facts and to consider the role played by basic science in understanding the processes that lead to cardiovascular disease and methods of preventing or reversing them; and the complementary role of robust clinical trials in ensuring that what should work in theory actually works in practice in the real mess of the real world of doctors and their patients.
Perhaps this is idealistic. The notion that the nation’s serum Daily Mail levels might be reduced by broadsheets offering free tutorials on the methods by which science arrives at robust knowledge and that knowledge is translated into benefits for patients is somewhat utopian. And yet I assume that most journalists (and their sub-editors who write those lethally misleading headlines) want to be able to look back on careers in which they have done more good than harm and advanced, rather than held back, the cause of truth. They are aware that anecdotes may be very effective at conveying the profundity of medical tragedies and the human importance of medical triumphs and disasters. But for making decisions about the absolute and relative values of medical treatments, and more broadly for determining policy, they are of no use whatsoever.
And so I would like to offer a few thoughts, which could be accommodated on a flash card, to help journalists deal responsibly with those anecdotes they and their readers value so much. Before I do so, however, I would like reflect a little more on the power that anecdotes have, which they owe to their central role they play in our making sense of the world in which we find ourselves. And this includes the kind of sense of the mediated world that I am trying to make in this talk – which, it will not have escaped your attention, has itself been rather heavily dependent on anecdotes.
Anecdotes are not an unredeemed evil. As we have already noted, medical science itself has been driven by accidental, observations, anecdotes, though that is merely a prompt to inquiry, the place where it begins, not where it ends. But beyond this, human beings have always told each other stories and storytellers have always commanded a hearing not only because we want to know what happens next but because what is recounted strikes us as standing for something that goes beyond the particular instance recounted.
We imagine beyond the events in the story to a whole class of similar events. ‘This could happen to me’ is extrapolated to ‘This could happen to anyone’; ‘This could happen any time’ to ‘This is happening all the time’. We are taught from earliest childhood to see stories as lessons, as parables: faulty induction is part of our ethical education. Identification with the stories that we are told, seeing them as potentially part of our own biography, is key to the development of that imagination which lies at the heart of sympathy, empathy and hence morality.
The power of stories to fuel extrapolation beyond themselves to an imagined series of general cases is greatly magnified by emotions – as Bertrand Russell pointed out in the passage I have just quoted/ for emotions are the means by which the world we think of ourselves to be surrounded by is illuminated. Angers – generous and mean-spirited – compassion, the sadness that we feel from entering into the particulars of another’s tragedy – these are the great drivers to an extrapolating imagination that takes us beyond what we know of the world to what we suspect of it; supporting that part of our world-picture – by far the greater part – which is not directly vouched for by our own experience. It helps that key figure within the very motor of our cognitive activity, the Professor of Data-Lean Generalisation who helps us to navigate through the world we live in, and to feel that we in some sense cognitively possess it, by being able to command a viewpoint on it. Emotion is the basis of that confirmation bias which is necessary to put together for ourselves a coherent narrative about the world or part of it. This journey, via emotion, from anecdote to world-picture is both our curse and our glory: it lies at the heart of our wider sense of responsibility – the desire to make a positive difference – and, equally, at the root of those generalising hatreds – racism, ‘classism’, all forms of ‘otherism’, which have made the world such an unpleasant place for some and potentially a dangerous place for all. The observation bias that is built into it, and the desire it has to confirm only itself (so that it will be satisfied with very few instances to support a statement of vast generality) enables us to be fluent about things of which we know little; to summarise, in the lounge bar of a pub, the character of an entire race of people in five ill-chosen words.
And it is round about here that we find the true greatness of great science. Like gossip and idle curiosity, it is manifestation of the fundamental human need to make sense of things, to build a picture of the world that vastly exceeds our experience so that we can understand and control it and, so by making the world a safer or more predictable place, to control our destinies. And at the same time, it does not allow the passion to determine what it sees.
It has, in short, managed to elevate its passion for understanding and for reliable knowledge - that will permit what Francis Bacon called ‘the effecting of all things possible’ – above those passions that normally permit us to transcend our limited individual consciousness and fuel our imagination. This is due in no small part to the fact that it is a collective passion that engages all mankind - an average issue of a leading scientific journal will have contributions from scientists from a dozen or more different countries and countless different backgrounds; that it is regulated by rules of inquiry that have been forged in bitter experience and bitter disappointment; and that takes place in the context of institutionalised scepticism.
This separation of passions is lies at the root of the ethic of science and is linked with what Claude Bernard advised as regards imagination: ‘Put off your imagination, as you put off your overcoat, when you enter the laboratory. Put it on again, as you put on your overcoat, when you leave’.
In the end, we must always begin with notions that are insecurely founded on a grossly inadequate database. Kingsley Amis once referred to the way we erect great belief systems on the basis of very little data as ‘the inverted pyramid of piss’ This is perhaps a little harsh, as the necessary starting point for belief formation in both everyday life and in science. In everyday belief formation it tends to be the end-point as well, whereas science moves on from there. While acknowledging the central role of stories, of anecdotes, in awakening us out of our own narrow consciousness to awareness of a world outside, we need equally to acknowledge that an understanding of the world rooted in anecdotes is cognitively underdeveloped. To give anecdotes too great a prominence in the debate on science, and medical science, and science policy, and healthcare policy, is to muddy those waters that it has cost scientist huge effort to clarify.
So, ye journalists, by all means offer your readers anecdotes but do not for one moment deceive either yourselves or your readers into thinking that this is science reporting or that one can move directly from anecdotes to robust generalisable knowledge or recommendations for application of science in healthcare or anywhere else or for the development of health science policy. Here, then, are some reminders that might be useful for journalists who find that they are reporting on science, particularly science as it applies to healthcare:
a) An anecdote is not a scientific datum, though it may justify a study to acquire data. Do not confuse experience – however intense and harrowing – with expertise.
b) The journey from a single observation to reliable, generalisable truth is long and arduous and full of pitfalls. Here are some of those pitfalls (all illustrated by the MMR furore:
Subscribing to the fallacy ‘Post hoc ergo propter hoc’: Event B happened after Event A therefore Event B is due to Event A.
Noting an association between Event A and Event B and concluding that the entire class of A-type events and B-type events will be associated. The propensity for confirmation bias means that you will preferentially notice, remember or be inclined to report, only those occasions in which there is an association.
Moving directly from observed association to imputation of causation. Both Event A and Event B may be the effects of some other event or factor.
Assuming that, because there is a causal link between Event A and Event B, or a certain factor seems to be operating in one instance of Event B, that the cause or factor is operating in all cases, or that it is the only cause or factor relevant to Event B.
These are hard lessons that mankind has learned reluctantly on the road to useful general truths. Do not be proud of unlearning them out of solidarity with folk common sense. Science in many respects is a triumph over the limitations of interested observers and often over common sense. Ask yourself again and again How typical is the story of the larger picture? How representative are these events? From what population of events are they drawn? For God’s sake give me denominators!
c) When discussing the beneficial or adverse effects of therapies, or more broadly technologies, consider the whole picture and the whole population. For example, even a treatment that has an adverse effect in some recipients may overall have a beneficial effect in the population. The anger of those who feel the treatment has injured them or their loved ones should prompt review of the evidence of overall benefit but should not count as evidence in itself.
d) When science fails to support Anecdote-Based Theory, do not assume that expert opinion:
is blinkered, or hobbled by ‘orthodoxy’. Science is in state of permanent reformation – that is how it advances so quickly;
is simply part of a lump called ‘the establishment’ that routinely conducts witch hunts against dissenters;
is in hock to the government, to Big Business or Big Pharma – or to other vested interests. Remember that those who point the finger at others’ vested interests – including you, the journalist - have vested interests themselves..
e) When ‘balance’ is sought, it is important that the two views given equal airing should have equal weight. One advocate of ABT, however impassioned, does not deserve as much of a hearing as one million data presented by someone who has acquired them in a way that has survived peer review by experts (who are in many cases professional competitors and on the lookout for errors). The 100% certainty of the anecdote-based theorist should not also be used to make the refusal to claim 100% certainty in science a weakness.
These principles should be known and embraced by all journalists – not just those charged with commenting on science but anyone whose comments may influence the views people may have about science, the application of science, science policy or the larger role of science in society. Indeed, it is outside the columns of science correspondents that the greatest damage to the collective understanding is done. The excellent work by people such as Geoff Watts, Georgina Ferry, Palab Ghosh, Mark Henderson is often overshadowed by the loud and more widely read ignorant certainties of characters such as Melanie Philips. Anyone who is going to make a comment about treatments for breast cancer in the General Features or about the heart attacks in GHQ should, at the very least, have some idea of the approaches to arriving at robust general truths that are employed in science; at the very least they should read Peter Medawar’s Pluto’s Republic, Lewis Wolpert’s The Unnatural Nature of Science, and Theodore Dalrymple’s Mass Listeria.
At any rate, media writers with a conscience should not collude with the cognitively primitive state in which we form most of our everyday beliefs. They should remember Bertrand Russell’s observation that while ‘popular induction depends upon the emotional interest of the instances, not upon their number’, reliable induction depends on numbers not emotional interest. If journalists give excessive weight to individual anecdotes they will trigger unnecessary worry and suffering, and sometimes cause diversion of resource from places where it would be of more use. Vox pop or even vox pop star is not vox dei.
Anecdotes are the first step on the road out of individual subjective consciousness to collective, objective knowledge. They are, however, only the first step. Science has gone far beyond this. If you are going to intervene in a debate on matters already subject to scientific investigation, you need to grow up out of your anecdotage. Otherwise journalism and the anecdotes it loves will get in the way of the collective clarity of thought that is going to be essential if we are to deal with the great challenges ahead of making this planet a safe, and reasonably comfortable, place for those who inhabit it.
The price of putting anecdotes in the driving seat is very high indeed. Uncritical focus on ‘the human story’ at the cost of ignoring or underplaying dull scientific facts may have inhuman consequences, as has been demonstrated by the recent return of measles and the first death from this disease for 14 years in the wake of the unfounded scare over MMR. For this I do not blame poor Jackie Fletcher but those journalists who, out of laziness and perhaps venality, gave her single experience and her heroic but misguided mission, more credence than it deserved.
So, ladies and gentlemen, there you have it: the curse of the media anecdote. It remains for me only to thank you again for the fantastic work you are doing, you dedication to quack-busting and this terrific honour. I shall display my award with pride, though I feel somewhat undeserving as well as proud.
- Gill Swain www.dailymail.co.uk/pages/live/articles/health/healthmain.html Last updated 11th July 2007.
- Friedrich Nietzsche The Gay Science Book 1, Section 2 (1982).
- Paul Valery Monsieur Teste translated by Jackson Matthews (London: Routledge & Kegan Paul, 1973), p. 65.
- Swain op cit
- I give examples of this in relation to MMR in Hippocratic Oaths. Medicine and its Discontents (London: Atlantic, 2004. See ‘Convictions’ p.109-127.
- See Tallis, ‘Convictions’ ibid. This chapter also covers the epidemiological evidence that demolished Andrew Wakefield’s claim of the link between MMR and autism.
- BBC ‘World at One’ 13th August 2007.
- See Lewis Wolpert The Unnatural Nature of Science (London: Faber, 1992)
- Quoted in Wolpert ibid p.19
- ‘First measles death for 14 years’ BBC News 24 3rd April 2006
Sir Iain Chalmers: for his critical contribution to EBM
Presenting the 2009 HealthWatch Award, Nick Ross said, “Iain Chalmers has saved more people’s lives than anyone else I can think of.” Iain Chalmers, editor of the James Lind Library, has for the last 30 years championed the need for health professionals and patients to have access to unbiased evidence on which to base clinical decisions. His talk on the development of fair tests of treatments in health care was illustrated with examples going as far back as 1500BCE, all taken from the James Lind Library’s extensive and publicly searchable online archives.
Iain Chalmers has devoted the last 30 years to efforts to help ensure that health professionals and patients have access to unbiased evidence on which to base their treatment decisions, most famously through his work as one of the co-founders of the Cochrane Collaboration. He was at the first ever HealthWatch meeting back in 1992, and since then he’s been a friend and valued critic, and prepared to make a pointed comment if he ever believes HealthWatch has failed to apply to itself the standards it expects of others. “It’s important to be even-handed, for us all to be judged by the same rules. If we depart from this, then we’ll be open to the accusation of double standards,” he said by introduction to his talk at the 2009 HealthWatch AGM. Chalmers, who received a knighthood in 2000 for services to healthcare, applies his passion for fairness now as editor of the James Lind Library , created to help people understand fair tests of treatments in health care.
The article below is based on the talk given by Iain Chalmers at the HealthWatch AGM 2009, and is a fuller version than the one which appeared in the print version of the newsletter, which was edited for reasons of space.
The subject of my talk is explaining fair tests of treatments in health care, and in this we have much unfinished business. I’d like to begin by introducing some of my special heroes in the field. Margaret McCartney, the Glasgow GP who writes a health column for the Financial Times every week , was a worthy recipient of last year's HealthWatch Award. I’d include the writers of some of my top books: Smart Health Choices by Judy and Les Irwig . Judy Irwig is a mother, Les Irwig a professor of clinical epidemiology, and their book explains clearly and authoritatively how not to be bamboozled by what you read in the media about health. Know Your Chances , in which American doctors Steve Woloshin and Lisa Schwartz explain how to interpret health statistics, is special because an early draft was itself subjected to a randomised controlled trial to see if it actually increased its readers’ knowledge of the subject. Another young British doctor, Ben Goldacre (HealthWatch’s 2006 Award winner), has shaken things up for science knowledge in this country with his Guardian column “Bad Science” . He writes, “Evidence-based medicine, the ultimate applied science… has saved millions of lives, but there has never once been a single exhibit on the subject in London’s Science Museum.”
However his fellow Guardian writer and HealthWatch Award winner, Polly Toynbee, did not make my hero list. As I pointed out in the April 2004 HealthWatch Newsletter , she once wrote that randomised clinical trials should be abandoned. “It may be a little less accurate scientifically,” she had written, “but if patients are allowed to choose which treatment they want and every detail of their condition, lifestyle, character and circumstances is fed into the trial data, I doubt if the results would be seriously distorted,” making clear her reluctance to agree to be a “guinea pig”. She completely failed to confront the fact that you often get very different results depending on design of the trial.
A few journalists—among them Nick Ross—understand evidence based medicine, and are prepared to battle against the stereotypes. On the 2nd April 2001 Nick was amongst fifty people who met to consider how to get the public to appreciate randomised controlled trials. There’s a problem with the name: it has so many negatives associated with it. “Randomised” suggests haphazard. “Controlled” implies controlling. “Trials” has legal connotations. It was Nick Ross who suggested, “why not call them fair tests?”
James Lind, a pioneer of fair tests, was a naval surgeon in the 18th Century and a member of the Society of Naval Surgeons (whose members went on to found the Medical Society of London). Like many who favour quantifying outcomes, he was something of an outsider. It’s harder to ask the question, “Is the Emperor wearing clothes?” when you’re a member of the Emperor’s establishment. It’s a problem that remains with us today. No matter how fair the test itself, the interpretation of science continues to be distorted by those who have a vested interest in the results, other than the well-being of patients.
The James Lind Library was launched by the Library of the Royal College of Physicians in Edinburgh in 2003. It has an online archive of illustrative records, from 1550 BCE to the present, illustrating how fair tests developed . These make clear that many of the principles of fair tests that we still use today go back hundreds, even thousands of years.
Conceptualising fair tests of treatments
In an extract of a letter written in 1364 , the Italian poet Francesco Petrarca wrote, “I solemnly affirm and believe, if a hundred or a thousand men of the same age, same temperament and habits, together with the same surroundings, were attacked at the same time by the same disease, that if one half followed the prescriptions of the doctors of the variety of those practising at the present day, and that the other half took no medicine but relied on Nature’s instincts, I have no doubt as to which half would escape.”
Treatments with dramatic effects
Even earlier, we have a surgical papyrus dated from around 1550 BCE which has been translated to reveal an explanation of how to reduce a dislocated mandible. It describes exactly what we do today, yet it was written more than 3,000 years ago. You don’t need carefully controlled trials to prove a treatment which is so clearly effective.
Recognizing the needs for controls
In the 10th Century CE, the Baghdad doctor Abu Bakr Muhammad ibn Zakariyya al-Razi (Rhazes), wrote on his experience of treating meningeal inflammation, noting the characteristic symptoms of photophobia, neck stiffness and headache. He wrote, “So when you see these symptoms, then proceed with bloodletting. For I once saved one group [of patients] by it, while I intentionally neglected [to bleed] another group. By doing that, I wished to reach a conclusion.” If this sounds rather barbaric remember it’s the way of thinking that’s important—he realised that he needed an untreated group in order to make an inference about the effects of his treatments.
The James Lind Library records a 16th Century example of a within-patient prospective controlled trial. “A kitchen boy fell into a cauldron of almost boiling oil…” wrote the French royal surgeon Ambroise Paré in 1575. “I went to ask an apothecary for the refrigerant medicines that one was accustomed to apply to burns. A good old village woman, hearing that I was speaking of this burn, advised me to apply for the first dressing raw onions crushed with a little salt… I was agreeable to trying the experiment and, truly, the next day, the places where the onions had been had no blisters or pustules, and where they had not been, all was blistered.”
During 18th Century naval campaigns more sailors were being killed by scurvy than by the fighting. One of several recommended treatments at the time was vitriol (sulphuric acid), which was favoured by the Royal College of Physicians of London. Of one of the earliest known reports of a clinical trial, the naval surgeon James Lind wrote in 1753, “…I took twelve patients in the scurvy… Their cases were as similar as I could have them. They all in general had putrid gums, the spots and lassitude, with weakness of their knees. They lay together in one place, being a proper apartment for the sick in the fore-hold; and had one diet common to all.” Lind allocated two sailors with scurvy to each of: “a quart of cider a day; twenty-five gutts of elixir vitriol three times a day; two spoonfuls of vinegar three times a day; a course of sea water… half a pint each day; two oranges and one lemon every day; the bigness of a nutmeg three times a day.” The most sudden and visible effects were seen amongst the seamen taking the fruit.
“Blinding” assessment of outcomes
The report of the homeopathic salt trials in Nuremburg in 1835 contains a detailed description of a randomized double-blind experiment in which participants were given either a homeopathic salt solution or pure distilled snow water. The details of which numbered bottles had contained which liquid were kept sealed until the end of the experiment. The experiences of the participants in the two groups were indistinguishable. One should bear in mind that homeopathic care in the late 18th and early 19th century was almost certainly safer than the bleeding, purging and use of heavy metals by orthodox practitioners.
Recognising the “law of large numbers” and the “limits of oscillation”
The idea of using numerical data to justify conclusions about treatments goes back at least three centuries. Pioneering work on how to apply inferential statistics to therapeutic data in order to make critical judgments on the value of therapies was published in Paris in 1840 by Louis-Dominique-Jules Gavarret. According to his beautifully written Principes Généraux de Statistique Medicale, “Average mortality, as provided by statistics, is never the exact and strict translation of the influence of the test medication but approaches it all the more as the number of observations increases. To be able to decide in favour of one treatment method over another, it is not enough for the method to yield better results: the difference found must also exceed a certain limit, the extent of which is a function of the number of observations.” Hence, the need to estimate what he calls “the limits of oscillation” (confidence intervals).
Confidence in results can be increased by examining the results of multiple trials. A key paper in the history of meta-analysis is Karl Pearson’s 1904 report in the British Medical Journal on “certain enteric fever inoculation statistics” which looked at correlations between typhoid and mortality and the inoculation status of soldiers serving in various parts of the British Empire.
In the early 20th Century important advances in study design were implemented in the USA in a programme of research to assess serum treatments for pneumonia. The trials in the programme demonstrated many of the important features of fair tests, involving large numbers of patients, allocation to treatment or control groups using an unbiased process (alternation), an assessment of the likelihood that observed differences could be explained by chance, and meta-analysis of the results of similar studies.
Recognising reporting bias
The English philosopher and statesman Francis Bacon, in his 1620 “New Instrument for the Sciences” commented, “It is a proper and perpetual error in Human Understanding, to be rather moved and stirred up by affirmatives than by negatives…” This is still as true today, and it can kill. Dr Cowley and his colleagues wrote in 1993 how, in an unpublished study done 13 years before, nine patients had died among the 49 assigned to an anti-arrhythmic drug (lorcainide) compared with only one patient among a similar number given placebos. “We thought that the increased death rate that occurred in the drug group was an effect of chance… The development of the drug was abandoned for commercial reasons, and this study was therefore never published; it is now a good example of ‘publication bias’. The results described here…might have provided an early warning of trouble ahead.”  In his 1995 book Deadly Medicine , the American author Thomas J Moore estimated that at the peak of their use in the late 1980’s, these widely-used anti-arrhythmic drugs killed as many Americans every year as were killed during the whole of the Vietnam war.”
Recognising the need for a cumulative science
In 1884 Lord Rayleigh, professor of physics in Cambridge and President of the British Association for the Advancement of Science, said, “If, as is sometimes supposed, science consisted in nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight… The work which deserves, but I am afraid does not always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out.”
In 1965 the English epidemiologist and statistician, Austin Bradford Hill, framed the four questions to which readers want answers when reading reports of research: Why did you start? What did you do? What answer did you get? And, what does it mean anyway?
An example that lives up to Bradford Hill’s expectations is the CRASH research into the effects of systematic corticosteroids in acute traumatic brain injury. The research was started because practice varied and a systematic review of existing studies (some of which had never been published) revealed important uncertainty about whether systematic steroids did more good than harm. To address this important uncertainty a large publicly-funded, multi-centre randomized trial—called the CRASH trial—was organised. The results, which were published in the Lancet in 2004  revealed that this treatment had been killing people since it was first used nearly 40 years previously.
The report of the CRASH trial is exemplary because it referred to current uncertainty about the effects of a treatment, manifested in a systematic review of all the existing evidence, and in variations in clinical practice; it noted that the trial was registered and the protocol published prospectively; it set the new results in the context of an updated systematic review of all the existing evidence; and it provided readers with all the evidence needed for action to prevent thousands of iatrogenic deaths.
In summary, science is cumulative, so researchers must cumulate scientifically, using methods and materials to reduce biases and the play of chance. Because researchers still do not do this routinely, people continue to suffer and die unnecessarily.
Editor, James Lind Library
- The James Lind Libraryhttp://www.jameslindlibrary.org/
- Margaret McCartney writing for the Financial Times http://blogs.ft.com/healthblog/
- Smart Health Choices by Les and Judy Irwig was published November 2007 by Hammersmith Press Ltd, paperback £12.99.
- Know Your Chances by Steve Woloshin was published November 2008 by California Press, paperback £11.95.
- Bad Science by Ben Goldacre was published in paperback edition April 2009 by HarperPerennial at £8.99.
- Chalmers I, HealthWatch Newsletter, issue 53, April 2004.
- The James Lind Library’s archives can be browsed on http://www.jameslindlibrary.org/trial_records/published.html
- This and the following texts can be accessed on the James Lind Library website by browsing the records, listed in chronological order.
- Cowley AJ, Skene A, Stainer, Hampton JR (1993). The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction. International Journal of Cardiology 40:161-166.
- Moore TJ (1995). Deadly Medicine. New York: Simon and Schuster
- The Lancet, Volume 364, Issue 9442, Pages 1321 - 1328, 9 October 2004.
Dr Margaret McCartney for her promotion of evidence-based medicine in general practice
Evidence-based general practice
Margaret McCartney is the 2008 winner of the HealthWatch Award. She is a GP in Glasgow who writes a stimulating and often provocative weekly medical column in the Financial Times Weekend section. She received her award at October’s Annual General Meeting. Here is the presentation she made on acceptance.
I am most certainly ‘for’ evidence based general practice. But evidence based general practice is not necessarily good general practice, and has brought a lot of new problems. What we surely want is not just any old EBM, but thoughtfully applied, professionally delivered, evidence based general practice. And while I do try and do my best to achieve this, I am not sure that I manage to. Bearing that in mind, I want to try and explain the problems and issues that EBM have led to in general practice, especially given the set of circumstances that medicine is now practised in.
I have a few personal guiding principles. More in medicine does not always mean better. Second, the most potent things about medical practice are not easily measurable. And last, harm is always possible.
‘Patient choice’ has been decided to be a good thing by politicians, and GPs are meant to offer choice to patients, for example, in where to be referred to for hospital care. GPs now spend large amounts of time discussing what clinic to send patients to, when perhaps what is of more importance lies elsewhere. The National Institute for Clinical Excellence offers to assess the evidence for treatment interventions and to try and measure its worth in terms of cost effectiveness. Yet NICE is often regarded as an enemy preventing people getting access to expensive interventions.
We have also seen the rise of the guideline. Guideline based medicine usually offers a distillation of evidence, but they are often used with a similar distillation of the limitations and uncertainties of that evidence. Guidelines have become goalposts, not necessarily a good thing.
GPs are usually independent contractors to the NHS. The GP contract essentially measures ‘Quality and Outcomes Framework points’. All sorts of things are measured. How many times blood pressures have been checked, how often people have been told to stop smoking, how many peak flows have been measured annually in people with asthma. Some have some evidence behind them—for example, there is a detectable decline in smoking where doctors offer help to quit. However, even here, it is still possible to do harm. For example, there are people who avoid their doctor because they are afraid that they will be chastised for being smokers still.
And of course doctors themselves are changing. GPs, and not just the female ones, are increasingly wishing to work part time. ‘Cradle to grave’ care is evaporating. Doing evening and weekends is in most places optional.
It is with these circumstances as the backdrop that a lady comes to see me. Statination is a word I use to describe the mechanism by which GPs currently seek to turn most people over the age of 50 into patients via prescribing a cholesterol lowering drug. Since her husband died three years ago, this lady has had episodes of anxiety and low mood which have been treated. She is an ex-smoker. Her high blood pressure is now in a satisfactory range. And she took up an offer to have her cholesterol checked for free at a high street chemist. The pharmacist was concerned. By the time my lady came to see me, she was agitated and worried. I went through her notes, and found that she had wanted to try a statin five years ago for a cholesterol of 6.9. There was a note about our discussion on lack of evidence for treating women in these circumstances. She had muscle pains and fatigue on taking the statin which resolved when she stopped it. We had decided then to concentrate on lowering cardiovascular risk overall via diet and exercise. This lady was now worrying that a heart attack was imminent. Have we really done her any favours?
As we know, there is good evidence and there is evidence which we can—and should—pick holes in. But even where evidence is favourable, it does not mean that we should necessarily start using the intervention. Statistical significance does not imply clinical significance.
What about the harms from something as commonplace as cervical screening? Every week or two I get a phone call from a lady who is distressed at being recalled for colposcopy. In fact the chances of being recalled unnecessarily—false positive—are far more likely than having a true positive. We should not underestimate the damage that this can do, both psychologically and in its wider effect on a woman’s family.
In recent years GP’s have been finding that smear results may include the additional information that the smear has shown ‘features consistent with HPV infection’. HPV has been in the news a lot recently because the new vaccination against it is now available. But when women have their smear test, the issue that signs of HPV might be reported in the smear result is not raised. When the result is available, GPs have a duty to share the information with the woman. But what then? The reason why we don’t know about the stress and damage caused by being told you have an untreatable infection which may—may—cause cervical cancer is because we have not adequately addressed the prospect of causing harm by our intervention. Yet such harm is witnessed in primary care and is most certainly done.
There are people in the audience who know much more about breast screening than I do. We have all seen on buses and billboards that the chances of getting breast cancer are ‘one in nine’. However this is not the case; this is only the risk if a woman is over 85, and most UK women don’t live that long. But this statistic wasn’t just used by breast cancer charities which, while they may have a useful role, also have an interest in getting media attention. I interviewed Julietta Patnick, the head of NHS screening for a piece in the Financial Times recently . I was asking her why the publicity for the NHS breast screening programme also uses the one in nine figure, and this is what she said. “I’ll tell you what happened: we were debating what to put on the leaflet and I was going to work, following a bus. And there it was, ‘1 in 9’, on an advert on a bus. I realized that if we didn’t put it on, we were going to confuse like mad.”
We know that women overestimate their chances of getting breast cancer, underestimate the benefits of treatment, and do not realize that the risk of breast cancer increases with age. It could be said therefore that all the breast cancer ‘awareness’ months as well as the NHS information have been associated with damage rather than benefit. When staff from the breast screening service published research last year showing that women are not aware that the risk of breast cancer increases with age, they did not suggest that their information should be improved—only that GPs should be more aware of the lack of knowledge.
Breastfeeding is another example of damage being done by well intentioned doctors. GPs are meant to encourage breast feeding, even at the pre-conception stage. If staff in the practice all agree that they will encourage women to breast feed then you can get a special certificate to put up on the wall.Yet we know that breastfeeding rates, despite the enormous publicity, haven’t actually changed that much in the UK over the last decade. And we have not looked adequately for evidence of harm that these campaigns have done. I see many women who are struggling to breast feed and feeling desperately unhappy if they stop. I have also been concerned that women who have had problems and stopped breastfeeding become reluctant to seek medical advice about other matters because they are afraid of being scolded; and I am concerned that ‘failure’ to breastfeed could fuel postnatal depression. But apart from my anecdotes, I have no definitive evidence—because no one has looked for the harms of the relentless drive to push women towards breast feeding.
Even the benefits of breast feeding may have been overemphasized by health professionals. There is a good Cochrane review showing that the benefits from breastfeeding are overall small gains. These are important—but we should also be aware of the harms our enthusiasm can do.
“Awash in information, patients face a lonely, uncertain future”. “We’re in the outer reaches of medical knowledge,” he said, “and none of us knows what you should do. So you have to make the decision, based on your values.” Ms Gaines, bald, tumour ridden, and exhausted from chemotherapy, was reeling. “I’m not a doctor—I’m a criminal defence lawyer. How am I supposed to know?”
It is the word ‘lonely’ that gets to me. Of course doctors must acknowledge and attempt to reduce uncertainty—even the GMC says so—and doctors must be able to explain why they have done certain things. But ‘evidence’, or lack of evidence, is sometimes used as a way of abdicating professional responsibility. ‘No evidence’ can say, ‘no evidence for what I have to offer being of benefit’ but it is also used as meaning, ‘not my job’. This is increasingly seen where protocols are used to decide who might benefit from a particular intervention. But if you don’t fit the protocol, who is going to look after you, and help you make difficult decisions steeped in uncertainty? I think it is telling that there are ‘advocacy services’ now who seek to get better medical care for patients. ‘Advocacy’ used to be firmly in the doctor’s job description. Sadly, this is not now the case.
Neither are doctors seen as being ‘in charge’ of informing people about health and disease. To a degree this is a welcome destruction of medical paternalism. But the void has been filled by other sources who are arguably even more ‘interested’.
A prostate cancer charity produces a leaflet aimed at women, encouraging them to persuade their man into having a ‘prostate check up’. It suggests that they, “leave information leaflets lying around in the bathroom, remote control or car seat” and has a guide to how to, “tug at his heartstrings”. I’m not even going to start on the misinformation there. Or we have a suggestion in a problem page that, “GPs are not trained to deal with skin problems” which is news to me; however it seems that the private clinics listed are. Then we have another company offering ‘MOTs’ via CT scanning, with nothing about the risks of radiation, or false positives or negatives, but which they tell us ‘could save your life’.
So who do we trust to support us through illness, to advise us fairly about preventive health services, and to give balanced information with due regard for risk as well as benefit? This should be the job of professional healthcare workers, but to do this, we have to put our own house in order. General practice is in a mess, because we are conflicted thanks to the GP contract. We are still in servitude to the pharmaceutical industry with our reliance on them for postgraduate medical education—pharma estimates it provides half. Physicians with private rooms may be guilty of using the media to stoke a market. Our correct enthusiasm for independent and unbiased evidence based medicine has not been accompanied by a similar need to be independent and unbiased ourselves.
Healthcare workers need to ensure that evidence based medicine is applied professionally as well as honestly and fairly. General practice can do more than provide fair information and advocacy. Everyone here will know about the benefits of the placebo effect, but also that using placebos are riddled with ethical conundrums. However, if we are to regard the psychologically mediated effects of good healthcare as ‘placebo like’ effects, there are lots which we can gainfully and ethically use. For example, in general practice, continuity of care, longer consultations, and confident professionals all improve measurable health outcomes. One intervention for COPD—pulmonary rehabilitation—uses peer support and graduated exercise to improve quality of life and reduce hospital bed days. It seems to me that these kind of ‘evidence based’ interventions are the ones which are underutilized and under acknowledged and yet which bridge the gap between a professional health service which cares about individuals and the unafraid application of evidence based practice. I think there is hope for evidence based general practice yet, but if all patients are going to gain from it, we need to be heightened in our sensitivity to harms, and to deliver that medicine with professionalism, compassion and care.
1. McCartney M. Reality check on breast cancer. Financial Times, 27th September 2008.
View online at: http://www.ft.com/cms/s/0/3f714b 0a-893c-11dd-8371-0000779fd18c.html?nclick_check=1
Tim Harford on behalf of the BBC More or Less team for their clear, honest and entertaining way of educating the public about the meaning of numbers
Junk, jigsaws and zombies: misleading stats in the news
The 2012 HealthWatch Award went to Tim Harford and the team behind BBC Radio 4’s “More or Less” programme. Tim received his award at the October AGM, and gave an entertaining presentation to HealthWatch members and patrons on the subject of misleading medical statistics. The article below is prepared from his presentation.
I’d like to begin by setting you a little test. Imagine you’re a doctor discussing a type of cancer screening with a patient. You see that test A increases the patient’s 5-year survival rate from 68 to 99%. Put your hands up if you think that would be a benefit? [audience hands were, hesitantly, raised] Test B, however, will reduce deaths from 2 per 1,000 to 1.6 per 1,000. Most people faced with comparing test A with test B would opt for test A.
In fact, only test B unambiguously saves lives: to be precise it saves 0.4 lives per 1,000 people. But what about the huge survival rate benefit of test A? To explain how this is, imagine a cancer that always strikes at the age of 60, but that shows no symptoms until age 68 … then kills at age 70. A screening test that accurately diagnoses the cancer in 62-year-olds would give them a 5-year survival rate of 100%. Yet they would still die at 70 if there was no treatment.
US psychologists recently put this test to 400 doctors. Eighty-two per cent thought that test “A” saved lives—which it didn’t. Only 60 per cent thought that test “B” saved lives, and fewer than one-third thought the benefit was large or very large—which is intriguing, because of the few people on course to die from cancer, the test saves 20 per cent of them. In short, the doctors simply did not understand the statistics on cancer screening.
In the course of this talk I’d like to share with you some of the things I’ve learnt about statistics in the news.
Mistakes are not always difficult to spot
An advertisement for “U-switch” internet service claims that 49% of British broadband customers are getting below-average broadband speed. Think about it … it’s like saying, “49% of NHS patients are getting below-average treatment”. Of course they are, and the rest are getting average or aboveaverage.
It’s saying nothing.
It’s easy to be wowed by big numbers
In 1997 Gordon Brown pledged to spend £300 million on pre-school provision over the following five years. Now, you need to peer beneath the numbers here. The need for pre-school provision affects about 1 million children every year. That figure boils down to about £1.08 per child per week. What exactly is going to be provided for £1 a week? So, don’t be impressed by the big numbers that aren’t.
Be wary of averages
We have reports of economic upturns—talking about average inflation, average incomes. Remember, on average a rainbow is white, yet it’s the colours that are important. The average is not the only thing you want to know to get the true picture.
A famous example of the misuse or misunderstanding of averages was when the financial crisis broke in August 2007, and the chief financial officer of Goldman Sachs commented that 25-standard deviation events had occurred several days in a row. We asked a professor of finance to calculate for us the likelihood of that actually occurring. He worked out that you might expect to see a 2-standard deviation event once ever 4-5 days. A 3-standard deviation event happens only every 3-4 years. A 4- standard deviation event once every 126 years. There could only have been one 5-standard deviation event since the last ice age. A 25-standard deviation event would be expected to happen only once every x years, where x is a number with 67 zeros.
Watch out for shifting definitions
A recent US report said that one in five students self-harm. Alarming to consider the possibility that a fifth of young people might hurt themselves by burning, slashing wrists, and attempting suicide. But is it true? Read the study behind the news and we find that 8,300 students were surveyed. Of those 3,000 chose to respond—could there be a bias here towards young people who already have an interest in self-harming? But it is the definitions that interested me. They included things like tugging at your own hair, scratching yourself. No doubt smashing your head onto your keyboard when reading an idiotic news story would have been included. According to the same study, only 49 of the responders reported causing themselves serious harm. That is 0.5% of the original sample. The story was not exactly false, but not really true either.
Sometimes you get statistics that have no merit whatsoever
We call them junk stats. We came across this one on the internet—four million US women are battered to death by their husbands or boyfriends. Four million? Common sense tells you that can’t possibly be true. We often find that the more serious the claim and the worthier the issue, the stupider the stats.
The dangers of junk stats
But it’s not only important issues that generate junk stats. We noticed an advertisement for a product that made lips 25% fuller. What does that mean? And a product that is 25% more “berrylicious” … we assume these have been cleared by the Advertising Standards Authority, but on what basis they proved the claims I can’t imagine.
Every year there is a Blue Monday. It’s a concept created to generate publicity for a charity that supports people with depression. A PR firm invented an equation that involves how many days you’ve just had off work and the length of time before your next holiday, and comes up with the most depressing day of the year. Is that OK? I’m inclined to think it makes a mockery of statistics, maths, and journalism. Someone with real evidence can be chucked in the same bucket as this kind of nonsense. Journalists using this kind of material don’t give their readers the tools they need to interpret stories properly.
There’s something we call “zombie stats”
These are the figures that can be shot to pieces over and over again and yet they still keep coming back. Here’s one. Public sector spend could be cut by 20% by reducing waste. Well, the biggest factor in public sector spending is salaries, not waste. So how can cutting waste reduce it by 20%? The figure came from a procurement consultancy that claimed to be able to get better deals—such as cheaper mobile ’phone bills. Of course they wouldn’t be able to make savings like that in all areas of spending, yet the government seemed happy to repeat the claim without any evidence. Ben Goldacre shredded the claim  we at More or Less shredded it, but like that zombie it just won’t lie down and die.
Recognise patterns that can fool us
In the cancer screening example that I began with, you could have known that I was going to try to fool you. But you don’t always see it coming. To take speed cameras, for instance. If the government try to put cameras at accident black spots, will they have an effect on the number of accidents that occur?
Some accident black spots are indeed dangerous places. But in some cases a cluster of accidents is just back luck. Bad luck doesn’t last. Put a speed camera there and, chances are, the run of bad luck ends, the accidents are reduced, and it seems to be down to the camera. I’m not saying that speed cameras don’t help, but they might help less than we thought.
What do we in the media need to do?
Sometimes we need expert help. We in the media can always find an expert who is generous enough to explain if we ask. And we need to know when we need to ask.
We need to get better at explaining risk. To take another example from recent news, we hear that if you eat a bacon sandwich every day, you have your risk of cancer increased by 20%. The questions we need to be answering here are, what kind of cancer are we talking about? How likely are you to get it anyway? In this case, we’re talking about cancer of the bowel. Under normal circumstances, 4 people in 100 get bowel cancer. If you eat a bacon sandwich every day, the risk increases to 5 in 100. Explaining risk in this way is helpful.
The BBC website’s top story at the moment is that old people can prolong their useful life not by going for more walks, but by doing jigsaw puzzles. Whether this results from a study or a systematic review, we don’t know. You need context in order to judge this kind of story. And there’s the recession. A deficit of £150 billion—it’s a meaningless big figure for most people. Until you calculate that it adds up to a bill of £2,500 per person per year. Is that higher or lower than the deficit for other European countries? Greece for example? A good party trick during party political conference season is to take all the numbers that appear in the media reports and divide them by the population of the country and see what you get.
And are the sources sound? Is it true? What exactly is being said? What is being compared? If we’re talking of treatments, were they tested on animals or people or in a petri dish? Get a sense of scale and context. If we can do that, we can use maths to tell stories that people can understand.
Broadcaster, author and journalist
Reference  http://www.guardian.co.uk/commentisfree/2011/jun/24/bad‐science‐local‐govermentsavings‐ ben‐goldacre
More about tim harford and “more or less”
Tim Harford is an author, columnist for the Financial Times and presenter of BBC Radio 4’s “More or Less”. The Royal Statistical Society has commended More or Less for excellence in journalism in 2010, 2011 and 2012; and the programme has won an award from Mensa. As a senior columnist for the Financial Times, Tim’s long-running “Undercover Economist” column reveals the economic ideas behind everyday experiences, while a new column, “Since You Asked”, offers a sceptical look at the news of the week. His first book, “The Undercover Economist” has sold one million copies worldwide in almost 30 languages. His writing has been published by the leading magazines and newspapers on both sides of the Atlantic. Tim won the Bastiat Prize for economic journalism in 2006 and has been named one of the UK’s top 20 tweeters by The Independent.
In BBC Radio 4’s “More or Less” programme, Tim Harford and his team investigate numbers in the news and try to make sense of the statistics which surround us. The half-hour programme is broadcast at 16:00 on Friday afternoons and repeated at 20:00 on Sundays on Radio 4.
For more about “More or Less” see the programme's page on the BBC website.
For Tim’s articles and blog, see his website.