Prof Raymond Tallis: for his thoughtful promotion of evidence-based medicine
Anecdotes, Data and the Curse of the Media Case Study
Let me say straight away how honoured I feel to here tonight to be talking to you and to be receiving your annual HealthWatch Award. This came totally out of the blue – which has very positively altered my attitude to the blue as a result. The blue is more usually a source of nasty surprises – the frozen block of urine dropped from a 747 – than nice ones. I feel not only honoured but distinctly undeserving, particularly when I think of the previous winners of your award. I am thinking of writers whom I admire enormously such as Geoff Watts and Polly Toynbee and the witty and courageous John Diamond; and great clinical scientists such as David Sackett and Richard Doll and Michael Baum. And as for Ben Goldacre – the recipient of last year’s award – what can I say, except that every time I read one of his columns, I punch the air with delight until it is bruised. When get my weekly fix of Bad Science all my exasperation evaporates. So thank you Ben, and thank you all my predecessors, for strewing a little broken glass in the path of charlatans, mountebanks, loblolly men, and all those alternative practitioners whose loud noises have threatened at times to drown the softer-voice of hard-won scientific truth. And thank you for inviting me to join their select company. I would like to add my congratulations to the students for their awards. ‘O brave new world that hath such people in it…’ Thank you also Nick and David and you other HealthWatch stalwarts who over the last 19 years have kept HealthWatch as a major force for busting quacks and standing more broadly for reason and evidence against the forces of unreason that flourish best in an evidence-free zone. I know you do this purely because you are passionate about the things David talked about in his address. As Nick, said, you don’t get anything back – though 50,000 – 70,000 hits is very impressive indeed.
I don’t know whether my frontal lobes are starting to lose their grip – or perhaps it’s my latest detox at the health farm – but I am starting to feel a little more optimistic and have the feeling that the quacks, at least in the UK, are perhaps on the retreat; that this has been a pretty good year for quack-busting. Perhaps it’s a matter of contrast-enhancement because we did start from a pretty low point when in September 2006 the MRHA (Medicines and Healthcare Products Regulatory Agency) gave homoeopathy a boost by licensing its products on the basis not of medical evidence but something called provings. I must admit, I had never heard of provings up till that time and this is what they are.
They’re conducted on volunteers in a reasonable state of health (provers) who generally have no knowledge of what substance they are taking. Doses are repeated until provers start to experience symptoms of a change in state. The provers record everything they experience, whether physical, emotional, mental, or even spiritual, as long as the change in state persists.
At the end of the proving, all the records are collated and compared to find the symptoms, states of mind, feelings and experiences the provers have in common and which can be reasonably attributed to the emerging signature resonance of the substance. Wendy Howard Homoeopathic Practitioner quoted in The Skeptic Express, 2006
It would be an interesting exercise for 6th Formers or even 12 year-olds to be challenged to say why this methodology is flawed. Anyway, the outrageous decision by the MRHA – which seems in this instance to have been influenced by populism rather than by science – galvanised opposition and the annus fairly mirabilis has resulted, with homoeopaths, and herbalists in particular, having a fairly rough time of it.
Many of you will be aware of the campaign fronted by Michael Baum, David Colquhoun, myself and others in partnership with Sense About Science that has had scored some palpable hits against homoeopathy. The campaign to persuade PCTS not to fund homoeopathic treatments on the NHS has attracted huge publicity and, what is more, prompted many PCTs to remove such treatments from their bill of fare. The latest scalp is the withdrawal of funding by the Tunbridge Wells PCT from their local homoeopathic hospital which consequently is likely to be mothballed. There are five Homoeopathic Hospitals in the UK: one down 4 to go.
And then there has been increasing awareness not only of the uselessness of most herbal remedies but, indeed, of their power to do harm. While it is possible that there are some benefits from some herbal remedies – after all, if you fire enough buckshot in the dark, you will sooner or later hit a mosquito’s legs and then say that that was what you were aiming at all the time – there is, as we all know, no evidence that the kind of mixtures dished out by herbalists in their shops do any good. There was huge publicity a couple of weeks ago arising from the study by Edzard Ernst – another person who I am very proud to have as a predecessor in the receipt of this award and who has, I believe, done more than anyone to force CAM to engage with the scientific method – recently published by the Postgraduate Medical Journal. This has provoked widespread demand for properly regulation of these medicines and of those who flog them. Let us hope that this time the MRHA doesn’t fumble the pass.
In short, much grounds for rejoicing. It is as if the nation’s serum Carol Caplan levels are falling and the parts per million of bullshit in the collective consciousness has dropped a notch or two. Now, if this speech was written by a computer, rather than written on a computer, and it had predictive texting, it would next print out the words – ‘There are, however, no grounds for complacency’. Or, as your Chairman, David Bender, no computer he, observed in his report last year – ‘We’, that is to say you, ‘are still needed’. And that’s certainly true. Unreason is always ready to break out, or to return, for reasons that go to the heart of what I shall be talking about this evening and, indeed, what it is about quack medicine that makes it more significant than might appear at first sight. Even as we speak, rejoicing in a year in which in the UK at least quackolysis has definitely had the upper hand over the quacks, herds of bulls are massing, ready to evacuate the contents of their ample colons.
Colonic material of a Taurine provenance is in infinite supply
One of the reasons for this is, of course, the way science and non-science are presented to the general public by the media. And the problems are epitomised by what is referred to in the title of my talk: the curse of the media anecdote. Much discussion of medical care – both orthodox and useless – tends to be revolve around, or even to be tethered to, anecdotes.
The media love anecdotes because those who work in them believe (usually correctly) that their customers do. Anecdotes – about a new miracle cure, a drug that is not being made available on the NHS, or the side effects of treatment, or some environmental hazard - sell product: they help to boost circulation, increase audience rating figures, and so on. Data, on the other hand, which take us towards the truth about these things, are less popular.
Anecdotes, however multiplied, do not point the way to reliable knowledge. As the aphorism says, ‘The plural of anecdote is not data’. For an anecdote does not even count as a datum. It may prompt a search for data, as when a chance clinical observation makes someone wonder whether A causes B, or C may have therapeutic benefits in condition D.
These deficiencies count less than they should. For anecdotes, unlike data, have a human face: they are closer to gossip, which is a universal human pastime, and to the things that shape belief formation in everyday life. This is something to which I will return presently. Scientific data - for all that science is one of the most human, as well as the greatest, of human achievements – lack this human face. Their very strength – that they are unattached to persons and personalities – is a PR weakness. At best, data belong to ‘the science community’; worse, to ‘the scientific establishment’; or, worst of all, when they are adopted by, and guide, those who make policy on our behalf, ‘the authorities’.
Let me illustrate the accursed power of the anecdote with one of the most potent examples of recent times: the story told by Jackie Fletcher, the founder of Justice, Awareness and Basic Support or JABS, a group devoted to campaigning on behalf of parents who believe that their children have been damaged by immunisation, most notably by the MMR triple vaccine. Her story is a tragic one. Nearly a decade after the MMR furore broke, it is still news and still being used to influence opinion.
In 2007, The Daily Mail devoted yet another article to her terrible experience, accompanied by a poignant picture of her and her son, ‘who developed autism after being given the MMR jab’:
The change in their bright, loving toddler was sudden. Ten days before his first fit, Robert had been vaccinated against measles, mumps and rubella (MMR). Unlike his older brothers, who had been given the jabs in single doses, Robert had been injected with three viruses at once.
To the Fletchers, the connection seemed obvious..
The Fletchers’ experience is unimaginably terrible: the transformation of a perfectly normal toddler into a child with autism, severe epilepsy, and incontinence, confined to a wheelchair, speaking only the words he knew as a baby, and requiring to be washed and fed. In their case, grief is envenomed with (unjustified) guilt. Mrs Fletcher says, ‘Deep down, I feel that I have caused all Robert’s problems by taking him for the MMR jab and I’m trying to fix it for him and I can’t’.
One would have to have a heart of stone to challenge Mrs Fletcher’s interpretation of what happened; her attempt to make sense of a senseless catastrophe; and her endeavour, through campaigning to protect others’ children against what she sees as a threat, to come to terms with a guilt she feels which, though unwarranted, is real. An anecdote such as this, set among data, is like a queen ant in a colony of drones. Stories have a power that bears no relation to their number or validity. The very fact that those who invoke anecdotes in support of their theories are so convinced by them, adds to their cognitive lustre.
Notwithstanding Nietzsche’s assertion that ‘Convictions are greater enemies of truth than lies’ , certainty, repeatedly asserted, and the anger and grief with which it is charged, increases the impact of the claim embedded in the story, on the reader or listener. When we are invited to assent to an empirical truth in an area in which we have no expertise, we are more inclined to do so if we identify with the person who is purveying it. It is a way of showing solidarity.
The contrast between the media anecdote and the way data are presented in scientific meetings or papers – where there is no reference to the feelings (anger, grief, conviction) of the authors and much concern about sources of error and uncertainty - underlines how disadvantaged the latter are. Of course there are passions behind science – who otherwise could stand the utter tedium of data gathering and learning the techniques of doing so reliably? – but the passions of themselves cut no ice. Indeed, they would be grounds for suspicion from those who are used to what Paul Valéry described as ‘the elementary courtesy of proof’.
Anecdotes also have the ability to attract other anecdotes: when the Fletchers took their stricken child to his many hospital appointments, ‘they encountered other parents who said things like “our child was fine until MMR”’.
This ‘multiplier effect’ is vastly scaled up when the anecdote comes from a celebrity in the media, or is endorsed by such a celebrity. A face that is everywhere is the vehicle for an opinion that is everywhere. The same story is replicated in a million places and has the impact of a million data. In a society where achievement in one area (for example, combining being a moderately competent actor with having an eye-popping figure) makes one an authority in many others, the views of celebrities are the equivalent of the union block vote.
From the point of view of the media, there are secondary benefits of the human story. When data falsifying the Anecdote Based Theory (ABT) start to look overwhelming, other, secondary, anecdotes emerge that nurture the primary one. There is talk of conspiracies, cover-ups by the establishment, and we are reminded of persecution of people (of genius and courageous) who have heretical views.5 The campaigners on behalf of ABT and their supporters look like plucky loners facing a mighty monolithic opposition. Powerful data are discredited simply because they originate from sources that are considered to be on the side of the Big Boys: the scientific establishment, the politicians (who have their own reasons for hiding the truth), and others, such as Big Pharma (who have vested interests). Even the sharing of these data with the public can be presented as an act of oppression. At the height of the MMR controversy, the Department of Health was accused of ‘bullying parents with facts’6.
The power of the anecdote is enhanced by the fact that, once it is in circulation, it looks like an entire half of the overall picture. The media claim that they are required, by their commitment to balance, to give equal hearing to both ABT and science. So Jackie Fletcher talking about her son Robert and giving her untrained critique of the epidemiological data is allowed as much airtime as a scientist bearing news of millions of data obtained under carefully controlled conditions. Given that controversy sells product, maintaining the myth that there are two equal sides to the story is desirable in itself.
The notion of balance is one that should be treated with care, the kind of care that it does not always, or even usually, receive in the media. If applied uncritically, it can result in giving data-free believers in the flatness of the earth the same amount of air-time and column inches as those who support the well-attested notion that the surface of the earth is curved. The Royal Society, quoted in an email by that doughty fighter against junk science, David Colquhoun, has made this point:
When obtaining specialist advice from experts, it is not diversity of opinion that should be balanced, but the weight of opinion accorded to various strands of scientific advice within that diversity. In the name of ‘balance’, the media invariably present opposing views on each side of an argument, regardless of the relative scientific merit of those opinions.
The price of so-called balance that does not take account of the ‘relative scientific merits’ of the opinions being given equal air-time, may be very high indeed. Most people in this room will be aware of Peter Duesberg the maverick molecular biologist who denied the link between AIDs and the human immunovirus. His views were largely discounted in the West and have, thank God have had no influence on science policy, public health policy and those scientific research programmes which have led to the discovery of highly active antiretroviral drugs which have transformed AIDS from a fatal to a chronic disease. Elsewhere, mankind has not been so lucky. Thabo Mbeki’s catastrophic AIDs policies have been hugely influenced by Duesberg’s views and the result is a pyramid of corpses sufficient to block out the sun.
Fortunately, the MMR controversy has a less calamitous outcome. When it was obvious that the link between MMR and autism was unsustainable, and that it never had any basis, some of the media showed signs of contrition. Not all of the press, of course. Melanie Philips still continues her one woman campaign against MMR and combination vaccines that she considers lethal. If her views were listened to, of course, and had any influence on public health policy, the result would be a catastrophe on the scale of what has happened in South Africa. The habit, however, of giving appealing individuals with their moving stories at least as much credence and coverage as unappealing data, of preferring faces to graphs, and vox pops to statistics, does not seem to have been kicked. The recent judgement by the National Institute of Clinical Excellence that certain drugs would not be cost-effective in the treatment of early Alzheimer’s disease on the basis of the evidence so far was dominated by human interest stories.
I found this particularly exasperating because (to declare an interest) I was a member of one of the NICE appraisal committees for three years. This, quite labour-intensive, unremunerated role brought with it two rewards: the knowledge that one was contributing in some small way to ensuring that the finite resources of the NHS would be spent in such a way as to maximise health gain; and the satisfaction, indeed pleasure, in participating in a very high-level, supremely well-informed discussion about the costs and benefits of medicines that took into account basic science, clinical evidence, health economics and social value judgements. Very little of this appeared in the media coverage. The World at One gave extensive air-time to a woman whose father had been given the drugs early in his disease and was ‘convinced’ that he would not have done so well had he been denied them.7
This example illustrates something that is common to much anecdote-based discussion of matters of public interest such as the benefits or harms of medical treatments or the impact of some environmental factor on health: anecdotes privilege the visible over the invisible. In the case of MMR, the millions of children who had been saved from harm by the vaccine were given little foreground; in the case of Alzheimer’s drugs, the impact on all other patients that would result from inappropriate allocation of resources to less cost-effective treatments, was hardly discussed.
I want to step back a little and look beyond their immediate consequences to another reason why the media anecdote is accursed. To think about this I want to rehearse some arguments I have had with people about the use of alternative medicine in the UK. I was recently discussing reflexology and it was pointed out to me that it is pretty harmless and that it doesn’t after all cost much. There was very little evidence that those who used it denied themselves the benefits of orthodox medicine. So why are you getting in such a pet? Well it seems to me that alternative remedies and the theories that are used to support them are a major contribution to the moronisation of the collective consciousness and, at some level, a betrayal of one of the greatest achievements of humanity, and so of humanity itself.
The journey humankind has taken to robust scientific knowledge of the world in which it finds itself has been a long and arduous one. For science has required not only that we somehow penetrate the opacity of nature but also that we overcome the prejudices, the intuitions, the assumptions, the convictions, the dogmas that we ourselves place in the path to reliable, usable knowledge. These obstacles are especially stubborn in medicine, not only because its object is the human body, that from which knowledge takes its rise, but also because its concern is with disease, where emotions, particularly those of hope and despair, tend to have the most powerful influence over our beliefs. The endeavour to cure illness is thus bound to be dogged by deception and self-deception. Hence medicine’s status as a parvenu among the sciences: it is, as Lewis Thomas said, ‘the youngest science’. The greatest miracle of scientific medicine is that medicine became scientific.
The contrast with traditional, or pre-scientific, medicine – the medicine that was all that was available for all but the last 100 years of the 300,000 years of H. sapiens sapiens - is extraordinary in many ways. One of the most important is the fact that scientific medicine, unlike traditional medicine, which is deeply self-satisfied with its knowledge and what it believes to be its effectiveness, is driven by an active uncertainty that is sceptical of received ideas and of authority and is continually seeking to improve on the status quo. It is an irony that scientific medicine is called ‘orthodox’ medicine because, unlike acupuncture, or homeopathy, is constantly scrutinising, challenging, overturning its own beliefs. Like the rest of science, it is in a permanent state of self-critique; and the authority of authorities goes no deeper than the facts that they can invoke. By contrast, acupuncture claims to derive its authority from individuals who laid down the law thousands of years ago. One of the most barefaced examples of proud intellectual stagnation must be the fact that homeopathy has not changed since Samuel Hahnemann offered it as an alternative to the poisoning and butchery that was prevalent at the time. Homeopath still rely on his book, first published in the 1790s, to guide their practice. What he spake, they shall do. There could be no more striking example of frozen orthodoxy and authoritarianism.
Alternative medicine has not progressed to anything approximating clinical science because it has not been prepared to challenge its own authority. So-called orthodox medicine, in an act of collective humility, has cultivated a routine distrust of its own practices. This humility has been almost as important in the development of effective therapies as the discoveries in basic biomedical science. The rejection of anecdote-based medicine in favour of treatments tested in randomised controlled trials and re-tested in meta-analyses etc has accounted for the remarkable advances particularly over the last 50 years, and the quickening pace of increases in life and health expectancy. The humility of shaping clinical practice in accordance with the unmanipulated outcomes of therapeutic trials is also connected with another layer of scepticism: that what looks good in theory (irrespective of how good the theory is) may not benefit patients in the real world.
Perhaps the most remarkable facet of this humility is the willingness of doctors to enter patients (with the latter's consent) into clinical trials run by other clinicians. Submission to the authority of such trials means subordinating one's own personal authority to that of other professionals, many of whom (such as statisticians and biomedical scientists) do not even belong to the medical profession. As Marc Daniels pointed out:[for clinicians] to be willing to merge their individuality sufficiently to take part in group investigations, to accept only patients approved by an independent team, and to submit results for analysis by an outside investigator involves considerable sacrifice.
The committment to minimising the play of chance, of bias, or of wishful thinking, is the supreme expression of the self- overcoming that scientific medicine requires. Avoiding beliefs guided by delusive hope, unfounded authority, superstition and plain stupidity, it cultivates an attitude of healthy scepticism towards itself to prevent its practitioners from misleading themselves or their patients. Its permanent strategy of active uncertainty, and the humility this implies, is the distinctive virtue of scientific medicine. In the world outside of scientific medicine, however, humanity has had little time to adjust to this almost inhuman scrupulousness.
Hence the fact that alternative medicine – and more broadly anecdote-based treatments - can still live, and even flourish, at least in terms of the number of customers, and the amount of cash that changes hands, side by side with medicine that actually makes patients better for reasons that are rooted in a scientific understanding of the human body and its diseases. Otherwise, it would seem unaccountable – like an ox-cart at stand in an airport next to a jumbo jet or an Asclepian temple fitted out with WiFi. Indeed, the constant self-critique of medicine and the biomedical sciences upon which it is based is often misunderstood. The self- dissatisfaction and self-transformation - resulting in ever more effective and, for the most part, less unpleasant, treatments - is seen as a kind of weakness. A discipline which is marked by a carefully nurtured scepticism towards itself is sometimes seen as arrogant or in disarray – unlike traditional healers who are proud never to have had a second thought in 5,000 years.
So this is perhaps the true curse of the media-propagated anecdote and anecdote-based beliefs about medical care: they represent a sickening regression to the cognitively primitive state from which medical science, science, and humanity has fought its way out.
So if we are going to free ourselves from the curse of the media-base anecdote, we need to have a major initiative explaining why anecdotes are misleading, why double-blind randomised controlled trials are necessary – how they work and how, over the last 50 years, they have played such a crucial role in the objectively observed advances in medical care.
A good place to start would be cardiovascular disease, in which randomised controlled trials have had almost as great a role in bringing about improvements in mortality as advances in basic biomedical science. A couple of statistics could introduce the topic. Firstly that cardiovascular disease mortality in men of all ages has fallen in 2005 to 30% of the age specific rate in 1968. Secondly, that deaths from coronary heart disease of people under 65 has fallen by 46% - by nearly a half – in the last ten years. People should be asked to celebrate these facts and to consider the role played by basic science in understanding the processes that lead to cardiovascular disease and methods of preventing or reversing them; and the complementary role of robust clinical trials in ensuring that what should work in theory actually works in practice in the real mess of the real world of doctors and their patients.
Perhaps this is idealistic. The notion that the nation’s serum Daily Mail levels might be reduced by broadsheets offering free tutorials on the methods by which science arrives at robust knowledge and that knowledge is translated into benefits for patients is somewhat utopian. And yet I assume that most journalists (and their sub-editors who write those lethally misleading headlines) want to be able to look back on careers in which they have done more good than harm and advanced, rather than held back, the cause of truth. They are aware that anecdotes may be very effective at conveying the profundity of medical tragedies and the human importance of medical triumphs and disasters. But for making decisions about the absolute and relative values of medical treatments, and more broadly for determining policy, they are of no use whatsoever.
And so I would like to offer a few thoughts, which could be accommodated on a flash card, to help journalists deal responsibly with those anecdotes they and their readers value so much. Before I do so, however, I would like reflect a little more on the power that anecdotes have, which they owe to their central role they play in our making sense of the world in which we find ourselves. And this includes the kind of sense of the mediated world that I am trying to make in this talk – which, it will not have escaped your attention, has itself been rather heavily dependent on anecdotes.
Anecdotes are not an unredeemed evil. As we have already noted, medical science itself has been driven by accidental, observations, anecdotes, though that is merely a prompt to inquiry, the place where it begins, not where it ends. But beyond this, human beings have always told each other stories and storytellers have always commanded a hearing not only because we want to know what happens next but because what is recounted strikes us as standing for something that goes beyond the particular instance recounted.
We imagine beyond the events in the story to a whole class of similar events. ‘This could happen to me’ is extrapolated to ‘This could happen to anyone’; ‘This could happen any time’ to ‘This is happening all the time’. We are taught from earliest childhood to see stories as lessons, as parables: faulty induction is part of our ethical education. Identification with the stories that we are told, seeing them as potentially part of our own biography, is key to the development of that imagination which lies at the heart of sympathy, empathy and hence morality.
The power of stories to fuel extrapolation beyond themselves to an imagined series of general cases is greatly magnified by emotions – as Bertrand Russell pointed out in the passage I have just quoted/ for emotions are the means by which the world we think of ourselves to be surrounded by is illuminated. Angers – generous and mean-spirited – compassion, the sadness that we feel from entering into the particulars of another’s tragedy – these are the great drivers to an extrapolating imagination that takes us beyond what we know of the world to what we suspect of it; supporting that part of our world-picture – by far the greater part – which is not directly vouched for by our own experience. It helps that key figure within the very motor of our cognitive activity, the Professor of Data-Lean Generalisation who helps us to navigate through the world we live in, and to feel that we in some sense cognitively possess it, by being able to command a viewpoint on it. Emotion is the basis of that confirmation bias which is necessary to put together for ourselves a coherent narrative about the world or part of it. This journey, via emotion, from anecdote to world-picture is both our curse and our glory: it lies at the heart of our wider sense of responsibility – the desire to make a positive difference – and, equally, at the root of those generalising hatreds – racism, ‘classism’, all forms of ‘otherism’, which have made the world such an unpleasant place for some and potentially a dangerous place for all. The observation bias that is built into it, and the desire it has to confirm only itself (so that it will be satisfied with very few instances to support a statement of vast generality) enables us to be fluent about things of which we know little; to summarise, in the lounge bar of a pub, the character of an entire race of people in five ill-chosen words.
And it is round about here that we find the true greatness of great science. Like gossip and idle curiosity, it is manifestation of the fundamental human need to make sense of things, to build a picture of the world that vastly exceeds our experience so that we can understand and control it and, so by making the world a safer or more predictable place, to control our destinies. And at the same time, it does not allow the passion to determine what it sees.
It has, in short, managed to elevate its passion for understanding and for reliable knowledge - that will permit what Francis Bacon called ‘the effecting of all things possible’ – above those passions that normally permit us to transcend our limited individual consciousness and fuel our imagination. This is due in no small part to the fact that it is a collective passion that engages all mankind - an average issue of a leading scientific journal will have contributions from scientists from a dozen or more different countries and countless different backgrounds; that it is regulated by rules of inquiry that have been forged in bitter experience and bitter disappointment; and that takes place in the context of institutionalised scepticism.
This separation of passions is lies at the root of the ethic of science and is linked with what Claude Bernard advised as regards imagination: ‘Put off your imagination, as you put off your overcoat, when you enter the laboratory. Put it on again, as you put on your overcoat, when you leave’.
In the end, we must always begin with notions that are insecurely founded on a grossly inadequate database. Kingsley Amis once referred to the way we erect great belief systems on the basis of very little data as ‘the inverted pyramid of piss’ This is perhaps a little harsh, as the necessary starting point for belief formation in both everyday life and in science. In everyday belief formation it tends to be the end-point as well, whereas science moves on from there. While acknowledging the central role of stories, of anecdotes, in awakening us out of our own narrow consciousness to awareness of a world outside, we need equally to acknowledge that an understanding of the world rooted in anecdotes is cognitively underdeveloped. To give anecdotes too great a prominence in the debate on science, and medical science, and science policy, and healthcare policy, is to muddy those waters that it has cost scientist huge effort to clarify.
So, ye journalists, by all means offer your readers anecdotes but do not for one moment deceive either yourselves or your readers into thinking that this is science reporting or that one can move directly from anecdotes to robust generalisable knowledge or recommendations for application of science in healthcare or anywhere else or for the development of health science policy. Here, then, are some reminders that might be useful for journalists who find that they are reporting on science, particularly science as it applies to healthcare:
a) An anecdote is not a scientific datum, though it may justify a study to acquire data. Do not confuse experience – however intense and harrowing – with expertise.
b) The journey from a single observation to reliable, generalisable truth is long and arduous and full of pitfalls. Here are some of those pitfalls (all illustrated by the MMR furore:
Subscribing to the fallacy ‘Post hoc ergo propter hoc’: Event B happened after Event A therefore Event B is due to Event A.
Noting an association between Event A and Event B and concluding that the entire class of A-type events and B-type events will be associated. The propensity for confirmation bias means that you will preferentially notice, remember or be inclined to report, only those occasions in which there is an association.
Moving directly from observed association to imputation of causation. Both Event A and Event B may be the effects of some other event or factor.
Assuming that, because there is a causal link between Event A and Event B, or a certain factor seems to be operating in one instance of Event B, that the cause or factor is operating in all cases, or that it is the only cause or factor relevant to Event B.
These are hard lessons that mankind has learned reluctantly on the road to useful general truths. Do not be proud of unlearning them out of solidarity with folk common sense. Science in many respects is a triumph over the limitations of interested observers and often over common sense. Ask yourself again and again How typical is the story of the larger picture? How representative are these events? From what population of events are they drawn? For God’s sake give me denominators!
c) When discussing the beneficial or adverse effects of therapies, or more broadly technologies, consider the whole picture and the whole population. For example, even a treatment that has an adverse effect in some recipients may overall have a beneficial effect in the population. The anger of those who feel the treatment has injured them or their loved ones should prompt review of the evidence of overall benefit but should not count as evidence in itself.
d) When science fails to support Anecdote-Based Theory, do not assume that expert opinion:
is blinkered, or hobbled by ‘orthodoxy’. Science is in state of permanent reformation – that is how it advances so quickly;
is simply part of a lump called ‘the establishment’ that routinely conducts witch hunts against dissenters;
is in hock to the government, to Big Business or Big Pharma – or to other vested interests. Remember that those who point the finger at others’ vested interests – including you, the journalist - have vested interests themselves..
e) When ‘balance’ is sought, it is important that the two views given equal airing should have equal weight. One advocate of ABT, however impassioned, does not deserve as much of a hearing as one million data presented by someone who has acquired them in a way that has survived peer review by experts (who are in many cases professional competitors and on the lookout for errors). The 100% certainty of the anecdote-based theorist should not also be used to make the refusal to claim 100% certainty in science a weakness.
These principles should be known and embraced by all journalists – not just those charged with commenting on science but anyone whose comments may influence the views people may have about science, the application of science, science policy or the larger role of science in society. Indeed, it is outside the columns of science correspondents that the greatest damage to the collective understanding is done. The excellent work by people such as Geoff Watts, Georgina Ferry, Palab Ghosh, Mark Henderson is often overshadowed by the loud and more widely read ignorant certainties of characters such as Melanie Philips. Anyone who is going to make a comment about treatments for breast cancer in the General Features or about the heart attacks in GHQ should, at the very least, have some idea of the approaches to arriving at robust general truths that are employed in science; at the very least they should read Peter Medawar’s Pluto’s Republic, Lewis Wolpert’s The Unnatural Nature of Science, and Theodore Dalrymple’s Mass Listeria.
At any rate, media writers with a conscience should not collude with the cognitively primitive state in which we form most of our everyday beliefs. They should remember Bertrand Russell’s observation that while ‘popular induction depends upon the emotional interest of the instances, not upon their number’, reliable induction depends on numbers not emotional interest. If journalists give excessive weight to individual anecdotes they will trigger unnecessary worry and suffering, and sometimes cause diversion of resource from places where it would be of more use. Vox pop or even vox pop star is not vox dei.
Anecdotes are the first step on the road out of individual subjective consciousness to collective, objective knowledge. They are, however, only the first step. Science has gone far beyond this. If you are going to intervene in a debate on matters already subject to scientific investigation, you need to grow up out of your anecdotage. Otherwise journalism and the anecdotes it loves will get in the way of the collective clarity of thought that is going to be essential if we are to deal with the great challenges ahead of making this planet a safe, and reasonably comfortable, place for those who inhabit it.
The price of putting anecdotes in the driving seat is very high indeed. Uncritical focus on ‘the human story’ at the cost of ignoring or underplaying dull scientific facts may have inhuman consequences, as has been demonstrated by the recent return of measles and the first death from this disease for 14 years in the wake of the unfounded scare over MMR. For this I do not blame poor Jackie Fletcher but those journalists who, out of laziness and perhaps venality, gave her single experience and her heroic but misguided mission, more credence than it deserved.
So, ladies and gentlemen, there you have it: the curse of the media anecdote. It remains for me only to thank you again for the fantastic work you are doing, you dedication to quack-busting and this terrific honour. I shall display my award with pride, though I feel somewhat undeserving as well as proud.
- Gill Swain www.dailymail.co.uk/pages/live/articles/health/healthmain.html Last updated 11th July 2007.
- Friedrich Nietzsche The Gay Science Book 1, Section 2 (1982).
- Paul Valery Monsieur Teste translated by Jackson Matthews (London: Routledge & Kegan Paul, 1973), p. 65.
- Swain op cit
- I give examples of this in relation to MMR in Hippocratic Oaths. Medicine and its Discontents (London: Atlantic, 2004. See ‘Convictions’ p.109-127.
- See Tallis, ‘Convictions’ ibid. This chapter also covers the epidemiological evidence that demolished Andrew Wakefield’s claim of the link between MMR and autism.
- BBC ‘World at One’ 13th August 2007.
- See Lewis Wolpert The Unnatural Nature of Science (London: Faber, 1992)
- Quoted in Wolpert ibid p.19
- ‘First measles death for 14 years’ BBC News 24 3rd April 2006