Obstacles to honesty in medical research
Dr Peter Wilmshurst, a consultant cardiologist, has spent the last two decades trying to expose research misconduct and has reported more than twenty doctors to the General Medical Council. In recognition of his dogged and selfless pursuit of the truth, Dr Wilmshurst was presented with the HealthWatch Award 2003.
I feel greatly honoured to receive the Health Watch Annual Award and I am grateful for the opportunity to speak to you about obstacles to honesty in medical research.
I have been interested in this subject for 20 years, since I first experienced research misconduct when I was a research registrar. I hope that a personal account of my experiences may explain why I believe this is a serious problem.
In 1986 I went to the Guardian Newspaper with the story after the medical and pharmaceutical regulators refused to take any action. I supplied the Guardian's lawyers with over 200 pages of documents and statements, which convinced them that they could successfully defend any legal action if sued. We were not.
My research was on heart failure. This is a common condition and it has a worse 5-year survival than many forms of cancer. Twenty years ago there were few treatment options to improve symptoms and none was proven to improve survival. I was offered the opportunity to do research on a promising new drug, named amrinone. It was patented by Sterling-Winthrop. Preliminary research looked promising. Research, mainly from the company, showed that the drug increased the strength of contraction of the heart in animals. But the most influential article and the one that persuaded me that the drug was worthy of research was on patients and was published in the New England Journal of Medicine in 1978.
The New England Journal is the most influential medical journal in the world. The article came from the Cardiology Department at Harvard and one of its five authors was the most well known cardiologist in the world and head of medicine at Harvard, Professor Eugene Braunwald. The paper was given extra prominence by being the first article in that issue of the Journal and it was accompanied by an editorial.
In a large series of experiments we showed that, although amrinone increased the strength of contraction of normal heart muscle, it did not affect contractility in patients with heart failure. We also found that amrinone frequently caused life threatening side effects.
With hindsight there were two things that should have raised my concerns when we started our research. The first were anomalies in the study from Braunwald's group. It was a small study, which made claims that were not substantiated by the observations reported.
Later I discovered that though the article stated that the 5 authors were employed in the Cardiology Department at Harvard Medical School, 2 were full-time employees of Sterling-Winthrop and had never worked at Harvard. Two of the three that worked at Harvard were paid consultants to the company. These conflicts of interest were not declared. In fact the New England Journal of Medicine had no policy on declaration of conflicts of interest at the time. The first statement on conflicts of interest was published in the New England Journal one month after I wrote to the Massachusetts Medical Society, which owns the Journal, complaining about the undeclared conflicts of interest in this case.[3,4]
The second thing that should have alerted me was a letter published in the New England Journal of Medicine from cardiologists in Los Angeles. The letter reported fatal side effects from amrinone. The first author, Dr Stanley Rubin, had a patient with severe heart failure. The patient's wife was a stock-broker. She saw the dramatic increase in the price of Sterling-Winthrop shares after the paper from Braunwald's group was published. She reasoned that this proved that amrinone was an important advance. She asked Dr Rubin to get amrinone for her husband. Rubin was able to persuade the company to let him have amrinone on a named-patient basis and the amrinone swiftly killed his patient. Rubin and colleagues sent the New England Journal the first report of side effects with amrinone. They did not tell Sterling-Winthrop that they had submitted the report. Within 48 hours Rubin was under pressure by the company to retract the report. The Journal admitted that it had sent Sterling-Winthrop a copy of Rubin's report. The Journal initially refused to publish the report but was forced to do so when Rubin said that if they did not he would go to the press.[3,6,7]
However the conflicts of interest involving the New England Journal, the Cardiologists at Harvard and Sterling-Winthrop did not end there. The company later produced a congener of amrinone, named milrinone. The initial human research on milrinone was also performed in Braunwald's department. Unusually it was agreed before the research had been completed that it would be published in the New England Journal. When the first 2 referees chosen by the journal to review the paper recommended rejection, the editor, Dr Arnold Relman sent the article to 2 more referees. They also recommended rejection, but the Journal published the paper on milrinone as previously agreed.[3,6,7] This says much about peer review in the World's most prestigious medical journal.
I discovered this much later. In the early days of our research my colleagues and I were more concerned that we could not confirm in our large number of experiments claims made in the small study from Braunwald's department.
We reported to Sterling-Winthrop that we were unable to find evidence that amrinone injections increased contractility in patients with heart failure and we reported our experience of serious adverse effects with the oral preparation of the drug. Company employees asked us to exclude some patients from the analysis. These were ones where there was a downward trend in contractility. The effect of excluding them would have been to produce an apparent but spurious increase in contractility in the remainder. We refused. My supervisor and I were then threatened with litigation. We published.
Our on-going research studies on amrinone ended when company employees removed the drug stocks from the pharmacy in the hospital and research institute. As a result, 2 of our publications contain statements pointing out that the studies were smaller than planned because Sterling-Winthrop had prematurely discontinued our trials without our agreement.[9,10]
A number of tactics were used to try to prevent my colleagues and I presenting our findings at meetings and to discredit us when we did present. One strange incident involved one of my colleagues, Alex Crowther, who was due to present some of our work on amrinone on the second day of a meeting in Luxembourg. He just managed to get on the last flight of the day that would permit him to attend the first session of the meeting. When he arrived he discovered that his talk had been rescheduled for the previous day. The organisers had received a forged letter that appeared to be from him asking for his talk to be brought forward a day. Those responsible were never identified.
When I presented our findings on side effects a company employee stood up and said that I had made up the findings. I had to point out that I was an independent investigator, but that my accuser was a company employee. I had nothing to gain by claiming that the drug was unsafe. I asked the chairman to appoint people to review our data. A few days after the meeting I received an apology from the company, but the hundreds who heard the allegations at the meeting would not be aware of the company's retraction.
At a number of other meetings at which I presented our findings, three eminent professors of cardiology, each of who was a paid consultant to Sterling-Winthrop, made public statements that they had tried to replicate our findings and failed. None of them acknowledged their affiliation to the company. Twenty years later none of those failures to replicate has been published. This tactic came to an end at a European Congress of Cardiology, in front of several hundred doctors. I pointed out that a professor who made these claims was a paid consultant to the company and that he had been making the claims for two years. I suggested that if he continued to make the claims without publishing his data people might think that he was lying. My findings were not challenged again.
At one point, my supervisor and I were asked to meet with the company and a different American professor of cardiology who is an opinion leader in the treatment of heart failure and who was a consultant to the company. The American professor told us that we were mistaken about the drug. He said that he was aware of finding by other investigators and that these entirely refuted ours. He advised us that we should not publish any more of our findings. He said that we would be found to be wrong and our reputations would be adversely affected. We went on to present 14 abstracts, and 15 publications.
One of the presentations was at the American Heart Association meeting in November 1982. I presented data, which showed that amrinone did not have the cardiac effects claimed. After my presentation, 3 professors of cardiology at separate American university hospitals told me that they had also obtained results similar to ours. They were unaware of each other's research or of our research. They informed Sterling-Winthrop. The company arranged meetings between each of them individually and the same professor of cardiology, who had told us that our findings were aberrant. He also told each of them the same thing. He persuaded two of them not to publish. The third did publish, after much soul searching because he was afraid that he would lose research contracts with Sterling-Winthrop and other pharmaceutical companies. After he published he received threats, including a threatening phone call at 2am.
The Netherlands Committee for the Evaluation of Medicines spotted our paper on the side effects of amrinone. There were major discrepancies when compared with the clinical record cards submitted by the company on our patients. We showed that the company had sent the Netherlands Committee forged clinical records for our patients with the information on adverse events deleted.
Because of this I contacted the UK Committee on Safety of Medicines and discovered that Sterling-Winthrop had also failed to notify the CSM of side effects in our patients. During discussions I discovered that contrary to statements made to us at the outset of our research, Sterling-Winthrop had not obtained a Clinical Trials Certificate for oral amrinone, though they had got a CTC for amrinone injection. This meant that the research with oral amrinone conducted by us as well as by doctors in the National Heart Hospital in London, in Newcastle-upon-Tyne and in Birmingham had been illegal.
When I raised this with the company, the senior vice president bragged that they were telling the government that if the company was prosecuted it would close down its large manufacturing plant near Newcastle upon Tyne. The company was not prosecuted for breaches of the Medicines Act.
I tried unsuccessfully to get sanctions against the company or its employees, but the Association of the British Pharmaceutical Industry, the Faculty of Pharmaceutical Medicine of the Royal College of Physicians and the General Medical Council were not interested. I spoke to editors of medical journals, including BMJ, Lancet and Nature. None disputed the facts but all were afraid to take on a multinational pharmaceutical company with unlimited financial and legal resources. One editor mentioned the loss of advertising revenue from the company.
The process of being rejected by all the official bodies that I believe should have dealt with the issues took nearly 5 years. While this was going on, in 1984, the company told a hearing of the Food and Drugs Administration in the USA that there had been over 1400 serious adverse events in 1200 patients given amrinone in trials and the company announced that they would cease trials and applications for product licences worldwide. Officially the drug was unsafe to take even on a doctor's prescription. Two years later, in 1986, I discovered that the company was still marketing amrinone in parts of Africa and Asia. In those countries it was being sold as an over the counter treatment for heart failure. I approached Oxfam, which had workers in the developing countries where this was happening. They collected evidence, which was presented at a meeting of the World Health Association in Geneva. Sterling-Winthrop was finally embarrassed into withdrawing the drug world wide in 1986.
It was my contact at Oxfam who put me in touch with James Erlichman, a Guardian reporter. He and the deputy editor, Peter Preston, were convinced by the evidence I had and so were the Guardian's lawyers. The paper covered the story on the front, back and the whole of an inside page of one issue and in follow-up stories in other issues.
I had seen how corporate greed and personal ambition had tended to distort scientific evidence. Sterling-Winthrop believed that my supervisor and I could be bribed or threatened into suppressing our data. Others, such a Drummond Rennie, Deputy Editor of the Journal of the American Medical Association, have documented this occurrence. Some professors preferred to suppress their findings rather than run the risk of losing prestige by appearing mistaken or losing lucrative contracts for future pharmaceutical research. Financial conflicts of interest caused some opinion leaders to behave dishonestly. Conflicts of interest, affected publication decisions at the New England Journal of Medicine. The institutions including government, which one might expect to help preserve research integrity, were not prepared to take on a multinational pharmaceutical company.
However these are not the only obstacles to honesty in medical research I have come across. In one case an eminent clinician, who was the president of his specialist society, and who had a large private practice doing a particular interventional procedure wished to publish a series of 400 cases. It was then the largest series in the United Kingdom. When the data was analysed it was found that his mortality rate for the procedure was unacceptably high compared with rates in other countries. If this became known it would have a disastrous impact on his private practice. So the mortality rate was falsified. However, they had already published an abstract at an obscure meeting at which amongst other things they reported the deaths in the first 254 patients. The number of deaths reported in the abstract was greater than in the 400 reported in the paper. This discrepancy became common knowledge in the specialty. I was present during a meal at which a junior doctor that was a co-author of the paper admitted that the falsification had occurred. He implied that he and other junior doctors had little option but to go along with their boss. Five other junior doctors heard the admission. I contacted the editor of the journal. It was part owned by the specialist society of which the senior author of the paper was the president. The editor knew of the rumours. He said that if I could get one of those who heard the incriminating admission to confirm it, he would act. I went back to those who had heard the admission. Now, years after those events, some have provided me with written statements confirming that they heard the admission, but at the time all said that they would not support my efforts to get the paper retracted. Some said that it would be bad for their careers. Some said that it would be bad for medicine or the specialty. One said that he thought that it was the sort of thing that any of us would do. Those 5 junior doctors went on to get consultant posts and one went on to be a president of the society himself.
My efforts to get the paper retracted were common knowledge in the specialty. I was asked to see the post-graduate dean who advised me to stop upsetting influential people. Until that point things had gone well in my career. As an undergraduate, I had obtained honours or distinction in 10 out of 11 subjects. I had been awarded an Honours degree overall, plus six undergraduate prizes and an Intercalated B.Sc. My house jobs were in my teaching hospital, and included the professorial medical job. Then I was senior house officer at the Hammersmith and in Oxford, medical registrar at Northwick Park, and cardiac registrar and senior registrar at St Thomas'. After these events, for the first time in my career, I had difficulty getting a job. I stopped counting the rejections after the 42nd. In many cases individuals with much less clinical and research experience were appointed. It was clear to me that loyalty, no matter how misplaced, was valued more highly in medicine than honesty.
I believe that obstacles to honesty in medical research generally fall into a few categories. One is personal ambition for promotion, advancement, money, kudos and power.
A second obstacle is that those who achieve success by becoming heads of departments or institutions can only maintain their position if their institution continues to succeed. Success is judged in many ways, but the most common measure of success is the balance sheet. Department heads are expected to pull in research grants. So money is another obstacle to honesty in research. This does not apply purely to pharmaceutical companies. I do not imagine that executives of Elsevier, which owns the Lancet, asks the editors much about the research published. I imagine that Elsevier asks how much was earned from drug advertising, how much was earned from sales to pharmaceutical companies of reprints of trials showing their drugs in a positive light and how the current citation rating will affect circulation profits. Of course academic institutions are the most mercenary of all.
However the greatest obstacle to honesty in medical research is the code of silence that pervades the medical profession and the research establishment. There is still considerable reluctance to shop another doctor, no matter how dishonest he is. In this setting of tolerance is there any wonder that ambitious young doctors, aware that to progress they need lots of publications with exciting findings, will embellish their findings and some will falsify the lot? Should we be surprised that a search for funding for their department and personal gain, from drug company consultancies, result in dishonest behaviour by senior academics and opinion leaders? Who will blow the whistle on them? Institutions seeking high rating in the research assessment exercise will try to suppress knowledge of dishonesty in their establishments, even to the extent of letting the guilty escape punishment. Those institutions demand success from their department heads and do not look too carefully at whether that success was achieved honourably or honestly. In this setting it is almost invariable that whistle blowers are damaged more than the guilty they expose. Academic institutions and journals do not want to be associated with dishonest research and treat harshly anybody that brings it to attention.
I have, with difficulty, persuaded a few journals to publish a small number of articles describing research misconduct.[3,13,14] Each article has been reviewed sentence by sentence by lawyers wanting evidence to support individual statements. This was because the editors of the journals were concerned that they might by sued if individuals or institutions were libelled. In a libel case it is no defence to say I am only the publisher not the author. This is in stark contrast to scientific publications. I have submitted many scientific articles for publication and many had implications for survival of patients, but no journal has ever asked me to prove that I got the results claimed. This might suggest that medical journal editors are more concerned with the reputations of academics and their institutions than the lives of patients. The simple truth is that editors are most concerned with money. Journals are never sued for publishing false results no matter how many patients died. In scientific research they can have the best of both worlds. They are absolved from blame if a study is wrong and gain an improved impact rating if the research is an important advance. A higher impact rating increases revenue from sales and advertising. Editors know that research can bring major reward to individuals and organisations, which may act as a temptation for dishonesty, but journals accept submissions on trust without checking their accuracy. Journals almost never retract work shown to be false. When they do, they make it clear that publication of the false research was entirely the fault of the authors. I would like to see whether the policies at journals changed if some were sued by patients harmed by implementation of treatments based on their publications.
There are few objective medical scientists, because they all know that success in their career is dependent on the results they obtain. Every one has a conflict of interest, everyone is human and some are venal.
Do academic institutions or journals recognise the humanity and venality of their staff? They do in some areas of activity. When paying wages, do any of these organisations leave out a bag of money and trust their staff to take the wages to which they are entitled? Of course they don't, because they realise that for some the temptation for dishonesty would be too great. The gains from dishonesty in research can be greater but institutions and journals trust researchers not to fall prey to these. We need to put in place robust checks on research. I believe that there should be random checks of raw data of work in progress and of submitted work. We know that use of performance enhancing drugs is common in competitive sports because of enforced drug checks without warning at sporting events and between events. If we did not have these checks we might mistakenly conclude that doping was not common in sport. I believe that the checks reduce the dishonesty in sport. We need a similar approach to research. The raw data could be demanded at a routine check during a visit to the research institution or when the research is submitted for publication. Failure to produce the raw data should be considered the equivalent of failing the inspection and should result in a ban on future research for a specified period and a review of previous research published. A finding that a department in an institution had falsified research should be a negative factor when assigning ratings in the research assessment exercise. In this setting justified whistle-blowing would be welcomed by institutions. Publication of dishonest research by a journal should affect its impact rating. The failure of a journal to publish a retraction of dishonest research should have a multiplied negative effect on the journals rating.
However the most important thing is that we must change the culture in medicine in which research success is viewed as the passport to success in ones career. For most clinicians only a limited experience of research is required to enable you to understand what you read in research articles and to participate in multicentre trials, organised by career medical scientists.
However there is a more fundamental problem, which is the issue of honesty. Most medical students start with high ideals. Research, which I hope is honest, has shown that as medical students go through medical school a progressively greater proportion believe that cheating in exams is acceptable. The institutions tolerate it. Three years ago Richard Smith wrote in the BMJ about a medical school that permitted a student caught cheating in the final exams to pass. I know of examples where Universities have refused to withdraw higher research medical degrees that are known to contain falsified research. I know of an academic institution in London in which senior officers know that one of their professors lied about his qualifications when he was appointed to that institution. Specifically he claimed to have a MD that he had not been awarded. The institution does not think he should be sacked and the GMC does not feel that he should appear before it. In that and other institutions there is tolerance of dishonesty at all levels. Only a sea change in opinion will produce the required improvement. I fear that it must be imposed from without because our leaders in medicine and academe lack the appetite to produce the required changes.
1. Erlichman J. Drug firm "made threats". Company tested heart drug with DHSS clearance. The Guardian 3rd November 1986; 1 and 6.
2. Benotti JR, Grossman W, Braunwald E, Davolos DD, Alousi AA. Hemodynamic assessment of amrinone. N Engl J Med 1978; 299: 1373-7.
3. Wilmshurst P. The politics of disclosure. Lancet 1997; 349: 510.
4. Relman AS. Dealing with conflicts of interest. N Engl J Med 1984; 310: 1182-3.
5. Rubin SA, Lee A, O'Connor L, Hubenette A, Tober J, Swann HJC. Thrombocytopenia and fever in a patient taking amrinone (letter). N Engl J Med 1979; 310: 1185.
6. Relman AS. The politics of disclosure. Lancet 1997; 349: 885.
7. Wilmshurst P. The politics of disclosure. Lancet 1997; 349: 1558.
8. Baim DS, McDowell AV, Cherniles J et al. Evaluation of a new bipyridine agent - milrinone - in patients with severe heart failure. N Engl J Med 1983; 309: 748-56.
9. Wilmshurst PT, Walker JM, Fry CH, et al. Inotropic and vasodilator effects of amrinone on isolated human tissue. Cardiovasc Res 1984, 18: 302-9.
10. Wilmshurst PT, Thompson DS, Juul SM, Dittrich HC, Dawson JR, Walker JM, Jenkins BS, Coltart DJ, Webb-Peploe MM. Effects of intracoronary and intravenous amrinone infusion in patients with cardiac failure and patients with near normal cardiac function. Br Heart J 1985; 53: 493-506.
11. Wilmshurst PT, Webb-Peploe MM. Side-effects of amrinone therapy. Br Heart J 1983; 49: 447-51.
12. Rennie D. Thyroid storm. JAMA 1997; 277: 1238-43.
13. Wilmshurst P. The code of silence. Lancet 1997; 349: 567-9.
14. Wilmshurst P. Institutional corruption in medicine. BMJ 2002; 325: 1232-5.
15. Smith R. Cheating at medical school. BMJ 2000; 321: 398.
16. Wilmshurst P. Doctors seem not to be punished for dishonesty in their cv. BMJ 2001; 323: 1309.
Medical journals: an extension of the marketing arm of drug companies?
Everyone knows that medical journals and drug companies are economically interdependent. Drug companies need journals to publish the results of their clinical trials showing the efficacy and safety of new drugs in order to obtain product licences for these drugs. Journals obtain substantial income from drug advertisements, reprints and supplements. "So what?" you may say, "isn't this how it is with all products advertised through the print and broadcast media, so why make a fuss?"
After accepting the 2004 HealthWatch Award Dr Richard Smith, formerly editor of the British Medical Journal, explained to his audience why it is very necessary to make a fuss, reports John Garrow.
The relationship between medical journals and the pharmaceutical industry is far more intimate and sinister than that between advertisers and the general media. It is not only Smith that says so. He cited some important commentators in support of his case: Lancet editor Richard Horton (1) claims, "Journals have evolved into information laundering operations for the pharmaceutical industry". Marcia Angell, former editor of the New England Journal of Medicine, has commented (2), "[The pharmaceutical industry] has moved very far from its original high purpose of discovering and producing useful new drugs. Now primarily a new marketing machine to sell drugs of dubious benefit, this industry uses its wealth and power to co-opt every institution that might stand in its way, including the U.S. Congress, the Food and Drug Administration, academic medical centres, and the medical profession itself." The promotional blurb of a book by Jerry Kassirer (3), another previous editor of the NEJM, reads, "Dr Jerome Kassirer offers an unsettling look at the pervasive payoffs that physicians take from big drug companies and other medical suppliers, arguing that the billion-dollar onslaught of industry money has deflected many physicians' moral compasses and directly impacted the everyday care we receive from the doctors and institutions we trust most."
These are eminent medical editors whose warnings we should heed. If, as Kassirer states, the enormous economic power and influence of drug companies has "deflected many physicians' moral compasses" to the extent that the public is now fed misleading information about the efficacy of drugs, this is something HealthWatch should mind about very much indeed. A few simple figures illustrate the great economic power and profitability of the drug industry. They are now producing relatively few new drugs, but concentrating their efforts more on marketing existing drugs. US companies spent $15.7 billion on drug promotion in the year 2000. That represents about $10,000 on each individual doctor. Since 1995 research staff numbers have been reduced by 2%, while marketing staff have increased by 59%. Researchers comprise just one in five of drug company staff - they are outnumbered two to one by marketing staff.
At present prescription drugs cannot be advertised in the general media (although this rule is being undermined by advertisements on the internet) so the industry needs to persuade doctors to promote and prescribe new drugs in place of older and less profitable ones. It has been shown that endorsement by doctors is more effective in altering prescribing practice than an equal expenditure on company representatives. Independent expert reviewers have found that about half of the advertisements for prescription drugs in medical journals are biased in favour of the drug, giving too great prominence to claims of efficacy, and too little to side effects, and that often the advertisement recommended the use of the drug in a patient group other than that in which it had been tested. It would be possible to peer review all advertisements in medical journals, but this would be very expensive and most editors would rather spend the money on maintaining as high a standard as possible in the research publications. It is not unknown for advertisers to strike a deal with editors, such as favourable editorial mention of a drug in return for placing an expensive advertisement. Some journals carry a section on "product news" which appears to be independent but is in fact "advertorials". And as we heard from the winner of the 2003 HealthWatch Award, Dr Peter Wilmshurst, it may be very difficult to publish reports of an adverse drug effect in major journals, because the makers of the drug will fight vigorously to suppress any such publication.
But perhaps the most important, and certainly the most subtle, way in which the drug industry can influence the opinion of doctors about the efficacy of a drug is by the publication of clinical trials in reputable journals. Systematic reviews of randomised controlled trial are the very best evidence on which Evidence Based Medicine is based. But the systematic reviews are only as reliable as the trials that the meta-analysts have analysed. Of course if trials are technically poor (bad randomisation, weak blinding, inappropriate statistical analysis, etc.) they are given little weight in good systematic reviews, but if the trial is technically perfect we must believe it, mustn't we?
It is unthinkable that reputable pharmaceutical companies and reputable medical journals would collude to publish fraudulent results about a drug trial, but we should bear in mind the pressures that operate when a major drug trial is submitted to a journal for publication. From the journal's viewpoint the financial benefits of publishing the trial are very large (see below). From the drug companies' viewpoint the stakes are even higher. They have already spent many £m in developing the drug, but if a major multicentre trial shows that the drug is relatively ineffective, or has serious side effects, that is a commercial disaster that must be avoided if that is at all possible. Initial drug trials are usually designed and funded by the manufacturer of the drug, the design of the trial is beyond reproach, and almost always the results are favourable to the sponsor. However, when independent researchers study the same drug the results are usually less favourable, and in some cases the drug is withdrawn because it is shown to be ineffective or unsafe. How can this occur?
Dr Smith led us through the methods that can enable companies to get the results they want without falsifying the data:
No need to falsify data: ways in which companies might use real trial results to get the results they want
The new drug can be compared with placebo, or too low a dose of a competitor drug, so the new drug is shown to be "effective", when really it is no better than an alternative treatment.
The new drug can be compared with too high a dose of a competitor drug, so it can be seen to have fewer side effects.
The new drug can be compared with a better (but more expensive) drug in a small trial so the results show "no significant difference" and the new drug appears good value for money.
The trial may have several different end-points, and the report cites those results in which the new drug performed well, but not those in which it performed badly.
The drug may be tested on a heterogeneous group of patients, some of whom did well and others badly. Select a group (eg. men over age 50) who did well and publish those results and forget the rest.
If there is no subgroup that does well do not publish that study at all.
If you have a good study, publish it more than once.
Sponsor multicentre trials, but publish only those centres that show favourable results.
Publish separately different outcome measures from the same trial.
Publish different follow-up periods, eg. results at 3 months, one year, two years...
Publish positive results in major journals and negative or neutral results in minor journals.
Combine results in ways that are favourable.
As an illustration he cited one particular drug about which there were publications describing 84 trials on 11,980 patients. In fact there were only 70 trials involving 8,645 patients, but 17% of the trials had been published more than once, though this was impossible to tell from the published studies. Smith used a Cochrane-type diagram to illustrate the way in which duplicated trials could increase apparent effectiveness. Initially 16 trials (group A) showed that the Number Needed to Treat (NNT) to obtain one favourable result was 9 patients (confidence interval 7-16). Three of the most favourable trials were duplicated (group B): now analysis showed only 4 NNT.
Next group B was duplicated again to give group C, so combining B+C gave 9 trials with 4 NNT, and finally combining all the trials and their selected duplicates there were (apparently) 25 trials giving 5 NNT (CI 4-6), which is a considerable improvement on the initial 9 (CI 7-16). Further examples were given of drugs that had apparently favourable clinical trial evidence (for example Cox-2 inhibitors vs NSAIDs, or HRT to protect against coronary heart disease) but scrutiny by independent experts showed the evidence was flawed. Setting the record straight about the efficacy of drugs is not a task for the faint-hearted: those with an interest in selling, say, Cox-2 inhibitors or HRT will fight fiercely to discredit any evidence that their value has been overstated.
At this stage of the address Dr Smith had convinced us that his title proposition was true: to a very great extent medical journals are an extension of the marketing arm of the drug companies, and someone should do something to correct the situation. But what about those pillars of society - physicians sworn to serve only the interests of their patients? Or the medical press - are they not part of the Fourth Estate, champions of the people, said by Edmund Burke to be more powerful than parliament itself? Why do they condone this scandalous abuse of public trust? As I looked around the audience, many of whom were, or had been (like myself) physicians and medical editors, there were no confident smiles to be seen. A lone representative of the pharmaceutical industry was not looking very happy. How had we got into this mess, and how could we get out?
The first question was easily answered. Reprints of important drug trials, or supplements sponsored by drug firms, are major sources of revenue for journals. Two-thirds of trials in major journals (Archives of Internal Medicine, JAMA, Lancet, New England Journal of Medicine) are funded by the drug industry - for the British Medical Journal it is only one third4. Editors of journals (or their publishers) cannot afford to reject everything that is commercially sponsored. In many fields of medical research (such as obesity, in which I have experience) industry is virtually the only source of funding to employ research registrars, or buy expensive equipment. Unless (like me) they are fortunate to have departments funded by a charity, the great majority of physicians depend upon industry for their research bread and butter, never mind the champagne and canapés that may also be on offer. Politicians call it "partnership with industry" but it is a partnership in which power lies with the commercial sponsor.
What is the solution? Dr Smith offered some answers: A register of trials, so unfavourable trials do not "disappear". Publication of online journals not beholden to commercial sponsorship, such as PLOS Medicine (http://medicine.plosjournals.org); critical review of trial protocols by independent experts; and of course much more public funding of clinically-important trials. Whether his call will be heeded remains to be seen, but it will be certainly receive support from members of HealthWatch.
At the end Dr Smith received a standing ovation, and coped with another 30 minutes of well-informed and pertinent questions. Truly he is a worthy recipient of the 2004 HealthWatch Award.
1. Horton R. New York Review of Books; March 11, 2004.
2. Angell, Marcia. The New York Review of Books; July 15, 2004; 51 (12).
3. Jerome P. Kassirer. On the Take: How Medicine's Complicity with Big Business Can Endanger Your Health. Oxford University Press, October 2004.
4. Egger M, Bartlett C, & Jüni P. Are randomised controlled trials in the BMJ different? BMJ, 2001; 323: 1253.
How the web has turned the tables on pseudo-science
HealthWatch newsletters go back to long before the recent proliferation of “skeptical blogs”. As far back as 1991, Issue 8 (Oct/Dec 1991) included an article about the use of the Vega Test to diagnose allergy. Since then the Vega test has been debunked again and again—for example in the BBC’s Inside Out programme in 2003,1 and three years later as “The great allergy con” in the DailyMail.2
Shortly afterwards I wrote about the test on my blog3, when I discovered it being offered at the private practice of a practitioner who had himself written a paper saying it didn’t work. And only a few days ago it was exposed on the BBC yet again, this time on Watchdog.4 Outrageously, consumer protection laws seem not to be being implemented in this country.
Outrage about pseudoscience is not new. Alfred Joseph Clark FRS held the established chair of Pharmacology at University College London from 1919 to 1926, when he left for Edinburgh. In 1938 he quoted, in his short book “PatentMedicines”, from a House of Commons Select Committee report on Patent Medicines that had been submitted to the House 24 years earlier:5
“For all practical purposes British law is powerless to prevent any person from procuring any drug, or making any mixture, whether patent or without any therapeutical activity whatever (as long as it does not contain a scheduled poison), advertising it in any decent terms as a cure for any disease or ailment, recommending it by bogus testimonials and the invented opinions and facsimile signatures of fictitious physicians, and selling it under any name he chooses, on payment of a small stamp duty. For any price he can persuade a credulous public to pay.”
His son relates what happened next, in his own memoir.6
“To AJ’s surprise and dismay he was sued for libel by a notorious rogue who peddled a quack cure for for tuberculosis. This man said that AJ’s remarks (such as “‘Cures’ for consumption, cancer and diabetes may fairly be classed as murderous”) were libellous and would damage his business. AJ was determined to fight, and he and Trixie decided to put their savings at stake if necessary. The BMA and the Medical Defence Union agreed to support him and they all went to lawyers. He was shocked when they advised him that he would be bound to lose for he had damaged the man’s livelihood! Finally, after much heart searching, he made an apology, saying that he had not meant that particular man’s nostrum.”
So are we making progress? After the irrationality of the 1960s the tide is beginning to turn. Today journalists know that if they write nonsense they’ll be dumped on fairly quickly by bloggers like Ben Goldacre (www.badscience.net), Quackometer (www.quack ometer.net) and Gimpyblog (http://gimpyblog.wordpress.com). But there’s still a long way to go, especially when our academic institutions continue to promote non-science.
The University of Westminster runs eleven alternative medicine degrees with titles including herbalism, chinese medicine, nutritional therapy, acupuncture and naturopathy. Middlesex University offers degrees in ayurveda, herbalism, traditional chinese medicine and acupuncture. Edinburgh’s Napier University offers degrees BA (Hons) in aromatherapy and reflexology (although the herbal medicine qualification disappeared after my enquiries under the Freedom of Information Act).
Wales is another example. The University of Wales Institute in Cardiff offers four degrees in complementary therapies, holistic massage, clinical aromatherapy and reflexology. Glyndŵr University offers degrees in traditional chinese medicine, reflexology and aromatherapy.And the University of Glamorgan offers two degrees in chiropractic.
Two years ago Polly Toynbee, writing in the Guardian,7 questioned the spending of public funds on alternative therapies and complained that the policy encouraged, “the burgeoning number of degrees and diplomas in complementary therapies offered by universities, such as the Thames Valley, Westminster or the University of Wales. Normal academic standards have been set aside for attracting new students. Legitimate fears that this gave a phoney scientific aura to humbuggery of all kinds are now proved right.” She duly received a letter from vice-chancellor and chief executive of the University of Wales, Professor Marc Clement, who invited her to the University to meet the validation staff so she could see for herself how their validation and monitoring procedures are applied and so to reassure herself regarding the academic standards.
In fact much information about the validation of courses at the University of Wales is readily available but I don’t know whether Ms Toynbee would be reassured by some of the information I’ve come across. On the university’s website, under the heading, “The Validation Unit”,8 we read,
“While the majority of the University’s students study in Wales, there is also an important international dimension to its work. It has in place a very successful and highly regarded international validation operation, which enables overseas institutions to offer the University of Wales degree at an equivalent standard to the degree offered in Wales itself.
“Validation is important in fostering links between Wales and other countries ... In 2008, more than 20,000 students were registered on validated courses of the University of Wales in 30 countries, covering a wide variety of academic disciplines.” In economic terms, it continues, “it is a significant export, each year generating overseas earnings of well over £2 million.”
But what is actually taught on these external courses? One course, though accredited by the University of Wales, was actually taught at the Northern College of Acupuncture in York. That is private and so not covered by the Freedom of Information Act (an increasing problem). I assumed that the accreditation committee would know what was taught, but the answers to my enquiries suggested not only that they didn’t but that they hadn’t even seen a detailed timetable. In June 2007 a press release promoted the new diploma/MSc course.
“The course uniquely combines the study of Western, naturopathic and traditional medicine approaches to nutrition—the best of East meets West—together with actual clinical practice of nutritional therapy. It covers the nutritional approach to a wide range of ailments, from acne to urinary infections and also incorporates meal planning, health foods, food preparation and nutritional research.”
Guest lecturers include Dr John Briffa, Professor Jane Plant MBE, and Patrick Holford. The course leader was clinical psychologist Jacqueline Young, author of “Complementary Medicine for Dummies”,9 who was famously quoted by The Guardian’s “Bad Science” columnist Ben Goldacre10,11 saying, “Implosion researchers have found that if water is put through a spiral its electrical field changes and it then appears to have a potent, restorative effect on cells.” Elsewhere she is said to recommend taking an “air bath”—“stand naked in a room at home or in your garden and walk around exposing your skin to different air flows and temperatures … do light exercises or skin brushing … continue walking for five to ten minutes but don’t let yourself get cold.”
The accreditation committee seemed quite unaware of this information. On 26 October 2008 my enquiries to the University of Wales resulted in a reply from its chair, Professor Nigel Palastanga, who wrote, “I personally am not familiar with her book and nobody on the validation panel raised any concerns about it … we would have considered [her CV] as presented in the documentation as part of the teaching team. In my experience of conducting degree validations at over 16 UK Universities this is the normal practice of a validation panel.”
The vice-chancellor, Marc Clement, failed to respond when I asked his opinion, as an engineer, of statements like, “Implosion researchers have found that if water is put through a spiral its electrical field changes and it then appears to have a potent, restorative effect on cells.”
In 2008, Palastanga was promoted to pro-vice-chancellor with responsibility for quality of teaching, and this year JacquelineYoung was awarded a teaching Fellowship at the University of Wales. The University of Wales validates no fewer than 11,675 courses altogether. Many of these are regular courses in universities in Wales, but they also validate 594 courses at non-Welsh accredited institutions, an activity that earned them £5,440,765 in the financial year 2007/8. It does seem a bit odd that St Petersburg Christian University, Russia, and the International Baptist Theological Seminary, Prague, should be accredited by the University ofWales. They also validate the International Academy of Osteopathy, Ghent (Belgium), the Osteopathie Schule Deutschland, the Istituto Superiore Di Osteopatia, Milan, the Instituto Superior De Medicinas Tradicionales, Barcelona, the Skandinaviska Osteopathögskolan (SKOS) Gothenburg, Sweden and the College D’Etudes Osteopathiques, Canada. The 34 UK institutions include the Scottish School of Herbal Medicine, the Northern College of Acupuncture and the McTimoney College of Chiropractic.
My Freedom of Information enquiry into the McTimoney course produced tons of accreditation documents but no teaching materials, on the grounds that they didn’t possess them. Only McTimoney had them. The University’s Freedom of Information officer replied, “The University is entirely clear about the content of the course but the day to day timetabling of teaching sessions is a matter for the institution rather than the University and we do not require or possess timetable information. TheAct does not oblige us to request the information but there is no reason you should not approach McTimoney directly on this.”
So the university doesn’t know the timetable. It doesn’t know what is taught in lectures, but it is “entirely clear about the content of the course.”
The university may be satisfied with what is taught about McTimoney Chiropractic. But the McTimoney Chiropractic Association, it seems, is not. On 8 June 2009 they sent a letter to their members urging them to take down their websites immediately because of fears that they might include unsubstantiated claims. They wrote, “If you have a website, take it down NOW.” The General Chiropractic Council itself, under pressure from over 600 complaints against its members, changed its mind in May 2010 about the very heart of the chiropractic myth, ‘subluxation’.12 The chiropractic vertebral subluxation complex is an historical concept but it remains a theoretical model. It is not supported by any clinical research evidence that would allow claims to be made that it is the cause of disease or health concerns.
This overturns much of what is taught to chiropractors. How did the University of Wales manage to miss it when accrediting the course? Why has the The Quality Assurance Agency for Higher Education not acted? Why has Universities UK (UUK), which represents UK university vice-chancellors, done nothing about it?
Could it be that they have been overtaken in the matter of intellectual integrity by what Ben Goldacre has called the “ragged band of bloggers”? The advent of the web has allowed anyone to be their own science journalist. Since about 2000, when Goldacre started to write his Thursday “Bad Science” column in The Guardian, there has been a rapidly increasing number of “skeptical bloggers”. Any journalist who writes rubbish can expect very rapid debunking. Now even the tabloid press have (some) good science. The web (together with the Freedom of Information Act) has made it very difficult to keep secrets. That is almost always a good thing.
Professor of Pharmacology, University College London
1. BBC Inside Out - South: Monday 17 February 2003. See http://www.bbc.co.uk/insideout/south/series2/food_sensitivity_ allergy_vega_tests.shtml
2. Dovey C, The great allergy con. The Daily Mail, 7 March 2006. See http://www.dailymail.co.uk/health/article-379166/The-great-allergy -con.html
3. David Colquhoun’s Improbable Science blog, 5 July 2006. See http://www.dcscience.net/?p=131
4. BBC Watchdog: Monday 18 October 2010. See http://www.bbc.co.uk/blogs/watchdog/2010/10/food_intolerance.html
5. Clark AJ. Patent Medicines. Published by FACT, London WC2, 1938. See http://www.dcscience.net/?p=257#more-257
6. Clark D. Alfred Joseph Clark. A memoir. C & J Clark Ltd, 1985. See http://www.dcscience.net/?p=257#more-257
7. Toynbee P. Quackery and superstition—available soon on the NHS. The Guardian, Tuesday 8 January 2008. See http://www.guardian. co.uk/commentisfree/2008/jan/08/politics.publicservices
8. The Validation Unit, on the website of the University of Wales. See http://www.wales.ac.uk/en/AboutUs/Structure/ValidationUnit.aspx
9. Young J. Complementary Medicine for Dummies. Published by John Wiley & Sons, 13 July 2007.
10.Goldacre B. What is an implosion researcher? Or an electric field of water? DrArbuthnot would like the BBC to say. The Guardian, Saturday 24 September 2005. See http://www.guardian.co.uk/science/ 2005/sep/24/badscience
11. Goldacre B. Oh, what a tangled web is being woven on the BBC health site. The Guardian, Saturday 1 October 2005. See http://www.guardian.co.uk/science/2005/oct/01/badscience
12. General Chiropractic Council. Advice on the research base for the chiropractic vertebral subluxation complex. See: http://www.gccuk. org/files/page_file/C-120510-11.pdf
Investigating Andy: the MMR scandal
The title of my talk for the HealthWatch 2011 annual general meeting was, “Regrets? I have a few—Inside the MMR investigation”. Being honoured with an award seemed like a moment to go off-script, and after more than a year of banging back and forth to give lectures in the United States, and dashing around Britain to “Skeptics in the Pub” meetings, I’d gotten a little jaded with my standard “Fixing the Link” PowerPoints. So I took a chance and said something more personal.
My first regret was a failure, years ago, which had nothing to do with vaccines. In 1990, which Nick Ross reminded everyone at the meeting was the year that HealthWatch was launched, I was ensconced on Potrero Hill as The Sunday Times man in San Francisco (don’t ask how I pulled off that stunt). And on a sunny day, across the Bay at UC Berkeley, I interviewed Dr Peter Duesberg. This was (and still is) the man who said (and still says) that HIV is not the cause of Aids.
At the start of the 1990s, there were big gaps in science’s grasp of how the retrovirus could do its mischief. Duesberg, at the time a distinguished retrovirologist, was onto them like flies on sheep. His analysis was fascinating, and I took a pile of notes, while we chatted at a campus picnic table. But when I solicited his views on the “real” cause of the pandemic, he seemed unconvincing, even irrational. So, to my lifelong regret, I never wrote the piece. This is among my more painful hindsights. Agree with him or not—and that wasn’t really my business—what he said, back then, was worth a thousand words in The Sunday Times. “Boffin thinks you can’t catch Aids.”
But, as I say, I didn’t write it. Couldn’t be bothered, to be honest. Which left a modest gap to be filled by someone else. Newspapers, like nature, abhor a vacuum. And in the office from which I’d departed to write dispatches from America’s roadsides, was a medical correspondent under pressure. He’d been advised by our managers to leaven the usual diet of “new treatment offers hope” reports to “get stories like Deer’s” that cause a fuss.
Within months of my failure, he waded in to the HIV issue with a notorious crusade. It was full-blooded denialism, but was allocated pages and pages by the then editor, Andrew Neil. And, bestowing on Duesberg the public credibility of a world class newspaper, it may well have fed into what was to occur in South Africa after 1999: president Thabo Mbeki’s denialist health strategy, which brought death and suffering on a horrifying scale.
If only I’d written a few well-picked words, conceivably things might have turned out differently. My colleague’s first pitch wouldn’t have survived the editor’s Tuesday conference. “Deer’s done that already,” Neil would have snapped.
Why is this relevant? Well, because journalists at the sharp end are bound to consider what plays out from their work. At the top, national editors are often supremely ideological, constantly thinking about the upshot of their coverage. But the most successful set this aside for the imperative to sell: to tell stories that people actually want to read. It’s an anarchic part of our culture, but is near the root of democracy. You tell stories, like climbing mountains, because they’re there.
My mistake with Duesberg was to evaluate his theories, as if I was the arbiter of his views. And when Andrew Wakefield came along as a topic of inquiry1 I made sure not to make the same error. I was determined primarily to be a servant of the story, which, for all my hard work, told itself. “Excuse me, ma’am, did you see what happened?” Like a train crash, or a robbery. Get the facts.
Although I didn’t have time for all this in my talk, if I’d had the choice, I’d sooner have revealed proof that MMR indeed causes autism. I’d have got more space, won more journalism awards, have been accorded more spots on TV, and earnt more money, than I did revealing the Wakefield scandal. Overshadowing these, moreover, and worth more than them all together, I’d have accomplished something remarkable for children. If vaccines caused autism, that’s important information to head off another kind of pandemic.
The truth I found was a lesser story, although I think one of consequence. People have told me it’s a justification for a life. But here’s another regret: the (perhaps understandable) spin which my editors sometimes sought to introduce. They were always going on about measles outbreaks and whether MMR causes autism or not. I don’t belittle those questions—which I’ve researched a great deal— but they were never what drove my inquiries. I simply wanted to set out what it was that happened: in short, what Wakefield did.
Recently, I was invited to lunch by the legendary reporter Bruce Page, who led The Sunday Times campaign over Thalidomide. He made me wince with envy at how he said he completed the investigative part by pulling research on the drug from the library. And he reminded me of an old adage, passed to him by someone else, that there are really only two stories in our tradition. The first, he said, was, “We name the guilty men.” The other: “Arrow indicates defective part.”
Mine was the first kind (the second would have been the science): nailing an all-singing-and-dancing charlatan. For my HealthWatch talk, I showed a few of my slides about how my investigation moved forward over the years. I put up tables which compared what the BMJ has calledWakefield’s “elaborate fraud” with the Piltdown Man hoax of 1912. And I presented a series of slides concerning others in the saga, which cause me worry about the integrity of medical science.
These included the graduate student who knew something was amiss, but had nowhere to go with his concerns; co-authors taking credit from a high-impact publication; and a reviewer who was disposed to believe the MMR data because of an old-boy connection.
Back to measles: of course I don’t minimise this disease, which is sometimes fatal and, more often, brain-damaging. Nor do I underestimate the importance of the science that lies behind vaccine safety. But alongside these issues, theWakefield scandal says much about the potential for wrongdoing in research. It reveals the ease with which misconduct can be effected in academia, and the difficulty, or reluctance, to tackle it. Bill Clinton once called the 21st century the “century of biology”. And after what I’ve seen can happen, just looking at MMR, I fear another Thalidomide-style catastrophe.
HealthWatch, to its credit, often touches on such issues, but I must admit to a small regret about some emphases. As with the amazing Skeptics in the Pub2 phenomenon, which has sprung up in the last few years, there seems to me to be a sporadic obsession with, say, rubbishing homeopaths, or brandishing the Daily Mail’s contradictions. By all means promote critical thinking, and call out absurdity, but, from my point of view, I’m not sure that it’s here that health requires watching most intently.
As I said in my talk, I think that—for young people especially— trying “complementary” medicines can be a rite of passage. They may be a way in which some explore their route to rationality, rather than a gullibility from which we must rescue them. To borrow a related thought from the former politician Denis Healey: “If you’re not a communist before you’re 30, then you’ve got no heart. But if you’re still a communist after you’re 30, you’ve no brain.”
People often ask about the abuse I get, having had the temerity to take on Wakefield. Here’s an example I quoted from my hate mail inbox, which produced laughter enough to delight any speaker. It was written by a lady. She wrote in bold capitals which, to avoid shouting, I’ve dropped to lower case:
“F... you. You are the scum of the earth. You are just trashing people that are totally dedicated to find cures for a horrible disease. Protecting your selfcentered ass, trashing others for selfgain, or protecting your buddies at a corrupted pharmaceutical industry. Well, now I’m trashing you right back. You are trash. We’re even.”
Upset, me? Nope. Understandable? Entirely. Wakefield has been encouraging people to be afraid of vaccines for 14 years. He was doing it at autism conferences long before I got involved. At one of these, I should point out, he gave his infamous “callous disregard” speech,3 about buying blood from children at a birthday party. In another speech, you won’t have heard, he betrayed an apparently sincere whistleblower,4 who voiced his vaccine safety concerns. If you watch the second video and listen to the audience, you’ll understand how Wakefield works the crowd.
And this brings me to the final regret which I raised at the meeting. It concerns people like the lady who wrote that email. That woman was suffering. She was clearly in pain—and a pain not inflicted by me. For years, some parents of children with developmental disorders have been led to believe that it was their own fault for not listening to Wakefield that a son or daughter is autistic or disabled. That can deepen a sense of guilt, from which springs a vulnerability which in some cases festers into hatred.
But my regret isn’t vicarious. It’s not mere bystander sympathy. It’s that I sometimes find my compassion feels strained. There was a day, for example, during Wakefield’s GMC hearing when a strange collection of people gathered outside. One was an overwrought woman bearing a placard with a picture of her young son who had undergone radical bowel surgery. But he wasn’t even involved in the Wakefield case and had been diagnosed at another hospital with a food intolerance.
Wakefield’s disciples, however, pretended that this boy was part of the case. It was a disgusting deception, for which those who knew should be ashamed. But I wondered, most uncharitably, about that mother. Why did she allow herself to be used in this way? And what about those who used her?
I don’t fear the abuse of those victimised by Wakefield. But I sometimes feel my reactions are misplaced. It’s easy to dismiss these suffering people as cranks or idiots. It used to happen in the controversy around Duesberg. But it’s a greater accomplishment to stay focussed in compassion, and on the needs which should be brought to the fore. There’s no right response to hatred in hatred. I need to try harder next time.
Brian Deer, Journalist
In addition to receiving the 2011 HealthWatch award, Brian Deer was named specialist journalist of the year in the 2011 British Press Awards. Brian Deer can be contacted through his website, http://briandeer.com/
Further reading from Brian Deer’s website on topics mentioned
Prof Raymond Tallis: for his thoughtful promotion of evidence-based medicine
Anecdotes, Data and the Curse of the Media Case Study
Let me say straight away how honoured I feel to here tonight to be talking to you and to be receiving your annual HealthWatch Award. This came totally out of the blue – which has very positively altered my attitude to the blue as a result. The blue is more usually a source of nasty surprises – the frozen block of urine dropped from a 747 – than nice ones. I feel not only honoured but distinctly undeserving, particularly when I think of the previous winners of your award. I am thinking of writers whom I admire enormously such as Geoff Watts and Polly Toynbee and the witty and courageous John Diamond; and great clinical scientists such as David Sackett and Richard Doll and Michael Baum. And as for Ben Goldacre – the recipient of last year’s award – what can I say, except that every time I read one of his columns, I punch the air with delight until it is bruised. When get my weekly fix of Bad Science all my exasperation evaporates. So thank you Ben, and thank you all my predecessors, for strewing a little broken glass in the path of charlatans, mountebanks, loblolly men, and all those alternative practitioners whose loud noises have threatened at times to drown the softer-voice of hard-won scientific truth. And thank you for inviting me to join their select company. I would like to add my congratulations to the students for their awards. ‘O brave new world that hath such people in it…’ Thank you also Nick and David and you other HealthWatch stalwarts who over the last 19 years have kept HealthWatch as a major force for busting quacks and standing more broadly for reason and evidence against the forces of unreason that flourish best in an evidence-free zone. I know you do this purely because you are passionate about the things David talked about in his address. As Nick, said, you don’t get anything back – though 50,000 – 70,000 hits is very impressive indeed.
I don’t know whether my frontal lobes are starting to lose their grip – or perhaps it’s my latest detox at the health farm – but I am starting to feel a little more optimistic and have the feeling that the quacks, at least in the UK, are perhaps on the retreat; that this has been a pretty good year for quack-busting. Perhaps it’s a matter of contrast-enhancement because we did start from a pretty low point when in September 2006 the MRHA (Medicines and Healthcare Products Regulatory Agency) gave homoeopathy a boost by licensing its products on the basis not of medical evidence but something called provings. I must admit, I had never heard of provings up till that time and this is what they are.
They’re conducted on volunteers in a reasonable state of health (provers) who generally have no knowledge of what substance they are taking. Doses are repeated until provers start to experience symptoms of a change in state. The provers record everything they experience, whether physical, emotional, mental, or even spiritual, as long as the change in state persists.
At the end of the proving, all the records are collated and compared to find the symptoms, states of mind, feelings and experiences the provers have in common and which can be reasonably attributed to the emerging signature resonance of the substance. Wendy Howard Homoeopathic Practitioner quoted in The Skeptic Express, 2006
It would be an interesting exercise for 6th Formers or even 12 year-olds to be challenged to say why this methodology is flawed. Anyway, the outrageous decision by the MRHA – which seems in this instance to have been influenced by populism rather than by science – galvanised opposition and the annus fairly mirabilis has resulted, with homoeopaths, and herbalists in particular, having a fairly rough time of it.
Many of you will be aware of the campaign fronted by Michael Baum, David Colquhoun, myself and others in partnership with Sense About Science that has had scored some palpable hits against homoeopathy. The campaign to persuade PCTS not to fund homoeopathic treatments on the NHS has attracted huge publicity and, what is more, prompted many PCTs to remove such treatments from their bill of fare. The latest scalp is the withdrawal of funding by the Tunbridge Wells PCT from their local homoeopathic hospital which consequently is likely to be mothballed. There are five Homoeopathic Hospitals in the UK: one down 4 to go.
And then there has been increasing awareness not only of the uselessness of most herbal remedies but, indeed, of their power to do harm. While it is possible that there are some benefits from some herbal remedies – after all, if you fire enough buckshot in the dark, you will sooner or later hit a mosquito’s legs and then say that that was what you were aiming at all the time – there is, as we all know, no evidence that the kind of mixtures dished out by herbalists in their shops do any good. There was huge publicity a couple of weeks ago arising from the study by Edzard Ernst – another person who I am very proud to have as a predecessor in the receipt of this award and who has, I believe, done more than anyone to force CAM to engage with the scientific method – recently published by the Postgraduate Medical Journal. This has provoked widespread demand for properly regulation of these medicines and of those who flog them. Let us hope that this time the MRHA doesn’t fumble the pass.
In short, much grounds for rejoicing. It is as if the nation’s serum Carol Caplan levels are falling and the parts per million of bullshit in the collective consciousness has dropped a notch or two. Now, if this speech was written by a computer, rather than written on a computer, and it had predictive texting, it would next print out the words – ‘There are, however, no grounds for complacency’. Or, as your Chairman, David Bender, no computer he, observed in his report last year – ‘We’, that is to say you, ‘are still needed’. And that’s certainly true. Unreason is always ready to break out, or to return, for reasons that go to the heart of what I shall be talking about this evening and, indeed, what it is about quack medicine that makes it more significant than might appear at first sight. Even as we speak, rejoicing in a year in which in the UK at least quackolysis has definitely had the upper hand over the quacks, herds of bulls are massing, ready to evacuate the contents of their ample colons.
Colonic material of a Taurine provenance is in infinite supply
One of the reasons for this is, of course, the way science and non-science are presented to the general public by the media. And the problems are epitomised by what is referred to in the title of my talk: the curse of the media anecdote. Much discussion of medical care – both orthodox and useless – tends to be revolve around, or even to be tethered to, anecdotes.
The media love anecdotes because those who work in them believe (usually correctly) that their customers do. Anecdotes – about a new miracle cure, a drug that is not being made available on the NHS, or the side effects of treatment, or some environmental hazard - sell product: they help to boost circulation, increase audience rating figures, and so on. Data, on the other hand, which take us towards the truth about these things, are less popular.
Anecdotes, however multiplied, do not point the way to reliable knowledge. As the aphorism says, ‘The plural of anecdote is not data’. For an anecdote does not even count as a datum. It may prompt a search for data, as when a chance clinical observation makes someone wonder whether A causes B, or C may have therapeutic benefits in condition D.
These deficiencies count less than they should. For anecdotes, unlike data, have a human face: they are closer to gossip, which is a universal human pastime, and to the things that shape belief formation in everyday life. This is something to which I will return presently. Scientific data - for all that science is one of the most human, as well as the greatest, of human achievements – lack this human face. Their very strength – that they are unattached to persons and personalities – is a PR weakness. At best, data belong to ‘the science community’; worse, to ‘the scientific establishment’; or, worst of all, when they are adopted by, and guide, those who make policy on our behalf, ‘the authorities’.
Let me illustrate the accursed power of the anecdote with one of the most potent examples of recent times: the story told by Jackie Fletcher, the founder of Justice, Awareness and Basic Support or JABS, a group devoted to campaigning on behalf of parents who believe that their children have been damaged by immunisation, most notably by the MMR triple vaccine. Her story is a tragic one. Nearly a decade after the MMR furore broke, it is still news and still being used to influence opinion.
In 2007, The Daily Mail devoted yet another article to her terrible experience, accompanied by a poignant picture of her and her son, ‘who developed autism after being given the MMR jab’:
The change in their bright, loving toddler was sudden. Ten days before his first fit, Robert had been vaccinated against measles, mumps and rubella (MMR). Unlike his older brothers, who had been given the jabs in single doses, Robert had been injected with three viruses at once.
To the Fletchers, the connection seemed obvious..
The Fletchers’ experience is unimaginably terrible: the transformation of a perfectly normal toddler into a child with autism, severe epilepsy, and incontinence, confined to a wheelchair, speaking only the words he knew as a baby, and requiring to be washed and fed. In their case, grief is envenomed with (unjustified) guilt. Mrs Fletcher says, ‘Deep down, I feel that I have caused all Robert’s problems by taking him for the MMR jab and I’m trying to fix it for him and I can’t’.
One would have to have a heart of stone to challenge Mrs Fletcher’s interpretation of what happened; her attempt to make sense of a senseless catastrophe; and her endeavour, through campaigning to protect others’ children against what she sees as a threat, to come to terms with a guilt she feels which, though unwarranted, is real. An anecdote such as this, set among data, is like a queen ant in a colony of drones. Stories have a power that bears no relation to their number or validity. The very fact that those who invoke anecdotes in support of their theories are so convinced by them, adds to their cognitive lustre.
Notwithstanding Nietzsche’s assertion that ‘Convictions are greater enemies of truth than lies’ , certainty, repeatedly asserted, and the anger and grief with which it is charged, increases the impact of the claim embedded in the story, on the reader or listener. When we are invited to assent to an empirical truth in an area in which we have no expertise, we are more inclined to do so if we identify with the person who is purveying it. It is a way of showing solidarity.
The contrast between the media anecdote and the way data are presented in scientific meetings or papers – where there is no reference to the feelings (anger, grief, conviction) of the authors and much concern about sources of error and uncertainty - underlines how disadvantaged the latter are. Of course there are passions behind science – who otherwise could stand the utter tedium of data gathering and learning the techniques of doing so reliably? – but the passions of themselves cut no ice. Indeed, they would be grounds for suspicion from those who are used to what Paul Valéry described as ‘the elementary courtesy of proof’.
Anecdotes also have the ability to attract other anecdotes: when the Fletchers took their stricken child to his many hospital appointments, ‘they encountered other parents who said things like “our child was fine until MMR”’.
This ‘multiplier effect’ is vastly scaled up when the anecdote comes from a celebrity in the media, or is endorsed by such a celebrity. A face that is everywhere is the vehicle for an opinion that is everywhere. The same story is replicated in a million places and has the impact of a million data. In a society where achievement in one area (for example, combining being a moderately competent actor with having an eye-popping figure) makes one an authority in many others, the views of celebrities are the equivalent of the union block vote.
From the point of view of the media, there are secondary benefits of the human story. When data falsifying the Anecdote Based Theory (ABT) start to look overwhelming, other, secondary, anecdotes emerge that nurture the primary one. There is talk of conspiracies, cover-ups by the establishment, and we are reminded of persecution of people (of genius and courageous) who have heretical views.5 The campaigners on behalf of ABT and their supporters look like plucky loners facing a mighty monolithic opposition. Powerful data are discredited simply because they originate from sources that are considered to be on the side of the Big Boys: the scientific establishment, the politicians (who have their own reasons for hiding the truth), and others, such as Big Pharma (who have vested interests). Even the sharing of these data with the public can be presented as an act of oppression. At the height of the MMR controversy, the Department of Health was accused of ‘bullying parents with facts’6.
The power of the anecdote is enhanced by the fact that, once it is in circulation, it looks like an entire half of the overall picture. The media claim that they are required, by their commitment to balance, to give equal hearing to both ABT and science. So Jackie Fletcher talking about her son Robert and giving her untrained critique of the epidemiological data is allowed as much airtime as a scientist bearing news of millions of data obtained under carefully controlled conditions. Given that controversy sells product, maintaining the myth that there are two equal sides to the story is desirable in itself.
The notion of balance is one that should be treated with care, the kind of care that it does not always, or even usually, receive in the media. If applied uncritically, it can result in giving data-free believers in the flatness of the earth the same amount of air-time and column inches as those who support the well-attested notion that the surface of the earth is curved. The Royal Society, quoted in an email by that doughty fighter against junk science, David Colquhoun, has made this point:
When obtaining specialist advice from experts, it is not diversity of opinion that should be balanced, but the weight of opinion accorded to various strands of scientific advice within that diversity. In the name of ‘balance’, the media invariably present opposing views on each side of an argument, regardless of the relative scientific merit of those opinions.
The price of so-called balance that does not take account of the ‘relative scientific merits’ of the opinions being given equal air-time, may be very high indeed. Most people in this room will be aware of Peter Duesberg the maverick molecular biologist who denied the link between AIDs and the human immunovirus. His views were largely discounted in the West and have, thank God have had no influence on science policy, public health policy and those scientific research programmes which have led to the discovery of highly active antiretroviral drugs which have transformed AIDS from a fatal to a chronic disease. Elsewhere, mankind has not been so lucky. Thabo Mbeki’s catastrophic AIDs policies have been hugely influenced by Duesberg’s views and the result is a pyramid of corpses sufficient to block out the sun.
Fortunately, the MMR controversy has a less calamitous outcome. When it was obvious that the link between MMR and autism was unsustainable, and that it never had any basis, some of the media showed signs of contrition. Not all of the press, of course. Melanie Philips still continues her one woman campaign against MMR and combination vaccines that she considers lethal. If her views were listened to, of course, and had any influence on public health policy, the result would be a catastrophe on the scale of what has happened in South Africa. The habit, however, of giving appealing individuals with their moving stories at least as much credence and coverage as unappealing data, of preferring faces to graphs, and vox pops to statistics, does not seem to have been kicked. The recent judgement by the National Institute of Clinical Excellence that certain drugs would not be cost-effective in the treatment of early Alzheimer’s disease on the basis of the evidence so far was dominated by human interest stories.
I found this particularly exasperating because (to declare an interest) I was a member of one of the NICE appraisal committees for three years. This, quite labour-intensive, unremunerated role brought with it two rewards: the knowledge that one was contributing in some small way to ensuring that the finite resources of the NHS would be spent in such a way as to maximise health gain; and the satisfaction, indeed pleasure, in participating in a very high-level, supremely well-informed discussion about the costs and benefits of medicines that took into account basic science, clinical evidence, health economics and social value judgements. Very little of this appeared in the media coverage. The World at One gave extensive air-time to a woman whose father had been given the drugs early in his disease and was ‘convinced’ that he would not have done so well had he been denied them.7
This example illustrates something that is common to much anecdote-based discussion of matters of public interest such as the benefits or harms of medical treatments or the impact of some environmental factor on health: anecdotes privilege the visible over the invisible. In the case of MMR, the millions of children who had been saved from harm by the vaccine were given little foreground; in the case of Alzheimer’s drugs, the impact on all other patients that would result from inappropriate allocation of resources to less cost-effective treatments, was hardly discussed.
I want to step back a little and look beyond their immediate consequences to another reason why the media anecdote is accursed. To think about this I want to rehearse some arguments I have had with people about the use of alternative medicine in the UK. I was recently discussing reflexology and it was pointed out to me that it is pretty harmless and that it doesn’t after all cost much. There was very little evidence that those who used it denied themselves the benefits of orthodox medicine. So why are you getting in such a pet? Well it seems to me that alternative remedies and the theories that are used to support them are a major contribution to the moronisation of the collective consciousness and, at some level, a betrayal of one of the greatest achievements of humanity, and so of humanity itself.
The journey humankind has taken to robust scientific knowledge of the world in which it finds itself has been a long and arduous one. For science has required not only that we somehow penetrate the opacity of nature but also that we overcome the prejudices, the intuitions, the assumptions, the convictions, the dogmas that we ourselves place in the path to reliable, usable knowledge. These obstacles are especially stubborn in medicine, not only because its object is the human body, that from which knowledge takes its rise, but also because its concern is with disease, where emotions, particularly those of hope and despair, tend to have the most powerful influence over our beliefs. The endeavour to cure illness is thus bound to be dogged by deception and self-deception. Hence medicine’s status as a parvenu among the sciences: it is, as Lewis Thomas said, ‘the youngest science’. The greatest miracle of scientific medicine is that medicine became scientific.
The contrast with traditional, or pre-scientific, medicine – the medicine that was all that was available for all but the last 100 years of the 300,000 years of H. sapiens sapiens - is extraordinary in many ways. One of the most important is the fact that scientific medicine, unlike traditional medicine, which is deeply self-satisfied with its knowledge and what it believes to be its effectiveness, is driven by an active uncertainty that is sceptical of received ideas and of authority and is continually seeking to improve on the status quo. It is an irony that scientific medicine is called ‘orthodox’ medicine because, unlike acupuncture, or homeopathy, is constantly scrutinising, challenging, overturning its own beliefs. Like the rest of science, it is in a permanent state of self-critique; and the authority of authorities goes no deeper than the facts that they can invoke. By contrast, acupuncture claims to derive its authority from individuals who laid down the law thousands of years ago. One of the most barefaced examples of proud intellectual stagnation must be the fact that homeopathy has not changed since Samuel Hahnemann offered it as an alternative to the poisoning and butchery that was prevalent at the time. Homeopath still rely on his book, first published in the 1790s, to guide their practice. What he spake, they shall do. There could be no more striking example of frozen orthodoxy and authoritarianism.
Alternative medicine has not progressed to anything approximating clinical science because it has not been prepared to challenge its own authority. So-called orthodox medicine, in an act of collective humility, has cultivated a routine distrust of its own practices. This humility has been almost as important in the development of effective therapies as the discoveries in basic biomedical science. The rejection of anecdote-based medicine in favour of treatments tested in randomised controlled trials and re-tested in meta-analyses etc has accounted for the remarkable advances particularly over the last 50 years, and the quickening pace of increases in life and health expectancy. The humility of shaping clinical practice in accordance with the unmanipulated outcomes of therapeutic trials is also connected with another layer of scepticism: that what looks good in theory (irrespective of how good the theory is) may not benefit patients in the real world.
Perhaps the most remarkable facet of this humility is the willingness of doctors to enter patients (with the latter's consent) into clinical trials run by other clinicians. Submission to the authority of such trials means subordinating one's own personal authority to that of other professionals, many of whom (such as statisticians and biomedical scientists) do not even belong to the medical profession. As Marc Daniels pointed out:[for clinicians] to be willing to merge their individuality sufficiently to take part in group investigations, to accept only patients approved by an independent team, and to submit results for analysis by an outside investigator involves considerable sacrifice.
The committment to minimising the play of chance, of bias, or of wishful thinking, is the supreme expression of the self- overcoming that scientific medicine requires. Avoiding beliefs guided by delusive hope, unfounded authority, superstition and plain stupidity, it cultivates an attitude of healthy scepticism towards itself to prevent its practitioners from misleading themselves or their patients. Its permanent strategy of active uncertainty, and the humility this implies, is the distinctive virtue of scientific medicine. In the world outside of scientific medicine, however, humanity has had little time to adjust to this almost inhuman scrupulousness.
Hence the fact that alternative medicine – and more broadly anecdote-based treatments - can still live, and even flourish, at least in terms of the number of customers, and the amount of cash that changes hands, side by side with medicine that actually makes patients better for reasons that are rooted in a scientific understanding of the human body and its diseases. Otherwise, it would seem unaccountable – like an ox-cart at stand in an airport next to a jumbo jet or an Asclepian temple fitted out with WiFi. Indeed, the constant self-critique of medicine and the biomedical sciences upon which it is based is often misunderstood. The self- dissatisfaction and self-transformation - resulting in ever more effective and, for the most part, less unpleasant, treatments - is seen as a kind of weakness. A discipline which is marked by a carefully nurtured scepticism towards itself is sometimes seen as arrogant or in disarray – unlike traditional healers who are proud never to have had a second thought in 5,000 years.
So this is perhaps the true curse of the media-propagated anecdote and anecdote-based beliefs about medical care: they represent a sickening regression to the cognitively primitive state from which medical science, science, and humanity has fought its way out.
So if we are going to free ourselves from the curse of the media-base anecdote, we need to have a major initiative explaining why anecdotes are misleading, why double-blind randomised controlled trials are necessary – how they work and how, over the last 50 years, they have played such a crucial role in the objectively observed advances in medical care.
A good place to start would be cardiovascular disease, in which randomised controlled trials have had almost as great a role in bringing about improvements in mortality as advances in basic biomedical science. A couple of statistics could introduce the topic. Firstly that cardiovascular disease mortality in men of all ages has fallen in 2005 to 30% of the age specific rate in 1968. Secondly, that deaths from coronary heart disease of people under 65 has fallen by 46% - by nearly a half – in the last ten years. People should be asked to celebrate these facts and to consider the role played by basic science in understanding the processes that lead to cardiovascular disease and methods of preventing or reversing them; and the complementary role of robust clinical trials in ensuring that what should work in theory actually works in practice in the real mess of the real world of doctors and their patients.
Perhaps this is idealistic. The notion that the nation’s serum Daily Mail levels might be reduced by broadsheets offering free tutorials on the methods by which science arrives at robust knowledge and that knowledge is translated into benefits for patients is somewhat utopian. And yet I assume that most journalists (and their sub-editors who write those lethally misleading headlines) want to be able to look back on careers in which they have done more good than harm and advanced, rather than held back, the cause of truth. They are aware that anecdotes may be very effective at conveying the profundity of medical tragedies and the human importance of medical triumphs and disasters. But for making decisions about the absolute and relative values of medical treatments, and more broadly for determining policy, they are of no use whatsoever.
And so I would like to offer a few thoughts, which could be accommodated on a flash card, to help journalists deal responsibly with those anecdotes they and their readers value so much. Before I do so, however, I would like reflect a little more on the power that anecdotes have, which they owe to their central role they play in our making sense of the world in which we find ourselves. And this includes the kind of sense of the mediated world that I am trying to make in this talk – which, it will not have escaped your attention, has itself been rather heavily dependent on anecdotes.
Anecdotes are not an unredeemed evil. As we have already noted, medical science itself has been driven by accidental, observations, anecdotes, though that is merely a prompt to inquiry, the place where it begins, not where it ends. But beyond this, human beings have always told each other stories and storytellers have always commanded a hearing not only because we want to know what happens next but because what is recounted strikes us as standing for something that goes beyond the particular instance recounted.
We imagine beyond the events in the story to a whole class of similar events. ‘This could happen to me’ is extrapolated to ‘This could happen to anyone’; ‘This could happen any time’ to ‘This is happening all the time’. We are taught from earliest childhood to see stories as lessons, as parables: faulty induction is part of our ethical education. Identification with the stories that we are told, seeing them as potentially part of our own biography, is key to the development of that imagination which lies at the heart of sympathy, empathy and hence morality.
The power of stories to fuel extrapolation beyond themselves to an imagined series of general cases is greatly magnified by emotions – as Bertrand Russell pointed out in the passage I have just quoted/ for emotions are the means by which the world we think of ourselves to be surrounded by is illuminated. Angers – generous and mean-spirited – compassion, the sadness that we feel from entering into the particulars of another’s tragedy – these are the great drivers to an extrapolating imagination that takes us beyond what we know of the world to what we suspect of it; supporting that part of our world-picture – by far the greater part – which is not directly vouched for by our own experience. It helps that key figure within the very motor of our cognitive activity, the Professor of Data-Lean Generalisation who helps us to navigate through the world we live in, and to feel that we in some sense cognitively possess it, by being able to command a viewpoint on it. Emotion is the basis of that confirmation bias which is necessary to put together for ourselves a coherent narrative about the world or part of it. This journey, via emotion, from anecdote to world-picture is both our curse and our glory: it lies at the heart of our wider sense of responsibility – the desire to make a positive difference – and, equally, at the root of those generalising hatreds – racism, ‘classism’, all forms of ‘otherism’, which have made the world such an unpleasant place for some and potentially a dangerous place for all. The observation bias that is built into it, and the desire it has to confirm only itself (so that it will be satisfied with very few instances to support a statement of vast generality) enables us to be fluent about things of which we know little; to summarise, in the lounge bar of a pub, the character of an entire race of people in five ill-chosen words.
And it is round about here that we find the true greatness of great science. Like gossip and idle curiosity, it is manifestation of the fundamental human need to make sense of things, to build a picture of the world that vastly exceeds our experience so that we can understand and control it and, so by making the world a safer or more predictable place, to control our destinies. And at the same time, it does not allow the passion to determine what it sees.
It has, in short, managed to elevate its passion for understanding and for reliable knowledge - that will permit what Francis Bacon called ‘the effecting of all things possible’ – above those passions that normally permit us to transcend our limited individual consciousness and fuel our imagination. This is due in no small part to the fact that it is a collective passion that engages all mankind - an average issue of a leading scientific journal will have contributions from scientists from a dozen or more different countries and countless different backgrounds; that it is regulated by rules of inquiry that have been forged in bitter experience and bitter disappointment; and that takes place in the context of institutionalised scepticism.
This separation of passions is lies at the root of the ethic of science and is linked with what Claude Bernard advised as regards imagination: ‘Put off your imagination, as you put off your overcoat, when you enter the laboratory. Put it on again, as you put on your overcoat, when you leave’.
In the end, we must always begin with notions that are insecurely founded on a grossly inadequate database. Kingsley Amis once referred to the way we erect great belief systems on the basis of very little data as ‘the inverted pyramid of piss’ This is perhaps a little harsh, as the necessary starting point for belief formation in both everyday life and in science. In everyday belief formation it tends to be the end-point as well, whereas science moves on from there. While acknowledging the central role of stories, of anecdotes, in awakening us out of our own narrow consciousness to awareness of a world outside, we need equally to acknowledge that an understanding of the world rooted in anecdotes is cognitively underdeveloped. To give anecdotes too great a prominence in the debate on science, and medical science, and science policy, and healthcare policy, is to muddy those waters that it has cost scientist huge effort to clarify.
So, ye journalists, by all means offer your readers anecdotes but do not for one moment deceive either yourselves or your readers into thinking that this is science reporting or that one can move directly from anecdotes to robust generalisable knowledge or recommendations for application of science in healthcare or anywhere else or for the development of health science policy. Here, then, are some reminders that might be useful for journalists who find that they are reporting on science, particularly science as it applies to healthcare:
a) An anecdote is not a scientific datum, though it may justify a study to acquire data. Do not confuse experience – however intense and harrowing – with expertise.
b) The journey from a single observation to reliable, generalisable truth is long and arduous and full of pitfalls. Here are some of those pitfalls (all illustrated by the MMR furore:
Subscribing to the fallacy ‘Post hoc ergo propter hoc’: Event B happened after Event A therefore Event B is due to Event A.
Noting an association between Event A and Event B and concluding that the entire class of A-type events and B-type events will be associated. The propensity for confirmation bias means that you will preferentially notice, remember or be inclined to report, only those occasions in which there is an association.
Moving directly from observed association to imputation of causation. Both Event A and Event B may be the effects of some other event or factor.
Assuming that, because there is a causal link between Event A and Event B, or a certain factor seems to be operating in one instance of Event B, that the cause or factor is operating in all cases, or that it is the only cause or factor relevant to Event B.
These are hard lessons that mankind has learned reluctantly on the road to useful general truths. Do not be proud of unlearning them out of solidarity with folk common sense. Science in many respects is a triumph over the limitations of interested observers and often over common sense. Ask yourself again and again How typical is the story of the larger picture? How representative are these events? From what population of events are they drawn? For God’s sake give me denominators!
c) When discussing the beneficial or adverse effects of therapies, or more broadly technologies, consider the whole picture and the whole population. For example, even a treatment that has an adverse effect in some recipients may overall have a beneficial effect in the population. The anger of those who feel the treatment has injured them or their loved ones should prompt review of the evidence of overall benefit but should not count as evidence in itself.
d) When science fails to support Anecdote-Based Theory, do not assume that expert opinion:
is blinkered, or hobbled by ‘orthodoxy’. Science is in state of permanent reformation – that is how it advances so quickly;
is simply part of a lump called ‘the establishment’ that routinely conducts witch hunts against dissenters;
is in hock to the government, to Big Business or Big Pharma – or to other vested interests. Remember that those who point the finger at others’ vested interests – including you, the journalist - have vested interests themselves..
e) When ‘balance’ is sought, it is important that the two views given equal airing should have equal weight. One advocate of ABT, however impassioned, does not deserve as much of a hearing as one million data presented by someone who has acquired them in a way that has survived peer review by experts (who are in many cases professional competitors and on the lookout for errors). The 100% certainty of the anecdote-based theorist should not also be used to make the refusal to claim 100% certainty in science a weakness.
These principles should be known and embraced by all journalists – not just those charged with commenting on science but anyone whose comments may influence the views people may have about science, the application of science, science policy or the larger role of science in society. Indeed, it is outside the columns of science correspondents that the greatest damage to the collective understanding is done. The excellent work by people such as Geoff Watts, Georgina Ferry, Palab Ghosh, Mark Henderson is often overshadowed by the loud and more widely read ignorant certainties of characters such as Melanie Philips. Anyone who is going to make a comment about treatments for breast cancer in the General Features or about the heart attacks in GHQ should, at the very least, have some idea of the approaches to arriving at robust general truths that are employed in science; at the very least they should read Peter Medawar’s Pluto’s Republic, Lewis Wolpert’s The Unnatural Nature of Science, and Theodore Dalrymple’s Mass Listeria.
At any rate, media writers with a conscience should not collude with the cognitively primitive state in which we form most of our everyday beliefs. They should remember Bertrand Russell’s observation that while ‘popular induction depends upon the emotional interest of the instances, not upon their number’, reliable induction depends on numbers not emotional interest. If journalists give excessive weight to individual anecdotes they will trigger unnecessary worry and suffering, and sometimes cause diversion of resource from places where it would be of more use. Vox pop or even vox pop star is not vox dei.
Anecdotes are the first step on the road out of individual subjective consciousness to collective, objective knowledge. They are, however, only the first step. Science has gone far beyond this. If you are going to intervene in a debate on matters already subject to scientific investigation, you need to grow up out of your anecdotage. Otherwise journalism and the anecdotes it loves will get in the way of the collective clarity of thought that is going to be essential if we are to deal with the great challenges ahead of making this planet a safe, and reasonably comfortable, place for those who inhabit it.
The price of putting anecdotes in the driving seat is very high indeed. Uncritical focus on ‘the human story’ at the cost of ignoring or underplaying dull scientific facts may have inhuman consequences, as has been demonstrated by the recent return of measles and the first death from this disease for 14 years in the wake of the unfounded scare over MMR. For this I do not blame poor Jackie Fletcher but those journalists who, out of laziness and perhaps venality, gave her single experience and her heroic but misguided mission, more credence than it deserved.
So, ladies and gentlemen, there you have it: the curse of the media anecdote. It remains for me only to thank you again for the fantastic work you are doing, you dedication to quack-busting and this terrific honour. I shall display my award with pride, though I feel somewhat undeserving as well as proud.
- Gill Swain www.dailymail.co.uk/pages/live/articles/health/healthmain.html Last updated 11th July 2007.
- Friedrich Nietzsche The Gay Science Book 1, Section 2 (1982).
- Paul Valery Monsieur Teste translated by Jackson Matthews (London: Routledge & Kegan Paul, 1973), p. 65.
- Swain op cit
- I give examples of this in relation to MMR in Hippocratic Oaths. Medicine and its Discontents (London: Atlantic, 2004. See ‘Convictions’ p.109-127.
- See Tallis, ‘Convictions’ ibid. This chapter also covers the epidemiological evidence that demolished Andrew Wakefield’s claim of the link between MMR and autism.
- BBC ‘World at One’ 13th August 2007.
- See Lewis Wolpert The Unnatural Nature of Science (London: Faber, 1992)
- Quoted in Wolpert ibid p.19
- ‘First measles death for 14 years’ BBC News 24 3rd April 2006