Trustworthy communication of risk and evidence: the battle against naughty numbers in the news
A talk by David Spiegelhalter, Chair of the Winton Centre for Risk and Evidence Communication, Centre for Mathematical Sciences, University of Cambridge; Non-Executive Director, UK Statistics Authority; President, Royal Statistical Society 2017-2018. Professor Spiegelhalter spoke at the Medical Society of London on 27th July 2021. He is joint winner of the 2021 HealthWatch Award. The following is an edited transcript. The full recording can be seen on the HealthWatch YouTube channel.
Tonight I'm not going to talk about the construction of the evidence but about the messaging - the storytelling around the numbers. The American statistician Nate Silva says the numbers have no way of speaking for themselves, we speak for them. That means the numbers have to be embedded in the context and there has to be a narrative around them in order to see what their value is.
I'm retired. I was going to have a nice gentle retirement and then Covid came along and suddenly everyone wants to talk to a statistician. But people will attack you. I was on Andrew Marr last week, and afterwards someone tweeted to him, “Well done, you've just put people off getting vaccinated by listening to that idiot David Spiegelhalter. You never question his findings which are completely wrong.”
I think that the statisticians and the public health organizations have done a great job. From my perspective there's a huge demand from the media for experts. There's been brilliant work by the UK Science Media Centre, an organization that puts scientists and terms in in touch with the journalists. Most journalists, particularly the specialist journalists, have acted very well. But it's tricky to remain non-aligned. The media love to split and blame and get arguments going. When I am asked, “why do you think this is happening?” and I reply, “I have no idea you'll have to wait and see” they won’t use that quote.
Putting heads above the parapet
I've written an article with Kevin McConway to help professional statisticians deal with the media. Some lessons: sound human; make friends with journalists; beware of “just a chat” - never have just a chat with a journalist, they're recording every word and they're going to use that one line where you slipped up and went all casual. You’ve just got to pick yourself up try to ignore the criticism and get going again.
We’re encouraging people from the Royal Statistical Society to get out there and put their heads above the parapet. Because I think the public deserve and need what I call non-aligned scientists.
The media are hopeless with denominators. Here’s one: “Only 0.005 percent of coronavirus deaths have been children.” That's completely wrong! Only 0.005 percent of children with the virus have died. So the denominator is not coronavirus deaths, it's children with coronavirus, in other words they've got a very low death rate.
That was the fault of the sub editor – the people who write the headlines. The journalist didn't write the headlines, he wrote a perfectly good article but someone else stuck a completely inappropriate headline on it.
I'm going to talk on the whole about mainstream media, but with a brief mention of some of the awful stuff on social media. One tweet said that a new study suggested that more than five million Britons have had the coronavirus (correct so far); and went on to say, given that 50,000 people had at the time died from it (correct) this means it has an infection fatality rate of less than 0.1 percent – wrong! 50 000 out of 5 million is one percent, not 0.1%. This person had been arguing for six months that Covid was no more dangerous than flu and he thought he'd found the statistics to prove that, buy by doing so he showed he was wrong the whole time.
Diehard skeptics of that variety have actually had very limited mainstream media coverage. They get lots of retweets among that particular coterie but actually in terms of the mainstream coverage it's very limited. Which is very reassuring.
I'd like to go on to the idea of pundits speaking outside their expertise. Lord Sumption, ex-Supreme Court judge was interviewed on the BBC Radio 4 Today programme on July 20th where he made three statements. The virus has not killed over 100,000 people; the people who died would probably have died within a year after anyway; and the number of people who have died without a sufficiently serious co-morbidity to appear on the death certificate is very small - a matter of hundreds and not thousands. All of these are totally wrong! Why wasn't he challenged at the time?
In the event, Full Fact, the fact-checking charity, destroyed the inaccuracies in that interview within hours. By that day 124,000 people had died with Covid as the underlying cause of death. People dying of Covid had lost about a decade of life on average.
Now, an area of dispute is around the fact that 90% of people who die with Covid do have underlying health conditions. In reality, Covid picks on any vulnerability you have and exaggerates it, it essentially takes the normal risk anyone has and pumps it up so if you have any vulnerability for any reason it'll hit it really hard. It’s a real bully.
But while 90% of people do have something else along with Covid on their death certificate, the other 10% haven't. So there were not a few hundred, but 15,000 death certificates that gave nothing else but Covid as the cause of death, just in England and Wales.
The fact that misinformation is appearing in that mainstream media is very bad, but fortunately there hasn't been that much of it.
Covid in context
A graph that came out this morning from Office for National Statistics shows how many deaths were registered over the last five years, which lets us see Covid in the context of the normal pattern. It shows that with the first peak of Covid there were also quite a lot of excess non-Covid deaths - almost certainly many of these did have Covid but it didn't go on the death certificate because they were in care homes, or weren't seen by the doctor.
Then in the second wave there is another spike of Covid deaths but here, the number of non-Covid deaths is running below average. The crucial question is, are the missing non-Covid deaths we saw in the second wave, just some of the people who died in the first wave but who would have died naturally a year later if they hadn’t?
There is something called mortality displacement. There are certainly fewer non-Covid deaths in the second wave because some of them died a year earlier in the first wave, but that doesn't fully explain what's going on in this graph. One of the reasons is: there's been no flu.
If you look at Covid hospital admissions last winter alongside flu hospital admissions, flu is a flat red line – there were practically none. It means we quite possibly could be hit badly by flu in this winter coming up. So the lockdown could have prevented 10,000 to 15,000 deaths from flu over the winter. I’m not saying we should always have a lockdown every winter, but it does seem to have saved lives from flu.
Let's talk about trust
Let’s finish off by talking about trust. Baroness Onora O’Neill did a fantastic series of Reith Lectures some years ago on the subject of trust. One of her great ideas is that organizations should not be trying to be trusted, instead there's a duty to demonstrate trustworthiness. This is a brilliant switch of the responsibility! For people to trust you, you’ve first got to demonstrate you can be trusted. She has had a huge impact. The Code of Practice for Statistics has trustworthiness as its first pillar.
In our Winton Centre we've been trying to codify what trustworthiness means in an era like this where you're trying to communicate complex evidence. We've got it down to five things:
- Inform people in order to empower them to make informed decisions rather than manipulate them to do what you want them to do. And if you do want to manipulate them, be honest with yourself that you are just advertising. Just own up.
- Balance, but not false balance - show the positives and the negatives but don't pretend they're equal.
- Disclose uncertainties.
- State the quality of the evidence. Be upfront if the evidence isn't very good if your statistics are unreliable.
- Pre-bunk misinformation. If people are spreading misinformation get in there hard and counter it and tell it to people even before they hear it. There is good evidence this can be effective. “Do you know there's people going around saying that vaccines make you infertile?” Get in there first and counter it.
Back in April, Jonathan Van Tam, deputy Chief Medical Officer for England, a trusted communicator, used our slides to tell people under 30 that they shouldn't get the Astra Zeneca vaccine because of the risk of blood clots. It's a complex argument, and the balance of risk changes depending on the likelihood of exposure to the virus. It may have changed because of how common the virus now is. They explained that the benefits of the vaccine for people my age in terms of preventing intensive care admissions, is huge. But when exposure levels are low, the benefits of getting the vaccine are much less for 20 to 29 year olds because that age group is very rarely going to end up in intensive care for Covid. For the Astra Zeneca vaccine, these rare clots are more common in younger people. So you've got these counteracting risks and benefits which change with age, and at some point the balance flips over and so they initially set the limit for recommending the vaccine at age 30 and above.
It would be quite reasonable now to change that flip point because exposure risk to Covid at the moment for young people is high. The risks of the vaccine stay the same but the risk of being exposed to the virus goes up. But that was our attempt to show balance. It went down very well.
Pre-debunking doesn't always go well
An example that didn't go down so well, was an attempt to pre-debunk misinformation. The official data for deaths from the delta variant up to June 21st this year, showed that among the over-50s there'd been 38 deaths among unvaccinated people, 18 among people with one dose of vaccine, but 50 among the people who had had two doses of the vaccine. So we wrote an article in The Guardian entitled “Why most people who now die with Covid in England have had a vaccination”, to pre-empt the inevitable misunderstanding that if the vaccinated are dying at a higher rate than the unvaccinated, vaccines don't work.
Really it’s because so many people have been vaccinated. The actual risk of dying among the vaccinated is roughly one-twentieth of the risk of dying for people who are not vaccinated, it’s the same for each age group. If everybody had been vaccinated then anyone who died would have been vaccinated so it's not an indication of the failure of the vaccine, it's an indication of the successful rollout of the vaccine.
We tried to explain, and I think we failed because we got picked up by vaccine sceptics, who only used the top title. I still believe it's right to pre-empt misinformation. Whatever you say will be manipulated but it's best to get it out there honestly rather than be on the back foot and reacting.
Scientists should rise above polarized policy debates
This article came out yesterday in The Conversation, “COVID: media must rise above pitting scientists against each other – dealing with the pandemic requires nuance.” I disagree. Why blame the media? Shouldn't scientists themselves rise above polarized policy debates? You should expect to be argued with don't expect to get away with it just because you’re a professor. If you're going to start telling people what to do you've got to be prepared to take the hit. Mainstream media have in general done a reasonable job at trying to fend off misinformation. Scientists have tried hard to explain but some have become advocates for policy and that's fine if you're representing a named organization, or if you're speaking as an individual, but this is not part of being a professional scientist – scientists should be speaking on behalf of science.
I feel very strongly that this polarization with scientists taking policy viewpoints has been one of the worst things of this pandemic and, if anything, more harmful than the misinformation campaigns that have been around.
Be careful speaking to the media. I did an interview with the Today programme before Christmas, and I was talking about precautions and how I can't sing in my choir anymore, and then I said “Maybe they should ban arguments over the Christmas table because shouting at each other in a small unventilated space is one of the worst things you can do.” So this was a joke. In the Daily Express next day we saw: “Christmas warning: families could be banned from arguing to prevent Covid spread, according to a leading British statistician.”
The HealthWatch Award 2020 was presented on 20th October to Professor Jennifer Rogers. Professor Rogers is Head of Statistical Research and Consultancy, PHASTAR and Vice President (External Affairs) at the Royal Statistical Society. It was a happy coincidence that her presentation was on World Statistics Day. A recording of her 50-minute presentation in full including the slide show and the questions and answers that followed can be seen online here. An adapted transcript of her talk follows.
News headlines are telling us what we should do, how we should live our lives, but headlines can be misleading and our own personal experiences can skew our understanding. Would it not be better to give people the tools they need to ask the right questions?
Let’s take for example the humble bacon sandwich, and warnings from news stories that eating bacon boosts your risk of cancer. Headlines like this in recent years caused bacon sales to plummet. But when we see headlines like this, we are looking at something called “relative risk”. It is not answering the real question, which what is our own individual “absolute risk”.
What is the chance of getting pancreatic cancer? The charity Cancer Research UK says that one person in 80 might develop it in their lifetime. If eating bacon is supposed to boost your risk of developing pancreatic cancer by 20%, that is actually increasing the absolute risk from 5 in 400 to 6 in 400 individuals. So, although 20% sounds scary, it’s 20% of what was quite a small number to start with.
2020 has been a really interesting year for medical statistics. Daily government briefings presenting data are being updated on a daily basis. There are all sorts of important questions about the virus that causes Covid-19 – how it is spread, where is the risk, what treatments are most effective. Amidst the flood of data, there has never been a more important time to use this data to inform decisions.
So now we’re going to look at some of the challenges behind even some of the easiest questions.
How many Covid-19 cases are there in the UK? We have become used to seeing daily reported number of cases, and terms like 7-day moving averages, but how can we know how prevalent the disease is in the general population? Are cases going up or down? Are we any worse off now than we were in March?
The number of reported cases can only ever be a proxy, to help estimate what is going on in the general population. Because the way we test people has changed so much over the last 10 months, we can’t easily compare case numbers detected now with what was detected in March. Prevalence is defined as the proportion of the population who are positive. But we don’t have the capacity to test everyone, so we have to make estimates based on assumptions. Let’s assume the test is 100% accurate, and we detect 100 positive cases amongst 1000 tests. That gives us 10% prevalence, which suggests that 10% of people might have the disease. But we can’t know that for certain. In reality the test will sometimes give us false positives, and false negatives. Also, we are assuming the sample of the population being tested is picked at random. But we’re not doing that. We are mainly only testing people who have symptoms or who have been in contact with others who have symptoms.
What effect does this have? Reported numbers of cases in the UK did not seem to come down as fast as they did in other countries. But if we look at the same period there was an increase in the number of tests. So, prevalence of the disease is likely to be lower now than it was in March, even though numbers of positive tests are higher.
Who we are testing has been changing. Firstly it was mainly people hospitalized with bad symptoms. Lots of the general population had the disease but were not tested. Then we started testing the general public, using drive in centres, and national surveillance data was being generated by random sampling of people. The test and trace system came in, with contacts of cases being invited for tests. Schools started, there are plenty of anecdotal stories of youngsters being sent home with a bit of a cold or fresher’s flu and not allowed to go back without a positive test, so the system has been inundated with people who were probably negative. This surge in demand for tests had such an impact on testing capacity, that many people who were positive had difficulty getting tested. So, figures on test results alone make it difficult to infer anything about the actual prevalence of Covid-19.
Can we use actual deaths as a proxy for case numbers? It is certainly a hard end point. Data from the Office for National Statistics uses information from actual death certificates. Look at graph of deaths over time, there is a reduction after the peak in April, then growth.
But the demographic has changed in that time. There are different risks associated with different demographics. In April, most cases of Covid-19 were being recorded in the older populations, who were turning up in hospital with severe symptoms, and who were at highest risk of dying from the disease. But at the end of September, much younger people were testing positive with the disease. The spread of the virus is changing, so we can’t use the death data to estimate the prevalence.
On the one hand, it is great that the government is being transparent about the figures, but when it comes to the number theatre of showing daily reported cases on the news, is it the most useful way to explain what is really going on? There have been instances of good communication at government press conferences, but I’ve also spotted a couple of things that I have taken issue with, and here are my biggest bugbears:
Rates among schoolchildren: On 30th September, Chris Whitty, when talking about figures on weekly test positivity, showed a graph that he said showed that rates among school age children are not going up. I am not sure I entirely agreed with him! The graph showed test positivity by different age groups, and it is true that the positivity rates among the younger groups didn’t seem to be changing based on the information in that graph. But that wasn’t necessarily true. Assume that we are testing children and the test positivity rate is 10%. Test more children and positivity remains 10%, but the number of cases will be going up. From beginning to end September shown the graph, what was actually happening with testing over that period? In fact, the actual number of tests over this period increased by a factor of 2 or 3. So just because the positivity rates stayed the same throughout, that doesn’t mean that the number of cases that were found to be positive also stayed the same. It seemed a slightly sneaky interpretation of the data that told a narrative that I think was quite convenient at the time.
Exponential growth: This is a phrase we hear quite a lot. 21st September we saw a plot that showed what would happen if cases would double every 7 days, and said that by the middle of October we could reach 50,000 new cases. On the 30th September when Patrick Vallance was questioned about this he said, “Doubling means things get very big quickly”. But really, “exponential growth” doesn’t necessarily mean “fast”. It is more concerned with the way speed is changing. The speed of growth is proportional to the size of the population. Think of it in terms of doubling times. If the doubling time is a day, that gets big really quickly. But with a doubling time of 10 years, that gets big a lot more slowly. In the month before lockdown on 23rd March, it is possible to calculate that cases were doubling every 2-3 days. Now, I don’t want to say that cases will not increase rapidly again. But exponential growth should not be a blanket term used to scare people. Different surveillance studies looking at growth rates give us different values. The REACT-1 study at Imperial College London is doing random population sampling of volunteers to learn about community transmission, and is using this to estimate rates of prevalence, and what growth and doubling time might look like. In the latest publication it estimated that between the 18th September and 5th October it was doubling every 29 days.
Now, the government figures given to us then projected doubling of the number of cases every 7 days. But if you make it every 29 days, the curve is much flatter, which gives us 6,200 cases daily. The government presented the 7-day doubling time data with no idea of measures of uncertainty. Look at what we actually ended up with in the middle of October: although there was a surge early on, there were lockdowns in place in some areas, and now it does seem like the doubling time is coming down.
There are attempts to get better estimates of the figures. The strategy of the REACT study is to invite people randomly to take part. The ONS survey is inviting households who have taking part in other surveys, doing home visits and offering reimbursement. This is not as random a sample as in the REACT study, and the estimated prevalence figures are different: REACT puts it at 45,000 new cases a day, while ONS estimates 17,200 a day. These differences may be due to the different sampling strategies that they have. But the findings for both will be interesting.
Covid has been a unique opportunity, with lots of challenges around messy data. We are in a pandemic that is brand new, and the knowledge landscape is changing constantly. There are many issues I’ve not been able to touch on here. One is the quality of the data, because there is always a balance between data that is available quickly but may be less accurate or harder to interpret meaningfully, and the data that we have to wait for but is more reliable. I’ve not mentioned risk data, but David Spiegelhalter has done some interesting work comparing risks of dying from coronavirus with annual general mortality rates at different ages. He has found that the risk of death from Covid-19 mirrors the way the risk of death generally increases with age.
This winter I’m going to be working with ITV as their Covid-19 statistician. I’ve been looking at how cases are distributed within cities, and finding that increases in cases in cities are being driven by increased numbers of students. It is also interesting that we are still seeing excess deaths at home but fewer deaths in hospital. There may be a conversation to be had not just about numbers of deaths but also quality of death. Test accuracy is another topic – new tests on the way may be more rapid, but will they be accurate? And of course, clinical trials for vaccines and for treatments, to be sure that anything we get on the market will be safe and effective.
Good data alone doesn’t help us make important decisions, we need that narrative and statisticians will continue to be essential in the fight against Covid. At the end of this we will have a new appreciation of the value of numbers, and we will be leaning on statisticians to lead the way through this.
Adapted from presentation by Jennifer Rogers, 20th October 2020
Investigative journalist and GP, Dr Faye Kirkland received the 2019 HealthWatch Award at the HealthWatch AGM in October. Here is her compelling presentation.
In my early 30s I was a GP partner in a practice I had wanted to join for years. A training practice, a very traditional surgery – where the doctors knew theirpatients and generations of families were all known by one doctor. This was the view of family practice I had been given as a medical student, seeing and treating people in the context of their families and communities.
But an internal voice started to grow in intensity. The pressure on the NHS, and the movement of care from hospital to GPs, often made continuity of care for patients more difficult. Qualified GP colleagues I knew left the UK altogether and went abroad, saying their quality of life was better, while I found myself staying at work later and later to provide the type of care I would want as a patient.
The nagging noise in my mind grew louder, and I started to wonder how I could use my knowledge in a different way. On a day off, I managed to get an interview at Cardiff University for a course in broadcast journalism, and to my surprise I got in. In just a week I had resigned from being a GP partner, moved my home into storage and my life from Brighton to Bristol, where I spent 18 months in my Dad’s spare room working as a GP at the weekend to fund the course in Cardiff.
I started to learn the craft of journalism, and realised I wanted to use investigative techniques and my knowledge to shine a light on areas of medicine that other journalists or patients might not see.
Access to diagnostic tests
A few years before starting the course, a close friend of mine had become seriously unwell. He had been to Accident & Emergency several times, collapsed at home and was even told by ambulance staff to pull himself together, that he had a virus. He was finally diagnosed with a brain tumour. Dif
ficultly in accessing imaging, and people not recognising symptoms, were both major hurdles to him getting his diagnosis. The skills I had acquired allowed me to investigate whether his experience was reflected across the country. I knew from when I had worked as a GP that in some areas I couldn’t even refer for an MRI scan, but in others I could. Despite the Department of Health saying that all should be able to access this, different clinical commissioning groups had different rules. I established a picture that showed this post code lottery. It was my final piece of work on my course but ended up as an hour of live radio for BBC 5 Live Investigates.(1)
Vitamin and mineral infusions
Within months of qualifying, I became interested in the world of vitamin and mineral infusions. While researching one programme I went to Harley Street undercover, posing as a patient.
I asked doctors if the vitamin and mineral infusions they advertised would treat my supposed eczema and anxiety. At one clinic a practitioner asked me to hold up my left hand, and with my right touch various organs of my body. The practitioner pressed down on my left arm and seemed to growl as if the diagnostic technique had revealed something about the state of my body. I was provided with vials of different vitamins to hold against various organs, and again asked to repeat the arm test. This time, it seems, I was stronger. This diagnostic revelation meant I needed vitamins – and fast. An urgent infusion was offered. I politely declined.
The cost of this miraculous arm test ran to hundreds of pounds and the infusions which, apparently, I needed every few weeks were approaching £200 each time.
The importance of my investigative work became apparent to me as I looked in a
room and saw a very frail man hooked up to two drips – one in each arm.
I found more than 100 webpages claiming that vitamin and mineral infusions such as these could help treat Parkinson’s Disease, asthma, depression and hepatitis, with no evidence to support the claims.(2) It was clear at the time – this was 2015 – there were huge gaps in regulation. Were these infusions, which were straight into your vein, a medicine or not? Up to then, no one had decided. The drugs regulator, the MHRA, took the investigation seriously. They had to write to each clinic, to analyse each of the vials to decide if these drugs were medicines or not. Eventually they decided medical claims could not be made against these products, as to do so, might imply they were a medicine. The claims, they say, had to be removed. The majority of the medical claims for these infusions have now gone from the websites – in doing so I hope protecting patients. However, unfortunately, un-evidenced generalised claims of improving your well-being or giving you a glow persist.
Online health care
That was the year I came across a burgeoning new area of medicine: care online, ever accessible, at a click of a button and often at a price.
Private, glossy websites often featuring celebrity doctors and well-known brands, offering care I would not give in the surgery. You filled in an online form with pre-set questions, send it off. A doctor reviews the form, prescribes medication for you without speaking or seeing you or having access to your NHS notes, and then a UK chemist would send the drugs to your home within 24 hours.
Four years ago, I found sites offering to cure sexually transmitted diseases after a few questions, with no warnings that their treatments were suboptimal and might not cure your infection.(3)
The familiar brand, Superdrug, was at that time making antibiotics available online for gonorrhoea, without the customer necessarily having proved they had it, and providing antibiotics which did not meet national guidelines. This was at a time when some patients were having to be treated for gonorrhoea in hospital, having acquired multi-resistant strains.(3) Following the investigation for BBC 5 Live Investigates, there was intervention from the British Association of Sexual Health and HIV, and a letter from The Chief Medical Officer and Chief Pharmaceutical Officer to all online clinics and GPs.(4) Since then there has been a massive change. Superdrug no longer provides antibiotics for this indication – instead they advise you, correctly, to seek treatment at a suitable clinic.
This experience made me wonder what else you might be able to buy online.
As with all such things, digital health care has stayed one step ahead of the regulators. In 2016, again with BBC 5 Live, we went undercover posing as patients. My colleague was able to buy drugs for an ear infection, supposedly of three days’ duratio
n, therefore probably viral and hence not curable by antibiotics. Some halfway through the questionnaire we were abruptly asked about sexually transmitted disease and a vaginal discharge. Following this, drugs were prescribed that I have never given for an ear infection. This was all delivered to the house and was signed off by a practising UK doctor.(5)
How could this be allowed to happen?
We asked the company which had provided this medication. They blamed a computer problem saying the questionnaire about vaginal discharge had got mixed up with the one for ears and actually we had been sent antibiotics for the wrong condition.
We asked the Care Quality Commission (CQC) – the health regulator in England – and the General Medical Council (GMC), the doctor’s regulator, to respond. The CQC brought forward a wave of inspections, with the clinic exposed being one of the first. It was suspended. The GMC investigated the doctors. Clear online health care standards for online companies were published by the CQC the following March.(6)
A step forward you may think? But last year, with the BBC’s Panorama, I found the sites were once again one step ahead. Not only were they supplying antibiotics but often drugs of misuse, such as opiates.
Some sites moved their headquarters out of England and therefore avoided regulation by the CQC, most choosing to base their companies in Romania. Now their online doctors’ care could not be regulated, and the prescriptions could be sent electron
ically from Romania to pharmacies in the UK and still delivered to your door.(7)
Panorama ‘Online doctors uncovered’
Still with Panorama, I wanted to see these Romanian companies – the supposed hub for these online doctor sites – for myself. But instead of finding a hive of doctors doing online consulting, I found empty flats. Just an address, that means the CQC cannot regulate and keep patients safe.
After the Panorama programme was broadcast last year I was contacted by a family whose daughter had been able to buy codeine from 18 online UK pharmacies, as well as continuing to obtain these drugs from her GP, who had not been informed. She had collapsed and died behind her front door. She was in her early 40s. Her inquest is yet to take place but her family believe the codeine contributed to her death.(8)
Also following the programme, and in part as a result of it, the General Pharmaceutical Council has also moved to stop pharmacies in Great Britain from dispensing certain high-risk medications to patients without their family doctors being informed.
Also in response to Panorama, the CQC asked the Government to change legislation. They are still waiting for this to happen.
Investigative journalism has taken me to places I never imagined, secret meetings, being given leaked documents and holding power to account. Continuing to be a doctor is a privilege but helping to create change on a national level can be an even greater one.
Freelance investigative journalist and GP, London
1. BBC Radio 5. 5 Live Investigates – Brain tumours. 26 Oct 2014. https://www.bbc.co.uk/programmes/b04mbk68
2. BBC Radio 5. 5 Live Investigates - Discharging mental health patients & Phishing emails, 18 Jan 2015. https://www.bbc.co.uk/programmes/b04yg8f1
3. Kirkland F. Concern over online gonorrhoea treatment. BBC News, 1 Mar 2015. https://www.bbc.co.uk/news/health-31649099
4. Gallagher J. Gonorrhoea 'could become untreatable'. BBC News, 27 Dec 2015. https://www.bbc.co.uk/news/health-35153794
5. BBC Radio 5. 5 Live Investigates - Online antibiotics. 2 Oct 2015 https://www.bbc.co.uk/programmes/b07xf2xr
6. Kirkland F. Buying medications online 'can put health at risk'. BBC News, 3 Mar 2017. https://www.bbc.co.uk/news/health-39134061
7. Kirkland F. Safety concerns over websites selling prescription drugs. BBC News, 6 Aug 2018. https://www.bbc.co.uk/news/health-45084555
8. Kirland F. Clampdown planned for British online pharmacies. BBC News, 16 Apr 2019. https://www.bbc.co.uk/news/health-47933346
From GP to MP: How to lose friends but try to influence people
Sarah Wollaston, MP for Totnes, Devon, spoke at the 2018 HealthWatch Annual General Meeting on Wednesday 31 October 2018. It had already been an eventful week. Our AGM took place the day after the House of Lords Science and Technology Committee had published the results of their inquiry: Research integrity: clinical trials transparency. Earlier that week, the Chancellor of the Exchequer had presented his 2018 Budget to Parliament. The following text is adapted from her presentation and the question and answer session that followed.
There is a real challenge ahead of us about how we present evidence within our politics, and value evidence. Something that I have noticed in my role chairing the Health and Social Care Select Committee, is how within politics you have to find a balance between the idea that everything should be based on popular demand, and presenting the evidence.
Nowhere is this more evident than in public health. When you go out and consult the public about where they would like to spend investment within the health service, public health is always at the bottom of the list. Yet the evidence shows that if you really want to tackle health inequalities and make a difference, that’s where the money should be going. But I’m afraid it always tends to get deprioritized, as we’ve seen over the last 24 hours, as we unpick what the Budget numbers mean. There’s been a bit of smoke and mirrors, in that some of the funding for NHS England is actually going to come out of public health budgets, which I think is a tragedy because it’s through public health that we’re going to really tackle the issues that governments all pay lip service to, about burning injustices, and reducing inequalities.
My point is that you’ve got to look at the evidence about how you’re going to achieve those aims. And in a week when we’ve got the Chancellor talking about how he wants to reduce the tragedy of lives lost to suicide, while we would all agree that’s extraordinarily important, you shouldn’t at the same time delay implementing evidence based policy around reducing gambling addiction because you’re caving into industry lobbying that is fighting the introduction of the maximum £2 fixed-odds stake for betting terminals. We absolutely need people in Parliament to make the case relentlessly for evidence and how you actually make a difference, rather than what the lobbying industry says makes a difference, or what is popular. And right now we are in the midst of the fight of our lives, against the “Trumpification” of politics, the downgrading of evidence, the ridiculing of any expertise, and I think nothing has exemplified that more than the referendum campaign and the way it was conducted.
That’s my job in parliament, to be rattling a few cages and to keep doing that! But there’s something you in HealthWatch can do. Never underestimate the effect of going to see your MP in person. If you write an e-mail, send a postcard, or re-tweet, you’re not actually going to shift the dial. And it doesn’t help for us to be talking to people who already agree with us. What we need to be doing is talking to the people that don’t agree with us, and having somebody actually turn up in your constituency surgery makes an MP think, I have to look at this. Tomorrow I’m launching our committee’s report into prison health care, and that for me came out of a grandmother coming to talk to me in tears about the death of her grandson in prison, and the authorities’ failure to follow up on reports that had been coming out from inspectors about circumstances that had led to that avoidable tragedy.
So, going to see your MP really has an impact. I hear time and again from colleagues that it is the person who takes the trouble to make that appointment that makes the lasting impression. Encourage your members to get out and talk to your MPs and talk to them about the importance of evidence and science.
Q: The Science & Technology Committee have done two reports on research integrity, but if you read them, nearly all the concerns are about medical research - I gave evidence to that committee, and it’s entirely about medical research. Should the Science and Technology Committee co-ordinate efforts more with other areas of research?
SW: Increasingly committees are doing joint enquiries where something is of mutual interest, for example, the report we recently published on anti-microbial resistance follows up on many of the points from previous committees. So we do try.
Q: At a meeting today at the Royal Society we heard about TARGIT (TARGIT IORT is targeted intraoperative radiotherapy treatment for breast cancer, see ), a well-researched innovation that has been reviewed and recommended by NICE, that would save NHS money, and which offers huge benefits to patients, yet has not been adopted by Public Health England. How can we free NICE so that their recommended good practice actually gets implemented?
SW: It’s a huge challenge in many areas of the NHS where you see good evidence based practice and it’s not being effectively rolled out. I would advise you (1) to go to your MP and try and get them to stand up and talk about it in Parliament; (2) there are all party parliamentary groups within parliament, for example there is one on breast cancer specifically, you could raise it with them; (3) although the Health and Social Care Select Committee tends not to look at single disease issues, we do hold regular accountability session with bodies like NICE, so there’s the possibility we could use a session to raise such an issue and ask whether we could help if their recommendations aren’t being implemented. You are very welcome to write to me, and although we wouldn’t hold a specific enquiry I might be able to raise in correspondence or queries issues that people raise with me.
Q: Do you have any thoughts on the matter of a people’s vote?
SW: Who would want to be wheeled into the operating theatre on the basis of a consent form we had signed two years ago, or in the case of younger voters, one their parents signed, and if you didn’t even know which operation you were going to have! And the surgeons are still arguing amongst themselves, and there is no majority for anything. The implications of Brexit will last for generations.
Q: At our recent HealthWatch symposium on debunking false health information, one of the issues was the level of confidence people have in their source of information – how do you and your committee decide whether a group or individual presenting you with evidence is credible or not?
SW: There are some markers for not being credible, for example, people who write to you with a sample size of ten to support a view that something is important should be implemented straight away. Essentially in an inquiry we publish everything, but we choose the witnesses we then hear from selectively. Unfortunately there are always more fantastically credible witnesses we’d like to hear from than we have time, so we have to prioritize based on what they write to us in their evidence. And we also try and get out of London and meet as many people from different areas as we can.
Q: The Science and Technology Committee report on clinical trials reported yesterday on Research Integrity; the govt now has two months to respond to this report, is there anything we can do to tip the scales towards a favourable response? Secondly, looking at the various NHS foundation trusts that sponsor clinical trials, the reporting rates are bad, and there’s now going to be a lot of inefficiency as each individual trust tries to figure out how to improve their clinical trial reporting – how can we ensure economies of scale to do this? Who is responsible in each trust and can they nominate someone to do it?
SW: Some people have got considerable expertise already – it is encouraging that Matt Hancock has appointed Ben Goldacre to be chair of his advisory panel, who has done fantastic work in this area and is not afraid to hold Public Health England to account. But it is not easy to identify who is responsible. When asking government to respond to a report, you can put in a recommendation to the government to identify who is responsible and who to hold to account.
Q: Part of the problem is that the Science and Technology Committee have put this back to UKRIO (the UK Research Integrity Office), the UK universities, who have consistently failed to self-regulate, and who have a concordat that only 30% of them adhere to, and none of the hospitals where human research is conducted are part of the concordat, so the committee has come up with a solution that has failed in the past and doesn’t anyway apply to medicine.
SW: It would be worth speaking to Norman Lamb MP about that. (Questioner's reply: “I have!”)
Q: Would you agree to introducing clear guidelines on minimum staffing levels for the NHS?
SW: I would agree that in an ideal world we would have minimum staffing levels, but the trouble is, there is overall a workforce staffing shortfall. If you were to impose a minimum staffing level on wards, if there was limited flexibility to move staff around you might find unintended consequences, such as you might not be able to transfer people out of ambulances into the emergency department, and end up with people being cared for in corridors or in ambulances. The workforce challenge across the NHS and across social care is extraordinary. In my area, there is a 8% vacancy rate in social care, and 7% of the workforce are from the EU, of the nursing workforce in Devon it is nearly 30%. What is going to happen if we make it much more difficult to recruit and people don’t feel welcome here?
Q: Granted, you can’t say, if you’re understaffed you’ve got to shut the doors and stop. But are there halfway houses? Supposing an acute hospital trust had to be on amber, so that if something goes wrong we can’t blame the person on the coalface?
SW: People make comparisons with airline safety but this is different in that you can ground an aircraft, but you can’t close a hospital - although there is a precedent even for that in critical areas - or you end up with people being cared for in sub-optimal conditions. I agree with the suggestion of a very clear amber alert where a hospital is understaffed, though if we imposed such measures too rigidly I would have concerns.
Q: Mass cancer screening – why don’t we have proper standards of informed consent in cancer screening? And how can we improve decision-making so that we can stop doing something that doesn’t work? On screening, there are also concerns over vested interests and charities running awareness campaigns that create fear and havoc and end up sending healthy people to add to their GPs workload?
SW: Margaret McCartney has done some amazing work highlighting this, and my congratulations to her for highlighting the losses and harms resulting from the rise of the private screening industry, with the costs falling back to the NHS, and harms to people who are persuaded to have these tests. Just yesterday I asked the Care Quality Commission what they are doing to ensure these clinics are operating in an ethical manner. Certainly there’s more that can be done. Drop me a line.
Q: Do you feel reassured about the provisions for medicines supply after we leave the EU?
SW: There has been a phenomenal amount of money diverted into contingency planning, which is going to feed back into the cost of drugs and healthcare. When you look at the amount of time it can sometimes take even now to source medicines – for example, my daughter is a junior doctor in pediatrics at the moment, and she recently spent two hours trying to source an epipen for a patient - I can’t begin to imagine what it will be like if these problems occur on an industrial scale after we leave the EU. The public is used to being able to take a prescription to the pharmacy and expect the medicine to be there. What is going to happen when the medicine is not there and no-one can find it for you? When my constituents get cross with me for banging on about Brexit, I say to them, when we are three months on after a hard Brexit, and these issues are still ongoing, I want to be able to look you in the eye and say that I tried my best to present the evidence of what would happen as a result. I hope these problems won’t happen, I hope I’m wrong, but even if I am, it’s been a most tragic waste of resources.
Q: Has the NHS become a religion? We are spending £150bn a year on the NHS alone, about a quarter of all public spending, yet it’s nothing like enough? Where is this going?
SW: There’s a huge success story we shouldn’t forget - we’re helping people live longer. Yes, we’re living with more years of ill health and multiple morbidity, so it’s a success story not a disaster. But did we adequately plan for this? For generations people have failed to take the long view and look at what longer life means for the health workforce, and people’s readiness to put money in. The trouble is, we promised people they can have all this for free. This week’s budget was an opportunity to say to people, honestly, this extra commitment has to be paid for somehow. But what we’ve seen is talk about Brexit dividends, and giving people a tax cut a year early, rather than being straight with people and saying that if we value our NHS we’re going to have to pay more. When something is a religion, criticism is not very welcome, and you can be vilified for it. We should guard against treating the NHS like a religion, because we should be able to challenge bad practice.
Q: A lot of what I do as a GP only results in marginal gains, we are given the impression we are saving lives, but in practice we are throwing money at the end of life.
SW: The biggest gains in healthcare have resulted from public health measures. If you were going to start somewhere, start with the first 1000 days of life – it is the subject of our current inquiry. Look at education, housing, dealing with poverty, the things that make the biggest difference to health.
Q: How do you cope with being an MP in Totnes?
SW: It is a wonderful and interesting place, but let’s take just one issue, that of vaccination. Totnes has the lowest vaccination rate in Devon, and there is a very strongly held belief by many people in town that homeopathic vaccines are the way to go. There are many children there who have not been vaccinated against anything at all, not even tetanus. This can be a challenge, because some people can become very cross if I point out that some of the claims made against vaccination are untrue, and they may say I have no business to represent Totnes when I do not support the alternative community. But my view is that it is absolutely my job, to speak the truth it as it is. The rise of the anti-vaccination movement and the resulting resurgence of measles is deeply worrying. Fortunately Totnes is much happier about my stance on Brexit.
Based on Sarah Wollaston’s presentation at the HealthWatch 2018 AGM, adapted by Mandy Payne.
Poking your nose in where it’s not wanted: the dark side of investigating healthcare
The 25th HealthWatch Award was presented to the former BMJ Investigations Editor Deborah Cohen, in recognition of her courageous reporting of medical issues in the face of attack from vested interests. She received her award from HealthWatch president Nick Ross at the 2017 HealthWatch AGM on Tuesday 17 October 2017, and the following text is adapted from her presentation.
It has meant a lot to me to know that others recognise that you’re reporting in the face of vitriol and attacks from vested interests.
The first big story I did was in 2009. HealthWatch members will already be familiar with the Tamiflu story and how the Cochrane Collaboration struggled to access clinical trial data for their review of this flu drug. The investigation into the missing data ended in my in-tray. I have degrees in medicine and medical journalism, so it was something to get my teeth into, and it turned out to be an interesting journey. As you’ll recall, the lead authors of the clinical trials had told us they didn’t have access to the primary data. The manufacturers, Roche, gave us observational data. But not the clinical trial data.
One of the privileges of working at the BMJ is you have access to some really good academics who will drop everything to help you out – they know who they are. So we set them to work making sense of this observational data that we had. Roche maintained that Tamiflu not only reduces the incidence of secondary complications by 67%, but it also reduced the rate of hospital admissions by 61%. But when we re-analyzed the observational data we found it didn’t say that. We published our report,(1) and the allegations we made were quite controversial. No satisfactory explanations had been given for the missing clinical trial data. Yet there were journalists who’d been covering the flu pandemic for a while, who’d taken it at face value that Tamiflu must be effective enough to merit the UK government stockpiling it – at a cost of £424m between 2006 and 2013.
Conflicts and controversies
In the course of our search we came across whistleblowers who had written up the clinical trial paper that was published in The Lancet. We were told they’d been instructed to include key words and phrases that were important for the marketing of the drug. It also became apparent that the people who were involved with the companies marketing the product were also key advisors to the World Health Organization (WHO), the European Medicines Agency (EMA), and others. Roche, together with some other manufacturers of influenza drugs, are the funders of a group called the European Scientific Working Group on Influenza (ESWI). This industry-funded group wrote WHO’s first influenza pandemic preparedness plan.
So, as a result of this investigation we did a follow up piece about the conflicts of interest of the advisors.(2) This proved even more controversial than the first, and we came under attack. The WHO would not release any information about whether their advisors declared conflicts of interest or not. The scientists said they had, but WHO would not confirm or deny.
Embarrassingly I ended up on an American “shock jock” show, where they took the line that this is a big pharma conspiracy, which was very far from what we had intended with this story. We’d been very careful to make the point that when you are making policy decisions, which could have huge impact in terms of finances and patient harm, should you really be using the people who have been marketing the drug? You would think not.
But the attacks continued. Nature did an exposé,(3) and people tweeted that our research had been discredited. In our article there had been a sentence that maybe could have been worded more carefully, so we ended up conceding one or two points in relation to the story, and the result was taken as an admission that our story was wrong, starting a back and forth between Nature and the BMJ. It became one of the most read articles on the BMJ website at the time.
The next thing we looked into was medical device regulation. We started with the European Union directives, which took some reading. Ours was a joint investigation between the BMJ and Channel 4’s Dispatches, so we had a whole team to help us go through the directives. Then in 2011 a very useful study was published by a group at the Cleveland Clinic, in Ohio. They looked at the US Food and Drug Administration (FDA)’s regulatory system for medical devices. The FDA have two processes, one is pre-market authorisation, and the other is called 510(k). To get a device onto the market using pre-market authorisation you have to have clinical evidence, such as from randomised control trials, or a case control study. But for 510(k) you only need to show something called equivalence, which means that your new device is “substantially equivalent” to an old one, a predicate device, that is already authorised.
It is possible to trace the history of some 510(k)-authorised devices, and to track the tweaks and changes that have been made to the design over time, so that a device approved in 2015 might not have had any clinical studies to support it, and gone through so many changes since, say, 1980, when the original version was approved, that you can end up with virtually a whole new device but without any clinical evidence for it. The Cleveland team’s study found that two thirds of device recalls were in respect of devices that had gone through the 510(k) route, that is, there was no direct clinical evidence for those versions of the devices.(4)
Working with the Centre of Evidence based Medicine at Oxford University, we thought it would be interesting to replicate this study in Europe. But we couldn’t. Because we kept coming up against the words: “commercial, in confidence”.
Devices come in different risk categories. In Class I, you have low-risk items like plasters and syringes; in Class II there are insulin pumps. Class III is for hip implants, pacemakers, things that need surgery to get them in place. I phoned the Medicines and Healthcare Regulatory Authority (MHRA) and asked if they had a list of Class III devices that are used in the NHS.
And they said “No”. No such list exists.
We knew that some implants must have been causing problems, so we put in a freedom of information request to look at reported harms. Again, “commercial in confidence”. We learned that under the Freedom of Information Act you can get clinical data related to pharmaceutical trials, but you can’t get it for medical devices.
So, we were thwarted from the outset by a lack of transparency. We wrote a story(5) about a hip implant called the ASR, which is made by DePuy, a subsidiary of Johnson & Johnson, and which was recalled in 2010. The ASR was used for two types of orthopaedic surgery – total hip replacement (ASR XL) and hip resurfacing (ASR resurfacing). The ASR in the United States had gone through 510(k) for total hip replacements. For the hip-resurfacing application it should have gone through the full pre-market authorisation process, but the FDA had called a halt to the approval because of problems with the trial. That didn’t happen in Europe. It has been used in tens of thousands of patients.
We later looked at all large diameter metal on metal total hip replacements. We reviewed the medical literature, and whole 510(k) daisy chains, and found that among an entire class of implants there was no publicly available clinical data to support their use. Now, bear in mind that these are made of cobalt and chromium. In the 1970’s and 1980’s surgeons were debating the toxic effects of cobalt and chromium, and in 2007 a comment piece in the Lancet(6) pointed out that little is known about the transport, distribution, excretion of metal ions in the body; toxic effect thresholds have not been characterised. So wouldn’t you think there would be a need to proceed with caution? We know a lot about the local effects of metal ions in the body, but about the systemic effects we know very little because if we’re talking about pharmacovigilance of medical devices like you have with drugs, well, it just doesn’t happen.
My question was, in this article, surely we should proceed cautiously with these devices, given that there is no known toxic threshold for these compounds? In the media it became, the BMJ hip replacement cancer scare, which had not been our intention at all. Suffice to say, that the bullying I encountered as a result gave me a glimpse into the business side of orthopaedic surgery and how things run when there is a great deal of money at stake, and some of it is very unsavoury.
After that, we went under cover. I should point out that it is actually really hard to get clearance to go under cover. There has to be a strong public interest reason, and you have to demonstrate that you can’t get the information any other way. We had a good reason – this was health. We created a fake metal hip. We called it the TMH (total metal hip) and tried to get our metal hip onto the market. We had a fake dossier, and gave it some disastrous data, failures, ions all over the place.
The system here in Europe is that you have to go through what are called notified bodies. There were about 70 of these, they are private organisations, and these are where manufacturers go to get a CE mark. Once they have this mark they can sell the device anywhere across Europe. The various regulators in the different countries then regulate those bodies. It’s an incredibly complex system.
You have to pay these bodies to review your dossier, before they decide if it should receive a CE certificate. That already seems like a conflict of interest, as these companies are paid for approving the device and then are paid again to do follow up audits every few years. An officer at one of the notified bodies in the Czech Republic admitted to us that it is “on the side of manufacturer and their products, not on the side of patients.”
We set up a fictitious company called Changi, and we hawked our new device around notified bodies, pretending to be their PR people. We found that not only are regulators out-sourcing the regulation of medical devices to private companies, but some of these private companies are further out-sourcing their device regulation activities to other private companies. We found ourselves at a branch of a Czech notified body which was in South Korea, where a small group of companies came together to operate as a one-stop-shop. Device manufacturers would go to one for advice about how to create their dossier, then another to receive the CE certificate, and then the next company offered marketing. We were told that it is easy to get products approved in Europe, and they are not worried about inspections by European officials, though they were really scared of the FDA.
We got our design approved for a CE certificate for our non-existent hip implant. That investigation had an impact.(7) We presented our evidence at the Science and Technology Committee and at the European Parliament when they were updating their medical device regulations.
Sugar and water
We were at a meeting just before the 2012 Olympics, when sports medics told us that sports drinks are one of the most controversial thing ever. What could be controversial about sweetened water? Again working with the Centre of Evidence based Medicine, we decided to take the top 10 best selling UK and US sports and fitness magazines, and pulled out the adverts to see what health claims they were making for sports drinks. Then we looked at the references to see what evidence we could find to support them. Then we tried the manufacturers’ websites to find references, before approaching the companies for evidence to support their claims.
We were told things like, water doesn’t quench thirst.
Anecdotally, we heard that one company was telling kids that if they drank water rather than sports drinks they would get cerebral oedema. An idea had been created, that of ‘exercise induced dehydration’. Now, any doctor who has treated someone with genuine dehydration, well, it’s not quite about jumping on the treadmill and getting a bit thirsty. But this idea had been created that if you exercise you get exercise-induced dehydration, and the solution is to drink more sports drinks. This then led to a genuine health concern which is, hyponatremia – a reduction in the body’s electrolyte levels due to drinking too much, which was leading to cerebral oedema, and there was a study in the New England Journal of Medicine that looked at the Boston Marathon and people who had had exercise-induced cerebral oedema, because they’d actually been taking in too much fluid.(8)
The backlash against our “The truth about sports drinks”(9) was incredible. The attacks kept on coming. One company held up what they said were over 100 clinical trials that underpinned their science. We critically appraised all of them, and found that most were poor quality and didn’t support the main claims made.
We did a programme with BBC’s Panorama about in vitro fertilisation (IVF).(10) You get to a certain age and lots of your friends are going through IVF, and if you listen to their stories you find they’re all having different kinds of treatments and also being sold extras, on top of IVF, so-called “add-ons”. We again teamed up with Oxford University, we scraped lots of websites from fertility clinics looking at the claims made for add-on treatments, and then looked at the evidence. We found very few were able to support their claims that these add-ons could actually improve your chances of having a baby. We used undercover again – and came under criticism for this – to look at the lack of fully informed consent for these treatments, and the lack of evidence, and how poorly regulated they are by the HFEA. Needless to say we experienced a backlash from private clinics in the UK.
There are times when you get home, shut the door, and you’re looking at the door, wondering what’s going to happen next. You can take the professional attacks, but worse than that is the sexual harassment. As a female reporter on a heavyweight subject I have my authority questioned, because I’m young and female, and I am constantly challenged about my ability to carry out these investigations. It’s a constant battle, and that is probably the biggest challenge in this area.
Based on Deborah Cohen’s presentation at the HealthWatch 2017 AGM, adapted by Mandy Payne
1. Cohen D. Complications: tracking down the data on oseltamivir. BMJ 2009;339:b5387. http://www.bmj.com/content/339/bmj.b5387
2. Cohen D, Carter P. WHO and the pandemic flu “conspiracies”. BMJ 2010;340:c2912. http://www.bmj.com/content/340/bmj.c2912
3. Butler D. Flu experts rebut conflict claims. Nature 2010;465:672-673. http://www.nature.com/news/2010/100608/full/465672a.html
4. Zuckerman DM1, Brown P, Nissen SE. Medical device recalls and the FDA approval process. Arch Intern Med. 2011 Jun 13;171(11):1006-11. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/227466
5. Cohen D. Out of joint: The story of the ASR. BMJ 2011;342:d2905. http://www.bmj.com/content/342/bmj.d2905.full.print
6. Learmouth ID, Case CP. Metallic debris from orthopaedic implants. The Lancet 2007;369(9561):542–544. http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(07)60251-7/fulltext
7. Cohen D. How a fake hip showed up failings in European device regulation. BMJ 2012;345:e7090. http://www.bmj.com/content/345/bmj.e7090
8. Almond CSD et al. Hyponatremia among Runners in the Boston Marathon. N Engl J Med 2005; 352:1550-1556.
9. Cohen D. The truth about sports drinks. BMJ 2012;345:e4737. http://www.bmj.com/content/345/bmj.e4737
10. BBC One Panorama. Inside Britain's Fertility Business. First broadcast Monday 28 November, 20:30. http://www.bbc.co.uk/programmes/b084ngkd (not currently available).