Guidance: Reporting Statistics and Data

Editorial Guidelines issues

This guidance note discusses how to report statistics and data, while avoiding some of the pitfalls. Advice in assessing the creditability of data-based stories; statistical checking or how to report statistics can be sought from Head of Data, BBC News and the BBC centres for data journalism in each Nation.

Key Points

• The same scepticism for numbers should be reserved as it would be for any fact or quote.
• The Editorial Guidelines state:

When output includes statistics, they should be in context, weighed, interpreted, challenged where appropriate and presented clearly. The BBC's use of figures should give them no more and no less weight than the evidence supports. (11.4.1)

• Claims made in press releases should be tested appropriately rather than taken at face value.
• Content makers should consider whether the results of data are 'statistically significant', in other words whether there is confidence the findings are not due to chance.
• Audiences may need to understand how the data originated to assess its importance. This may include understanding study-design; sample size; representativeness; how the data was collected; geographical relevance or time periods and the methodology used to analyse the data.

Introduction

Statistics are a great source of information which can lead content makers to strong stories, provided the right questions are asked. The same scepticism for numbers should be reserved as it would be for any fact or quote, whether the figures are from the BBC's own journalism or other organisations and individuals.

The provider of the statistics should be able to justify their figures and conclusions and explain any assumptions upon which they are based.

There are a few top-level questions content makers should ask:

• WHO has produced the statistics? How reliable is the source?
• WHY have the statistics been produced and why now? What question is the statistic an answer to? Does the source have a vested interest or hidden agenda?
• HOW have the statistics been compiled? What exactly has been counted? Are the underlying assumptions clear?
• WHAT does the statistic really show? Does the study really answer what it set out to test? What are the producers of the statistics not telling you? Might they be hiding negative results or failed studies? Avoid automatically taking statistics at face value.
• WHERE can you find the underlying data and is it available?

The Editorial Guidelines state:

When output includes statistics, they should be in context, weighed, interpreted, challenged where appropriate and presented clearly. The BBC's use of figures should give them no more and no less weight than the evidence supports. (11.4.1)

Sources

For guidelines about assessing statistics from sources, see Editorial Guidelines 11.4.3 – 11.4.6.

The Editorial Guidelines caution that:

Peer-review of research published in scientific journals is an indication of reliability but does not guarantee it. Scientific research that has not yet been peer reviewed should be treated with extra caution. (11.4.6)

Content makers should be mindful that peer reviewed research may contain bias. Selection bias, observer bias and reporting bias may exaggerate the effects of an intervention. Publication bias may also create a misleading bias in the overall published literature, as studies with positive findings are more likely to be submitted to, and published by journals, than ones where no effect was shown or were inconclusive.

The Editorial Guidelines also warn that:

Although press releases can be the source of useful content, they can also contain exaggerations about research, using statistics selectively. Such claims should be tested appropriately rather than taken at face value. (11.4.6)

Content makers should consider looking beyond the headlines in press releases, to reveal stories other than those based on the averages or the UK population as a whole. Comparisons between different sectors of the economy, or different groups in society, for example, may reveal different aspects of economic growth. Geographic breakdowns at national, regional or local levels, may also strengthen reporting on the devolved UK. For guidelines about comparisons see Editorial Guidelines 11.4.12.

For further discussion about how to evaluate statistics from sources, such as surveys and polls see separate guidance note.

(See guidance: Opinion Polls, Surveys, Focus Groups and Straw Polls: Surveys by other organisations)

Contextualising Statistics

For guidelines about the importance of contextualising statistics see Editorial Guidelines 11.4.1-2 and 11.4.7.

For guidelines about reporting percentages and percentage changes; projections; uncertainty; correlation or causation; comparisons, graphs & charts etc and risk see Editorial Guidelines 11.4.8 - 11.4.14.

Averages

Averages are useful ways of summarising lots of numbers in one figure, providing the correct average measure is used in the right context.

The Editorial Guidelines caution that:

Content makers should take care with their use of language, such as in demonstrating an average, or a mean, or a median point. Using such terms interchangeably or in the wrong context can result in misleading the audience. (11.4.7)

Avoid using the term 'average' to mean 'typical', 'ordinary', 'normal' or 'most people', unless it means that.

The mean (the sum of all the numbers, divided by how many numbers there are) is a useful calculation to show the 'average' (or central value), if the sample is representative and sufficiently large.

Using the mean may be misleading, where distributions might be skewed, such as with salaries, where some wages may be much higher or lower than most. For example, if the mean income of ten middle-managers in a pub is £50,000 and a billionaire, walks in, the mean income suddenly shoots up to £100 million. The mean income is now far higher than the actual earnings of everyone in the pub, other than the billionaire. Using the mean here gives a false impression of the 'typical wage'.

Where a dataset contains a few extremely high or low values and is unrepresentative or insufficiently large, then the median may be a more accurate and appropriate way to represent the central value. (The median income is the one halfway through the list of incomes lined up from smallest to largest.)

Care should be taken when reporting a change in an average as it does not necessarily mean a change for an individual. For example, if average wages rise, this does not mean that all people in the distribution get paid more; some people may not have seen an increase in their wage at all because a mean value does not reflect income distribution.

Big and Small Numbers

Content makers should be mindful that just because a number is very big or small does not make it substantial. Millions or billions are not part of everyday experience, so it is not easy for the audience to judge if they are big or not. (See Being Clear About Significance below)

Content makers must be careful not to make a story more dramatic by the use of extreme numbers. Big and small numbers should be put into context and divided by the number of items to which they relate or people they affect.

For example^[2], a government promise to spend £300m over five years to create a million new childcare places may seem a lot of money equalling £300 per place. But when calculated per year (divide by 5), it's £60 annually and only £1.15 extra per week (divide by 52).

Outliers

Outliers, or the most extreme and unexpected numbers (large or small) that don't fit the mould in a dataset, should be treated with an additional level of scrutiny.

Often outliers can be chance phenomena or due to experimental abnormality, data error or measuring mistake. As such, they may not reveal anything unusual or scientifically significant at all and a story based on such an outlier may need to be rejected.

But not all outliers are mistakes, and these unrepresentative numbers might mark something significant. So, consideration should be given to whether outliers are credible, given existing evidence. If in doubt, ask the producer of the statistics and check with other experts in the field.

Rising or falling numbers

Care should be taken when reporting rising or falling numbers, explaining what they rose to or fell from.

Content makers should recognise that numbers can go up as well as down and avoid attaching too much importance to chance results. A high number could also be part of a falling trend, so care needs to be taken when drawing conclusions from a peak, as it may not represent an upward curve. When a number reaches an unusual high, it's likely to fall to a more typical number next (unless, for example, it represents the start of an epidemic). When exceptional high or low values return to more typical values over time, statisticians call this 'regression to the mean'.

Regression to the Mean

Content makers should be mindful that unusually high or low measurements in repeated data tend to be followed by measurements that are closer to the mean (see Averages above). This is known as 'regression to the mean' and happens because most values are closer to the mean than the extreme ones.

Consideration should be given to regression to the mean as a possible cause of an observed change. For example, the reduction in the incidence of an illness following a vaccination programme may be explained by regression to the mean, particularly if the intervention was at the height of an outbreak.

Consideration should also be given to other factors which may have influenced a change. For example, the introduction of a speed camera following a spike in car crashes may appear to explain the reduction in accidents the following year. However, this fall back to the norm (regression to the mean), may have happened anyway regardless of the presence of the speed cameras, due to chance or the improvement in road layout and car safety.

Comparisons

Comparisons can help numbers which may be meaningless in isolation, make more sense. For example, reporting that German GDP has increased by 0.3% is more meaningful if audiences are told which time periods are being compared or how large German GDP is, or how the change compares with other European countries.

Failure to look at comparisons can highlight other contextual problems. For example, 584 unwanted pregnancies from one type of contraceptive is not so significant when compared with the much higher failure rates of other contraceptives, making it possibly the most effective form of contraception to use.

But comparisons of any kind are often fraught with difficulties. To avoid bogus comparisons make sure the same groups are being compared over the same time period and that the activity being compared is also the same. Consider the comparison carefully before accepting it as evidence.

Beware that changes in measuring systems or recording standards can invalidate comparisons over time. For example^[2], an apparent spike in violent crime in 2008/09 can be explained by changes introduced in 2002/03 to the way some offences were logged by police; it was not part of a rising trend when compared to violent crime in the late 90s. Any comparison of police recorded crime statistics over time without explaining this qualification is likely to mislead.

Take care with league tables such as hospitals or schools. A single statistical measure is unlikely to be a valid basis for comparing one hospital or school with another. A teaching hospital may have a worse score, but only because sicker patients are referred to it. A school may perform better because it reflects the socio-economic intake of the pupils.

Exercise additional caution with international comparisons where what is being counted may be measured in different ways.

Risk

For guidelines about Risk see Editorial Guidelines 11.4.14.

Changes in Risk

The Editorial Guidelines state:

Where the actual level of risk remains small, despite an apparently large relative increase (eg a tripling), the editorial justification of reporting such a story must be considered. Where there is editorial justification for reporting changes in risk, the level of risk should also normally be sought and included as well as the relative change. (11.4.14)

For example^[3], a report suggesting a 20% increase in the risk of getting colon cancer from eating an extra ounce of bacon a day sounds dramatic. But it omits vital information. It's not enough to tell audiences how the risk of getting colon cancer changes if we eat extra bacon everyday (the relative risk); the audience also needs to know what the risk of getting colon cancer was originally (the absolute or baseline risk). If the likelihood of developing colon cancer at all, is 5%; a 20% increase of that baseline risk is only 1 percentage point, meaning that the lifetime (absolute) risk of getting colon cancer is now 6%. Knowing that, may mean the audience chooses not to give up eating bacon every day.

To make levels of risk more meaningful, content makers should consider expressing that risk in human terms rather than as a percentage. For example^[4] about 5 men in 100 are likely to get colon cancer during their life. If they all ate an extra rasher of bacon every day, about 6 men would. So only 1 extra man per 100 men will get colon cancer if they eat extra bacon daily and the other 99 men won't.

(It should be noted that 1,000 out of 10,000 sounds like a higher risk than 1 out of 10 and should be avoided. If comparing risks, the same denominator should be used. For example, 2 out of 100 compared with 10 out of 100, rather than 1 in 50 compared with 1 in 10.)

Reporting of contested issues

The Editorial Guidelines advise that

The reporting of a range of contested issues, such as the economy or the extent of climate change, can be complex and rely on an understanding of accurate and meaningful statistics. Advice should be sought, where appropriate, from the BBC's specialist journalists.

In presenting such issues, it is important to ensure that legitimate choices about policy, such as on tax and debt, or on energy consumption, are not presented as imperatives or inevitabilities. Relevant context should be made clear, including where there are trade-offs in policy choices. The BBC should report uncertainty when it is relevant to an understanding of the full context. (11.4.15)

Presenters and programme-makers should be properly briefed about statistical information before they conduct interviews. This should include briefings about statistical information available from independent sources which may challenge a contributor's argument.

The UK Statistics Authority has the statutory role to safeguard and promote the production and publication of Official Statistics. It should be noted that where the Authority publishes correspondence from its Chair, it is providing an independent assessment of statistics used in the public domain. We should be alert to correspondence where the Chair is particularly critical.

(See Editorial Guidelines Section 2 Impartiality)

Statistical Significance - How sure are we?

For guidelines about Uncertainty, see Editorial Guidelines, 11.4.10.

Content makers should consider whether the results of data are "statistically significant", in other words whether there is confidence the findings are not due to chance. Reporting should highlight any caveats or doubts about significance, taking care not to overstate statistical significance.

Confidence Intervals / Margin of Error

Statisticians express significance using 'confidence intervals', better known as 'margins of error'. These explain how well the sample results from an experiment, a survey or an opinion poll represent what may be actually happening.

For example, an opinion poll may try to predict the results of a general election based on a sample of the voting population. Pollsters work out how close their predictions might be by calculating a 'margin of error'. For a typical 1000-person poll, the margin of error is plus or minus 3%. So, if the headline figure for a party's support is 32%, the poll is providing evidence that suggests support is between 29% and 35%. 19 times out of 20, a poll will be accurate to within 3%. ie with 1 in 20 polls, the true answer will lie outside the margin of error (though out of those 20 polls, it can't tell you which one).

Usually, the smaller the sample, the larger the margin of error and the less likely the result is robust. Also, results which fall well within the margin may not indicate anything at all. For example^[5], we cannot be confident unemployment has actually fallen over a three month period when the level of the fall, 79,000, is within the margin of error of plus or minus 81,000. A statistically insignificant figure is practically meaningless.

Content makers must report the margin of error in graphics if the result falls within the margin to enable audiences to judge the significance of the data.

For more discussion about surveys, opinion polls, questionnaires, votes and straw polls see Editorial Guidelines Section 10: Politics and Public Policy: Opinion Polls and Surveys and Guidance: Opinion Polls, Surveys, Votes, Focus Groups and Straw Polls

Practical Significance

Content makers should also consider whether the data, which may be statistically important, is relevant to the audience. For example, do the short-term changes in unemployment figures reveal how the labour market has changed, or is an examination of the long-term trends required?

Data

For guidelines about Data: data quality; how data is sourced; how to access analyse, present, report and store data, see Editorial Guidelines, 11.4.16 – 11.4.24.

Audiences may need to understand how the data originated to assess its importance. This may include understanding study-design; sample size; representativeness; how the data was collected; geographical relevance or time periods.

The Editorial Guidelines state:

The methodology used to analyse any dataset should normally be available to the audience, in a format appropriate to the content, including any relevant uncertainty or margins of error. (11.4.23)

The more detail a report includes of the data analysis, the more detail should be provided about the methodology, taking account of where the content will be published.

For example, a BBC investigation about illegal sewage spills during a dry period published online may devote 300 words and an explanatory graphic to the methodology that would not be possible in a short TV or radio report.

• [1] The Tiger That Isn't: Seeing Through a World of Numbers, (Profile Books) Michael Blastland & Andrew Dilnot, p18/19 ↩
• [2] Statistics for policy professionals, Good Practice Team, Government Statistical Service, Jan 2017, p13↩
• [3] Sense About Science and Straight Statistics, Making Sense of Statistics, Michael Blastland, p13 ↩
• [4] The Tiger That Isn't: Seeing Through a World of Numbers, (Profile Books) Michael Blastland & Andrew Dilnot, p108-110 ↩
• [5] Why do we report unemployment every month? Anthony Reuben, BBC News ↩