The worst statistic

“The economic value of this is …”  This is the worst statistic ever, produced at the instigation of interested parties who want to show how valuable their work is, as part of the case that there should be more government funding.

I hear it most days. Here is a typical example from today’s BBC news.

The world’s national parks and nature reserves receive eight billion tourist visits a year, generating around $600bn of spending, according to research.

After emitting a little steam from my ears, I thought ‘let’s blog on it’ and then ‘I’d better check what I’m talking about first’.

The research was produced by PLOS biology. The full article is here. The study was supported by the Natural Capital Project and there is a disclaimer which says

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

The paper gives a detailed account of how they used a range of resources to build a detailed (and, to my inexpert perusal, convincing) model of each of a stratified sample of 500 parks, to identify how many visitors each receives per year, and extrapolate from those models to the full population.

On the $600bn figure, they say

Economists working on tourism distinguish two main, non-overlapping components of value [12]: direct expenditure by visitors (an element of economic impact, calculated from spending on fees, travel, accommodation, etc.); and consumer surplus (a measure of economic value which arises because many visitors would be prepared to pay more for their visit than they actually have to, …

I’ll stick to the simpler part – ‘direct expenditure by visitors … on fees, travel, accommodation’.  So that will include the train ticket to go to the park and the drinks, meals and hotel nights bought in the national park  (but presumably not drinks, meals and hotel night from estabishments outside the park). Won’t those people have eaten meals and drunk coffees and beers, quite likely for similar value, had they not gone to the park at all?  Also it is looking at the income side of the consumer outlets in the parks – but not at their expenditure. If expenditure is greater than costs, and the outlet is on the way to bankruptcy, should it still be counted as providing economic value? According to this approach, yes. The authors are not economists, and when it comes to translating visits to dollars, the quality of the research drops.

Of course the researchers, and the BBC, both know that $600bn is meaningless unless compared with some other figure. So what are possible points of comparison? In the BBC article we have:

The tourism income vastly outweighs the $10bn a year spent safeguarding them, says a Cambridge University team.

(which is a shorter version of the final paragraph of the journal paper.)

Comparing this figure with the $600bn is like comparing the number of strawberries I grow in my garden, with the number that Morocco exports. Both about strawberries, but that’s about as far as it goes.

The other comparison we are offered is from Prof. Andrew Balmford, lead author of the report, who is quoted by the BBC as saying

the $10bn a year currently spent on these areas is small when compared with the quarterly profits of the likes of the computing giant Apple. Stopping the unfolding extinction crisis is not unaffordable. Three months of Apple profits could go a long way to securing the future of nature.

which strikes me as a very, very, very bad argument. I trust my daughter will make no arguments like that in her philosophy A-level exam.

I hear ‘the economic value is …’ statistics on the news most days, be it for the value of the arts, or the savings to the health service if speed limits are reduced, or the benefits to the economy that immigration brings. The tiny bit of research I’ve done this morning only goes to confirm my suspicion that we might as well disembowel cockerels and read their entrails.

Of course, it hurts. The researchers and news teams are doing their best to give us some basis for understanding the situation, all as part of the public discussion that, we hope, will help policy makers make better policies. And this post is purely critical: I have no clever method of converting, eg, visits to parks into ‘economic value’.   And I’m sorry to have picked on what is probably useful and (except for the money bit) valid research. But all the same: those ‘economic value of ’ statistics: don’t swallow them!

Published by

Adam Kilgarriff

I'm a scientist who has set up and runs a small company. I'm married (to Gill Lamden) with three children, Boris (22), Maddie (18) and Raffie (9) (as at today, 28 January 2015, in case I forget to update!) We live in Brighton, UK. Last November (2014) I found I had bowel cancer (stage 5; not curable; only 'manageable'). We've been adjusting to that since (and it is what provoked me to start the blog) My scientific area is linguistics, with my specialisms being corpus linguistics, computational linguistics, and lexicography - or, best of all, the intersection of all three. Since 2004, my company, Lexical Computing Ltd., has been providing a web service, the Sketch Engine, to linguists and lexicographers wanting to find out about words, using corpus-driven methods. Customers include Oxford University Press, Cambridge University Press, Collins, Macmillan, Le Robert, and around a hundred universities worldwide.

8 thoughts on “The worst statistic”

  1. I was (possibly still am) a Chartered Patent Agent. Media coverage of intellectual property is invariably inaccurate and incomplete (III). I don’t doubt for a minute that media coverage of other subjects is III. We have to remember that The Friday Ad, Queen’s Park News (Brighton residents will know what these are) and The Manchester Guardian are all advertising media. The editorial content if any is there to get you to buy it or pick it up off the doormat and look at the advertisements. Next time you open The Manchester Guardian pretend it’s The Friday Ad.

  2. Hi Adam,
    On Friday I was in Stirling for Vaclav Brezina’s workshop on statistics for Corpus Linguistics (BAAL CL SIG) – pitched at beginner level which suited me just fine! He was very good at showing how deceptive stats can be. When we were looking at various mind-boggling equations I thought of you and SketchEngine’s argument for using logDice for collocation.. in the end lay people like me tend to trust the experts rather than look ‘under the bonnet’. I’m quite happy to accept logDice without fully understanding it. However it’s good to be reminded to look out for the ‘angle’ since, as your post so eloquently proves – stats can be used to prove just about any point!

  3. As Mark Twain so eloquently put it, “There are three kinds of lies: lies, damned lies and statistics.” Numbers make most people cross-eyed, and they are then unable to think clearly, or at all. But thanks for so clearly putting this particular bit of popular media chatter into proper perspective. Oh, and thanks for the link to Storm, too – well worth the 10 minutes indeed.

  4. I agree that statistics get abused very regularly, but I think the blog is a bit harsh on the particular example quoted. The researchers claim $600bn of spending in the national parks. That may or may not be an accurate figure, but at least it is capable of being measured (ie money really is being spent in the parks, and one could in theory count it all up). And, granting that at national level one should do some netting off for deadweight costs of food etc that would have been consumed anyway, for a local economy there is a real difference, because most of that hypothetical deadweight consumption wouldn’t have taken place in or around the parks. Also, the existence of the parks will have persuaded some people to visit the country when they wouldn’t otherwise have done so. Real money is therefore being redistributed, either within a country or between countries, as a result of the investment made in creating and maintaining the parks, and while it’s difficult to calculate the net total amount, it’s certainly not inherently impossible. And the contrasting fortunes of, say, Blackpool or Margate compared to the Lake District or the Yorkshire Dales over the last 50 years suggests this can make a real difference, so it’s worth trying to understand it. The difficult bit, and where logic and statistics do tend to be jettisoned rapidly, is when people try to extrapolate forward, and claim – for example – that if $10bn dollars “investment” in parks is associated with $600bn spending by visitors, then a $20bn investment will double the spending. Much money has gone down many drains as a result of such wishful thinking.

    1. Martin, yes I see what you mean. But you remind me of a point I missed out in the post: the slippage from the straightforward talk about ‘spend’ to the altogether trickier one of ‘value’, with ‘the value to the economy’ being a common expression. The sentence in the BBC piece about this is

      It is based on assigning economic value to the natural environment in order to better preserve it in the future.

      This is where the leap of faith happens, from easy-to-measure to hard-to-measure, from pennies in pockets to the betterment of mankind, from the mundane to the sacred.
      Also you defend the article in terms of transfer of expenditure away from non-park areas into park areas but that isn’t mentioned once in the piece.

      1. Hmmm…Adam, I think this turns out to be a polysemy lecture: clearly your view of “economic value” is more philosophical than BBC’s one and, to be honest, mine as well. There is no scientific variable of “economic value” within economics, and if even you picked (any) one of the many quantitative measures like GDP, it would not correspond to “betterment of mankind” (this is to me one of the biggest outstanding issues of economics, actually! Anybody interested shall read the very impressive Economics of Good and Evil).

        And now the polysemy lesson:

        economic value = money being spent

        (there is nothing beyond that, sadly, both in practice and in theory)

      2. I’d certainly agree that “economic value” isn’t the same as absolute “value”, and that plenty of things are valuable without being economically valuable. Viewing a sunset in a national park would fit that category, (unless the mosquitoes are biting too hard for it to be appreciated). But the reverse holds true as well; having “absolute value” (by being beautiful, sublime, awe-inspiring, or any other good Romantic epithet) doesn’t preclude having an economic value as well – and the economic value, unlike the absolute value, can be measured. So it provides a valid framework for study and discussion, within its own terms and limitations, and provided one recognises these, then it’s worth trying to measure accurately. In other words, I agree with Milos Jakubicek below.

  5. Great stuff Adam, thanks for sharing your thoughts.

    “The environment is part of the economy and needs to be properly integrated into it so that growth opportunities will not be missed” – this was Dieter Helm,chair of the government’s natural capital committee, getting his ideas precisely inside-out so far as I can tell (as reported by George Monbiot here:

    I imagine one of the reasons that even nature-lovers like Tony Juniper, whose book Monbiot was savaging, fall for this inversion is because money can be handy. But Monbiot quotes
    George Lakoff, who I think hits the nail on the head. Lakoff has “explained that attempts to monetise nature are a classic example of people trying to do the right thing without understanding frames: the mental structures that shape the way we perceive the world”.

    “As Lakoff points out, you cannot win an argument unless you expound your own values and re-frame the issue around them. If you adopt the language and values of your opponents “you lose because you are reinforcing their frame”. Costing nature tells us that it possesses no inherent value; that it is worthy of protection only when it performs services for us.” Services one might add that are already recognised and quantifiable.

    Basically it is a short step from valuing nature in cash, to understanding it to be tradable. Which IMO it totally isn’t.

    Conservationist and prolific blogger Miles King is also interesting on this I think. also responding to Tony Juniper, he writes (

    “I would suggest that the risk with Natural Capital approaches is that delivery of ecosystem services will be achieved by the lowest necessary value ecosystem. While an ancient woodland might be a very good carbon store, a newly planted conifer plantation stores more carbon more quickly, and so will outcompete the ancient woodland for carbon credits, or whatever monetary system is being used. The non-use value of the ancient woodland (its history, its meaning, its beauty) will be undervalued.”

Comments are closed.