PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents?

Lutz Bornmann, Robin Haunschild, Werner Marx

arXiv:1512.07071v2physics.soc-phcs.DL

TL;DR

Measuring societal impact is difficult, and the study asks whether policy-document mentions can help assess research’s policy-related impact. Using climate-change publications and Altmetric mention data, it finds that only 1.2% had a policy mention, with mentions especially associated with particular journals and fields.

  • Problem

    Societal impacts are complex, non-linear, and unpredictable, making them difficult to measure in ways that capture significant, enduring change.

    Source: §2 · p. 8

  • Method

    The study examines Altmetric policy-document mentions for a climate-change publication set and compares the characteristics of papers mentioned in policy documents.

    Source: §5 · pp. 22–23

  • Results

    1.2% (n=2,341) of 191,276 climate-change publications had at least one policy mention, and Nature and Science papers and work in two named fields were especially relevant in policy contexts.

    Source: §5 · pp. 22, 24

  • Takeaways & Limitations

    Policy-document mentions are a potentially useful source for studying the broader impact of research on policy-related areas.

    Source: §1 · p. 3, §6 · p. 26

    The low mention rate may reflect either limited Altmetric coverage or that only a small part of the literature is policy relevant.

    Source: §5 · p. 23

Abstract

from arXiv · show

In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA) societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric - a start-up providing publication level metrics - started to make data for publications available which have been mentioned in policy documents. We regard this data source as an interesting possibility to specifically measure the (societal) impact of research. Using a comprehensive dataset with publications on climate change as an example, we study the usefulness of the new data source for impact measurement. Only 1.2% (n=2,341) out of 191,276 publications on climate change in the dataset have at least one policy mention. We further reveal that papers published in Nature and Science as well as from the areas "Earth and related environmental sciences" and "Social and economic geography" are especially relevant in the policy context. Given the low coverage of the climate change literature in policy documents, this study can be only a first attempt to study this new source of altmetric data. Further empirical studies are necessary in upcoming years, because mentions in policy documents are of special interest in the use of altmetric data for measuring target-oriented the broader impact of research.

1 Introduction

Societal impact has become part of national research evaluation, but measuring it remains a challenge. This study examines whether policy-document mentions of climate-change publications can serve as an indicator of research impact on policy-related areas.

1 Introduction

  • In the UK REF and Australia’s ERA, societal impact is an inherent part of national research evaluation.The introduction also describes a shift toward expecting research outcomes to be socially relevant and applicable to users.Source: §1 · p. 3
  • Interviewees described societal impact as research outcomes that make a change or difference, or create economic benefits.A cited definition also describes impact as recorded influence on non-academic organizations or actors outside the university sector.Source: §1 · pp. 3–4
  • Altmetric began making data available on academic publications mentioned in policy documents to uncover interaction between science and politics.The study asks whether these records can offer a way to measure research’s societal impact in policy-related areas.Source: §1 · p. 4
  • The study tests whether Altmetric’s policy-document mentions can help measure research impact on policy-related areas, using climate-change publications as its example.The authors focus on papers read by political actors and mentioned in policy-related documents as a potentially measurable form of science-policy interaction.Source: §1 · pp. 4–5
  • Climate change was selected because Altmetric continuously analyzes corresponding policy sites and the topic has long been policy-relevant.The authors investigate whether mentioned papers differ by journal, publication year, or document type.Source: §1 · p. 5

2 Literature overview

Societal impact is difficult to measure because impacts are complex and delayed, while feasible indicators may not capture lasting change. Prior work includes qualitative case studies and quantitative indicators; this study examines policy mentions as a possible measure of policy enactment.

2 Literature overview

  • This study examines policy-document mentions as a possible source for measuring policy enactment of research.Policy enactment includes research impacts on policies, laws, and regulations.Source: §2 · pp. 5–6
  • Case studies can convey complex impact information but require substantial effort, while some researchers favor more comparable indicators.The literature also cautions that qualitative approaches may emphasize prominent examples and overlook more modest contributions.Source: §2 · pp. 6–7
  • Altmetrics count online attention and activity around publications, while prior societal-impact work also considered patent and clinical-guideline citations.The cited literature presents patents as indicators of research contributions to industry innovation and guideline citations as a way to track knowledge toward clinical practice.Source: §2 · p. 7
  • Research impacts are complex, non-linear, and unpredictable, creating a tendency to count what is easy to measure rather than significant, enduring change.Proposed measurement models may be multidimensional but lack a suitable balance between comprehensiveness and feasibility.Source: §2 · p. 8
  • Research evidence may take around 15 years to reach practice, and impact assessment also faces barriers involving sociocultural adoption, cumulative research, and attribution.Other cited barriers include positive and negative effects and difficulty assessing transferability to unintended problems.Source: §2 · p. 8
  • Prior climate-policy studies examined research usability, interactive assessment models, and science-policy actors and advisory forms.The cited work includes studies of decision-making usability, regional integrated assessments, and Austrian climate-policy advice.Source: §2 · p. 9

3 Methods

The study uses Altmetric’s policy-document tracking and a climate-change publication set to examine policy mentions. The set yielded 2,341 papers with at least one mention, while the tracked sources were limited to major organizations in North America and Europe.

3.1 Analyses of policy documents at Altmetric

  • Altmetric’s Policy Miner uses text mining to discover publication mentions in policy documents.The monitored sources included organizations such as EFSA, GOV.UK, IPCC, ICRC, WHO, IMF, MSF, NICE, Oxfam, and UNESCO.Source: §3.1 · p. 10
  • The current policy-tracking sources are limited to major organizations from North America and Europe.Altmetric planned to curate and add sources, particularly from Asia and Africa.Source: §3.1 · p. 10

3.2 Climate change publication set

  • The climate-change publication set contains articles and reviews published from 1980 through 2014, assembled using an interactive query formulation method.The authors used Web of Science DOIs and sought additional DOIs through CrossRef, discarding records whose bibliographic values disagreed.Source: §3.2 · pp. 10–11
  • 1.2% (n=2,341) of 191,276 climate-change publications with DOIs had at least one policy mention.The DOI set covered 86.1% of the full publication set; policy-mention data were retrieved from Altmetric via its API on 16 November 2015.Source: §3.2 · p. 11
  • The Food and Agriculture Organization of the United Nations and the Intergovernmental Panel on Climate Change account for the most mentions of climate-change papers.Table 1 lists policy sources that mention climate-change papers.Source: §3.2 · p. 11, Table 1 · p. 12

4 Results

Policy-document mentions were concentrated among a small number of climate-change papers, and mention patterns varied by journal, document type, and subject area. Among the most-mentioned papers, adaptation and vulnerability topics with practical societal relevance were prominent.

4.1 Climate change papers mentioned in policy documents

  • 92% of climate-change papers mentioned in policy documents received only one or two mentions, although some received up to 18.Table 2 reports the number of papers with at least one policy-document mention.Source: §4.1 · p. 12, Table 2 · p. 13
  • Policy documents take longer to mention papers than scientific papers typically take to reach their citation peak, in part because policy documents emerge over longer periods.The authors contrast recurring policy-document cycles such as IPCC reports, which appear every six to seven years, with a typical scientific paper arising within one year.Source: Table 3 · p. 13, §4.1 · p. 14
  • Reviews accounted for 12.1% of mentioned papers versus 5.5% of the climate-change paper set, while research articles accounted for 87.9% versus 94.5%.The authors suggest that reviews may be more useful to policy-document authors because they summarize results spread across specific papers.Source: §4.1 · pp. 16–17, Table 5 · p. 16
  • Earth and related environmental sciences and Social and economic geography were overrepresented among policy-mentioned papers.The authors connect the geography category's overrepresentation with growth in research on climate adaptation, impacts, and vulnerability.Source: Table 6 · p. 17, §4.1 · pp. 18–19

4.2 Climate change papers mentioned in policy documents most frequently

  • Five of the nine climate-change papers with at least ten policy mentions were published in Nature or Science.The authors associate these journals' prominence with their prestige, visibility, and more general paper topics.Source: §4.2 · p. 19, Table 7 · p. 19
  • Eight of the nine most-mentioned papers concerned adaptation and vulnerability, including agriculture and fisheries, and the authors characterize these topics as having practical consequences for society.They note that agriculture accounted for about 9% of climate-change literature, while policy documents appeared to focus on this practically relevant subset.Source: §4.2 · p. 19, Table 7 · p. 19

5 Discussion

Policy-document mentions were sparse even for climate-change research, while cited papers skewed older and toward reviews, certain journals, and two subject areas. The authors treat the data as promising but provisional, given coverage and interpretation limits.

5 Discussion

  • Only 1.2% (n=2,341) of 191,276 climate-change publications had at least one policy mention.The authors note that the percentage may reflect Altmetric's recent, still-low policy-document coverage, among other possible explanations.Source: §5 · pp. 22–23
  • Low coverage may reflect recent Altmetric indexing, policy documents' failure to mention every important source, or barriers and low interaction between researchers and policy makers.Source: §5 · p. 23
  • The authors leave open whether mentions measure paper relevance, researcher engagement, or organizations' inclusion practices; unmentioned papers may simply be unknown to policy makers.They suggest analyzing mention contexts or surveying document authors to investigate the reasons for mentions.Source: §5 · p. 23
  • Policy documents cited older research than researchers did, and reviews were overrepresented among policy-cited papers.The authors suggest reviews may connect research lines and summarize primary studies.Source: §5 · p. 23
  • Papers in Nature and Science and in Earth and related environmental sciences and Social and economic geography were especially relevant in the policy context.Source: §5 · p. 24
  • Policy mentions are time-dependent, so evaluating entities with broad outputs across years and areas will require time- and area-specific normalization.Source: §5 · p. 24
  • Altmetric includes assessment reports as well as government documents, but the authors treat reports as preliminary evidence of political impact; duplicate multilingual documents can be counted more than once.Source: §5 · pp. 24–25

6 Conclusions

The authors argue for target-oriented societal impact measures and identify policy documents as a useful source for measuring research’s impact on politics. They also stress that this source needs greater transparency and that mention counts alone provide a limited account of science-policy interaction.

6 Conclusions

  • Impact measurement should be target-oriented because counts without identifiable recipients leave the kind of impact measured uncertain.The authors contrast target-oriented measures with broad approaches that combine altmetric sources.Source: §6 · pp. 25–26
  • Policy documents are among the few altmetric sources suited to target-oriented measurement, including research impact on politics.The authors call for this source to be studied further.Source: §6 · p. 26
  • Interpreting policy-mention results requires an up-to-date, detailed list of the organizations and document sources Altmetric analyzes.The homepage lists only examples, and sources such as ministries or regional governments are not identified; fuller coverage information would support assessment of database inclusiveness.Source: §6 · p. 26
  • Many questions remain before policy-document mentions can be considered for evaluation practice such as the REF or ERA.The authors characterize the source as not yet ready for prompt consideration as a metric.Source: §6 · pp. 26–27
  • Policy-mention counts measure how often papers are mentioned, not how research is discussed or the different forms of science-policy interaction.The analysis represents interaction as policy actors reading academic papers they consider important.Source: §6 · p. 27

Related papers

Loading 1512.07071v2…