PaperlayerView exact revision on arXiv

Source-linked AI summary

Trends in Russian research output indexed in Scopus and Web of Science

Henk F. Moed, Valentina Markusova, Mark Akoev

arXiv:1805.02434v1cs.DL

TL;DR

The paper examines how Scopus and Web of Science coverage affects measures of Russian research output. It compares publication trends across the databases and finds that coverage changes account for substantial growth in Scopus counts, warranting caution in interpreting these indicators.

  • Problem

    Because Scopus and WoS indicators inform research assessment, the paper asks how database coverage differences affect measured Russian publication output.

    Source: §1 · p. 2

  • Method

    The study compares Russian publication counts in Scopus and WoS over time, disaggregating trends by publication characteristics and sources.

    Source: §4 · p. 7, §4.7 · p. 15

  • Results

    About three quarters of Russian-language journals covered by Scopus in 2016 were not indexed there in 2012, explaining a substantial part of the 2012–2016 increase.

    Source: §5.1 · p. 18

  • Takeaways & Limitations

    The authors advise caution in using WoS, and especially Scopus, to measure changes in research performance from an international perspective or assess Project 5-100 objectives.

    Source: §5.1 · p. 20

    The study does not assess individual institutions because the authors lacked accurate, verified information on Russian institutions.

    Source: §1 · p. 2

Abstract

from arXiv · show

Trends are analysed in the annual number of documents published by Russian institutions and indexed in Scopus and Web of Science, giving special attention to the time period starting in the year 2013 in which the Project 5-100 was launched by the Russian Government. Numbers are broken down by document type, publication language, type of source, research discipline, country and source. It is concluded that Russian publication counts strongly depend upon the database used, and upon changes in database coverage, and that one should be cautious when using indicators derived from WoS, and especially from Scopus, as tools in the measurement of research performance and international orientation of the Russian science system.

1. Introduction

The paper examines how database coverage affects the measurement of Russian publication output, a concern because Scopus and Web of Science indicators inform research assessment and university rankings. It compares their longitudinal coverage and publication counts amid the Project 5-100 policy context.

1. Introduction

  • Project 5-100 sought to improve Russian higher education’s prestige and place at least five participating universities among the top 100 in three global rankings.The government launched initiatives to increase university competition and international status and visibility.Source: §1 · p. 1
  • The study compares Russian publication output indexed in Scopus and WoS to clarify how database coverage differences affect bibliometric indicators.Such indicators play a key role in university rankings and research assessment.Source: §1 · p. 2
  • The central questions concern Scopus’s sharp recent increase, its comparison with WoS, and the roles of research performance and changing database coverage.The study also asks how trends vary by document type, language, discipline, and country.Source: §1 · p. 2
  • The study focuses on publication counts and database coverage, not citation impact or institution-level assessment.The authors lacked high-quality information on Russian institutions for verified institution-level assessment.Source: §1 · p. 2

2. Background

The background describes Russian science reforms and the policy setting for Project 5-100, including a 2012 output target and funding for selected universities.

2. Background

  • Reforms shifted government attention and resources toward higher education, while the Russian Academy of Sciences faced a 20% research-personnel reduction and universities received new research designations.The government established a Federal University in each of ten federal districts and awarded National Research University status to 28 universities in 2008.Source: §2 · p. 3
  • A 2012 decree set a target for Russian research output to reach 2.44% of global output and for five universities to enter a top-100 ranking by 2015.Project 5-100 began in January 2014, with funding for 2013 transferred to selected universities.Source: §2 · p. 3
  • In 2013, legislation reformed three government academies, and a decree required research-organization evaluations to include paper counts, citation scores, and impact factors from Web of Knowledge or Scopus.The academy reform was controversial and met strong resistance from the research community.Source: §2 · p. 3
  • Project 5-100 allocated 44 billion rubles for 2013–2016 and selected universities for funding tiers; later competitions expanded participation and added a regional-development initiative.The 2017 regional initiative selected 51 universities from 121 applicants, including ten Project 5-100 participants.Source: §2 · p. 4

3. Data collection

The study assembled Scopus and WoS publication counts using different data sources and collection procedures, while excluding other Russian citation indexes from its analysis.

3.1 Scopus

  • Scopus counts were collected manually from its online platform in October 2017 and entered one-by-one, following changes that impeded bulk extraction.Online Scopus field analyses returned at most 160 entries, limiting yearly source-count generation.Source: §3.1 · pp. 4–5
  • Scopus counts included articles, reviews, and conference papers; duplicate article/proceedings records were considered too few to substantially affect results.The study assumed the low fraction of duplicate records did not materially alter findings.Source: §3.1 · p. 5

3.2 WoS

  • WoS counts came from seven Core Collection indexes, mostly extracted with permission from InCites; its document types included articles, reviews, and proceedings papers.About 30,000 documents had both article and proceedings-paper assignments, which the authors assumed did not affect conclusions.Source: §3.2 · p. 5

3.3 Other relevant datasets

  • The analysis used WoS and Scopus, not the Russian Index of Science Citation or the Russian Science Citation Index at Clarivate.The latter is a national Russian-literature index on the WoS platform, but its sources are outside the WoS Core Collection.Source: §3.3 · p. 6

4. Results

For 2012–2016, Russian document counts rose exponentially in both databases, but growth differed by document type; removing discontinued Scopus journals leaves the exponential pattern intact.

4.1 Counts per document type

  • Both Scopus and WoS showed approximately exponential growth in Russian documents during 2012–2016.Figure 1 presents annual counts by document type, with Scopus curves in black and WoS curves in grey.Source: §4.1 · p. 7, Figure 1 · p. 7
  • Scopus journal articles and reviews grew at a 13.5 CAGR versus 6.4 in WoS during 2012–2016, while proceedings papers grew at 21.9 versus 40.2.Total-document CAGR was 15.2 in Scopus versus 12.9 in WoS.Source: Table 1 · p. 7, §4.1 · p. 8

4.2 Recent changes in Scopus coverage

  • Removing documents in the 424 Scopus journals discontinued in 2018 leaves Russian publication growth exponential, despite a substantial effect on counts.The discontinued-journal set contained 29,271 Russian articles, reviews, and conference papers; its removal did not affect the overall conclusions.Source: §4.2 · p. 8, Table 2 · p. 8
  • The discontinued-journal set was 99.5% English-language and about 0.4% Russian-language, so its removal did not affect the publication-language analysis.Source: §4.2 · p. 8

4.3 Counts per publication language

Russian publication-language trends differ across Scopus and WoS, with Russian-language journals contributing substantially to Scopus growth. The databases also showed little overlap among their most-published Russian-language journals in 2016.

4.3 Counts per publication language

  • Scopus CAGR was 7.6 versus 4.7 for WoS for all articles and reviews during 2006–2016, and Russian-language documents contributed considerably to Scopus growth.For non-Russian-language documents, the corresponding CAGRs were 6.4 for Scopus and 4.6 for WoS.Source: §4.3 · p. 9
  • Only one journal appeared in both databases’ 2016 top 25 Russian-language journals: Terapevticheskii Arkhiv.This suggests substantial differences in Scopus and WoS coverage of Russian-language journals that year.Source: §4.3 · p. 9
  • The analysis examined Scopus Russian-language journals newly covered in 2016 but not 2012, and whether VINITI indexed them in its Abstract Journal.VINITI coverage was used as an indication of subject-field significance; the Abstract Journal does not cover social sciences and humanities and only partially covers medical literature.Source: §4.3 · pp. 9–10

4.4 Analyses of conference proceedings documents

Conference proceedings form a changing share of Russian output, and the databases differ in some trends and citation patterns. Their 2016 coverage of leading proceedings titles was nevertheless similar.

4.4 Analyses of conference proceedings documents

  • In 2016, conference papers were 23% of Russian documents in Scopus and 32% in WoS.In WoS, the proceedings-paper share fell from 20% in 2007 to 12% in 2011, then rose sharply during 2014–2016.Source: §4.4 · p. 10, Figure 3 · p. 10
  • In 2016, Scopus and WoS showed strong similarities in proceedings coverage, including absolute counts and overlap among titles in their top 25 lists.The comparison used the 25 proceedings volumes with the most Russian-institution papers in each database.Source: §4.4 · pp. 10–11, Table 4 · p. 11
  • Citation outcomes for the top proceedings volumes were statistically similar between Scopus and WoS; one-off volumes had very low citation impact outside Russia.High Russian shares in one-off conference volumes suggest they were held in Russia and attended mainly by Russian participants.Source: §4.4 · pp. 11–12, Table 5 · p. 11
  • For conference series, the Russian share tended to be much lower than for one-off volumes, while their papers’ citation impact outside Russia remained very low.That impact was somewhat higher than for papers in one-off volumes, in both databases.Source: §4.4 · p. 12

4.5 Analysis by discipline

Scopus counts grew faster than corresponding WoS counts in most disciplines, but the databases distributed Russian output differently across fields. Classification differences and changes qualify comparisons of discipline growth.

4.5 Analysis by discipline

  • Scopus CAGRs exceeded those of the most similar WoS areas in most disciplines during 2012–2016, with the largest differences in seven named disciplines.These were Economics, Social Science, Arts & Humanities, Pharmacology & Toxicology, Neuroscience, Environmental Science, and Computer Science.Source: Table 6 · p. 12, §4.5 · p. 13
  • Discipline growth rates may also reflect classification changes, such as journals moving between categories.The paper notes this might apply to WoS Neuroscience & Behaviour, which showed negative growth rates.Source: §4.5 · p. 13
  • Scopus showed relatively stronger Russian representation in Social Science & Humanities, while WoS showed stronger representation in Physics and Chemistry.The distributions differ partly because the databases define disciplines differently; multiple assignments also mean discipline totals include double counts.Source: §4.5 · p. 13

4.6 Analysis of BRIC countries and most productive countries

Russian publication growth during 2012–2016 was unusually high in Scopus relative to WoS and comparison countries. The authors caution that database coverage and processing affect interpretation.

4.6 Analysis of BRIC countries and most productive countries

  • During 2006–2016, Russia’s articles-and-reviews CAGR was 7.6% in Scopus versus 4.2% in WoS.The comparison also considered BRIC countries and the ten most productive countries; the country table reports growth rates for 2006–2016 and 2012–2016.Source: §4.6 · p. 14, Table 7 · p. 14
  • The 2016 counts for the ten most productive countries were about 1.5% higher when indexed through April 2018 than through October 2017.The difference reflects processing delay.Source: §4.6 · p. 14
  • During 2012–2016, Russia’s Scopus growth was disproportionately large relative to WoS, the other BRIC countries in Scopus, and the ten most productive countries.The authors suggest Scopus prioritized Russian output and WoS prioritized Chinese publications; the Russian Citation Index on the WoS platform was excluded from their WoS counts.Source: §4.6 · p. 14

4.7 Analysis by journal in Scopus

The Scopus journal analysis finds different patterns for English- and Russian-language journals. Whether Russian output exceeds the 2.44% norm depends on database and document and language filters.

4.7 Analysis by journal in Scopus

  • In 50 high-volume English-language journals, 13,236 Russian documents represented 23% of 2016 output, and 73% appeared in journals used by Russian researchers in 2012.The sample is biased toward voluminous journals; the authors suggest increased publishing in familiar journals explains a substantial part of the English-language growth, but cannot estimate it precisely. Three discontinued journals accounted for 773 documents, less than 6% of 2016 Russian output.Source: §4.7 · p. 15
  • Of 160 Russian-language journals used in Scopus in 2016, 117 (73%) had no Russian-institution documents in 2012 and supplied about 60% of 2016 Russian-language-journal documents.The authors could not determine whether these journals were newly founded or newly indexed; they were new in Scopus coverage relative to 2012.Source: §4.7 · p. 15

4.8 The 2.44 per cent norm for the share of Russian publication output

  • 2.44%: Russian output exceeds the norm in both databases when proceedings are counted; without proceedings, it exceeds the norm in Scopus but not WoS, and falls to 2.18% when Russian-language papers are excluded.After excluding 424 discontinued Scopus journals, the share remains above the norm.Source: §4.8 · p. 16, Table 8 · p. 16

4.9 Trends in domestic and international collaboration

The share of domestic collaboration grew in both databases, while international-collaboration trends differed between Scopus and WoS. The authors relate Scopus’s decline partly to expanded coverage of Russian journals.

4.9 Trends in domestic and international collaboration

  • The share of domestic collaboration grew significantly in both Scopus and WoS.Figure 4 presents the domestic and international collaboration trends based on Russian publications covered by the two databases.Source: §4.9 · p. 16, Figure 4 · p. 17
  • Scopus showed a declining international-collaboration share, which the authors partly attribute to increased coverage of Russian journals.They suggest these journals may not be among the more prestigious journals preferred by foreign coauthors.Source: §4.9 · p. 16, Figure 4 · p. 17
  • In WoS, the international-collaboration share fell during 2006–2011, rose during 2011–2013, then fell again; the authors say the last decline could relate to Russian Academy of Sciences reform.Source: §4.9 · p. 16, Figure 4 · p. 17

5. Discussion and conclusions

Russian publication counts and growth rates depend on the database and its coverage changes, complicating their use to assess research performance and international orientation. The paper concludes that WoS, and especially Scopus, should be used cautiously for this purpose and calls for a framework for independent database assessment.

5.1 Conclusions

  • Database choice and coverage changes substantially affect Russian publication counts and growth rates, so WoS and especially Scopus require caution as measures of research performance and international orientation.The share increase during 2012–2016 depended on expanded proceedings coverage in both databases and, for Scopus, additional Russian-language journals.Source: §5.1 · pp. 17, 20
  • Scopus and WoS showed statistically similar 2012–2016 document growth rates: 15.2% for Scopus against 12.9% for WoS.The analysis also found a similar exponential pattern in WoS.Source: §5.1 · pp. 17–18
  • During 2006–2016, the CAGR of Russian articles and reviews indexed in Scopus was about 80% higher than the corresponding WoS rate, with Scopus leading in most disciplines.The difference was especially evident in Economics, Social Science, Arts & Humanities, Pharmacology & Toxicology, Neuroscience, Environmental Science, and Computer Science.Source: §5.1 · p. 18
  • Russian-language journal coverage helps explain Scopus growth: Russian-language papers rose from 4.8% to 14.8% in Scopus during 2006–2016, while WoS’s share fell from 6.5% to 3.0%.About three quarters of Russian-language journals covered in Scopus in 2016 were not indexed there in 2012; another part of the journal-output increase came from researchers publishing more in journals already familiar to them in 2012.Source: §5.1 · p. 18
  • Conference-proceedings article CAGR during 2006–2016 was around 10% in both databases, while the 2016 conference-paper shares were 23% for Scopus and 32% for WoS.WoS’s proceedings share increased during 2012–2016, catching up with Scopus; the increase in Russian papers’ share during 2012–2016 depended in both databases on expanded proceedings coverage.Source: §5.1 · pp. 18–19
  • The study found low citation impact for Russian proceedings papers in both databases, while many citations to proceedings titles were country self-citations.High self-citation percentages combined with low citation levels may produce outliers in relative citation rates; the authors report 1–2 institutions per publication year above 3.0 and three cases above 5.Source: §5.1 · pp. 19–20
  • The extent to which database differences affect performance assessments depends on the methodology and indicators used; ranking effects are difficult to predict without detailed analysis.The paper also questions whether growth in Russian-language journals or conference papers reflects genuine internationalization and notes that displacement of publications from national journals outside Scopus and WoS cannot be assessed with its data.Source: §5.1 · pp. 18–19

5.2 Concluding remarks

  • The authors call for a framework for critical, independent assessment of databases and their use in research evaluation, supported by comparable classifications, dedicated software, and database expertise.They note that current studies, including theirs, tend to be ad hoc and that the large-scale datasets needed for systematic database comparison are mostly unavailable.Source: §5.2 · p. 20

Appendix

The appendix presents tables of high-volume proceedings titles and English-language journals associated with Russian publication output.

Appendix

  • Table A1 identifies the 25 proceedings titles in Scopus and WoS with the largest number of papers from Russian institutions in 2016; displayed entries are type 2, or one-off volumes.Source: Table A1 · p. 23, Appendix · p. 25
  • The proceedings entries include named conference volumes such as Eiconrusnw 2016 and Geomodel 2016.The table legend identifies type 2 as a one-off volume and marks titles included in both databases with Y.Source: Appendix · pp. 24–25
  • Table A2 is captioned as the top 50 English-language journals with the largest number of papers from Russia in 2016.Its listed titles include Physics of the Solid State and Bulletin of Experimental Biology and Medicine.Source: Table A2 · p. 25, Appendix · p. 26
Loading 1805.02434v1…