Source-linked AI summary
Large-Scale Analysis of the Accuracy of the Journal Classification Systems of Web of Science and Scopus
Qi Wang, Ludo Waltman
TL;DR
Journal classification systems are important in bibliometrics, but their accuracy had not been systematically compared. This paper develops direct-citation criteria to identify questionable journal-category assignments in WoS and Scopus. Under these criteria, WoS performs significantly better than Scopus.
Problem
The accuracy of the WoS and Scopus journal classification systems had not been systematically compared despite their importance in bibliometric analyses.
Method
The study uses direct citation relations between journals and categories to define criteria identifying weak assigned connections and strong connections to unassigned categories.
Results
Web of Science performs significantly better than Scopus in classification accuracy according to the citation-based criteria.
Takeaways & Limitations
The criteria identify journals whose category assignments may be questionable in WoS and Scopus.
Takeaways & Limitations
The analysis relies only on direct citation relations, so other perspectives may yield different conclusions, especially for journals with few citations or fields where books are important.
Abstract
from arXiv · showhide
Journal classification systems play an important role in bibliometric analyses. The two most important bibliographic databases, Web of Science and Scopus, each provide a journal classification system. However, no study has systematically investigated the accuracy of these classification systems. To examine and compare the accuracy of journal classification systems, we define two criteria on the basis of direct citation relations between journals and categories. We use Criterion I to select journals that have weak connections with their assigned categories, and we use Criterion II to identify journals that are not assigned to categories with which they have strong connections. If a journal satisfies either of the two criteria, we conclude that its assignment to categories may be questionable. Accordingly, we identify all journals with questionable classifications in Web of Science and Scopus. Furthermore, we perform a more in-depth analysis for the field of Library and Information Science to assess whether our proposed criteria are appropriate and whether they yield meaningful results. It turns out that according to our citation-based criteria Web of Science performs significantly better than Scopus in terms of the accuracy of its journal classification system.
1. Introduction
Journal classification is central to bibliometric research, yet the accuracy of the Web of Science and Scopus systems had not been systematically compared. This study examines and compares their accuracy.
- Journal classification supports research-area delineation, cross-field impact comparison, and interdisciplinarity studies.
- Web of Science and Scopus both provide journal classification systems for multidisciplinary bibliographic research.
- The paper therefore focuses on examining and comparing the accuracy of the two systems.
2. Background
WoS and Scopus are multidisciplinary, journal-level classification systems used widely in bibliometrics, but their accuracy has lacked systematic large-scale evaluation and their construction methods are insufficiently documented.
- Multidisciplinary classification systems broadly cover research areas and typically classify journals rather than individual publications.
- Other multidisciplinary systems include Science-Metrix, NSF, UCSD, and ANZSRC, with differing levels of aggregation and algorithmic or expert involvement.
- WoS and Scopus both allow journals to have multiple classifications, while differing in journal coverage, collection policy, and classification.
- Documentation does not describe the methodology used to construct WoS and Scopus journal classification systems at a reasonable level of detail.
- Scopus’s ASJC has 304 bottom-level categories and 27 top-level categories, compared with about 250 categories in WoS.
- Previous work explored ways to validate or improve WoS- and Scopus-based classifications, but no systematic large-scale accuracy analysis existed.
3. Methodology
The paper evaluates classification accuracy through direct citation relations between journals and categories. Two criteria flag assignments with weak within-category connections or missing assignments to strongly connected categories.
- The study chooses a citation-based bibliometric approach rather than expert-based or text-based assessment.
- Direct citation relations are used because bibliographic coupling and co-citation provide more indirect relatedness signals.
- The approach assumes journals should have stronger citation connections within their assigned category than outside it.
- Relatedness r_i,c counts citations between journal i and category c, while t_i counts all incoming and outgoing citations of journal i.
- Criterion I flags assigned categories with r_i,c ≤ α, whereas Criterion II flags unassigned categories with r_i,c ≥ β.
- A journal satisfying either criterion is considered to have a questionable category assignment.
- The analysis retains journals with t_i < 100 because excluding them could distort category relatedness, although the criteria provide limited evidence for evaluating them.
4. Data
The analysis uses CWTS WoS and Scopus data for journals publishing during 2010–2014, excluding conference proceedings and books. Scopus includes more journals, categories, and category assignments per journal than WoS.
- The dataset comes from CWTS in-house WoS and Scopus databases and includes journals and book series, excluding conference proceedings and books.
- The analysis covers journals with publications during the five-year period 2010–2014.
- Category assignments changed during the analysis period, so the study used each journal’s most recent assignments.
- Scopus journals outnumber WoS journals by almost two to one and Scopus includes 80 more categories.
- The average number of categories per journal is 1.6 in WoS and 2.1 in Scopus.
- Almost 60% of WoS journals belong to one category, whereas more than 60% of Scopus journals belong to two or more categories.
5. Results
The results show that Scopus has substantially more questionable Criterion I assignments than WoS, while both databases perform reasonably well on Criterion II. Detailed examples and the LIS analysis further illustrate misclassifications and differences in database coverage.
- Criterion I: Scopus performs substantially worse than WoS on Criterion I, with more questionable journals and assignments across all tested α thresholds.For α = 0.05 and α = 0.1, the corresponding Scopus percentages are more than twice those for WoS.
- Criterion I: 16% of WoS journals have one or more questionable category assignments when α = 0.1.This result remains despite WoS performing better than Scopus on Criterion I.
- Criterion II: For Criterion II at β = 0.9, no WoS journals and only two Scopus journals satisfy the criterion; at β = 0.5, fewer than 5% do so in either database.Both systems therefore show relatively limited cases in which journals are strongly connected to unassigned categories.
- Criterion II: At β = 0.6, WoS and Scopus show similar categories containing journals that should have been assigned there, including economics and electrical and electronic engineering.These categories are reported in the respective database tables with corresponding naming differences.
- Combined Criteria I and II: The combined criteria identify one questionable WoS assignment versus 32 in Scopus.Australian Journal of Management is weakly connected to WoS MANAGEMENT but strongly connected to BUSINESS, FINANCE; Scopus examples include Cooperation and Conflict and Mobilization.
6. Discussion and Conclusions
Using direct citation relations, the study finds that Web of Science classifies journals more accurately than Scopus, while both systems can be overly permissive and lack transparency. The authors also identify methodological and structural limitations that constrain interpretation of the comparison.
- Research findings: WoS performs much better than Scopus under Criterion I: Scopus has more than twice the percentage of journals and assignments satisfying the criterion at α=0.05 and α=0.1.Criterion I identifies journals weakly connected to their assigned categories.
- Research findings: Less than 5% of journals in both systems satisfy Criterion II, indicating that strongly connected categories are typically assigned.WoS performs somewhat better than Scopus under this criterion.
- Research findings: Only one WoS journal satisfies the combined criteria, compared with 32 journals in Scopus, confirming WoS’s significantly better overall result.The combined criteria identify assignments that may be questionable for both weak and missed strong connections.
- Research findings: Both databases, especially Scopus, appear too lenient in assigning categories, suggesting stricter policies supported by citation analysis.The authors also report that a significant share of journals appear assigned to too many categories.
- Research findings: Scopus sometimes uses confusingly similar category labels, while both systems lack proper documentation of how classifications are constructed and updated.Examples include LINGUISTICS & LANGUAGE versus LANGUAGE & LINGUISTICS.
- Limitations and future research: The analysis relies only on direct citation relations, so questionable assignments are conclusions from a citation perspective that other approaches may revise.Text-based and expert-based approaches may be more effective for journals with few citation relations, including newly established journals.
- Limitations and future research: Direct citation relations may omit important scientific communication in book-oriented fields, although a Cultural Studies test found no clear evidence that bibliographic coupling is more suitable.The direct-citation approach found 61% of WoS Cultural Studies journals satisfying Criterion I at α=0.1, while bibliographic coupling produced a similar result.
- Limitations and future research: Differences in database coverage and category size may affect the comparison, while journal-level systems fit multidisciplinary journals increasingly poorly.Relative statistics correct for Scopus’s larger coverage, but possible differences in field distributions remain uncorrected.
Appendix
The appendix provides category tables and examples of journals whose citation patterns differ sharply across categories, including Library and Information Science cases.
- Appendix: The appendix lists WoS multidisciplinary categories and Scopus journals satisfying both Criterion I and Criterion II under α=0.1 and β=0.6.The Scopus table excludes journals with t! < 100.
- Appendix: Several listed journals show low values for one category and higher values for another, including Computer Graphics Forum, Cooperation and Conflict, and Cultural Studies of Science Education.The reported pairs include 0.01/0.64, 0.01/0.67, and 0.04/0.77, respectively.
- Appendix: Additional examples include Current Bladder Dysfunction Reports, Economics of Governance, Filozofia, and Geotechnique Letters with category-pair values ranging from 0.60 to 0.72.The entries report contrasting category values such as 0.00/0.70, 0.03/0.72, 0.06/0.63, and 0.03/0.60.
- Appendix: The appendix also lists examples involving applied mathematics, theoretical computer science, marketing, sociology, education, and economics categories.Examples include Journal of Cryptology, Journal of Personal Selling and Sales Management, Mobilization, Multicultural Perspectives, and Perspektiven der Wirtschaftspolitik.
- Appendix: Portal: Libraries and the Academy shows 0.00 for Development and 0.83 for Library and Information Sciences.
- Appendix: Public Services Quarterly shows 0.00 for Accounting and 0.80 for Library and Information.