Source-linked AI summary
How journal rankings can suppress interdisciplinary research. A comparison between Innovation Studies and Business & Management
Ismael Rafols, Loet Leydesdorff, Alice O'Hare, Paul Nightingale, Andy Stirling
TL;DR
The paper examines whether journal rankings used in research evaluation disadvantage interdisciplinary research. Using publication and citation data, it compares Innovation Studies units with leading Business & Management schools and finds that ranking-based assessment favours disciplinary research, with implications for how esteem and resources are allocated.
Problem
Research evaluation may be biased against interdisciplinary research, despite its perceived contributions to scientific breakthroughs, societal problem-solving, and innovation.
Method
The study uses publication and citation data with plural, conditional metrics and converging partial indicators to compare interdisciplinarity and research performance across Innovation Studies units and leading Business & Management schools.
Results
The analysis finds that journal rankings favour disciplinary research and can produce lower assessments of interdisciplinary Innovation Studies research despite citation-based indicators not supporting that lower assessment.
Takeaways & Limitations
Using journal rankings to allocate esteem and resources can suppress interdisciplinarity and underscores the need for metrics that capture multidimensional research qualities.
Takeaways & Limitations
The generalisability of the findings beyond Business & Management and Innovation Studies remains an open question.
Abstract
from arXiv · showhide
This study provides quantitative evidence on how the use of journal rankings can disadvantage interdisciplinary research in research evaluations. Using publication and citation data, it compares the degree of interdisciplinarity and the research performance of a number of Innovation Studies units with that of leading Business & Management schools in the UK. On the basis of various mappings and metrics, this study shows that: (i) Innovation Studies units are consistently more interdisciplinary in their research than Business & Management schools; (ii) the top journals in the Association of Business Schools' rankings span a less diverse set of disciplines than lower-ranked journals; (iii) this results in a more favourable assessment of the performance of Business & Management schools, which are more disciplinary-focused. This citation-based analysis challenges the journal ranking-based assessment. In short, the investigation illustrates how ostensibly 'excellence-based' journal rankings exhibit a systematic bias in favour of mono-disciplinary research. The paper concludes with a discussion of implications of these phenomena, in particular how the bias is likely to affect negatively the evaluation and associated financial resourcing of interdisciplinary research organisations, and may result in researchers becoming more compliant with disciplinary authority over time.
Highlights
The study compares Innovation Studies units with Business & Management schools and finds that journal rankings favour disciplinary research over interdisciplinarity.
- Innovation Studies units are more interdisciplinary than Business & Management schools according to various metrics.
- Business & Management schools show higher performance according to indicators based on journal rankings.
- Citation-based indicators do not support the higher performance assessment produced for Business & Management schools.
- The analysis suggests that journal rankings are biased against interdisciplinarity.
1. Introduction
The introduction frames interdisciplinary research as valuable but potentially disadvantaged by evaluation systems, and sets up a quantitative comparison of Innovation Studies and Business & Management in the UK.
- Interdisciplinary research is associated with scientific breakthroughs, societal problem-solving, innovation, creativity, and new research avenues.
- Quantitative evidence on evaluation bias against interdisciplinary research remains limited, despite qualitative studies reporting disadvantages.
- Innovation Studies is a diverse, ambiguously bounded, problem-oriented area that often does not fit discipline-based assessment panels.
- The UK provides a suitable setting because it has an established Innovation Studies community, a comparatively homogeneous higher-education system, and a long assessment history.
- Journal rankings are promoted as objective assessment tools but have been criticised as inappropriate for evaluating research quality.
- The study compares three UK Innovation Studies centres with three leading Business & Management schools using journal rankings and article-based citation indicators.
- The paper claims to provide a firm quantitative demonstration that bias against interdisciplinarity can arise in ostensibly objective journal rankings, not only peer review.
- The article also develops bibliometric concepts, operationalisations, visualisations, citing-side normalisation, and multiple indicators for multidimensional assessment.
2. The evaluation of interdisciplinarity research
The literature review describes persistent evaluation barriers for interdisciplinary research, while noting that previous quantitative findings have been ambiguous and that disciplinary standards create additional hurdles.
- The review notes that claims about the benefits of interdisciplinarity often rely on limited, success-selected case studies and are difficult to prove systematically.
- Interdisciplinary research faces coordination difficulties and systemic barriers from the disciplinary organisation of science.
- Reported barriers include poorer career prospects, lower collegial esteem, reviewer discrimination, and difficulty publishing in prestigious journals.
- Interdisciplinary research is widely viewed as facing a key barrier in research evaluation.
- In a survey of 5,505 respondents, 51% of researchers, 68% of department heads, and 48% of RAE panel members viewed the 1996 UK RAE as inhibiting interdisciplinarity.
- Interdisciplinary evaluation is difficult because research between or beyond disciplines may face contrasting quality criteria from multiple fields.
- Earlier quantitative studies produced ambiguous or contradictory findings, leaving the evidence on evaluation bias inconclusive.
3. Methods and underlying conceptualisations
The study treats interdisciplinarity as multidimensional and assesses it through diverse indicators and mappings that capture knowledge integration, diversity, coherence, and intermediation.
- Performance and interdisciplinarity are multidimensional concepts that cannot be fully captured by a single indicator.
- The methodological framework uses plural, conditional metrics and converging partial indicators to improve assessment under uncertainty and ambiguity.
- Knowledge integration concerns linking diverse bodies of expertise, whereas intermediation concerns work positioned outside dominant disciplinary structures.
- Figure 1 represents sub-disciplines as nodes, disciplinary clusters through shared colours or shapes, interactions through green lines, and organisational activity through node size.
- Figure 2 conceptualises intermediation as activity in interstitial disciplines or sub-disciplines between dominant disciplinary areas.
- Knowledge integration and intermediation are distinct processes that may overlap but need not occur together.
- Diversity is assessed through variety, balance, and disparity across publication sub-disciplines.
Rao-Stirling diversity
The section situates interdisciplinarity measurement within multiple bibliometric approaches, emphasizing diversity, coherence, and intermediation while noting important classification and citation-normalisation limitations.
- Coherence: Coherence measures whether categories are connected within the subset, using observed versus expected average distances of cross-citations.The formulation compares citation distances as they occur with expected distances and treats coherence as distinct from diversity.
- Intermediation: Intermediation captures how far publications lie from dense central disciplinary areas, using journal-level network measures because classification artefacts can distort the analysis.The proposed measures include a weighted clustering coefficient and average similarity across journals.
- Performance indicators: Citation-based performance indicators require normalisation because publication and referencing norms vary substantially across research specialties.Citing-side normalisation can avoid dependence on disciplinary classifications, although it may not correct differences in publication rates.
- Measurement limitations: Field normalisation remains problematic because journals can be allocated to disciplines in multiple contrasting but equally plausible ways.The study relies on Web of Science Subject Categories for data-availability reasons, despite their limited reliability for individual papers.
4. Results
Innovation Studies units occupy more interdisciplinary positions than Business & Management schools, while ABS journal rankings favour the latter despite citation-based measures yielding much less decisive differences.
- Interdisciplinarity: Innovation Studies units cite across more distant Subject Categories than Business & Management schools, linking otherwise uncommon combinations such as management with biomedical sciences.ISSTI links management and biomedical sciences, while SPRU connects economics and planning with ecology, environment and energy.
- Interdisciplinarity: Business & Management units concentrate activity in dense management and economics areas, whereas Innovation Studies units publish, reference and receive citations in interstitial journals spanning management, economics and natural sciences.Examples include Research Policy, Social Science and Medicine, and Energy Policy.
- Disciplinary bias in ABS rankings: ABS’s 4* journals narrowly focus on management, business and finance, while lower-ranked journals span broader social sciences, engineering-related fields and application areas.The ranking structure may partly contribute because ranks 4* and 4 contain 21 and 73 journals, respectively, versus more than 100 in ranks 3 and 2.
- Performance assessment: ABS-based scores rank all three Business & Management schools above Innovation Studies units, with interdisciplinarity strongly correlated with lower ABS-based performance.The reported Pearson correlations are -0.78 for Rao-Stirling diversity, -0.88 for coherence, and 0.92 for intermediation measured by clustering coefficient.
- Performance assessment: Citation-based results remain exploratory because citations were counted through October 2010 without a fixed citation window, while citation rankings also have standard errors of 8–18%.These uncertainties make ranking units problematic and support treating the findings as indicative rather than definitive.
- Performance assessment: Citation-based comparisons do not reproduce the ABS ranking: total citations do not favour Business & Management, and alternative normalisations produce similar, reversed or statistically indistinguishable results.Citing-side normalisation identifies ISSTI as the best performer, while field normalisation places Imperial and LBS only slightly ahead of SPRU without a statistically significant lead.
5. 1 Mechanisms of bias amplification
Several mechanisms can amplify journal-ranking bias against interdisciplinary research: narrower journal coverage, nonlinear resource allocation, and incentives that reinforce disciplinary publication patterns.
- Journal coverage: IS researchers may need to publish eight articles for four ABS-listed outputs when evaluated by a Business & Management panel.The alternative is shifting publication patterns away from IDR toward a more disciplinary focus.
- Journal coverage: ABS-listed journals cover a much smaller percentage of IS publications than BMS publications.This difference is identified as the first mechanism amplifying bias against IDR.
- Resource allocation: Rank-based resource allocation maps a 50% performance difference between MIoIR and LBS into a 120% resource difference.The quasi-exponential scale assigns multipliers of 0, 1, 3, and 9 to ranks 1 through 4, respectively.
- Institutional incentives: Cumulative assessment exercises create strong incentives to change publication, hiring, and promotion patterns.The paper links these incentives to the amplification of bias against IS and IDR.
- Generalisability: Business & Management scholars compete for publication in a small number of similar journals, limiting how far these findings may generalise.The paper identifies the field’s narrow publication pattern as a possible field-specific feature.
- Performance indicators: ABS rankings correlate more closely with field-normalised mean Impact Factor than with citations per paper.The authors interpret this pattern as supporting an Impact Factor-based bias against IDR.
6. Conclusions
The study finds that ABS journal rankings systematically disadvantage interdisciplinary Innovation Studies research relative to more disciplinary Business & Management research. It argues that evaluation systems should recognise multidimensional performance and that the finding requires testing beyond this setting.
- Conclusions: IS units are more interdisciplinary than leading BMS across multiple maps and metrics, yet ABS rankings assess their research as lower quality.Citation-based indicators, considered more robust measures of performance, do not support that lower assessment.
- Conclusions: Citation-based indicators do not support the lower quality assessment assigned to interdisciplinary IS research by ABS rankings.The comparison is presented as evidence that journal rankings bias evaluation toward disciplinary research.
- Generalisability: The findings require testing in wider contexts, especially the natural sciences, to establish their robustness and generalisability.The paper notes marked differences in social institutionalisation across and within fields.
- Contribution: The study provides strong quantitative evidence that journal rankings can bias research assessment against interdisciplinary research.It is presented as one of the first and most thorough quantitative explorations of this issue.
- Implications: Journal rankings can suppress interdisciplinarity when used to allocate esteem and resources.The paper connects this problem to metrics that cannot intrinsically reduce multidimensional qualities to a single indicator.
- Caveats: The study’s performance measures are limited by a short post-publication citation period and conventional mean-based metrics.Citation data were collected without a fixed citation window.