PaperlayerView exact revision on arXiv

Source-linked AI summary

Citations Are Late: Reading epistemic instability from what papers believe, years before the citation graph catches up

Andrey Ustyuzhanin, Ekaterina Trofimova, Denis Zuenko

arXiv:2610.07397v1cs.DL

TL;DR

Citation-based disruption measures require years of forward citations, while changes in stated modelling beliefs may appear earlier. The paper tests an abstract-based content signal against CD5 on documented shifts. It finds an NLP observability lead of about 25 quarters, with the carrier signal varying by shift.

  • Problem

    Citation-based disruption indices are structurally delayed and content-blind, leaving a need to test whether stated beliefs can reveal paradigm shifts earlier.

    Source: §1 · p. 1

  • Method

    The paper extracts reliance, contestation, and applicability-stress judgements from abstracts, aggregates quarterly content signals, and compares their pre-registered detections with citation-based measures.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    The content signal leads the CD5 observability wall by about 25 quarters in NLP, but leads the breakthrough by five quarters there and is contemporaneous in vision.

    Source: §1 · p. 2, §10 · p. 9

  • Takeaways & Limitations

    The defensible claim is that reading beliefs makes paradigm shifts observable at publication time while citation-based indices cannot report for years.

    Source: §10 · p. 9

    Both documented cases are arXiv computer-science subfields, so generality to unresolved shifts in distant disciplines remains untested.

    Source: §9 · p. 8

Abstract

from arXiv · show

Paradigm shifts in science are visible in what researchers assert and contest before they are visible in the citation graph. We ask whether a cheap, content-level signal of epistemic instability, derived from the changing distribution of stated modelling beliefs in paper abstracts, can anticipate a paradigm shift earlier than the dominant citation-based disruption index (CD5). On the displacement of recurrent networks by Transformers in NLP, a pre-registered content signal crosses its detection threshold in 2016-Q1, whereas a real-time CD5 monitor cannot even observe the 2017 breakthrough until 2022-Q2, since CD5 needs a five-year forward-citation window: a lead of about 25 quarters. The flat citation baseline is not an artifact of one index, as our CD5, a reference-normalised variant, and an authoritative precomputed index all sit near zero across the shift. The lead is also not a faster proxy: at equal latency the signal beats the content competitors tested, including a learned CD-from-text model and an embedding disruption measure, and the decomposition sees what a keyword cannot (keyword-blind AUC of about 0.9, reproduced on human labels). The lead over CD5 is an observability lead. Which signal carries it depends on the shift: belief adoption in NLP, contestation and applicability stress in vision. Measured against the breakthroughs themselves, the signal leads by five quarters in NLP and is contemporaneous in computer vision.

1 Introduction

Citation-based disruption measures register shifts only after forward citations accumulate, while researchers’ stated beliefs can change earlier. The paper tests an abstract-based content signal against that lag and reports a lead across two documented shifts.

1 Introduction

  • The content signal detects the RNN-to-Transformer shift in 2016-Q1, about six years before a CD5 monitor could observe it, and reproduces a lead on a vision shift.The vision replication reports a 14–19-quarter lead over the CD5 observability wall.Source: §1 · p. 2
  • Three citation-disruption measures remain near zero across the NLP shift, while tested content competitors do not onset for it at equal latency.The comparison includes CD5, a reference-normalised variant, and SciSciNet’s precomputed index.Source: §1 · p. 2
  • A component decomposition recovers the shift where a keyword counter is blind and provides a low-dimensional, drift-stationary statistic.It also recovers the incumbent axis without an oracle.Source: §1 · p. 2
  • Human annotation supports the paradigm-reliance construct and reproduces the decomposition’s keyword-blind advantage on human labels.The in-scope items have κ ≈0.83 agreement and 100% verbatim warrants.Source: §1 · p. 2

2 Related Work

Prior work measures disruption through citation networks or text change, but a pre-registered head-to-head on documented shifts with lead-time estimates was missing. This paper instead tracks stated reliance on modelling paradigms.

2 Related Work

  • The CD index [6] and its science variant classify disruption through later citations to a focal paper instead of its predecessors.The literature also documents sensitivity to citation inflation, reference counts, database coverage, and implementation choices [13] [16].Source: §2 · p. 2
  • Text-based signals, including novelty, citation framing, and language-model surprise, can be computed at publication time, but lacked a direct pre-registered comparison on documented shifts.The missing comparison included a lead-time estimate.Source: §2 · p. 2
  • Word- and topic-drift methods register both paradigm shifts and vocabulary fashions, whereas this signal targets incumbent-paradigm reliance by method role.A keyword or embedding-drift signal can count “LSTM with attention” as change when the method-role signal does not [7] [12].Source: §2 · pp. 2–3

3 Method

The method extracts modelling-belief judgements from paper abstracts and aggregates them into quarterly signals. It uses pre-registered detection rules and compares the resulting content signals with citation-based measures.

3 Method

  • A pre-registered pipeline uses arXiv titles and abstracts, a keyword pre-filter, and a language model to extract architecture family, incumbent reliance, contestation, and applicability stress.The reported NLP filter precision is 0.72 on a 50-paper hand check; no full text or external knowledge is used downstream.Source: §3 · p. 3
  • A second model independently re-extracts judgements, while prompts require verbatim evidence spans and forbid outside knowledge to bound hindsight.The validation pass uses a model from a different family.Source: §3 · p. 3
  • Four quarterly signals—adoption, contestation, applicability stress, and Bayesian surprise—are smoothed over two quarters and detected after exceeding baseline µ + 2σ for two consecutive quarters.The comparison defines CD5’s observability wall as the flashpoint quarter plus five years and also reports reference-normalised and precomputed disruption indices.Source: §3 · p. 3

4 Case 1: Transformers displacing recurrence in NLP

In NLP, belief adoption crosses its pre-registered threshold years before a real-time CD5 monitor can observe the Transformer breakthrough. The shift is also not reflected in three citation-disruption measures.

4 Case 1: Transformers displacing recurrence in NLP

  • The architectural shift is reflected in recurrent models falling from 43% in 2016 to 7% in 2022 as Transformers rise from 0% to 47%.On the matched control topic, no content signal crosses threshold; a verbatim-warrant audit found no unsupported positive labels among 38 cases.Source: §4 · pp. 3–4
  • 2016-Q1 detection of belief adoption precedes the 2022-Q2 CD5 observability wall by 25 quarters, as the figure’s top and bottom panels show the content signal and citation measures.The CD5 wall follows the 2017-Q2 flashpoint by five years.Source: §4 · p. 3, Figure 1 · p. 4
  • All three disruption measures stay close to zero without a Transformer-era spike, so the lead is not an artifact of the chosen citation index.The measures are CD5, reference-normalised CDnok, and SciSciNet’s precomputed D; Vaswani is unmeasurable in OpenAlex due to record fragmentation.Source: §4 · p. 4, Figure 1 · p. 4
  • Belief adoption alone carries the NLP lead; contestation and applicability stress remain sub-threshold, while Bayesian surprise fails under the pre-registered rules.Surprise is noise-dominated when quarterly counts are small and near-silent during the gradual 2018–2020 shift.Source: §4 · p. 4

5 Is it more than a faster proxy? Equal-latency content baselines

The equal-latency comparison tests whether the belief signal offers more than a faster CD or keyword proxy. No tested baseline onsets for the shift or separates it from a matched control, whereas the belief signal does both.

5 Is it more than a faster proxy? Equal-latency content baselines

  • Signal-relevant extraction distinctions agree at 0.85–0.93, but analyst gold was not repeated for the cs.CV replication.The table caption notes that raw multiclass agreement is depressed by the no/not-applicable labelling boundary.Source: Table 1 · p. 5
  • No equal-latency baseline onsets for the shift or separates it from the matched control, while the belief signal does both.The comparison uses the same pre-registered onset rule and evaluates AUC against a vocabulary-matched control.Source: §5 · p. 5, Table 2 · p. 5
  • The learned CD-from-text model reproduces CD’s flatness, while the embedding disruption measure never rises for the shift.Both are among the strongest named content predictors tested.Source: §5 · p. 5
  • On the same focal papers, belief-signal AUCs are 0.96 for A, 0.88 for C, and 0.91 for X, versus 0.38 for B7.The full-corpus belief AUC and citation-based successors use different populations; the matched-focal-paper comparison addresses that asymmetry for discrimination.Source: §5 · p. 5
  • The raw departs share is U-shaped, while the method-family component is monotonic and leads the late, sparse “transformer” keyword.The early hump comprises pre-neural non-recurrent work.Source: Figure 2 · p. 6

6 The decomposition sees what a keyword cannot

The decomposition distinguishes the emerging method family from keyword presence, including where challenger vocabulary is constant. It also yields a fixed, lower-dimensional monitor rather than a drifting keyword basis.

6 The decomposition sees what a keyword cannot

  • Without being given the incumbent belief, open-assumption clustering recovers a recurrence-necessity axis: 78% of its statements mention recurrence, RNNs, or LSTMs, and reliance curves correlate at r ≈0.45.The axis is not dominant among discovered assumptions (size-rank 11/16).Source: §6 · p. 6
  • On challenger-term papers, the method-family component reaches AUC ≈0.92 on model labels and 0.79 on human-adjudicated gold, versus 0.5 for the uninformative keyword.The literal term appears in only ≈25% of departing papers; on a non-circular control, the methods-aware component scores 0.83 versus 0.64 for the flat reliance signal, with ∆AUC ≈+0.04 after partialling out both.Source: §6 · p. 6, Table 3 · p. 7
  • The raw departs share is U-shaped because its early hump is pre-neural non-recurrent work, while the method-family component is monotonic and isolates the emerging paradigm.The decomposed component leads the keyword.Source: §6 · p. 6
  • A keyword model’s AUC falls from 0.91 to 0.71 when trained through 2017 and tested from 2019 onward, with ≈97% of its top predictive vocabulary replaced.The component basis is instead a fixed rule, and relies/departs can be defined before Transformers exist, unlike a keyword monitor keyed to the shift-marking term.Source: §6 · pp. 6–7

7 Human validation

A human-annotated cs.CL slice tests whether paradigm reliance is recoverable beyond model labels. The method-family component’s keyword-blind advantage reproduces on human labels, while the off-scope boundary remains a measurement issue.

7 Human validation

  • Assessors agree on the substantive relies-versus-departs judgment at Cohen’s κ ≈0.83, with 92% raw agreement among papers both consider in scope.The lower overall κ of 0.55 is driven almost entirely by the off-scope (na) boundary.Source: §7 · p. 7
  • All assessor warrant spans are verbatim abstract substrings, and the adjudicated 97-item gold yields LLM-panel accuracy of 74% overall and 88% on in-scope papers.The residual accuracy gap lies at the na boundary rather than in the reliance judgment.Source: §7 · p. 7
  • On challenger-term papers, the method-family component reaches AUC = 0.79 (n = 56) on human labels, versus 0.5 for the keyword.This reproduces the decomposition’s advantage where keyword presence is uninformative by construction.Source: §7 · p. 7
  • The gold set’s na boundary is codebook-defined rather than consensus-defined: a second adjudication pass agreed on only 16/29 pre-selected hardest disagreements.The validation is a first slice of 97 NLP abstracts labelled by two assessors and adjudicated.Source: §7 · pp. 7–8

8 Case 2: Deep CNNs displacing hand-crafted features in vision

In computer vision, the same pipeline detects the hand-crafted-feature to deep-CNN shift before the CD5 observability wall. The leading signal differs from NLP, and the control is less clean.

8 Case 2: Deep CNNs displacing hand-crafted features in vision

  • The hand-crafted-feature share falls from 45% in 2010 to 1% in 2019, while the deep-CNN share rises from 0% to roughly 79%.The cs.CV corpus covers 2010–2019 and contains 35,265 papers, including 19,243 treatment papers.Source: §8 · p. 8
  • Contestation crosses threshold at 2013-Q1, Bayesian surprise at 2014-Q1, and applicability stress at 2014-Q2, leading the AlexNet-cohort CD5 observability wall by 19, 15, and 14 quarters, respectively.Belief adoption tracks the outgoing hand-crafted-feature assumption and declines; contestation and stress lead.Source: §8 · p. 8
  • Bayesian surprise fires in this more abrupt shift, while the authors specify that naive surprise’s failure is specific to gradual shifts, not surprise in principle.The content-signal family reproduces the lead even though the individual carrier changes with the shift’s geometry.Source: §8 · p. 8
  • The vision control is not a clean null because deep learning also prompts eventual contestation in control papers; treatment nevertheless leads control by roughly 19 quarters and three to four times the magnitude.Extraction validation rests on cross-model agreement of 0.60 to 0.80 and an audit with no unsupported labels (0 of 41); a cleaner control is left to future work.Source: §8 · p. 8

9 Limitations

The evidence is bounded by two outcome-selected arXiv shifts within computer science, abstract-only extraction, and limited validation; the positive result remains a two-case pilot.

9 Limitations

  • Both documented shifts were selected on the outcome and come from arXiv within computer science, so generality beyond cross-subfield cases remains untested.A prospective test on an unresolved shift in a distant discipline is identified as the needed test of generality.Source: §9 · p. 8
  • Extraction is abstract-only and follows a keyword pre-filter with 0.72 precision; human validation covers a 97-abstract NLP gold set and two assessors.The vision replication is validated only by cross-model agreement, and its control is contaminated rather than clean.Source: §9 · p. 8
  • The central negative result holds across three disruption indices and two learned successors, while the positive result is offered as a pilot on two cases.The verbatim-evidence audit bounds hindsight without eliminating it, and the fixed µ + 2σ rule has no interval on the lead-time.Source: §9 · pp. 8–9

10 Conclusion

The paper frames the content signal’s advantage as publication-time observability rather than a faster citation measure. Its carrier differs by shift, and more cases are needed to establish anticipation as a general pattern.

10 Conclusion

  • CD5’s five-year forward-citation window makes it unavailable for years after a shift, while belief-based monitoring is observable at publication time.The lead over CD5 is largely its definitional lag; the lead over the breakthrough itself is smaller and varies by case.Source: §10 · p. 9
  • In NLP, belief adoption turns five quarters before the 2017-Q2 flashpoint; in vision, the earliest content signal fires one quarter after AlexNet.Adoption carries the NLP lead, while contestation and applicability stress carry the vision lead.Source: §10 · p. 9
  • Establishing anticipation as the rule rather than the NLP case requires more shifts, and the authors release the construct, labels, and case studies for application elsewhere.Source: §10 · p. 9

A Reproducibility

The reproducibility materials support regenerating the headline figures, while sensitivity, landmark-score, vocabulary-control, and baseline analyses qualify how the results should be read.

A Reproducibility

  • The released annotation materials, sampled-paper identifiers, labels, adjudication, gold set, and scripts support corpus reconstruction and regeneration of headline figures.Models are version-pinned, total LLM cost is under $11, and citation records were cross-checked across databases.Source: A Reproducibility · p. 11

B Detection sensitivity

  • 74% of the 108 detection settings detect belief adoption between 2015-Q1 and 2016-Q1; the pre-registered default detects at 2016-Q1.Non-detections occur only at the strictest settings; neighboring-quarter confidence intervals support a clear rise, while the exact onset remains uncertain.Source: B Detection sensitivity · p. 11

C Landmark-paper disruption scores

D Vocabulary-matched control

  • The term-matched, topically close control yields AUC ≈0.99 for the belief signal, while a broad keyword counter collapses to AUC = 0.00.The control has challenger-term prevalence 1.00 versus 0.26 in the shift-core; the strict term “transformer” is not treated as a discriminator.Source: D Vocabulary-matched control · p. 12

E Baseline implementations

  • None of the equal-latency baselines onsets for the shift; all use the same per-quarter grid, onset rule, and shift-versus-control AUC harness.The baselines include term prevalence, lexical and embedding surprise, reference atypicality, a learned CD predictor, and embedding disruptiveness.Source: E Baseline implementations · pp. 12–13

Related papers

Loading 2610.07397v1…