Drift Inspector: Exploring and Measuring Scientific Drift with Atomic Contribution Claims

1 Introduction

• Existing literature tools count topics and terms in papers or abstracts, although motivation can lag behind what papers actually contribute.An older thread may persist in motivation while a paper advances a different topic.Source: §1 · p. 1
• Drift Inspector extracts decontextualized, falsifiable Atomic Contribution Claims from abstracts, then clusters them across years and tracks their prevalence.Its interactive views link aggregates back to the claims and source papers.Source: §1 · p. 1
• The paper releases pre-extracted EMNLP and ACL Anthology results, human-validated claims, and a visualization tool for exploring drift across years, authors, and conferences.The contribution list also reports validation against gold reference topics.Source: §1 · p. 2

2.1 Literature exploration systems

• Existing literature dashboards index papers, fields, authors, and term frequencies, while topic models analyze raw abstracts rather than distinguishing contributions from mentions.The same word can describe a paper's motivation or its actual contribution.Source: §2.1 · p. 2

2.2 Diachronic analyses of NLP

• Unlike prior diachronic work that classifies contributions into a fixed taxonomy, Drift Inspector lets clusters emerge bottom-up and packages the measurement loop as a reusable tool.The loop includes extraction, validation, clustering, statistics, and visualization.Source: §2.2 · p. 2

2.3 Atomic semantic units

• Drift Inspector repurposes claim atomization for scientometric counting, checking claims against their abstracts while leaving their truth to the source papers.The workflow maps abstracts to claims, then embeds and clusters those claims across years.Source: §2.3 · p. 3, Figure 2 · p. 3

3 System Description

• The system workflow separates offline extraction, embedding, clustering, and drift statistics from the online interactive interface.The same canonical clustering supplies both per-cluster trajectories and the map.Source: §3 · p. 3

3.1 Atomic Contribution Claims

• Atomic Contribution Claims are abstract-derived, single-proposition, decontextualized, falsifiable assertions about a paper's contributions.The authors use abstracts for consistent availability across six years and a single LLM pass per paper.Source: §3.1 · p. 3

3.2 Semantic Topology

• After reducing claim vectors with UMAP and clustering with HDBSCAN, about 36% of claims remain unclustered noise, rising from 31–34% in 2020–2022 to 42% in 2025.These claims are kept in the interface as a background layer rather than discarded.Source: §3.2 · p. 3

3.3 Drift Quantification

• Cluster prevalence is the fraction of a year's papers with at least one claim in that cluster, and the system reports endpoint shifts, yearly trajectories, and relative drift.Counting papers rather than claims prevents papers with many similar claims from inflating a topic's share.Source: §3.3 · pp. 3–4

3.4 The Drift Inspector Frontend

• The frontend links map, trend, cluster, cohort-comparison, and methods views, with displayed numbers traceable to their claims and papers.The map supports year filtering, drift coloring, and claim search.Source: §3.4 · p. 4

3.5 Implementation and Availability

• The pipeline caches artifacts and seeds stochastic stages, making released-claim clustering reproducible; re-running extraction reproduces aggregate rates, not individual claim strings.Code is MIT-licensed and the claim dataset is CC BY 4.0.Source: §3.5 · p. 4

3.6 Scalability and Cost

• The full ACL Anthology run covers 80,144 abstracts, 423 venues, and 346,010 claims; extraction costs about $0.50 per thousand abstracts with gpt-oss-120b.Embedding, projection, and clustering take well under an hour on laptop CPU at either tested scale.Source: §3.5 · p. 4, §3.6 · pp. 4–5

4.1 Human Validation of Claim Quality

• Annotators rated 97.8%, 91.8%, and 90.5% of sampled claims Good, flagged all 44 planted negatives by majority, and agreed robustly (Fleiss’ κ = 0.844 dropping Unsure).A blind independent LLM judge matched majority consensus with 94.5% accuracy (κ = 0.863) and flagged 43 of 44 negatives.Source: §4.1 · p. 5

4.2 Cluster Validity Against an External Taxonomy

• On the 653-paper SToP overlap, claim-level purity (0.689) exceeds abstract-level BERTopic-SPECTER2 (0.324), and ACC leads or ties on coverage-robust metrics.Coverage adjustment addresses ACC’s lower hard-partition coverage when all-noise papers are excluded.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 Robustness of Headline Findings

• All 15 headline shifts are significant under two-proportion tests, with bootstrap 95% intervals excluding zero; trajectories are largely monotone across six years.The sampling analysis used 2,000 paper-level resamples; parsing declined from 5.1% to 0.1% of papers across the six annual snapshots.Source: §4.3 · p. 6
• Across nine alternative representations, at least 11 of 15 shifts retain their signs (median 14/15); two sign-blind matching alternatives never reduce per-encoder counts.The representations include eight other encoders and a TF-IDF + SVD baseline.Source: §4.3 · pp. 6–7
• Re-extraction with gpt-oss-120b preserves all 15 signs and keeps ACC clustering ahead of abstract-level baselines on SToP (purity 0.638, V-measure 0.541).The alternative extractor yields 4.6 claims per paper versus 3.7.Source: §4.3 · p. 7
• Across 36 hyperparameter settings, clusterings agree with the reported result at ARI 0.79–0.95 and AMI 0.93–0.97, while all 15 signs are preserved.The sweep produced 55–116 clusters.Source: §4.3 · p. 7

5 Case Study: EMNLP 2020–2025

• Across 4,488 EMNLP papers, parsing falls by 4.9 percentage points while multimodal and vision-language modeling gains 9.1 points; agents rise more than tenfold.Machine translation, dialogue, and question answering each shrink by 3.7–4.4 points; declining task claims thin and disperse rather than simply vanish.Source: §5 · p. 7

5.1 Use Cases

• The BERT cohort’s residual agenda concentrates in claim verification and biomedical NLP, while the larger LLM cohort shifts toward vision-language modeling and efficiency.The comparison reports yearly topic mix as a percentage of cohort claims for 197 BERT papers and 3,253 LLM papers.Source: Figure 5 · p. 7
• Compare ranks clusters by cohort paper-share gaps, while Trends ranks clusters by their 2020–2025 shifts and claim search reveals where residual matches fall.The ACL–COLING comparison contrasts classic-NLP topics with efficient reasoning, multilingual LLMs, and vision-language evaluation; retrieval’s leading cluster rises from 0.7% to 3.3% of papers.Source: §5.1 · p. 7

6 Conclusion

• The system is open-source and corpus-agnostic, and its proposed next step is a hierarchical theme system that aggregates clusters into navigable super-themes.The code is MIT-licensed and the data are CC BY.Source: §6 · p. 8

Limitations

• Abstracts omit technical nuance and negative results, while some contributions appear only in paper bodies; validated results cover EMNLP, and broader-corpus cluster figures warrant more caution.The larger six-venue and anthology extractions are LLM-judge-monitored rather than human-revalidated; clustering boundaries and labels may also vary with model and hyperparameters.Source: Limitations · p. 8

• Cluster prevalence reflects sampling, extraction, and clustering choices, so it should be read as a descriptive signal about abstracts.Source: Ethics Statement · p. 8
• A decline in standalone parsing contributions does not mean parsing is solved or worthless.Source: Ethics Statement · p. 8
• The analysis uses publicly available ACL Anthology abstracts under CC BY 4.0 and involves no private data.Source: Ethics Statement · p. 8
• The authors document parameters and release artifacts to keep the measurement inspectable.Source: Ethics Statement · p. 8

• The extractor returns JSON claims about contributions introduced, proposed, evaluated, demonstrated, or established in titles and abstracts.Source: A Extraction and Judge Prompts… · p. 10
• Prompts include methods, datasets, resources, tools, evaluation setups, and established findings, while excluding background, motivation, prior-work claims, and raw metric-only claims.Source: A Extraction and Judge Prompts… · p. 10
• Each claim must express one faithful, self-contained proposition without meta-language or raw numbers; outputs usually contain 1–5 claims, or none.Source: A Extraction and Judge Prompts… · p. 10
• The extractor self-checks candidates, while a separate LLM judge labels each as Good, Bad, or Unsure and assigns a main-issue category.Source: A Extraction and Judge Prompts… · p. 10
• Human validation used 180 stratified items: 136 claims from the 16,576-claim output and 44 deliberately invalid controls, shown without identifying controls to annotators.Source: A Extraction and Judge Prompts… · p. 10

• The balanced EMNLP corpus contains 4,488 papers and 16,576 claims; the six-venue corpus retains 69,950 claims, 88 clusters, and papers from 2018–2026.Source: B Pipeline Configuration · p. 10
• EMNLP extraction used qwen3-235b-a22b-thinking-2507, while the anthology used gpt-oss-120b with medium reasoning effort and a strict JSON schema.Source: B Pipeline Configuration · p. 10
• Claims were embedded with SPECTER2 using claim text and CLS pooling, then clustered with UMAP and HDBSCAN using the specified parameters.Source: B Pipeline Configuration · p. 10
• The portable single-file interface embeds its assets and data, with web fonts as its only external request; Figure 7 presents Compare’s bump-chart mode.Source: B Pipeline Configuration · p. 10

• Figure 6 plots yearly paper-level document frequency for the eight most declining and eight most growing clusters, with significant headline endpoint shifts.Source: Figure 6 · p. 11
• Across nine alternative representations, each re-embedded and re-clustered on 16,576 claims with the identical UMAP/HDBSCAN pipeline, at least 11 of 15 headline-shift signs were preserved.Source: C Encoder Ablation · p. 10, Table 3 · p. 11
• SPECTER2 is the canonical clustering because it has the top purity among the compared representations.Source: C Encoder Ablation · p. 11
• Figure 7 ranks EMNLP clusters by yearly document frequency: vision–language modeling reaches rank 1 by 2024, while summarization falls from 7 to 31.LLM efficiency climbs from rank 14 to ranks 3–5.Source: Figure 7 · p. 11