Source-linked AI summary
More Computational Resources Do Not Ensure Higher Scholarly Impact: Evidence from Leading NLP Conference Papers
Shuai Chen, Tong Bao, Jitong Peng, Chengzhi Zhang
TL;DR
The paper asks how reported GPU resources relate to scholarly impact in NLP. It analyzes 13,921 ACL, EMNLP, and NAACL main-conference papers by extracting and standardizing reported GPU configurations, then links them to impact and metadata. Reported resources are associated with impact but provide little standalone explanatory power, with GPU count showing more consistent associations than hardware generation.
Problem
It remains unclear whether differences in computational resources are systematically associated with scholarly impact across the broader NLP community.
Method
The study extracts reported GPU models and counts from 13,921 papers, standardizes the largest reported configurations, and estimates adjusted associations with citation and award outcomes.
Results
Reported resources show positive but limited alignment with impact: the annual top 20% held 83.9%–89.9% of reported capability but only 27%–32% of citations and 20%–33% of awards.
Takeaways & Limitations
Reported GPU resources may expand experimental possibilities but neither ensure nor are necessary for scholarly impact.
Takeaways & Limitations
The estimates are conditional associations because impact measures are incomplete proxies and unobserved author, institutional, and project characteristics may remain correlated with resources and impact.
Abstract
from arXiv · showhide
Computational resources are increasingly central to NLP research, but how closely reported GPU capability aligns with scholarly impact remains unclear. We analyze 13,921 ACL, EMNLP, and NAACL main-conference papers published between 2020 and 2025, using GPU resources as our operational measure of computational resources. From full texts, we extract GPU models and counts, standardize each paper's largest reported configuration into a comparable hardware-capability measure, and link these data to citation, award, topic, and institutional metadata. GPU reporting became more common but remained incomplete, while reported capability increased mainly through newer hardware generations and medium-scale multi-GPU configurations. Resource concentration substantially exceeded impact concentration: the annual top 20% of GPU-quantifiable papers accounted for 83.9%-89.9% of reported GPU capability, but only 27%-32% of citations and 20%-33% of paper awards. In adjusted models, a tenfold increase in aggregate reported GPU capability was associated with a 3.52-percentage-point increase in within-NLP topic-year citation percentile, but increased model R^2 by only 0.0042. GPU count showed more consistent positive associations with citation and award outcomes than newer hardware generation. Overall, reported GPU resources are associated with scholarly impact but provide little standalone explanation of research influence.
1 Introduction
This study asks whether reported computational resources are associated with scholarly impact across NLP papers, and finds a positive but limited alignment: resource concentration far exceeds citation and award concentration, while GPU measures add little explanatory power.
- The study operationalizes computational resources using reported GPU models and counts, standardized as each paper’s largest reported configuration.The measure captures reported hardware capability rather than realized consumption.
- 83.9%–89.9% of reported GPU capability was concentrated in the annual top 20% of GPU-quantifiable papers, versus 27%–32% of citations and 20%–33% of awards.
- A tenfold increase in aggregate reported capability was associated with a 3.52-percentage-point increase in primary citation percentile but only a 0.0042 increase in R2.
- GPU count showed more consistent associations with citation outcomes and awards than newer hardware generation.
- GPU resources were positively associated with scholarly impact but were neither necessary nor sufficient for high impact and explained little additional variation.
- 13,921 ACL, EMNLP, and NAACL main-conference papers published between 2020 and 2025 form the study corpus.
2 Related Work
Prior research identifies unequal access to AI compute across industry, academia, and institutions, alongside growing reproducibility burdens for compute-intensive experiments.
- Prior studies document industry–academia asymmetries, elite-institution advantages, and reporting and reproducibility burdens in AI compute.
3 Methodology
The methodology builds a corpus of leading NLP conference papers, enriches it with bibliographic and award metadata, extracts and normalizes reported GPU information, and analyzes reporting and capacity patterns.
- 3.1 Data Collection: The corpus contains ACL, EMNLP, and NAACL papers from 2020 to 2025, with 13,921 papers after excluding inaccessible PDFs.
- 3.1 Data Collection: Paper full texts were parsed and linked through DOI to OpenAlex metadata, official conference award records, and GPT-4o-mini NLP topic labels.
- 3.2 GPU Usage Extraction: GPU usage extraction combined a human-validated evaluation set with LLM-based extraction of GPU models and counts, achieving Cohen’s κ of 0.94 on valid GPU-resource evidence.
- 3.3 GPU Normalization: Raw GPU mentions were cleaned and mapped to a canonical hardware catalog using exact matches, aliases, and rules for model and memory variants.
- 3.3 GPU Capacity Estimation: The analysis distinguishes model-reported and strict samples based on whether standardized GPU models and explicit counts are observable.The model-reported sample contains 6,900 papers, while the strict sample contains 5,360 papers.
- 3.3 GPU Capacity Estimation: Reported GPU capacity uses theoretical peak Tensor FP16/BF16 throughput per GPU and selects the largest reported configuration rather than summing configurations.This avoids double counting hardware across experiments, stages, or runtime environments.
4 Results
Reported GPU resources became more visible and capable over time, with strong institutional and topical concentration. Their concentration substantially exceeded citation and award concentration, while adjusted associations with impact were positive but added little explanatory power.
- GPU Reporting Completeness and Trends: Reporting rose from approximately 30% to 57% for GPU models and from 15% to 49% for both models and counts between 2020 and 2025.Capacity analyses remain conditional on papers with positive, quantifiable GPU configurations.
- The Evolution of GPU Scale in NLP: Reported capacity shifted toward newer generations and medium-scale multi-GPU configurations, while nine-or-more-GPU configurations remained uncommon at 11.7% in 2025.One- or two-GPU configurations declined from 49.5% in 2020 to 35.9% in 2025.
- Institutional Variation in GPU Capacity: Industry-involved papers reported higher median capacity and appeared more often in the annual high-capacity tail, although institutional indicators overlapped substantially.Joint models retained a strong positive industry association, while collaboration coefficients were attenuated.
- GPU Capacity Across Research Topics: Higher reported capacity concentrated in LLM agents, code models, and language modeling, whereas syntax, sentiment analysis, and discourse and pragmatics generally reported lower-capacity configurations.The displayed topic figure covers a readable subset of 29 categories; complete results are reported separately.
- 4.2 Concentration of GPU Capability and Scholarly Impact: 83.9%–89.9% of reported capability came from the annual top 20% of papers, compared with 27%–32% of citations and 20%–33% of awards.The gap indicates that capability concentration substantially exceeded impact concentration.
- 4.3 Limited Scholarly Impact of GPU Resources: A tenfold increase in aggregate reported capability was associated with a 3.52-percentage-point increase in the primary citation percentile but only a 0.0042 increase in model R2.GPU count showed more consistent positive associations across citation outcomes and awards than newer hardware generation.
5 Discussion
Reported GPU resources align positively with scholarly impact, but resource concentration greatly exceeds citation and award concentration. These resources neither ensure nor are necessary for influence, and better reporting would help distinguish capability from actual consumption.
- 83.9%–89.9% of reported GPU capability belonged to the annual top 20% of papers, versus 27%–32% of citations and 20%–33% of awards.The upper tail contained most reported capability but much smaller shares of scholarly-impact measures.
- High-capability papers were more likely to enter the citation top 10%, yet most were not highly cited and most highly cited papers fell outside that group.
- Reporting GPU models, counts, runtime, utilization, and externally provided compute would help distinguish available capability from actual consumption.
- API-only studies may depend on substantial upstream computation while reporting no local hardware, limiting coverage and potentially understating compute dependence in application-oriented research.This issue is more consequential for recent descriptive trends than for the main citation models using 2020–2023 data.
6 Conclusion
Across 13,921 ACL, EMNLP, and NAACL main-conference papers from 2020–2025, reported GPU capability became more common and concentrated, but its alignment with scholarly impact remained limited. Adjusted associations varied by outcome, with GPU count more consistently related to citations and awards than aggregate capability or hardware generation.
- 13,921 ACL, EMNLP, and NAACL main-conference papers from 2020–2025 form the paper-level dataset of reported GPU configurations.
- GPU reporting became more common but remained incomplete, while capability increased mainly through newer hardware generations and medium-scale multi-GPU configurations.
- GPU count showed the most consistent associations across citation outcomes and the only award association supported by both linear-probability and rare-event models.
- Reported GPU resources are important infrastructure for contemporary NLP research but provide only a limited standalone explanation of research influence.
Limitations
The study is limited by incomplete and heterogeneous GPU reporting, a capability measure that omits important dimensions of computation, observational impact measures, and a restricted conference corpus.
- Reported resources and sample selection: Only explicitly reported, standardizable GPU models and counts are observed, excluding some local and remotely managed computation.Only 92 of 240 GPU-reporting papers (38.3%) contained a consumption-related signal, and those signals were too heterogeneous for a comparable measure.
- Hardware-capability measurement: Aggregate reported GPU capability uses the largest observed configuration and theoretical peak throughput, not runtime, utilization, efficiency, or cumulative experimental use.Identical configurations can represent short inference runs or prolonged training.
- Impact measures and observational design: Citations, high-citation status, and awards are incomplete proxies, while unobserved author, institutional, and project characteristics may confound the conditional associations.The estimates should not be interpreted causally.
- Impact measures and observational design: Citation analyses reduce but do not eliminate citation-window and field differences, and topic normalization relies on a single assigned primary topic.OpenAlex field-normalized estimates are sensitive to the reference set.
- Corpus and metadata scope: The corpus includes only ACL, EMNLP, and NAACL main-conference papers from 2020 to 2025, limiting generalization to other venues and publication types.Affiliation metadata and full-counting rules also cannot identify resource ownership, researcher mobility, or access to shared infrastructure.
Ethics Statement
The study uses public scholarly articles, metadata, award records, and hardware specifications, processing released paper text without confidential submissions or peer-review materials. Its ethics and data practices emphasize aggregate linkage, public-source constraints, and validation of automated extraction.
- Public articles, ACL Anthology and OpenAlex metadata, official award records, and public hardware specifications supply the study’s data.
- Author names and affiliations were used only for bibliographic linkage and aggregate analysis, without collecting private communications, reviewer information, or protected demographic attributes.Geographic variables refer to institutional locations rather than authors’ nationality or ethnicity.
- Only publicly released paper text was processed with MinerU and LLM-based systems; confidential submissions and peer-review materials were not provided.
- The measures represent reported configurations rather than actual consumption, ownership, researcher ability, or scientific quality, and observed associations are not causal.
- GPU extraction was evaluated against human annotations, with Cohen’s κ of 0.9409, 90.83% exact GPU-model agreement, and 87.50% exact GPU-count agreement.
B.4 Extraction Evaluation and Large-Scale Extraction
The paper evaluates automated GPU-resource extraction and constructs analysis samples from standardized hardware reports, while emphasizing that capacity results remain conditional on observable reporting.
- Extraction Evaluation: F1 scores were 0.882 for Gemini-3-Flash-Preview and 0.879 for DeepSeek-V3.2 under exact match.DeepSeek-V3.2 was selected for full-corpus extraction after balancing performance against processing cost.
- Large-Scale Extraction: 13,921 ACL, EMNLP, and NAACL papers from 2020–2025 were processed using full-text GPU-resource extraction.Introductions and related work were removed, and papers were split into overlapping text chunks to reduce noise from cited or prior work.
- Hardware Normalization: Unresolved records included generic hardware terms, capacity-only descriptions, cloud instances, and non-hardware model or framework names.CPU-related records were removed from GPU catalog analyses.
- Organizational Predictors: Industry and collaboration indicators were generally not statistically significant predictors of reporting, with cross-sector collaboration interpreted only as a weak signal.The number of institutions had negative but unstable and nonsignificant coefficients.
- Scope: The analyses describe observable, extractable, and standardizable GPU information rather than all true computational resource use.This conditional interpretation follows incomplete reporting and exclusions such as TPU-containing papers.
C.2 Audit of Compute-Consumption Reporting
The audit found that compute-consumption information was visible in fewer than half of sampled GPU-reporting papers and was not standardized enough for corpuswide GPU-hour estimates.
- Consumption Audit: 38.3% of audited papers contained a potential consumption-related signal, while 61.7% contained none.The audit covered 240 papers sampled across 16 venue–year strata.
- Reporting Heterogeneity: Reported consumption signals varied across duration, run counts, token usage, and related quantities.These subsets generally could not be combined into a common GPU-hours measure without additional assumptions.
- GPU Scale: Single-GPU papers declined as medium-scale configurations of 3–4, 5–8, and 9–16 GPUs became more common.Configurations of 33–64 or 65+ GPUs remained rare throughout the period.
- Hardware Generations: V100 led reported GPU models from 2020–2022, while A100 became the leading model from 2023 onward.Later years also showed greater visibility of A100 80 GB, RTX A6000, and H100 PCIe models.
D.3 Reported GPU Memory Capacity
Reported GPU memory and configuration capacity increased over time, with higher-capacity hardware concentrated in particular research topics and a substantial right tail across papers.
- Memory Trends: The median maximum GPU memory rose from about 16 GB in 2020–2021 to nearly 48 GB in 2025.The P90 increased from about 32 GB to 80 GB, indicating growing prevalence of high-memory GPUs.
- Memory Categories: After 2023, 40–48 GB and 64–80 GB configurations increased markedly as reporting shifted toward devices such as A100 and H100 GPUs.Total paper-level GPU memory showed a similar trend as both counts and per-GPU memory increased.
- Configuration Capacity: Reported peak configuration capacity had a median of 624 TFLOP/s and a P95 of 6,048 TFLOP/s.The distribution was strongly right-skewed, with most papers in a medium-capacity range and a small high-capacity tail.
- Research Topics: Reported capacity was above the overall median in LLM agents, code models, human-centered NLP, language modeling, and multimodality.Syntax/parsing, discourse/pragmatics, sentiment/argument mining, information extraction, and semantics had lower shares in the P90-threshold group.
- Impact Associations: A tenfold increase in reported GPU capability was associated with a 3.52-percentage-point increase in the primary within-NLP topic–year citation percentile.The aggregate model added only 0.0042 to model R2, while GPU count showed the more consistent positive citation pattern.
E.4 Rare-Event Robustness for Paper Awards
Rare-event robustness analyses preserve a dimension-specific award pattern: reported GPU count is positively associated with awards, whereas newer hardware generation is not supported.
- Rare-Event Design: 111 of 5,357 papers received an award label in the strict 2020–2025 sample.Because awards were rare, the authors re-estimated the joint model using Firth penalized logistic regression.
- GPU Count: A tenfold increase in reported GPU count was associated with 1.713 times the odds of receiving an award.The 95% CI was [1.189, 2.437], with Holm-adjusted p = 0.0085.
- Hardware Generation: The Ampere-or-newer hardware association was close to zero and statistically imprecise.The results support an association with deployment scale, not a uniform association across hardware dimensions.
- Interpretation: The authors interpret the finding as a rare-event robustness association rather than evidence that increasing GPU count causes award recognition.The observational design and small number of award-positive papers limit causal interpretation.
E.5 Robustness to Expanded Observable Controls
Expanded pre-publication controls attenuate the reported GPU-capability association with citation impact, but the association remains positive and adds little explanatory power.
- Robustness to Expanded Observable Controls: The robustness analysis uses a common complete-case sample of 2,077 papers while retaining the original fixed effects and team-structure controls.Additional specifications add author citation history, team publication experience, institutional citation visibility, collaboration structure, and public-artifact availability.
- Robustness to Expanded Observable Controls: 2.65 percentage points remains the estimated association after adding artifact availability to the expanded-control specification.The estimate declines from 3.13 percentage points in the common-sample baseline to 2.74 and then 2.65 percentage points.
- Robustness to Expanded Observable Controls: 0.0022 is the incremental R2 after adding pre-publication controls and artifact availability.Incremental R2 declines from 0.0032 in the common-sample baseline to 0.0023 and then 0.0022.
- Robustness to Expanded Observable Controls: The OpenAlex field-normalized estimates remain small and statistically imprecise after expanded controls.The log-citation estimate is also attenuated in the expanded-control specifications.
F Generalizability Beyond Main-Conference Papers
Analyses extending beyond Main Conference papers reproduce the paper’s central pattern: reported GPU capability is positively associated with citation impact, but the alignment is limited.
- Sample coverage and resources: 23,838 papers form the combined corpus after adding 9,917 Findings papers to 13,921 Main Conference papers.Findings papers report standardized GPU models and explicit counts more often, but have lower median reported GPU counts and aggregate capability.
- Sample coverage and resources: 58.7% of Findings papers report a standardized GPU model and 42.2% report both a standardized model and explicit GPU count.The corresponding Main Conference rates are 49.6% and 38.5%.
- Replication of citation-impact associations: A tenfold increase in reported GPU capability is associated with a 4.6-percentage-point increase in the primary citation percentile among Findings papers.The corresponding Main Conference estimate is 3.5 percentage points, and the pooled estimate is 3.9 percentage points.
- Replication of citation-impact associations: Incremental R2 values range from 0.0009 to 0.0093 and remain below 0.01 in both publication tracks.These values are numerically larger in the Findings sample for each outcome, but the differences are descriptive.
- Overlap between capability and impact: 15.8% of high-capability Findings papers are highly cited versus 9.0% of other Findings papers, yielding a descriptive risk ratio of 1.75.Among Main Conference papers, the corresponding rates are 14.5% and 9.1%, with a risk ratio of 1.59.
- Overlap between capability and impact: 84.2% of high-capability Findings papers are not in the top 10% of the citation distribution.High-capability papers account for only 30.6% of highly cited Findings papers, so high capability is neither sufficient nor necessary for high citation impact.