PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM4Impact: Integrating Heterogeneous Information for Scientific Impact Prediction

Yong Cao, Markus Flicke, Haoyu He, Katrin Renz, Andreas Geiger

arXiv:2610.10138v1cs.CL

TL;DR

New-paper impact prediction must rely on heterogeneous evidence available at publication time, whose value varies by paper and context. LLM4Impact integrates and calibrates these sources, outperforming strong baselines across yearly and monthly tasks. The paper also reports a 10.13% reduction in in-distribution five year RMSE.

  • Problem

    Cold-start impact prediction must infer a newly published paper’s future impact from publication-time information, while existing approaches often use one source or combine sources without accounting for their different predictive roles.

    Source: §1 · p. 2

  • Method

    LLM4Impact integrates semantic, heterogeneous graph, and LLM evidence with continuous temporal representation, context-aware gating, and context-conditioned calibration.

    Source: §1 · p. 2

  • Results

    LLM4Impact consistently improves over strong semantic, graph based, and LLM based baselines across year and month prediction; five year RMSE drops 10.13% in distribution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Ablation and perturbation studies indicate that evidence sources provide complementary information and remain useful across changing publication contexts.

    Source: §6 · p. 9

    The study covers a finite set of scientific domains, publication years, and prediction horizons, leaving broader settings for future investigation.

    Source: H LIMITATIONS AND FUTURE WORK · p. 27

Abstract

from arXiv · show

Predicting the future impact of a newly published paper is challenging because it must be inferred from heterogeneous evidence available at publication time. Existing approaches often rely on a single source of information or combine multiple sources without accounting for their different predictive roles. In this paper, we present LLM4Impact, an evidence-aware method for scientific impact prediction that learns to represent, integrate, and calibrate heterogeneous information. LLM4Impact combines semantic, graph, LLM, and temporal representations, and injects graph information into a frozen LLM through continuous prefix tokens. A context aware gating mechanism adaptively weights different evidence, while a separate calibration module accounts for domain and temporal variation in citation scales. We further construct a large-scale benchmark dataset with 2 million papers, leakage-safe point-in-time heterogeneous ego graphs, temporal splits, and both year-level and month-level citation targets. Experiments show that LLM4Impact consistently outperforms strong semantic, graph, and LLM based baselines, with a 10.13% reduction in year RMSE on the in distribution test set and a 6.87% reduction under out-of-domain distribution. Our results reveal that the value of such evidence is context dependent: different papers benefit from different sources, while domain and publication time affect how evidence translates into citations. This finding motivates adaptive evidence selection and context-conditioned calibration rather than simply richer representations. We will release our code, benchmark, and an interactive web demonstration upon publication.

1 INTRODUCTION

Cold-start impact prediction must use evidence available at publication time, but the value of different sources varies across papers and contexts. LLM4Impact integrates and adaptively calibrates heterogeneous evidence, and is reported to outperform strong baselines across prediction horizons and temporal shifts.

1 INTRODUCTION

  • Semantic, graph, temporal, and LLM signals are complementary, but their predictive value varies across papers, domains, and publication periods.The paper argues that prediction requires determining when and how to use each evidence source, not simply adding more information.Source: §1 · p. 2
  • LLM4Impact combines semantic, heterogeneous graph, and LLM evidence, injecting graph information into the LLM through continuous prefix tokens.A continuous temporal encoder supports unseen years; domain- and time-aware gating weights branches per instance, and context-conditioned calibration adjusts for citation-scale variation.Source: §1 · p. 2
  • 10.13% lower five year RMSE in distribution and 6.87% lower under temporal shift; monthly QWK also improves over strong semantic, graph based, and LLM based baselines.LLM4Impact improves monthly QWK by 40.77% in distribution and 24.44% under temporal distribution shift; ablations indicate complementary gains from LLM, graph, and calibration components.Source: §1 · p. 2
  • The paper’s interactive demo supports controlled what-if probing of how changes to authors, venues, references, or publication time affect predicted impact.It holds other information fixed while varying contextual factors.Source: §1 · pp. 2–3
  • The benchmark contains 2 million papers, leakage-safe point-in-time heterogeneous ego graphs, temporal splits, and both five-year and monthly citation targets.Its temporally shifted test set examines generalization to publication years outside the training range.Source: §1 · p. 2

2 RELATED WORK

Prior work predicts scientific impact from bibliometric features, scholarly networks, and textual or LLM-derived signals. These approaches capture different aspects of impact, while network methods underuse textual semantics.

Predicting Scientific Impact

  • Traditional bibliometrics describe established rather than future impact, motivating prediction from early signals; initial handcrafted-feature methods struggled to capture impact dynamics.Later data-driven approaches include temporal citation-growth models and cold-start prediction from pre-publication metadata and bibliographic networks.Source: §2 · p. 3

Heterogeneous Scholarly Networks

  • Heterogeneous scholarly networks represent diverse entities and relations and support multi-scale impact analysis, but network models underuse textual semantic richness.Methods include meta-path analysis, ranking over content and publication networks, and temporal or relational GNNs.Source: §2 · p. 3

LLMs for Scientific Understanding

  • LLMs and scientific-text embeddings capture topical relevance and manuscript qualities associated with later impact, complementing metadata-driven models.The cited work spans content-only impact prediction, identification of highly cited papers, and qualitative evaluations correlated with later citations and Altmetric scores.Source: §2 · p. 3

3 DATASET CONSTRUCTION

The dataset section presents a large-scale scientific impact benchmark built from Semantic Scholar with temporal splits and point-in-time graph features. It includes year- and month-level citation targets, with month-level labels motivating additional evaluation metrics.

3.1 CONSTRUCTION PIPELINE

  • The benchmark samples 2 million eligible query papers and splits them into historical (2005–2017) and fresh (2018–2020) sets to evaluate generalization to more recent papers.Query papers have a non-empty reference list and an available abstract; the sample is drawn without field-based stratification.Source: §3.1 · p. 4, §3 · p. 4, Table 1 · p. 4
  • Each paper’s two-hop ego-graph expands its references with their top-25 cited references, truncating citation counts to values available by the query paper’s publication year.This construction is described as preventing temporal leakage.Source: §3.1 · p. 4

3.2 DATASET STATISTICS

  • The benchmark provides year- and month-level citation targets, and month-level counts remain zero for most papers at every horizon.Because near-zero predictions can have low regression error without distinguishing cited papers, the authors add threshold-sensitive classification metrics for monthly prediction.Source: §3.2 · p. 4, Figure 2 · p. 4

4 METHOD

LLM4Impact addresses cold-start scientific impact prediction by combining structural, semantic, and temporal evidence through multi-branch, context-aware fusion. Its design targets unseen publication times, paper-specific evidence reliability, and domain- and time-dependent citation scales.

4 METHOD

  • LLM4Impact integrates structural, semantic, and temporal signals through multi-branch, context-aware fusion to address cold-start scientific impact prediction.Its three components target unseen publication years, instance-dependent evidence reliability, and domain- and time-dependent citation scales.Source: §4 · p. 5

4.1 TASK FORMULATION

  • Predictions use only graph information available at publication time and estimate cumulative citations over five yearly or monthly horizons.The query paper has no citation history, and neighboring papers use point-in-time citation counts.Source: §4.1 · p. 5

4.2 MULTI-SOURCE REPRESENTATION LEARNING

  • Semantic features encode the title and abstract, while a citation-aware GNN summarizes a two-hop heterogeneous ego-graph into a graph representation.The graph branch uses type-specific projections and attention pooling; training also contrasts clean and popularity-augmented graph views.Source: §4.2 · p. 5

4.3 GRAPH-TO-LLM PREFIX INJECTION

  • Continuous graph-prefix injection into a frozen LLM consistently outperforms textual serialization of the same structural information for citation-impact prediction.The graph representation is projected into the LLM hidden space and prepended to the instruction, title, and abstract in embedding space.Source: §4.3 · pp. 5–6

4.4 DIAGNOSING AND FIXING TEMPORAL ENCODING

  • A continuous, extrapolable time encoder addresses unseen publication years, which lookup embeddings leave untrained in the fresh test split.Error–graph-density relationships are nearly identical across historical and fresh splits; unseen timestamps use the continuous base representation.Source: §4.4 · p. 6

4.5 DOMAIN- AND TIME-AWARE GATED FUSION

  • A gate conditioned on publication time and research domain assigns normalized branch weights, adapting which evidence source to trust for each paper.The context combines the encoded publication time with the mean-pooled category embedding.Source: §4.5 · pp. 6–7

4.6 DECOUPLED PREDICTION CALIBRATION

  • A separate context-conditioned calibration module rescales predictions for domain and publication-time variation, while a composite objective adds auxiliary supervision.The auxiliary terms provide citation-bucket supervision, graph-level contrastive learning, and branch-wise deep supervision.Source: §4.6 · p. 7

5 EXPERIMENT AND ANALYSIS

The experiments show LLM4Impact outperforming strong baselines across yearly and monthly prediction, including under temporal shift. Ablations and controlled perturbations examine the contributions and behavior of its evidence sources, while further analyses explore sensitivity and alternative citation metrics.

5.1 EXPERIMENT SETTING

  • The study compares SciBERT, graph and LLM baselines, and adapted citation-prediction methods, using RMSE for year-level and multiple metrics for month-level prediction.The month-level metrics include AUROC, PR-AUC, QWK, RMSE, and NDCG@20; zero-shot LLMs are reported separately.Source: §5.1 · p. 7, Table 2 · p. 8, Table 3 · p. 8

5.2 MAIN RESULTS

  • Month-level results include a 40.77% QWK improvement in distribution and an 11.73% NDCG improvement under temporal shift.The 7B model achieves the best trained-model performance across reported month-level metrics on both tests.Source: §5.2 · pp. 7–8
  • 10.13% improvement over NAIP in average year-level RMSE on previous test; under temporal shift, the improvement is 6.87%.LLM4Impact reports average RMSE of 0.710 versus NAIP’s 0.790 on previous test and 0.786 average RMSE on fresh test.Source: §5.2 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 8

5.3 ABLATION STUDY

  • Removing the LLM branch raises average RMSE by 12.6% on previous test and 12.9% on fresh test, the largest component ablation effect.Removing calibration raises RMSE by 9.6% and 10.4%; removing the graph branch raises it by 6.6% and 4.2%, respectively.Source: §5.3 · p. 8, Table 4 · p. 8
  • Venue perturbations produce larger prediction effects than author perturbations, while sentence shuffling affects predictions less than content removal or replacement.The controlled studies examine 20 computer science papers and also perturb publication year, references, truncation, and topic.Source: §5.3 · p. 9, Figure 4 · p. 9

5.4 MORE ANALYSIS AND DISCUSSION

  • The interactive demo lets users vary publication year, references, venue, authors, institutions, or research categories while holding other information fixed.Users can compare the resulting citation trajectories across models for controlled what-if analysis.Source: §5.4 · p. 9, Figure 4 · p. 9
  • Predictions from raw citation counts and TNCSI are strongly positively correlated, while the fresh test remains more challenging than the previous test.The analysis compares RMSE and Spearman correlation for the two citation targets.Source: §5.4 · p. 9

6 CONCLUSION

LLM4Impact integrates semantic, heterogeneous graph, LLM, and temporal evidence for cold-start scientific impact prediction. It improves over strong baselines across yearly and monthly tasks, while ablations and perturbations indicate that evidence sources provide complementary information across changing publication contexts.

6 CONCLUSION

  • Table 5 compares RMSE and correlation between raw citation counts and TNCSI on the fresh test set.Source: Table 5 · p. 9
  • LLM4Impact integrates semantic, heterogeneous graph, LLM, and temporal evidence for cold-start scientific impact prediction.The framework combines continuous temporal representation, context-aware evidence gating, and context-conditioned calibration.Source: §6 · p. 9
  • The framework consistently improves over strong baselines across yearly and monthly prediction tasks.The conclusion also reports that ablation and perturbation studies show complementary evidence sources remain useful across changing publication contexts.Source: §6 · p. 9

AI USE STATEMENT

The paper reports its LLM use, reproducibility plans, and benchmark construction. The dataset uses a natural, unstratified S2AG sample, with temporal and domain characteristics described alongside its scale and coverage.

AI USE STATEMENT

  • LLMs were used during early-stage writing for wording, grammar checking, and searching for missing literature, but not in later writing iterations.The authors do not consider LLMs significant contributors to the paper.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

A DATASET CONSTRUCTION AND STATISTICS

  • The benchmark uses 2 million query papers and 39.3 million-node ego graphs, unifying multi-grained impact labels with author, venue, and institution metadata.The authors describe it as the first resource at this scale with these combined features, and explicitly note domain shift from unstratified sampling.Source: A DATASET CONSTRUCTION AND STATISTICS · p. 13, Table 6 · p. 13
  • The source is a single Semantic Scholar snapshot; the query window stops at 2020 partly because field-of-study coverage falls to roughly 32% from 2021 onward.The snapshot contains 237,026,483 papers spanning 1970–2025, and the query set is sampled without field-based stratification.Source: §A.1 · p. 14

A.1 METADATA STATISTICS

  • Medicine tags 75.6% of query papers, compared with 26.6% for Biology, reflecting the sample’s non-uniform, unstratified field distribution.Field shares are multi-label and therefore do not sum to 100%.Source: §A.1 · p. 14, Table 7 · p. 14, Table 8 · p. 14
  • Train, validation, and previous-test splits share the 2005–2017 historical pool, while fresh test consists exclusively of disjoint 2018–2020 publication years.The fresh-test block is separated in publication-year space by construction.Source: §A.1 · p. 15

A.2 SPLIT DATASET DISTRIBUTION

The fresh test has higher citation labels than the overlapping historical splits, reflecting a systematic shift in citation levels. The section also describes the loss components and evaluation metrics used to assess predictions.

A.2 SPLIT DATASET DISTRIBUTION

  • At year 5, fresh test has median 6.0 vs. 5.0 citations and month-5 mean 0.68 vs. 0.43, unlike the overlapping train, validation, and previous-test trajectories.The fresh test is higher at every horizon and both granularities.Source: §A.2 · p. 15, Figure 6 · p. 15
  • Train, validation, and previous test share the 2005–2017 publication-year pool, while fresh test occupies a disjoint 2018–2020 block.Source: Figure 5 · p. 15
  • The later test labels reflect a citation-rate regime shift: newer papers fall in more actively growing subfields amid rising publication volume.The section frames this as a systematic generalization gap, not merely a different test window.Source: §A.2 · p. 15

B LOSS FUNCTION DETAILS

  • The composite objective combines citation regression with auxiliary impact-bucket classification, graph contrastive learning, and branch-specific regression supervision.Direct branch supervision is intended to keep a branch from being ignored by the gate during early training.Source: B LOSS FUNCTION DETAILS · p. 16

C EVALUATION METRICS DEFINITION

  • Evaluation covers five horizons at both year and month granularities, using log(1+x) targets and RMSE plus month-level classification and ranking metrics.Source: C EVALUATION METRICS DEFINITION · p. 17

C.1 YEAR-LEVEL METRICS

  • Year Avg and Month Avg are unweighted means of per-horizon RMSE, while NDCG@20 evaluates top-ranked papers over the complete test split.Global computation reassembles predictions and labels across shards before ranking.Source: §C.1 · p. 17

C.2 MONTH-LEVEL METRICS

Month-level evaluation addresses the many zero-citation labels with discrimination metrics and assesses ordinal agreement across citation-impact levels. Results are reported separately for overlapping historical and later publication periods.

C.2 MONTH-LEVEL METRICS

  • With 85.2% of valid previous-test month labels equal to zero, evaluation supplements regression with binary discrimination metrics.Source: §C.2 · p. 18
  • AUROC compares cited with uncited papers by predicted score, while PR-AUC summarizes precision–recall performance and is informative under severe class imbalance.Scores are reported by month horizon and aggregated across five horizons.Source: §C.2 · p. 18
  • QWK measures ordinal agreement between predicted and realized citation-impact buckets, assigning larger penalties to larger category disagreements.The same fixed thresholds define low, medium, and high categories across models and splits.Source: §C.2 · p. 18
  • Previous test overlaps the training publication period, whereas fresh test contains papers published after the training cutoff and evaluates temporal extrapolation.Source: §C.2 · p. 18

D MORE EXPERIMENTAL RESULTS

The reported ablation improvements are statistically significant, domain-level temporal gaps vary, and supplementary materials provide an interactive demo. The section also describes TNCSI and the limits of its original scope.

D.1 SIGNIFICANCE TEST

  • All ablation comparisons yield p < 0.0001, with 95% confidence intervals above zero on both previous and fresh test sets.Paired bootstrap comparisons use 2,000 corpusid-matched resamples.Source: §D.1 · p. 18, Table 9 · p. 19

D.2 MORE CASE STUDY

  • The interactive demo includes real-paper prediction cases, dataset examples, and views for metadata and abstract perturbations.The case gallery includes wins, typical cases, and losses alongside five baselines and observed citation counts.Source: §D.2 · pp. 18–19

D.3 DOMAIN PERFORMANCE DIFFERENCES

  • Computer Science has the largest fresh-test RMSE gap (+0.13), while History’s gap is essentially zero (−0.00); the pattern holds across 22 domains.Several small-sample humanities domains also have gaps under +0.05.Source: §D.3 · p. 19

E TNCSI CALCULATION

  • TNCSI compares an article’s citation impact within its research field on a 0–1 scale, but its original focus on review papers and cumulative citations limits suitability for new or regular research articles.Source: E TNCSI CALCULATION · p. 19

F TRAINING PARAMETERS

The training setup uses frozen LLM backbones adapted with LoRA and a graph branch that supplies prefix tokens. Prompt materials describe citation prediction, metadata normalization, and interactive examples.

F TRAINING PARAMETERS

  • The full-dataset setup uses frozen Qwen3-0.6B, with a Qwen2.5-7B-Instruct variant for comparison against NAIP, and adapts both with LoRA.The graph branch encodes a 1-hop ego-tree and projects its embedding into 10 soft prefix tokens prepended to the LLM input.Source: F TRAINING PARAMETERS · pp. 19–20
  • Figure 7 organizes full-model RMSE by field and labels the fresh-minus-previous gap; Figure 8 illustrates metadata, references, and year/month citation targets.Source: Figure 7 · p. 20, Figure 8 · p. 21

G PROMPT

  • The citation-prediction prompt combines structural context with the paper’s title and abstract to request cumulative citation counts at future time points.Source: G PROMPT · p. 21
  • Interactive figures compare predicted and observed trajectories and expose metadata and abstract perturbation studies.Source: Figure 9 · p. 22, Figure 10 · p. 23, Figure 11 · p. 24
  • Venue and affiliation normalization prompts standardize formatting and duplicates, while mapping meaningless or nonmatching strings to “unknown.”Source: G PROMPT · pp. 26–27

H LIMITATIONS AND FUTURE WORK

The study is limited to finite domains, publication years, and prediction horizons, leaving broader settings for future work.

H LIMITATIONS AND FUTURE WORK

Loading 2610.10138v1…