PaperlayerView exact revision on arXiv

Source-linked AI summary

SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

Amanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey, Sergey Feldman

arXiv:2211.13308v4cs.CLcs.AIcs.IRcs.LG

TL;DR

Scientific document-representation benchmarks have not captured the diversity of tasks needed to assess generalization. The paper introduces SciRepEval and uses it to develop format-specific document embeddings, finding that this approach improves performance over single-embedding methods.

  • Problem

    Existing benchmarks provide limited evidence about scientific document-representation performance across diverse task formats.

  • Method

    The paper introduces SciRepEval and trains multi-format representations using format-specific objectives, control codes, and adapter modules.

  • Results

    Format-specific representations substantially improve generalization, while existing models struggle across task types and simple multi-task training does not significantly improve them.

  • Takeaways & Limitations

    SciRepEval provides a diverse standardized basis for evaluating scientific document representations, and SPECTER2 supplies representations tailored to different task formats.

  • Takeaways & Limitations

    The tasks and training techniques depend on paper titles and abstracts, and performance may be subpar when abstracts are absent.

Abstract

from arXiv · show

Learned representations of scientific documents can serve as valuable input features for downstream tasks without further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diversity of relevant tasks. In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. It includes 24 challenging and realistic tasks, 8 of which are new, across four formats: classification, regression, ranking and search. We then use this benchmark to study and improve the generalization ability of scientific document representation models. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats, and that simple multi-task training fails to improve them. However, a new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance. We experiment with task-format-specific control codes and adapters and find they outperform the existing single-embedding state-of-the-art by over 2 points absolute. We release the resulting family of multi-format models, called SPECTER2, for the community to use and build on.

1 Introduction

Existing scientific document-representation benchmarks do not adequately test diverse task generalization. SciRepEval addresses this gap and shows that format-specific representations improve generalization where existing models and simple multi-task training struggle.

  • Existing benchmarks focus on document similarity or highly correlated tasks, limiting evaluation of scientific document-representation generalization.
  • SciRepEval provides 24 practical tasks across classification, regression, proximity-based ranking, and ad-hoc search, including eight new tasks.
  • SPECTER and SciNCL struggle to generalize across multiple task types, while simple multi-task training does not significantly improve them.
  • SPECTER2 learns multiple document representations tailored to different task formats rather than relying on one representation for all tasks.
  • Format-specific representations improve generalization and outperform prior work by over 2 points absolute.

2 Background

Scientific document representations build on domain-pretrained language models and citation-based contrastive learning. This work extends the setting toward multi-task, format-specific embeddings evaluated directly as downstream features.

  • Scientific document encoders commonly use language models pretrained on scientific corpora as foundations for dense document embeddings.
  • SPECTER and SciNCL fine-tune SciBERT with triplet loss based on citation relationships, and both are evaluated on SciDocs.
  • SciDocs has important limitations, motivating SciRepEval’s more challenging and diverse tasks for cross-task generalization.
  • SciRepEval trains representations across eight large scientific datasets spanning four task formats instead of dividing tasks primarily by semantics.
  • Format-specific representations are explored using control codes or separate parameter-efficient adapter modules.

3 SciRepEval

SciRepEval is a standardized benchmark of diverse scientific document-representation tasks spanning four formats and realistic use cases. It combines existing and new datasets, with training tasks separated from held-out evaluation tasks.

  • SciRepEval covers classification, regression, proximity-based ranking, and ad-hoc search across 24 scientific document-representation tasks.
  • The benchmark emphasizes diverse, practical tasks and standard training and evaluation datasets to support comparisons between methods.
  • SciRepEval includes SciDocs but adds realistic search, author-disambiguation, and paper-reviewer-matching tasks.
  • Datasets with at least 200,000 instances are used for training, while the remaining datasets are reserved for evaluation.
  • Ad-Hoc Search: Ad-hoc search ranks candidate papers for short textual queries, using TREC-CoVID and NFCorpus for testing and a Semantic Scholar clickthrough dataset for training.
  • Proximity: Proximity tasks rank candidate papers against paper queries and support retrieval, recommendation, paper similarity, and author disambiguation applications.
  • Classification and Regression: Classification uses embeddings with linear support vector classifiers and binary or macro F1 evaluation, while regression uses linear support vector regression and Kendall’s τ.

4 Multi-format representation learning

The paper proposes learning distinct document embeddings for each task format to balance generalization with specialization. It implements this idea through control codes or adapters, trained with format-appropriate objectives.

  • 4 Multi-format representation learning: A single embedding may be insufficient across diverse tasks, while task-specific embeddings limit generalization and increase storage costs.The proposed middle ground is one embedding per task format rather than one per task or one for every document.
  • 4 Multi-format representation learning: Format-specific representations are produced using either a control code associated with the format or adapter blocks attached to the model.Figure 1 presents these as the two mechanisms for generating multi-format embeddings.
  • 4 Multi-format representation learning: The model learns an embedding for each (paper, format) pair from labeled data covering ad-hoc search, proximity, classification, and regression.These embeddings support lightweight classifiers and regressors as well as nearest-neighbor tasks on held-out training and new tasks.
  • 4.1 Model: The base model is a pretrained transformer encoder that receives a document’s title and abstract separated by a [SEP] token.The model is further trained with objectives suited to the four task formats.
  • 4.1 Model: Control codes prepend a special format token, whose final-layer embedding becomes the document representation supplied to the task-specific head.The alternative adapter approach uses separate network modules for each task format.
  • 4.2 Training: Training uses task-heterogeneous batches with linear heads for classification and regression, applying cross-entropy, binary cross-entropy, or mean squared error as appropriate.The linear heads are discarded after training.
  • 4.2 Training: For proximity and ad-hoc search, each training example is a query, a higher-scoring positive candidate, and a lower-scoring negative candidate optimized with triplet margin loss.The loss uses Euclidean distance between query and candidate embeddings, with margin ϵ set to 1 based on preliminary experiments.
  • 4.2 Training: The search setup additionally provides publishing venue and year as input information.This detail applies specifically to the Search task.

5 Experiment Setup

The experiments train multi-format models on large in-train tasks and compare control-code, classification-only, adapter, and fusion-based setups. Search and proximity data are converted into sampled triplets for training.

  • Training Data: Multi-format models are trained on 8 large in-train tasks, with up to 5 proximity or ad-hoc search instances created per query.Positive and negative papers are sampled from each query’s candidate pool.
  • Training Data: Training is capped at 600K samples per task, producing 3.27M training instances and 446K validation instances.Using smaller samples capped at 400K per task yielded relatively poor performance.
  • Training Data: The released training data includes 6.2M triplets and 176K validation triplets, including SciNCL triplets as a subset.The triplets use candidate papers cited by a query paper’s citations but not by the query itself.
  • Transformer Baselines: The study compares MTL CTRL, MTL CLS, BERT PALs, and Fusion architectures, with control-code and adapter methods producing multiple representations per document.MTL CTRL uses control codes, whereas MTL CLS produces a single shared representation.

6 Results

Multi-format representations outperform single shared embeddings on the diverse SciRepEval benchmark, with control codes and adapters combining strong performance and efficiency. The format-specific approach remains effective across alternative base models and aligns best with corresponding task formats.

  • Overall Results: Multiple-representation approaches outperform the single-embedding MTL CLS model by 1.4 to 2 points on average.Pre-fine-tuned multi-format SPECTER2 variants outperform baseline models, while SPECTER2 Base and MTL CLS are on par.
  • Efficiency: Control codes and adapters are the most efficient approaches in model size and computation runtime.Control codes require one new embedding per format without affecting training time; adapters train fewer parameters than PALs.
  • Evaluation Protocol: Table 2 separates citation-trained Base, single-embedding MTL CLS, format-specific MTL CTRL and adapter variants, and their averaged ensemble.Results for trained models report means and standard deviations across five random-seed runs, with ANOVA and Tukey testing at α = 0.05.
  • Ensembling: Averaged control-code and adapter embeddings consistently outperform the individual models across in-train, out-of-train, and overall-average settings.Concatenating the embeddings produced similar results but doubled the embedding size.
  • Alternative Base Models: The task-format-specific advantage persists when SPECTER and SciNCL replace the original base model.MTL CTRL and adapters still substantially outperform MTL CLS across these alternative bases.
  • Cross-Task Analysis: Table 3’s diagonal contains the best results for corresponding formats in both in-train and out-of-train tasks.This pattern indicates that format-based partitions produce representations suited to their corresponding formats.

7 Analyses

Format-specific representations outperform randomly grouped partitions, while combined training helps mainly related formats. SPECTER2 also presents efficiency trade-offs and achieves a new state of the art on MDCR, with a transductive caveat.

  • Task-format specialization: Format-specific control codes produced the best results for tasks matching their trained format, both in-train and out-of-train.The evaluation sampled one in-train and one out-of-train task per format.
  • Task-format specialization: Over 2.7 points on average favored task-format partitions over random partitions across in-train and out-of-train tasks.The advantage held across the evaluated task formats.
  • Multi-format training: Combined training on proximity and ad-hoc search improved performance only on related tasks, whereas training on all tasks performed better on average across formats.
  • Efficiency: Control codes require one new embedding per format without affecting training time, while PALs, Fusion, and ensemble variants add computation costs.Adapters have similar runtime to MTL CTRL and slightly lower task performance than the more expensive ensemble.
  • MDCR evaluation: The multi-task format-based models established a new state of the art on MDCR, with Adapters yielding the best results.However, 23% of dataset papers appeared in SPECTER2 training data, potentially giving the model a transductive advantage.

8 Conclusion

The paper introduces SciRepEval, a 24-task benchmark spanning four scientific document-representation formats. It finds that separate representations for each format substantially improve performance over a single representation, while identifying open evaluation and efficiency questions.

  • SciRepEval: SciRepEval evaluates scientific document representations on 24 tasks spanning classification, regression, ranking, and search.
  • Conclusion: Learning a separate document representation for each task format substantially improves task performance compared with learning a single representation for all tasks.
  • Future work: Future work should examine partitioning schemes beyond task format, higher-fidelity metrics, and accuracy relative to computational and storage cost.

Limitations

The paper’s benchmark and models have several scope and evaluation limitations, including dependence on short textual inputs, incomplete benchmark coverage, and constrained task partitioning. Some evaluations also rely on static, silver-labeled, or reduced datasets and additional metadata or infrastructure.

  • Dependence on short text features: Performance may be subpar when a paper lacks an abstract, because the models depend on titles and abstracts as textual features.SPECTER2 Base processes titles alone for 5% of papers during pre-training to improve robustness.
  • Dependence on short text features: The benchmark does not evaluate full-text methods, since full text is less consistently available and increases transformer computational costs.The authors suggest exploring full-text methods and supplementing textual features with metadata in future work.
  • Potential benchmark extensions: Several benchmark components have limited coverage or staleness: Fields of Study has few manually annotated samples, while some OpenReview tasks use papers from 2017–2022.The authors propose increasing gold labels and periodically updating static task data.
  • Other task partitioning methods: Alternative task partitions may outperform the format-based partition used for multi-format training, which did not produce the best overall benchmark results.The authors report that incorporating Same Author Detection into proximity tasks worked better than maintaining a separate author-based format.

B Implementation details

The implementation assigns separate task-format signals and trains multi-format models using format-appropriate objectives, control codes, or adapters. Evaluation compares these approaches with single-embedding multi-task learning and ASPIRE on relevant tasks.

  • Control codes: Tasks sharing a format share task-format-specific parameters during pre-training, while four special tokens provide format signals.Random initialization performed best on average among the tested initialization schemes.
  • Control codes: Four control codes map classification, regression, proximity, and ad-hoc search to distinct input formats.The codes are [CLF], [RGN], [PRX], and [QRY], with query/document inputs for ad-hoc search.
  • Adapters: Adapter variants attach one adapter per task family, whereas PALs add one layer per task format before training the network.Single-adapter and fusion stages use separate training durations and configurations.
  • Downstream evaluation: Classification and regression downstream evaluation fits a linear SVM, tuning regularization C by grid search and using one-vs-all settings for multi-class and multi-label tasks.
  • Comparisons: Multiple-embedding control-code and adapter approaches outperform simple multi-task learning with a single document embedding across the benchmark.ASPIRE results are reported only on proximity tasks because that is the setting for which its representations were designed.

E SciRepEval Domain Distribution and MDCR Evaluation

SciRepEval covers substantially more domain data than the SPECTER citation-triplet training set, and its task relationships vary across the benchmark. On MDCR, task-format-based models establish strong results across most scientific fields.

  • Domain distribution: SciRepEval contains 105x more documents on average per domain than the SPECTER citation triplets.Medicine and Computer Science remain the largest domain groups in SciRepEval.
  • MDCR evaluation: MDCR results show that task-format-based models outperform BM25 and establish the new state of the art.
  • MDCR evaluation: The ensemble model matches or exceeds BM25 across all reported scientific domains except Geology and History.
  • Ablation: Removing the citation objective lowers in-train performance from 61.9 to 61.8 and out-of-train performance from 57.9 to 57.5.The ablation therefore reports a larger drop for out-of-train evaluation.
  • Task relationships: Task-performance correlations span a wider range across all SciRepEval tasks than among the tasks in SciDocs.DRSM, Biomimicry, and S2AND are uncorrelated with most other tasks.

H Related Tasks for MTL Training

The paper frames multi-task training around task relatedness because negative transfer can make task combinations harmful. Exhaustively testing all combinations is computationally prohibitive, motivating selective or broader pre-fine-tuning strategies.

  • Task relatedness: Multi-task training requires task combinations that avoid negative transfer.
  • Task relatedness: Testing all 2^T − 1 combinations for T tasks may be computationally prohibitive.
  • Task relatedness: Prior work suggests pre-fine-tuning on many tasks simultaneously can offset negative transfer among subsets of those tasks.
  • Evaluation: Task-relatedness analysis compares training on one or more task formats with MTL CTRL using per-format and overall average performance.
  • Evaluation: Figure 2 reports Pearson correlations between task-performance metrics, with SciDocs showing the highest within-benchmark correlations.
Loading 2211.13308v4…