Explaining Relationships Between Scientific Documents

1 Introduction

• Document-relationship explanations could reduce literature-review time and support personalized research assistance, writing assistance, and explanations of citation recommendations.These applications are proposed as potential uses of automatically describing inter-document relationships.Source: §1 · p. 1
• Representing important document information, handling long-tailed technical vocabulary, and expressing varied relationships are central challenges for this task.The same document may need different explanations depending on its paired document.Source: §1 · p. 1
• Citing sentences serve as naturally occurring proxy explanations, while the task generates from general document representations rather than specific citation locations.This formulation is intended to facilitate applications such as search and writing assistance.Source: §1 · p. 1
• The paper contributes a relationship-explanation dataset, a domain-adapted GPT-2 model, SCIGEN, and expert evaluation of generated technical text.The authors also explore document views and entity- and term-based representations for conditioning and ranking explanations.Source: §1 · p. 2

2 Related Work

• Unlike citation-intent classification, relationship explanation uses freeform generation to express varied relations and can describe works that do not cite one another.The task is supervised with readily available citing sentences rather than task-specific annotations.Source: §2 · p. 2
• Unlike prior scientific summarization focused on abstracts or reviews, this work learns to express the specific relationship between two documents from citing sentences.The authors describe this as the first work focused on that objective.Source: §2 · p. 2
• Unlike Xing et al. (2020), which inserts a citing sentence into a particular document context, this task describes general relationships independent of citation location.The authors connect this location-independent framing to applications in search and summarization.Source: §2 · pp. 2–3
• The modeling approach applies domain-specific pretraining before fine-tuning GPT-2, while selecting document content to fit its limited context window is a central challenge.The section situates this approach within transfer learning and scientific-document modeling.Source: §2 · p. 3

3 Task Overview

• The task uses in-text citation sentences as proxy explanations for document relationships because explicit examples are difficult to find and technical annotation is costly.The sentences provide partial supervision for generating a natural-language relationship explanation.Source: §3 · p. 3
• For principal document S and cited document C, the model predicts explanation t from S without t and C, maximizing P(t | S′, C).At inference, the goal is to generate a sentence that accurately describes the relationship between the documents.Source: §3 · p. 3
• Because scientific documents exceed GPT-2’s 512- or 1,024-token context window, the paper explores ways to represent document content for generation.The evaluation combines expert human judgments with BLEU and ROUGE automatic metrics.Source: §3 · pp. 3–4
• The dataset contains 622K single-reference explanations from 154K connected computer-science articles in S2ORC.Sentences citing more than one reference are omitted; validation and test sets each contain 5,000 sentences.Source: §3 · p. 3

4 Models

• The model-development setup fine-tunes a pretrained language model on the relationship task, adds scientific-text pretraining, and includes an approximate-nearest-neighbor retrieval baseline.The retrieval baseline seeks plausible human-authored explanations from training data.Source: §4 · p. 4

4.1 Neural Text Generation

• SCIGEN concatenates selected principal- and cited-document tokens, separated by a marker, and predicts the explanation one token at a time.The model is fine-tuned on this paired-document context; at inference it generates explanations with nucleus sampling.Source: §4.1 · p. 4
• The generation experiments compare abstracts, introductions, and sampled non-citing sentences as context, then explore representing the cited document with important concepts.These context choices address the limited window available for long scientific documents.Source: §4.1 · p. 4
• SCIGPT2 continues GPT-2 pretraining on scientific texts, and replacing GPT-2 with SCIGPT2 significantly improves output quality and held-out perplexity.The additional training uses 75,000 updates over 4.8 million abstracts and body paragraphs; qualitative improvements were also observed.Source: §4.1 · p. 4

4.2 Retrieval with Approximate Nearest Neighbors

• The retrieval baseline finds nearby training document pairs using cosine distances between SciBERT-encoded abstract vectors, then returns the associated human-authored citing sentence.The distance weights can be set equally or tuned on validation BLEU with MERT.Source: §4.2 · p. 5

5.1 Automatic Evaluation

• The principal-document introduction yields a slight, statistically significant automatic-metric improvement over its abstract, while cited-document representations show no clear winner among abstract, introduction, and random sample.The introduction-versus-abstract difference has p < 0.01 after Bonferroni correction.Source: §5.1 · p. 5

5.2 Human Evaluation

• The study compares the principal introduction × cited abstract SCIGEN setting with tuned retrieval and gold explanations using expert judgments of correctness and specificity.Thirty-seven NLP researchers provided over 800 judgments; gold explanations were included for calibration.Source: §5.2 · p. 6
• The retrieval systems tend to produce incorrect explanations more often than not.The automatic evaluation selects the principal introduction paired with the cited abstract for human evaluation, although retrieval scores slightly better in some settings.Source: §5.1 · p. 5, §5.2 · p. 6
• Most SCIGEN outputs are judged correct, but their specificity falls further behind gold explanations than their correctness does.Judges often found SCIGEN generations vague or generic, unlike gold texts, which scored highest across text-quality dimensions.Source: §5.2 · p. 6

6 Using IE-Extracted Term Lists

• Generated explanations often omit specific model or dataset names because these phrases tend to appear beyond abstracts and introductions, where naive full-text sampling misses them.The paper turns to information extraction to construct denser cited-document context.Source: §6 · p. 6
• The cited-document term list uses tf-idf scores for unigrams and entities extracted with a scientific named-entity recognition system.The approach uses document-specific salient words and phrases as conditioning context for SCIGEN.Source: §6 · pp. 6–7

6.1 Entity-Based Ranking

• SCIGEN generates candidate explanations, extracts their entities, and selects the candidate with the highest mean reciprocal rank against entities from the cited document.Soft alignments account for textual variation when matching candidate entities to input entities.Source: §6.1 · p. 7

6.2 Manual Analysis

• The IE-based model shows only a small correctness increase over the sentence-based model in manual analysis.Two authors assessed 50 datapoints from each model using a three-way correctness scale.Source: §6.2 · p. 7

6.3 Automatic Evaluation

• Intro × tfidf contexts outperform sentence-based, retrieval, and other IE-based models on automatic metrics, with a slight advantage for entities over terms.These gains occur when the principal document is represented by its introduction and the cited document by a term or entity list.Source: §6.3 · p. 7

• One example correctly links two papers through their shared use of pinyin for Chinese-character transliteration.Source: §7 · p. 8
• A topical match can fail when a paper analyzes discourse-relation identification but does not train a model.Source: §7 · p. 8
• A “Too Vague (but not incorrect)” explanation mentions distant supervision but misses how the principal paper uses the cited work.Source: §7 · p. 8
• Citation-sentence proxies underrepresent concurrent work, while explanations of multi-hop relationships are missing from the training data.The discussion notes that contemporaneous papers cite each other less often.Source: §7 · p. 8
• Table 5 presents document titles and SCIGEN outputs, including a final example whose papers do not cite each other.Source: Table 5 · p. 8, §7 · p. 8
• Citation sentences can summarize the cited work or be too specific, motivating better-curated training sets for relationship explanations.Source: §7 · p. 8
• SCIGEN usually produces syntactically correct, topical explanations despite the difficulty of the domain’s vocabulary.The discussion says this is consistent with findings on domain-specific language models.Source: §7 · pp. 8–9
• Factual accuracy remains difficult to enforce, especially when language-model inference uses sampling; knowledge graphs are proposed as a possible aid to modeling cited documents and improving outputs.The discussion also suggests knowledge graphs could provide distant supervision for concurrent work.Source: §7 · p. 9

• The conclusion presents explaining relationships between scientific texts as a task connected to facilitating researcher productivity.Source: §8 · p. 9
• The authors use a large, publicly available scientific-document dataset to train a domain-adapted left-to-right language model for text generation.Source: §8 · p. 9
• They compare abstracts, introductions, sampled sentences, and informative terms or entities through human and automatic evaluations.Source: §8 · p. 9

A Training Details

• SCIGPT2 received 75k additional pretraining steps on 4.8 million abstracts and body paragraphs, followed by 10-epoch finetuning to build SCIGEN variants.The pretraining batch size was 64, and the finetuning took approximately 100k steps.Source: A Training Details · p. 12

B Dataset Construction

• The dataset pipeline extracts principal context, cited context, and citation-sentence triplets, then concatenates the contexts and target sentence with separator tokens.Citation sentences are truncated to 100 tokens and contexts to 450 tokens.Source: B Dataset Construction · p. 12

D Human Evaluation Details

• Human evaluation used 37 NLP researchers, with each judge assessing 15 datapoints per quality and system outputs assigned to avoid showing one judge multiple systems for a datapoint.Judges could mark uncertain examples with “?” or skip them; those responses were ignored.Source: D Human Evaluation Details · p. 12

E Validity of Human Judgments

• Judges sometimes accepted explanations paired with mismatched documents, but at a lower rate than when both documents were correct.The mismatched pairs had topical coherence, and the study found no indication that either document exerted a stronger influence on correctness judgments.Source: E Validity of Human Judgments · pp. 12, 14

F Further Detail on Automated Metrics Results

• The no-principal × cited-abstract model makes retrieval decisions without principal-document information, demonstrating the importance of relational information.Table 9 reports automated evaluation for all systems.Source: F Further Detail on Automated Metrics… · p. 14

G Error Analysis

• 41.1% of gold explanations with grammatical errors were classified as incorrect, compared with 25.4% of those without errors.The authors say this may partially explain why evaluators rated some gold sentences incorrect.Source: G Error Analysis · p. 14

H Analysis of Token-Wise Overlap

• The best-performing SCIGEN systems, with or without ranking, used context representations with the largest token-wise overlap with gold sentences.The authors caution that larger overlap does not guarantee better performance.Source: H Analysis of Token-Wise Overlap · p. 14, Table 10 · p. 15

I Oracle Study

• With 20 independently generated samples per document pair, an ideal ranker could reach a BLEU score of 19.50, while the proposed ranker remained far below that bound.The proposed ranking mechanism achieved higher BLEU in some cases, motivating further work on ranking.Source: I Oracle Study · pp. 14–15, Figure 5 · p. 14

J Auto-Completion

• Providing SCIGEN the first three words of a relationship sentence raised the generated continuation's BLEU score to 21.38.The authors suggest this could support more personalized relationship sentences.Source: J Auto-Completion · p. 15

K Explaining without Citations

• The paper presents SCIGEN explanations for concurrent NAACL 2018 paper pairs that did not cite each other, without supervision from citations between those papers.The examples include post-processing that replaces first-person pronouns with “principal.”Source: K Explaining without Citations · p. 15, Table 11 · p. 15