Source-linked AI summary
Target-aware Abstractive Related Work Generation with Contrastive Learning
Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Rui Yan, Xin Gao, Xiangliang Zhang
TL;DR
Related-work generation needs to compare a target paper with its references, but existing methods rely on comparative sentences that are hard to obtain. TAG combines target-aware graph encoding, keyphrase-guided hierarchical decoding, and contrastive objectives; it outperforms strong baselines on two scholar datasets in automatic and tailored human evaluations.
Problem
Existing related-work methods rely on comparative sentences written in advance and hard to obtain, while prior multi-citation generation does not incorporate the target paper.
Source: §1 · p. 1Method
TAG encodes target–reference relationships in a target-centered graph, decodes hierarchically with keyphrases as semantic indicators, and uses contrastive objectives to favor references over non-references.
Source: §1 · p. 2Results
TAG outperforms state-of-the-art baselines on two scholar datasets in ROUGE metrics and tailored human evaluations.
Source: §1 · p. 2Takeaways & Limitations
Generated related work can serve as a draft that contextualizes a target paper among references and reduces authors’ writing time and effort.
Source: §1 · pp. 1–2In the image-retrieval case study, all models make mistakes when describing differences between the target work and related works.
Source: §5.3 · p. 9
Abstract
from arXiv · showhide
The related work section is an important component of a scientific paper, which highlights the contribution of the target paper in the context of the reference papers. Authors can save their time and effort by using the automatically generated related work section as a draft to complete the final related work. Most of the existing related work section generation methods rely on extracting off-the-shelf sentences to make a comparative discussion about the target work and the reference papers. However, such sentences need to be written in advance and are hard to obtain in practice. Hence, in this paper, we propose an abstractive target-aware related work generator (TAG), which can generate related work sections consisting of new sentences. Concretely, we first propose a target-aware graph encoder, which models the relationships between reference papers and the target paper with target-centered attention mechanisms. In the decoding process, we propose a hierarchical decoder that attends to the nodes of different levels in the graph with keyphrases as semantic indicators. Finally, to generate a more informative related work, we propose multi-level contrastive optimization objectives, which aim to maximize the mutual information between the generated related work with the references and minimize that with non-references. Extensive experiments on two public scholar datasets show that the proposed model brings substantial improvements over several strong baselines in terms of automatic and tailored human evaluations.
1 INTRODUCTION
Related work generation aims to contextualize a target paper among its references, but existing approaches rely on comparative sentences that are difficult to obtain. TAG generates abstractive, target-aware related work and outperforms state-of-the-art baselines on two datasets in automatic and tailored human evaluations.
1 INTRODUCTION
- Related work generation produces a draft that contextualizes a target paper among cited papers and can reduce authors’ writing time and effort.The section compares and connects the current work’s innovation and novelty with previous studies.Source: §1 · p. 1
- Existing methods rely on comparative sentences written in advance, which are hard to obtain, while prior multi-citation generation omits the target paper.Without target-paper information, generated text cannot describe how the target relates to cited works.Source: §1 · pp. 1–2
- TAG uses a target-aware graph encoder, a keyphrase-guided hierarchical decoder, and contrastive objectives that favor references over non-references.The encoder models direct paper-to-paper and keyphrase-mediated information flow.Source: §1 · p. 2
- TAG outperforms state-of-the-art summarization and related-work baselines on both scholar datasets in ROUGE metrics and tailored human evaluations.Human evaluations also find most TAG outputs more constructive than baseline outputs.Source: §1 · p. 2
2 RELATED WORK
Prior related-work generation methods range from extractive selection to abstractive generation, but limitations around advance-written comparisons and target-paper information remain. Related work generation also differs from multi-document summarization because it must identify individual contributions and arrange them comparatively.
2 RELATED WORK
- Extractive methods select sentences from references or citing papers, but comparative discussion still depends on sentences scientists must write in advance.The cited approaches include topic-biased sentence selection and extraction of citation sentences.Source: §2 · p. 2
- Abstractive models generate new text, but single-citation models cannot handle multiple citations, while a multi-citation model omits target-paper information.Without that target information, the model cannot generate comparative discussion about the target paper.Source: §2 · p. 2
- Unlike multi-document summarization, related-work generation must identify individual papers’ contributions and arrange them in comparative discussion.Multi-document summarization instead synthesizes similar information and removes redundancy.Source: §2 · p. 3
- This paper’s contrastive learning uses citation-graph relatedness for generating comparative discussion, differing from prior contrastive-learning applications.The passage contrasts this use with work on sentence representations, code summarization, and paper representations.Source: §2 · p. 3
3 THE PROPOSED TAG MODEL
TAG formulates related-work generation from a target paper and its references, then combines a target-aware graph encoder, hierarchical decoder, and contrastive learning to produce contextual comparisons.
3.1 Problem Formulation
- The task uses the target paper’s abstract and its references’ abstracts to generate related-work text, trained against a ground-truth related-work section.The target and reference abstracts represent the input papers; the difference between generated and ground-truth text supplies the training signal.Source: §3.1 · p. 3
3.2 Model Overview
- TAG encodes target–reference relationships directly and through keyphrases, then uses keyphrase-guided hierarchical decoding and contrastive learning to generate informative comparisons.The model seeks higher mutual information between generated text and related papers than between generated text and unrelated papers.Source: §3.2 · p. 3
3.3 Target-aware Graph Encoder
- The graph centers on the target paper, linking it to references and connecting references through keyphrases to model global and topic-specific information flow.Keyphrase representations are updated through self-attention and cross-attention with adjacent paper nodes.Source: §3.3 · pp. 3–4
3.4 Hierarchical Decoder
- The hierarchical decoder first attends to keyphrases, then uses their context to select relevant content from the target paper and references for vocabulary prediction.It concatenates the resulting source context vectors with the decoder state.Source: §3.4 · p. 5
3.5 Contrastive Learning
- Local and global matching objectives use mutual information to favor generated text that matches related papers over unrelated works.The local network matches individual paper samples with generated text, while the global network compares pooled representations.Source: §3.5 · pp. 5–6
4 EXPERIMENTS
The experiments evaluate target-aware related-work generation on two scholar datasets using automatic and human assessments. TAG outperforms the compared baselines on both types of evaluation.
4.1 Datasets
- TAS2 contains 107,700 training, 5,000 validation, and 5,000 test cases; TAD contains 208,255 training, 5,000 validation, and 5,000 test cases.The datasets average 4.26 references per TAS2 case and 5.10 per TAD case; 57 of 60 sampled cases matched original paper content.Source: §4.1 · p. 6
4.2 Baselines
- Baselines include extractive and abstractive systems; their with.Target versions take target and reference papers as multiple inputs without distinguishing them.The comparison includes LexRank, NES, BertSumEXT, BertSumABS, MGSum, EMS, and RRG.Source: §4.2 · p. 6
4.3 Evaluation Metrics
- Automatic evaluation uses full-length F1 for ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-SU; human evaluation tests factual retention and rates informativeness, coherence, and succinctness.Three computer-science PhD students assessed 40 TAD cases; the human-evaluation criteria use a 1–3 scale.Source: §4.3 · pp. 6–7
4.5 Experimental Results
- 4.66% on ROUGE-1 over RRG w.T on TAS2 and 7.00% on ROUGE-L on TAD are reported improvements for TAG; all improvements over the best baseline are significant (p-value < 0.05).The authors also report that TAG outperforms previous abstractive models on all metrics.Source: Table 2 · p. 7, §4.5 · p. 8
4.4 Implementation Details
- Experiments initialize the graph module’s Transformer with BERT base and consider five reference papers and 20 keyphrases per generation.The setup uses hidden size 768, six attention heads, and beam search of size 5.Source: §4.4 · p. 7
4.5 Experimental Results
- TAG answers 4.4% more questions than RRG w.T, scores 0.17 higher on Informativeness and 0.07 higher on Coherence, and leads on all human-evaluation metrics.Average Kappa statistics were 0.42, 0.45, and 0.41 across the evaluations, indicating moderate annotator agreement.Source: Table 3 · p. 7, §4.5 · p. 8
5 ANALYSIS AND DISCUSSION
The analyses support contrastive learning’s utility both within TAG and as an add-on, while ablations associate graph and keyphrase components with better scores. Case and visualization analyses show useful topic coverage alongside errors describing the target-work distinction.
5.1 Ablation Study
- Removing the graph encoder drops TAS2 ROUGE-1 by 1.14; removing the hierarchical decoder drops TAG ROUGE-L by 0.72, while removing contrastive learning causes substantial drops across all metrics.The authors associate these ablation results with target-centered graph interaction, keyphrase guidance, and information from negative samples.Source: §5.1 · p. 8
5.3 Case Study
- All models capture image-retrieval topic details, but each misstates the target contribution; TAG says spatial relations, while the ground truth combines spatial and textural features.The error analysis describes unfaithful target-work distinctions as a common summarization problem.Source: Figure 6 · p. 9, §5.3 · p. 9
5.2 Effectiveness of add-on Contrastive Learning
- 28.78 to 29.19 ROUGE-1 for EMS w.T and 26.46 to 26.96 ROUGE-L for RRG w.T on TAS2 with contrastive learning.Both baselines perform substantially better with the add-on module, though both still underperform TAG.Source: Figure 7 · p. 9, §5.2 · p. 9
5.4 Visualization Analysis
- The local matching loss converges as matching improves; decoding attention shifts from “network proximity” early to “load balancing” late, aligning topic coverage with the target distinction.In the visualization, the x-axis is decoding step and the y-axis is selected keyphrases.Source: Figure 8 · p. 9, §5.4 · p. 10
6 CONCLUSION
TAG combines target-centered graph encoding and target-guided hierarchical decoding with contrastive learning. Experiments show it outperforms the latest baselines in automatic and tailored human evaluations.
6 CONCLUSION
- TAG combines target-centered graph encoding, target-guided hierarchical decoding, and contrastive learning, and outperforms the latest baselines in automatic and tailored human evaluations.The paper identifies a multilingual related work generation system as future work.Source: §6 · p. 10