Causal Intervention for Abstractive Related Work Generation

1 Introduction

• Prior generators use citation context, citation networks, or target-centered attention to encode inputs and relations between reference papers and the target paper.These approaches have made progress in generating coherent related work.Source: §1 · p. 1
• Existing models can exploit frequent wording and writing habits as shortcuts, which may harm quality and generalizability, especially under train–test distribution shift.Such patterns may not contribute intrinsically to the task and can impair robustness.Source: §1 · p. 1
• Transitional phrases such as “for example” and “later” can reflect writers’ ordering habits rather than semantic relations between referenced papers.When models learn those habits instead of document relations, generated transitions may be unsuitable and affect later content.Source: §1 · pp. 1–2
• Causal intervention examines how sentence order and document relations affect transitional content and the whole generated section, aiming to reduce spurious correlations.The authors argue this lets causal relations exert greater influence on generation.Source: §1 · p. 2
• CaM models sentence order, document relations, and transitional content in a causal graph, then uses intervention components and Transformer fusion to propagate intervened information.The components are Primitive Intervention, Context-aware Remapping, and Optimal Intensity Learning.Source: §1 · p. 2
• CaM outperforms state-of-the-art approaches on two related-work generation datasets, supporting the effectiveness of introducing causality into the task.The authors also describe the model as producing higher-quality related work.Source: §1 · p. 2

2 Problem Formulation

• Related-work generation produces a section Y-hat from reference papers D to minimize its difference from ground-truth section Y, using each reference paper’s abstract as its representation.The passage defines each reference paper and the ground-truth section as word sequences.Source: §2 · p. 2

3 Methodology

• The methodology identifies causal relations in related-work generation, models them with a causal graph, and integrates CaM with Transformer to influence generation.The paper presents the overall model structure in Figure 3.Source: §3 · pp. 2–3, Figure 3 · p. 4

3.1 Causal Modeling for Related Work Generation

• Sentence order can guide transitional content as a writing habit, but models may overprioritize it over document relations, creating unreliable transitions under distribution shift.The proposed graph cuts c →x while preserving the paths from sentence order and document relation to transitional content.Source: §3.1 · p. 3

3.2 Causal Intervention Module for Related Work Generation

• CaM combines Primitive Intervention, Context-aware Remapping, and Optimal Intensity Learning to remove spurious correlations, smooth intervened embeddings, and learn intervention strength.The module is integrated with Transformer, as shown in Figure 3.Source: §3.2 · p. 3, Figure 3 · p. 4

3.2.3 Optimal Intensity Learning

• Optimal Intensity Learning derives intensities for original, intervened, and remapped embeddings to learn the overall intervention strength and produce final embeddings.The paper notes that maximum intervention intensity may conflict with pre-training strategies and need not improve performance.Source: §3.2 · p. 3, §3.2.3 · pp. 4–5

3.2.1 Primitive Intervention

• Primitive Intervention uses do-calculus and the backdoor criterion to cut the c →x path and estimate p(y|do(x)) from sentence-order-specific effects.The method adds sentence-order information to input embeddings before estimating the relevant probabilities.Source: §3.2.1 · pp. 3–4

3.2.2 Context-aware Remapping

• Context-aware Remapping addresses embedding-space mismatch and possible context damage by updating intervened embeddings with a fixed-window attention process.The method scans the sequence with consecutive windows and retains renewed embeddings.Source: §3.2.2 · p. 4

3.3 Fusing CaM with Transformer

• CaM is placed between Transformer decoder layers; a mask preserves non-sentence-start embeddings while retaining intervention on sentence-start words.The model is trained with cross-entropy loss between predicted and ground-truth output.Source: §3.3 · p. 5

4.1 Datasets

• The evaluation uses public S2ORC and Delve datasets, and compares CaM with eight extractive and abstractive baselines; S2ORC spans domains while Delve focuses on computer science.The settings use ROUGE-1, ROUGE-2, and ROUGE-L F1 metrics; baseline results are taken from the RRG paper using the same dataset settings.Source: §4.1 · p. 5, §4.3 · p. 6, §4.2 · p. 6

4.4 Overall Performance

• CaM achieves the best performance on both S2ORC and Delve, outperforming the state-of-the-art approaches.The comparison uses ROUGE scores against extractive and abstractive baselines.Source: Table 2 · p. 6, §4.4 · pp. 6–7, §4.3 · p. 6

4.5 Ablation Study

• PI+RMP+OPT achieves optimal results; PI alone generally slightly reduces performance, while PI+RMP outperforms RMP.The study attributes the full combination’s results to OPT exploiting information across different representations.Source: §4.5 · p. 7

4.6 Human Evaluation

• In human ratings, CaM scores best for informativeness and coherence, with coherence its strongest advantage; its succinctness is slightly below RRG.Fifteen graduate students rated 40 S2ORC samples on three metrics from 0 to 3; the authors planned a larger MTurk evaluation for the final version.Source: §4.6 · p. 7, Table 3 · p. 7

4.7 Fusing Strategy Comparison

• Four CaMs yield the best performance on both datasets; too few may lack continuous intervention, while too many may crowd Transformer learning and introduce noise.CaMs are distributed evenly across Transformer layers, with one always placed at the end.Source: §4.7 · p. 7, Figure 6 · p. 7

4.8.1 Testing with Reordered Samples

• On reordered samples, CaM performs better than Transformer and loses less on ROUGE-1 and ROUGE-2; it also improves ROUGE-L relative to Transformer.The reordered evaluation uses manually rearranged cited-paper and sentence orders, with transitional content removed.Source: §4.8.1 · p. 8, Figure 7 · p. 8

4.8.2 Testing with Migrated Test Set

• When trained on Delve and tested on S2ORC, CaM retains and enlarges its advantage over Transformer despite performance dropping after migration.The comparison uses Relative Outperformance Rate because CaM already outperforms Transformer before migration.Source: §4.8.2 · p. 8, Figure 8 · p. 8

4.9 Causality Visualization

• The visualization contrasts CaM’s attention to contextually relevant cited-paper words for “however” with Transformer’s experience-based generation of “the.”The authors interpret CaM’s attended words as supporting document-relation comparisons and the resulting transition.Source: §4.9 · pp. 8–9, Figure 9 · p. 8

• CaM models relations in related work generation with a causal graph to capture causalities.Source: §5 · p. 9
• CaM is fused with Transformer to integrate intervened information throughout the generation process.Source: §5 · p. 9
• Experiments show CaM’s superiority over the latest models and demonstrate its effectiveness.Source: §5 · p. 9

• CaM intervenes on sentence-start words, making its improvement inconclusive when those words are generated inaccurately.Source: Limitations · p. 9
• The authors report that CaM can improve large-scale or pre-trained models, but its effectiveness with small-scale models trained from scratch might not be ensured.Source: Limitations · p. 9
• Improving CaM’s effectiveness when sentence-start words are inaccurate is identified as future work.Source: Limitations · p. 9

A.1 Related Work Generation

• Recent abstractive generators use citation context, reference-paper relations, or target-paper information to produce related work.Chen et al. (2022) uses target-centered attention and target-representative words; the authors did not compare it as a baseline because it uses target-paper information.Source: §A.1 · p. 11

A.2 Causal Intervention

• Causal methods in recommendation and NLP address cross-scenario information, data bias, counterfactual reasoning, and spurious text correlations.The cited examples include Wang et al. (2022a, 2022b), Feng et al. (2021), and Wang and Culotta (2020).Source: §A.2 · p. 11

• The section provides additional analysis of experiments from Section 4.9.Source: B Experiment Result for Causal… · p. 11

B.1 Generated Related Work Comparison

• CaM generates enriched content whose meaning is closer to the ground truth than Transformer output.The passage describes this as a comparison of generated related works.Source: §B.1 · p. 11
• When introducing problems and new approaches, CaM begins the sentence with “however” and produces a more accurate, seamless transition.Source: §B.1 · p. 11
• Transformer generates the high-frequency word “the” at the same position, which the passage characterizes as preference- and experience-driven rather than an effective decision.Source: §B.1 · p. 11

B.2 Visualization Result Analysis on Full Text

• Figure 10 visualizes cross-attention for “however” in CaM and “the” in Transformer; darker blue marks greater attention to an input source word, and vertical lines separate cited papers.The passage describes CaM’s attention as more targeted and focused on more important information.Source: §B.2 · p. 11, Figure 10 · p. 12
• Transformer’s attention to “the” covers a wider range of information but lacks emphasis, with normal blue covering more of the visualization.The passage characterizes Transformer’s decision process as unclear and ineffective.Source: §B.2 · p. 12