Explaining Relationships Among Research Papers

• Researchers need concise, customized summaries of new papers because keeping up with rapid publication growth requires curating, reading, and digesting each feed.Source: §1 · p. 1
• Prior citation generation often produces one citation sentence in isolation, missing relationships among cited papers and the expository transitions needed for a coherent multi-document story.Source: §1 · p. 1
• A feature-based LLM-prompting approach generates citation texts and transition sentences at paragraph level, using extracted related-work sections as evaluation targets.Source: §1 · p. 2
• The paper proposes features about relationships between cited and citing papers and among co-cited papers, composed into one prompt to generate multiple citations and transitions.Source: §1 · p. 2
• The study also tests a human-provided plan of high-level relationships to guide citation organization in generated paragraphs.Source: §1 · p. 2
• Expert evaluation found a strong correlation between human preference and integrative writing, suggesting readers prefer abstract citations connected by transitions into an overall story.Source: §1 · p. 2

• Extractive approaches concatenate salient sentences, limiting coherence, stylistic variation, transitions, and links back to the target paper.Source: §2 · p. 2
• Abstractive approaches typically generate one citation at a time from cited-paper abstracts, assuming the rest of the target paper and related-work section are available (AbuRa’ed et al., 2020; Xing et al., 2020; Ge et al., 2021; Luu et al., 2021; Li et al., 2022).Source: §2 · p. 3
• Chen et al. (2021) generate multiple citations using a graph network, whereas this paper uses interpretable natural-language features for paper content, relationships, citation discourse roles, and writing style.Source: §2 · p. 3
• Prior methods use different datasets and slightly different task definitions, so there is no standard benchmark for direct comparison (Li and Ouyang, 2022).Source: §2 · p. 3

3 Approach

• Multi-stage prompting extracts features from cited papers before composing a prompt for a paragraph of citations with expository and transition sentences.Source: §3 · p. 3

3.1 Citation Network Features

• The citation network represents papers as nodes and relationships as edges, using faceted paper summaries and natural-language descriptions of relationships and citation intent or usage.Faceted summaries capture objectives, methods, findings, contributions, and keywords; enriched citation intent distinguishes why a paper is cited and whether it is dominant or reference-type.Source: §3.1 · pp. 3–4

3.2 Target Paper Features

• The target paper’s title, abstract, introduction, and conclusion provide reader-perspective context, while a human-provided plan summarizes the main ideas to guide generation.The plan is derived from the gold related-work section with citations omitted; automatic plan generation is left to future work.Source: §3.2 · p. 4

3.3 Related Work Paragraph Generation

• The prompt can generate a paragraph, subsection, or full review in one pass; section-level generation is most robust, but input-length limits sometimes require manually chunking and concatenating output.Source: §3.3 · p. 4

3.4 Enhancing Details with CTS

• To add detail beyond summaries, the system retrieves cited text spans using a newly generated candidate citation span as query, ranks cited-paper sentences by average ROUGE-1 and ROUGE-2 recall, then augments the prompt and regenerates the paragraph.Source: §3.4 · p. 5

4 Experimental Settings

• Because no prior baseline can generate long texts for this task and GPT-4-powered Bing Chat is not competitive, the experiments compare input-feature variations.Source: §4 · p. 5

4.1 Implementation Details

• Feature extraction uses Chat-GPT and generation uses GPT-4 with an 8k-token input limit; the study uses no training, fine-tuning, or dataset.Prompts were developed using the first author’s prior publications, with judges’ nominated papers reserved for testing.Source: §4.1 · p. 5

4.2 Human Evaluation Settings

• Domain experts familiar with the cited papers evaluated candidate reviews for fluency, coherence, relevance, factuality, usefulness, writing style, and overall quality.Evaluators nominated papers related to their own work; about half the evaluated papers were NLP and almost all were published after September 2021.Source: §4.2 · p. 5

4.3 Input Feature Variations

• Variants vary the human-provided main-idea plan, target-paper TAIC, cited-paper summaries, enriched citation intent and usage, paper-pair relationships, and CTS regeneration.Variant B omits the main-idea plan and is the only fully automatic variant; H adds CTS-based regeneration.Source: §4.3 · pp. 5–6

5 Results and Analyses

• Most variants achieved decent ROUGE scores, while variant B without the main-idea plan scored significantly lower and variant C without target-paper context was next lowest.The evaluation describes most variants as mostly on-topic.Source: §5 · p. 6

5.1 Importance of Input Features

• Baseline A outperformed variant B without the main-idea feature, which was not captured by any other feature.The authors interpret this as evidence that a human-provided main idea is important for guiding a satisfactory story.Source: §5.1 · p. 6
• Baseline A outperformed variant C, which lacked the target paper’s title, abstract, introduction, and conclusion.The authors conclude that this target-paper context provides crucial context for generated paragraphs.Source: §5.1 · p. 6
• Access to either enriched citation usage or paper-to-paper relationships showed a weak helpful trend, with the latter slightly preferred; using both or neither underperformed.The authors suggest the features are mutually redundant because citation usage summarizes a cited paper’s relationships, and using both can lead the LLM to overemphasize this information.Source: §5.1 · pp. 6–7
• Using cited-paper abstracts instead of faceted summaries improved fluency, coherence, writing, and overall ratings, while baseline A scored better on informativeness and factuality.The authors hypothesize that abstracts’ human-written, narrative-style sentences may explain the preference on the former dimensions.Source: §5.1 · p. 7
• Regenerating with cited text spans produced highly variable human scores: 44% of judges reported improved writing style, 26% improved informativeness, and 30% decreased factuality.The authors advise using CTS cautiously.Source: §5.1 · p. 7

5.2 Analysis of Writing Style

• Gold related-work sections mainly used transition and narrative sentences with reference-type citations, whereas generated variants used more explicit single-summary sentences and dominant-type citations.The generated paragraphs were therefore mostly descriptive summaries rather than coherent stories integrating cited works.Source: §5.2 · p. 7

5.3 Correlation Among Human Preference, ROUGE, and Writing Style

• Human preference correlated with ROUGE-L scores (Kendall’s τ=0.592), and ROUGE scores correlated with the proportion of reference-type citations (Kendall’s τ=0.691).Judges also identified factual errors and failures of emphasis and paragraph organization; the authors report that organization and flow between citations matter to judges.Source: §5.3 · p. 7, §5.4 · pp. 7–8

6 Conclusion

• The authors’ pilot study uses related-work sections as evaluation targets for generating citations and transition sentences at the paragraph level.The approach is presented as a first step toward customized daily-feed summaries.Source: §6 · p. 8
• The approach prompts LLMs to extract interpretable features about cited and target papers and their relationships, then uses a human-supplied main-idea plan to guide a coherent story.Related-work sections serve as proxies for short, target-customized daily-feed summaries.Source: §6 · p. 8
• Human judges dislike reviews that concatenate cited-paper summaries and strongly dislike generations that wrongly emphasize less impactful papers.The authors conclude that understanding relationships among papers matters beyond accurately describing each paper’s methodology.Source: §6 · p. 8

Limitations

• The system does not help readers find additional related papers, and integrating citation-list optimization is left for future work.Several judges wanted help discovering papers to read.Source: Limitations · pp. 8–9
• The 8k-token input limit required iterative generation and concatenation for nearly half of related-work sections, significantly harming coherence between subsections.The experiments used gpt-4-0314 and lacked access to gpt-4-32k.Source: Limitations · p. 9
• Missing cited works—because of unsupported source types, incomplete citation lists, failed retrieval, or paywalls—limit system performance.The pipeline could not parse books or websites and sometimes missed or could not download cited papers.Source: Limitations · p. 9
• Heuristic extraction can make author names and years inaccurate, while generation passes can mix author-year and numbered citation styles.The authors propose directly supplying the bibliography to an LLM in future work.Source: Limitations · p. 9
• Intermediate features lack gold references and additional human evaluation, while this preliminary study also omits post-processing for fact-checking, correction, and plagiarism avoidance.The influence of intermediate-feature quality on final output is left for future work.Source: Limitations · p. 9
• Prompts rely on proprietary OpenAI models that may be deprecated, affecting reproducibility and requiring prompt updates; tested models also produced variable results.Claude and later GPT versions produced qualitatively similar texts, text-bison had less satisfactory styles, and LLaMA-2 70B Chat failed the task.Source: Limitations · p. 9
• Evaluated papers and prompt design concentrate on computer science, particularly NLP, with only one geology paper evaluated outside that concentration.The authors cite the cost and difficulty of recruiting experts and tuning prompts in other disciplines.Source: Limitations · p. 9

Ethics Statement

• LLMs may produce toxic or non-factual statements and may plagiarize cited papers, though the authors consider plagiarism less concerning when reviews are shown only to feed owners.The stated use is to help readers curate a daily paper feed.Source: Ethics Statement · p. 9
• Using related-work sections as targets could enable cheating in publications, and careless use may harm the scientific community or conflict with future rules.The authors advise against using the system to write related-work sections for publication and urge care around potential regulations.Source: Ethics Statement · p. 10
• Generated reviews remain noticeably worse than human-written ones in organization and style, and fully automated output performs very poorly without a human-provided main-idea plan.The authors say readers can distinguish and criticize AI-generated content.Source: Ethics Statement · p. 10

A Extractiveness of Generations

• Coverage and density metrics are used to investigate how each feature contributes to generated related-work sections.The passage names Grusky et al. (2018)’s metrics.Source: A Extractiveness of Generations · p. 13
• Coverage is high for features including faceted summaries, cited-paper abstracts, and CTS, but overlapping information means feature scores need not sum to one.Faceted summaries and abstracts contain highly similar information.Source: A Extractiveness of Generations · p. 13
• Cited-paper abstracts and CTS retain high coverage even when absent from prompts, while only variant B contains human-provided main ideas unavailable in other features.The authors suggest the absent-input coverage indicates that abstracts and CTS contain information valued by the LLM.Source: A Extractiveness of Generations · p. 13
• Despite similar coverage, faceted summaries have significantly higher density than cited-paper abstracts, indicating that the LLM more directly copies from faceted summaries.Density measures the length of extracted fragments.Source: A Extractiveness of Generations · p. 13
• Main ideas and paper-pair relationships are also highly extractive, while the authors consider plagiarism less concerning because these features come from feature-extraction summaries rather than cited papers.The sentence continues in the passage describing the source of the highly extractive features.Source: A Extractiveness of Generations · pp. 13, 15
• Density varies across variants: ablating a feature lowers its density while increasing the remaining features’ scores, as variant G illustrates for main ideas and faceted summaries.The LLM cannot reconstruct an exact substring absent from its input.Source: A Extractiveness of Generations · p. 15

B Examples of Features and Generated Variants

• Section 2 is the target example, with only one example per feature shown because covering every feature across cited papers would be extremely long.The related-work section is reproduced in variants A–H, with the two paragraphs in each variant generated sequentially due to the LLM input-length limit.Source: B Examples of Features and Generated… · p. 15
• Human-rated related-work sections are tabulated by field of study and publication year.The supplied captions identify Tables 11 and 12 as distributions of those characteristics.Source: Table 11 · p. 15, Table 12 · p. 15

B.1 Citation Network Features

• The evaluation captions identify factual-error counts, extractiveness coverage and density for n = 38, average human scores, and variation with the best overall human-evaluation score.The captions do not provide the underlying values.Source: Figure 4 · p. 15, Figure 5 · p. 16, Table 13 · p. 16
• Hoang and Kan (2010) describe ReWoS, which uses hierarchical keywords for general and specific extractive summaries and studies argumentative patterns; initial human evaluation improved over generic multi-document baselines.The work introduces automated related-work summarization as a task.Source: §B.1 · pp. 15–16
• The examples connect Xing et al. (2020) to See et al. (2017) through an adapted pointer-generator architecture and characterize Xing et al.’s citation intent and usage.Xing et al. is described as cited for related work and as a baseline in citation-sentence generation.Source: §B.1 · pp. 16–17

B.2 Target Paper Features

• Prior work includes extractive related-work generation and abstractive citation-text approaches, which generally produce a single citation sentence or span; the passages also note the lack of a standard benchmark for comparing methods.The described abstractive approaches mostly rely on cited-paper abstracts.Source: §B.2 · p. 17

B.3 CTS

• The quoted Hoang and Kan (2010) material describes article data as including a reference related-work summary and input research articles, then proposes summarization for a given topic tree.Source: §B.3 · p. 17

C Example Generated Variants

• Variant A is the baseline, while Variant H is the baseline with CTS enhancement.Source: Figure 6 · p. 18, Figure 13 · p. 25
• The related-work text contrasts prior systems with an LLM workflow intended to produce a full section comparable to a human first draft.Earlier approaches are described as extractive systems that can lack coherence and stylistic variation.Source: C Example Generated Variants · pp. 19, 21
• Variants B, C, and D ablate the main idea, target-paper TAIC, and faceted summaries in favor of cited-paper abstracts, respectively.Source: Figure 7 · p. 19, Figure 8 · p. 20, Figure 9 · p. 21
• Variants E, F, and G ablate citation intent/usage, paper relationships, or both, respectively.Source: Figure 10 · p. 22, Figure 11 · p. 23, Figure 12 · p. 24