Source-linked AI summary
Relatedly: Scaffolding Literature Reviews with Existing Related Work Sections
Srishti Palani, Aakanksha Naik, Doug Downey, Amy X. Zhang, Jonathan Bragg, Joseph Chee Chang
TL;DR
Scholars must synthesize growing scientific literature, while related work sections are scoped to individual papers and can be difficult to navigate together. Relatedly scaffolds exploration of these sections with generated headings and cues for prioritizing new information and deemphasizing redundancy. In a within-subjects study, participants produced better-quality outlines than with a paper-list baseline.
Problem
Scholars need to connect information across many papers as literature grows, while individual related work sections are scoped to a single paper and can be difficult to navigate together.
Source: §1 · p. 1Method
Relatedly scaffolds exploration of multiple papers’ related work sections with descriptive headings and cues that prioritize unexplored, dissimilar information and deemphasize redundant information.
Source: §7 · pp. 14–15Results
Participants produced better-quality outlines with Relatedly than with a paper-list baseline, as rated by topic experts blind to condition.
Source: §1 · p. 3Takeaways & Limitations
Five participants later adopted Relatedly to support upcoming paper submissions, without obligations or compensation.
Source: §7 · p. 15The study compared Relatedly with a paper-list baseline rather than citation-graph visualization tools, which the authors identify as a comparison for future work.
Source: §7.1 · p. 16
Abstract
from arXiv · showhide
Scholars who want to research a scientific topic must take time to read, extract meaning, and identify connections across many papers. As scientific literature grows, this becomes increasingly challenging. Meanwhile, authors summarize prior research in papers' related work sections, though this is scoped to support a single paper. A formative study found that while reading multiple related work paragraphs helps overview a topic, it is hard to navigate overlapping and diverging references and research foci. In this work, we design a system, Relatedly, that scaffolds exploring and reading multiple related work paragraphs on a topic, with features including dynamic re-ranking and highlighting to spotlight unexplored dissimilar information, auto-generated descriptive paragraph headings, and low-lighting of redundant information. From a within-subjects user study (n=15), we found that scholars generate more coherent, insightful, and comprehensive topic outlines using Relatedly compared to a baseline paper list.
1 INTRODUCTION
Reviewing a growing, interdisciplinary literature requires scholars to connect ambiguous information across many papers, while existing tools offer limited sensemaking support. Relatedly scaffolds exploration of multiple papers’ related work paragraphs; a within-subjects study found better-quality outlines than a paper-list baseline.
Motivation
- Related work sections offer timely, synthesized views but reflect a single paper’s scope, so combining sections could broaden topic coverage and perspectives.Survey papers can be unavailable or quickly outdated, while search engines find relevant papers without presenting higher-level themes.Source: §1 · pp. 1–2
Formative study
- In a formative study, scholars preferred reading related work sections but struggled to prioritize and track paragraphs with overlapping and diverging references and foci.The study used a text search engine over related work sections and interviewed 10 scholars.Source: §1 · p. 2
System
- Relatedly retrieves relevant paragraphs and headings, generates descriptive headings when needed, and supports drilling down into similar paragraphs.It dynamically re-ranks remaining paragraphs and highlights unexplored references that diverge from the reader’s history, while low-lighting sentences about papers cited in already-seen paragraphs.Source: §1 · pp. 2–3
Evaluation
- Participants produced better-quality outlines with Relatedly than with a paper-list baseline, as rated by topic experts blind to condition.Logs showed more paragraphs and papers explored despite equal time and access to the same papers; participants also preferred Relatedly.Source: §1 · p. 3
Contributions
- The paper reports a novel approach to reviewing literature through related work sections, findings from a formative study with 10 scholars, and a system for scaffolded exploration.It also reports a within-subjects study with 15 participants on literature discovery and synthesis.Source: §1 · p. 3
2 RELATED WORK
The related work situates Relatedly against literature-review practices and tools that provide limited support for connecting papers into meaningful topic themes. Relatedly builds instead on the synthesis already present in authors’ related work sections.
2.1 How Scholars Conduct Literature Reviews
- Literature reviews seek to identify patterns, gaps, themes, and sub-themes, but reading individual papers to build an overview can be time-consuming and cognitively overwhelming.Survey papers can help, yet they may be unavailable or become outdated as new research emerges.Source: §2.1 · p. 3
- Related work sections provide context and structure, but each reflects its author’s specific point of view rather than exhaustively covering a broader topic.Scholars seeking a broader picture may need to read multiple sections across papers.Source: §2.1 · p. 3
- Relatedly draws on information foraging theory by showing potential information gain for paragraphs and letting users explore similar paragraphs for deeper views of a subtopic.These cues support switching between exploring and examining a discovered subtopic.Source: §2.1 · p. 3
2.2 Tools for Supporting Literature Review
- Search engines help users find papers, but synthesizing themes and connections, prioritizing reading, tracking information, and assessing progress remain largely user tasks.Faceted search can also depend on unavailable metadata or labels too general to give meaningful topic insight.Source: §2.2 · pp. 3–4
- Visual paper clusters can be incoherent or hard to interpret because they may conflate dimensions and lack semantic relationships or high-level theme descriptions.Automatic graph explanations may be inaccurate or incomplete, while manual and crowd-powered summaries take time and effort to produce.Source: §2.2 · p. 4
- Relatedly avoids generating new connections and clusters by using the synthesis authors already created in related work sections.Its challenge is helping readers explore overlapping clusters and differing perspectives across papers.Source: §2.2 · p. 4
3 FORMATIVE STUDY & DESIGN GOALS
The formative study examined how scholars explore related work paragraphs and identified challenges in connecting, prioritizing, and tracking literature. Its findings informed design goals for supporting topic organization and paragraph exploration.
3.1 Formative User Study Method
- The formative study used semi-structured interviews with 10 people experienced in scientific literature search, reading, and writing.Participants represented several research domains and varied in education, roles, and literature-review experience.Source: §3.1 · p. 4
- Participants discussed their workflows and explored related work paragraphs on misinformation for 20 minutes, followed by questions and mock-up feedback.Interviews were transcribed and coded through an open-coding process to identify challenges and design goals.Source: §3.1 · p. 4
3.2.1 Current Literature Review Workflows and Challenges
- All 10 participants used scholarly search engines and read papers individually; half also gathered paper lists socially.Three annotated PDFs and two saved papers to bibliography managers.Source: §3.2.1 · p. 4
- All 10 participants reported difficulty connecting papers, and 8/10 described unknown unknowns such as missing keywords or latent subtopics.Seven had trouble tracking what they had read, and five had difficulty deciding what to read first.Source: §3.2.1 · pp. 4–5
3.2.2 Preference for Exploring Related Work Sections and Challenges
- Nine of 10 participants preferred related work paragraphs to papers for quickly gaining topic vocabulary, references, takeaways, and research threads.They also valued the summaries’ context for deciding whether to open referenced papers.Source: §3.2.2 · p. 5
- Participants still wanted help prioritizing paragraphs, distinguishing their similarities and differences, identifying central papers, and tracking reading progress.Seven of 10 noted unclear paragraph similarities and differences; five wanted help choosing reading order.Source: §3.2.2 · p. 5
- Most participants wanted early paragraphs to map diverse subtopics, and six of 10 said similar paragraphs should not appear on the same page.They also valued paragraphs with broad coverage and sources spanning recent and foundational work.Source: §3.2.2 · p. 5
3.3 Design Goals
- The design goals were to support meaningful topic organization and subtopic exploration, fluid prioritization across paragraphs, and tracking of read paragraphs and references.These goals were guided by the formative study.Source: §3.3 · p. 5
4 THE RELATEDLY SYSTEM
Relatedly is presented as an approach to literature review guided by the formative study. The system section covers its scenario, features, heading-generation pipeline, and implementation.
System overview
- Relatedly aims to help users build a broader and more insightful overview of a research topic.The approach was guided by formative-study insights.Source: §4 · p. 5
- The system is introduced through an example user scenario and a walkthrough of its main features.Source: §4 · p. 5
- The system section also explains and evaluates its automatic section-heading generation pipeline and describes implementation details.Source: §4 · p. 5
4.1 Example User Scenario
A researcher uses Relatedly to move from an unfamiliar topic overview into focused reading across paragraphs about a subtopic. Dynamic ranking and progress cues help her continue exploring beyond the initial overview.
4.1 Example User Scenario
- Relatedly’s Overview View uses descriptive headings to reveal high-level research foci, such as Fact Checking Datasets and Social Media, News, and Misinformation.The headings help her skim the paragraphs and get a sense of their focus.Source: §4.1 · p. 5
- The Similar Paragraphs View surfaces other paragraphs about the selected subtopic, letting her read how different authors summarize its prior work.She explores paragraphs from different source papers, including more recent ones.Source: §4.1 · p. 5, Figure 2 · p. 6
- After she explores material, Relatedly re-ranks paragraphs toward those with more unexplored and dissimilar references, while low-lighting sentences that refer to papers she has already explored.A progress bar tracks how much of the paragraphs and references she has explored.Source: §4.1 · pp. 5–6
4.2 System Features
Relatedly organizes related-work paragraphs into topic overviews and subtopic drill-downs, then helps users prioritize novel material and track exploration. Its features address the overlapping and diverging references and foci that make paragraph reading difficult to prioritize.
4.2.1 [D1] Infer Topic Overview + Drill-Down to Subtopics.
- The Overview View retrieves relevant paragraphs and ranks them to cover a broad range of subtopics; users can open a Similar Paragraphs View to drill down on a selected subtopic.The selected paragraph is pinned beside similar paragraphs, and users can access other sections from its source paper for context.Source: §4.2.1 · pp. 6–7
- MMR re-ranking balances query relevance with information novelty, using unexplored references to favor paragraphs that jointly cover unique references and diverse subtopics.Relatedly first retrieves paragraphs with BM25 and then re-ranks them using MMR [9].Source: §4.2.1 · p. 6
- Relatedly extracts descriptive author headings when available and uses a BART-based model to generate titles for paragraphs with absent, generic, or short headings.The titles are intended to describe each paragraph’s focus and serve as subtopics.Source: §4.2.1 · p. 7
- For a selected paragraph, Similar Paragraphs are ordered first by shared references and then by semantic similarity between referenced papers.Semantic similarity uses Euclidean distances between SPECTER paper embeddings [13].Source: §4.2.1 · p. 7
4.2.2 [D2] Prioritize and explore similarities and differences across
- Relatedly supports prioritization by surfacing unexplored-reference counts and highlighting unexplored references that are semantically different from papers already explored.The count updates as users interact with references, and the highlights use distances between SPECTER paper embeddings [13].Source: §4.2.2 · p. 7
- A reference timeline shows the publication years of a paragraph’s cited papers, helping users triage paragraphs for recent or older, potentially fundamental work.Its blue dots represent publication years; the timeline spans the earliest to latest papers referenced across all paragraphs.Source: §4.2.2 · p. 7
- Citation frequency badges show how often a paper is referenced in result paragraphs, while self-reference icons identify text where authors situate their own work.The badges are intended to indicate a paper’s centrality or importance to the topic.Source: §4.2.2 · p. 7
4.2.3 [D3] Keeping Track Exploration Progress over Papers and Para-
- Relatedly low-lights sentences tied to previously encountered references and removes paragraphs marked explored from the Overview View.Marking a paragraph explored counts its references as encountered and triggers re-ranking based on unexplored papers, dissimilarity to what has been read, and query relevance.Source: §4.2.3 · pp. 7–8
4.3 Automatic Section Heading Generation
Relatedly generates descriptive headings for paragraphs whose source headings are missing or uninformative. Human ratings found its BART-generated headings comparable to long, descriptive author headings and better than headings that were not descriptive.
4.3 Automatic Section Heading Generation
- Headings help users survey research threads, so Relatedly generates them when source headings are absent or generic.The heading model was trained on paragraphs with heuristically identified descriptive titles.Source: §4.3 · p. 8
4.3.1 Method.
- The authors trained large BART and T5 models on 23,957 paragraphs with heuristic-filtered descriptive titles, then selected BART for generating headings beyond the filtered set.The dataset was split 80%/10%/10% for training, validation, and testing; BART headings were judged more detailed and descriptive in a small sample.Source: §4.3.1 · p. 8
4.3.2 Human Evaluation.
- Two authors rated 400 paired headings while blind to their condition, after calibration rounds yielded Krippendorff’s alpha of 0.90 for S1 and 0.74 for S2.The rated sets covered author- and model-generated headings, both for descriptive headings in the test split and for paragraphs without descriptive headings.Source: §4.3.2 · p. 8
- Model-generated headings were comparable in quality to long, descriptive author headings (S1: p=0.40; S2: p=0.32; n=[100, 100]) and significantly higher quality when descriptive author headings were unavailable (both p<0.001***; n=[100, 100]).Screening 200 generated titles found repetition in 5/200 (2.5%) and hallucination in 9/200 (4.5%).Source: §4.3.2 · p. 8
4.4 Implementation Details
Relatedly is implemented as a web application with paragraph search and behavior logging. Its evaluation dataset comprises extracted paragraphs from papers in five HCI and NLP conferences published from 2016 to 2021.
4.4 Implementation Details
- Relatedly uses a ReactJS front end, a Flask back end, and PostgreSQL for dataset access and study behavior logging.The implementation also uses Whoosh for BM25 full-text paragraph search and encrypts communications while anonymizing session and user IDs.Source: §4.4 · p. 9
4.4.1 Dataset.
- The dataset contains 63,144 paragraphs from 11,382 papers, extracted from full papers in five HCI and NLP conferences published between 2016 and 2021.About 49,975 paragraphs came from related work sections; most remaining paragraphs came from introductions and discussions.Source: §4.4.1 · p. 9
- Paragraphs were selected by requiring at least three references, and self-referential phrases were resolved to the source paper.Topics were selected for convenience to support development and recruitment of participants likely to engage with them.Source: §4.4.1 · p. 9
5 USER EVALUATION STUDY DESIGN
The study compared Relatedly’s topic-centric exploration of related-work paragraphs with a scholarly-search baseline, asking participants to create survey outlines. A within-subjects design, controlled tasks, and expert ratings supported comparison of their review processes.
5 USER EVALUATION STUDY DESIGN
- The within-subjects study compared Relatedly with a standard paper-search engine while 15 participants reviewed literature and created survey outlines.Participants used each system on a different topic, with system and topic assignments counterbalanced.Source: §5 · pp. 9–10, §5.1.2 · p. 11
5.1.1 The Baseline Conditions.
- The baseline used BM25 to return papers matching query terms in titles or abstracts, with access to metadata and related-work sections containing three or more references.Both conditions provided access to the same data, and the baseline included a button to reduce interaction costs for opening related-work sections.Source: §5.1.1 · p. 10, Figure 4 · p. 10
5.1.2 Tasks.
- Participants were asked to build an outline that captured terms, concepts, perspectives, and relevant research for a future survey on their assigned topic.They completed the task once per system, using different topics to limit learning carryover and counterbalancing system and topic order.Source: §5.1.2 · pp. 10–11
5.1 Experimental Setup
- Participants used both systems on two interdisciplinary topics selected for their complexity and representation in the HCI and NLP dataset.The topics were misinformation, fake news and fact checking, and crowdsourcing.Source: §5.1.2 · p. 11
5.1.3 Participants.
- The study recruited 15 research-active participants, mostly research scientists, postdoctoral researchers, and graduate students, from labs at three universities.Participants were 19–32 years old, and the study was conducted remotely.Source: §5.1.3 · p. 11
5.2 Study Procedure
- Each participant completed two 20-minute review tasks after tutorials, creating an outline while thinking aloud.Participants had five minutes to explore each system using a test topic before beginning its task.Source: §5.2 · p. 11
5.3 Measures
- Experts blind to system condition rated outline coherence, insightfulness, and detail and counted research themes; log and qualitative analyses captured system use and experience.The outline ratings used five-point scales, and the study also included a post-task survey of system values.Source: §5.3.1 · p. 11, §5.3.2 · p. 11, §5.3.3 · p. 11
6 FINDINGS
In a 15-participant within-subjects study, Relatedly produced higher-quality outlines than a paper-search baseline. Participants explored more paragraphs, often preferred Relatedly, and five volunteered to use it for real-world work afterward.
6.1 Higher Quality Synthesized Outlines
- 10.53 vs 8.07 was the mean overall outline-quality score for Relatedly and Baseline, respectively, with a significant difference between conditions.Blind experts rated outlines on coherence, insightfulness, and detail; the combined measures also differed significantly in the MANOVA and Wald-type tests.Source: §6.1 · pp. 11–12, Figure 5 · p. 12
- All participants switched between overview and similar-paragraph views, moving between broad topic exploration and drilling down into subtopics.Participants described breadth-first exploration and using the unexplored-reference badge to prioritize paragraphs.Source: §6.1 · p. 12
6.2 Paper- vs Topic-Centric Exploration
- 16.85 vs 7.21 paragraphs were explored on average in Relatedly and Baseline, respectively.Relatedly participants interacted with more than twice as many paragraphs; the difference was statistically significant (p=0.001).Source: Figure 6 · p. 12, §6.2 · pp. 12–13
- Participants used highlights to prioritize novel references, low-lighting to deprioritize previously read references, and descriptive headings to organize topics.Citation-frequency tags were also used to identify papers participants considered important.Source: §6.2 · p. 13
6.3 Participants Preferred Relatedly
- 13 of 15 participants preferred Relatedly; they rated it higher on five review-support statements, with no significant difference in perceived workload.The statements concerned finding relevant research, understanding relationships, bringing sources together, prioritizing reading, and tracking information.Source: §6.3 · p. 13
6.4 Volunteered Use in the Wild
- Five scholars volunteered to use Relatedly for upcoming submissions, and in follow-up accounts described using it alongside search engines, PDF readers, and reference managers.Participants said traditional search engines remained useful for looking up specific papers or authors.Source: §6.4 · pp. 13–14
7 DISCUSSION
Relatedly uses existing related-work paragraphs to scaffold exploration, prioritizing novel, dissimilar information while de-prioritizing redundancy. In the study, participants explored more and produced higher-quality outlines than with the baseline; the discussion also identifies study and adoption boundaries.
7 DISCUSSION
- Participants explored significantly more paragraphs with Relatedly, which the authors suggest could help explain their higher-quality learning outlines.The discussion connects exposure to multiple explanations with the possibility of making sense of complex topics, while presenting the explanation as potential rather than established causation.Source: §7 · p. 15
- Participants reported that Relatedly helped them find relevant research, understand relationships, bring together perspectives, prioritize reading, and track what they had read.The authors also report significantly more themes or subtopics in outlines and more domain-specific terms discovered with Relatedly.Source: §7 · p. 15
- Participants produced higher-quality outlines with Relatedly than with the baseline, and 13 of 15 preferred Relatedly for literature reviews.Five participants adopted it for upcoming submissions after the study; the authors describe this as a promising indication of real-world benefits.Source: §7 · p. 15
- Relatedly re-ranks paragraphs and highlights sentences to prioritize information dissimilar to what users have already explored, while de-prioritizing redundant information.Its overview and similar-paragraph views support switching between exploring diverse subtopics and reading details about a specific subtopic.Source: §7 · p. 15
- The approach is demonstrated with scientific texts and related-work sections; the authors propose possible applications in policy, legal research, news, and programming.These other-domain applications are presented as future research avenues, not evaluated uses.Source: §7 · pp. 15–16
7.1 Limitations and Future Work
- The approach assumes high-quality related-work sections; the dataset used papers from leading HCI and NLP venues, and broader coverage may need source controls or writing-quality assessment.The authors also identify assessing reference importance as a possible way to mitigate bulk and passing citations.Source: §7.1 · p. 16
- The evaluation compared Relatedly with a paper-list baseline rather than citation-graph visualization tools, which the authors leave for future comparison.They chose the paper-list baseline to keep sessions realistic and under 90 minutes while limiting participant fatigue.Source: §7.1 · p. 16
- The formative interviews mostly involved junior scholars, Relatedly is designed for scholarly users, and the lab study examined a single topic and query.The authors identify broader user support and exploration across multiple topics and queries as future work.Source: §7.1 · p. 16
8 CONCLUSION
Relatedly guides literature review through related-work paragraphs gathered across papers, with reading cues and information-gain tracking. A comparative user study found that scholars synthesized topics in broader, more coherent, and more insightful ways than with the baseline.
8 CONCLUSION
- Relatedly guides users through subtopics using related-work paragraphs extracted across multiple papers rather than focusing on individual papers one by one.It adds reading-support cues and information-gain tracking to help users cover similar and distinct topics across paragraphs.Source: §8 · pp. 16–17
- In a comparative user study, scholars using Relatedly produced broader, more coherent, and more insightful topic syntheses than with the baseline.Participants also discovered more domain-specific terms and preferred Relatedly.Source: §8 · p. 17