PaperlayerView exact revision on arXiv

Source-linked AI summary

Scim: Intelligent Skimming Support for Scientific Papers

Raymond Fok, Hita Kambhamettu, Luca Soldaini, Jonathan Bragg, Kyle Lo, Andrew Head, Marti A. Hearst, Daniel S. Weld

arXiv:2205.04561v3cs.HC

TL;DR

Researchers need better support for skimming scientific papers between searching and reading. Scim uses automatic, faceted, evenly distributed, configurable highlights, and studies found that it reduced information-seeking time while supporting high-level understanding, especially for dense or unfamiliar papers.

  • Problem

    Researchers need intelligent interfaces that support skimming scientific papers, a task between searching and reading that is difficult amid growing literature.

  • Method

    Scim combines formative research with automatic highlights that are faceted by content type, distributed across papers, and configurable globally and locally.

  • Results

    Scim reduced information-seeking time without changing perceived task difficulty, while diary-study readers found it useful for dense texts and unfamiliar domains.

  • Takeaways & Limitations

    The studies suggest that intelligent highlighting can support rapid, high-level skimming of scientific literature.

  • Takeaways & Limitations

    Highlights provide a single pathway and do not address other sensemaking aspects of skimming without more personalized controls and complementary support.

Abstract

from arXiv · show

Researchers need to keep up with immense literatures, though it is time-consuming and difficult to do so. In this paper, we investigate the role that intelligent interfaces can play in helping researchers skim papers, that is, rapidly reviewing a paper to attain a cursory understanding of its contents. After conducting formative interviews and a design probe, we suggest that skimming aids should aim to thread the needle of highlighting content that is simultaneously diverse, evenly-distributed, and important. We introduce Scim, a novel intelligent skimming interface that reifies this aim, designed to support the skimming process by highlighting salient paper contents to direct a skimmer's focus. Key to the design is that the highlights are faceted by content type, evenly-distributed across a paper, with a density configurable by readers at both the global and local level. We evaluate Scim with an in-lab usability study and deployment study, revealing how skimming aids can support readers throughout the skimming experience and yielding design considerations and tensions for the design of future intelligent skimming tools.

1 INTRODUCTION

Scim is an intelligent interface for skimming scientific papers that balances highlight importance, content diversity, and distribution while giving readers control over highlight density. Formative studies and evaluations show that Scim supports faster information finding and realistic skimming, while suggesting directions for future assistants.

  • Motivation: Scim addresses researchers’ need to rapidly review expanding scientific literatures, positioning intelligent skimming between searching for papers and reading them in depth.Skimming provides a faster but coarser understanding of paper contents and has become more widespread with digital publishing, despite being difficult.
  • Formative studies: Formative interviews, observations, and prototype studies found that effective skimming highlights should cover diverse and important content while remaining evenly distributed across a paper.Readers also wanted influence over the quantity and distribution of highlights, revealing a tension between passage importance and desirable highlight distribution.
  • Scim design: Scim highlights objectives, novelty, methods, and results with distinct colors, distributes them throughout papers, and lets readers control highlight quantity globally and locally.The design balances important, diverse, and evenly distributed content while addressing readers’ desire to influence highlight quantity and distribution.
  • Evaluation: In a lab usability study, Scim enabled readers to locate requested information significantly faster than a standard document reader, with comparable effort and accuracy.The study examined readers’ ability to search for specific kinds of information in papers.
  • Evaluation: A two-week diary study examined Scim in realistic use and suggested future improvements, including highlighting background for later highlights and integrating authors’ typographical emphases.These findings extend the design space for intelligent skimming assistants beyond automated highlighting alone.

2 RELATED WORK

Prior work characterizes skimming as rapid, goal-directed reading that is cognitively and physically demanding, especially for scientific papers. Existing support includes scrolling adaptations, typographical cues, scientific-paper augmentations, and summaries, but summaries can be unreliable and lack interactive access to the full paper.

  • Skimming and Reading Processes: Skimming is rapid, goal-directed reading in which researchers seek a general understanding while prioritizing relevant information and skipping the irrelevant.The growing volume of digital papers has increased the need for efficient skimming.
  • Skimming and Reading Processes: Skimming is cognitively demanding because readers build and integrate a mental model across sentences, while eye-movement limits make sustained rapid gaze placement physically demanding.Readers are also generally inaccurate at identifying goal-relevant information within text.
  • Skimming and Reading Processes: Scientific-paper skimming is often hasty and incomplete, with readers attending to visual content and section headers while using document structure, typographical cues, and visualizations for rapid comprehension.These macro- and microstructures help readers navigate information during rapid reading.
  • Tools for Supporting Skimming: Reading-support systems have augmented scrolling, attention allocation, typographical cueing, and scientific-paper exploration through contextual information, linked text and visualizations, and interactive document features.Content-aware systems have reordered or resized content and pinned headings and figures, while highlighting systems focus attention and can improve retention.
  • Summarization and Augmented Reading: Summaries offer a shortened alternative to full-paper skimming, but automated versions remain error-prone, susceptible to hallucination, unreliable alone, and unable to support interactive exploration of the full paper.Augmented reading interfaces retain access to the source as readers’ goals and interests change.

3 DESIGN GOALS

Formative interviews, observations, and prototype evaluations with academic readers yielded seven design goals for intelligent, highlight-based skimming interfaces. These goals emphasize augmenting existing practices with diverse, accurate, well-distributed highlights while minimizing distraction and supporting control and personalization.

  • Formative research: Formative research with eight academic readers and iterative prototype evaluations informed seven design goals for intelligent highlight-based skimming interfaces.Readers were observed skimming papers, describing their goals, strategies, and difficulties; themes were validated through discussion and review with a second author.
  • Design goals: Highlights should augment readers’ existing skimming practices and support diverse goals, including learning techniques, relating work to prior research, finding directions, and gaining high-level understanding.Readers used strategies shaped by their goals, papers, and familiarity, often consulting abstracts, introductions, contributions, results summaries, and conclusions.
  • Design goals: Highlights should direct readers to important content in lengthy middle sections that conventional beginning-and-end reading strategies may miss.Readers sometimes transitioned into deep reading, while important content could occur in the middle of paragraphs.
  • Design goals: Interfaces should minimize visual distraction, provide enough highlights across the paper, and distribute them broadly rather than concentrating them in only the introduction or conclusion.The formative research suggested roughly one highlight per paragraph as a rule of thumb, while highlighting techniques differed in how effectively they attracted attention without distracting readers.
  • Design goals: Faceted highlights require high accuracy because classification errors distracted readers and reduced their confidence in the tool.Readers noticed when rhetorical categories conflicted with their expectations, such as results passages labeled as methods.
  • Design goals: Skimming tools should support user control, personalization, and changing comfort as readers become familiar with intelligent highlights, motivating longitudinal evaluation.Readers wanted to adjust highlight amounts or content manually or adaptively, and one longitudinal diary study was motivated by comfort changing over time.

4 SCIM

Scim supports scientific-paper skimming by directing attention to salient passages through colored, distributed highlights. Readers can configure highlight density and content focus globally or within paragraphs, while annotations and a linked passage list support navigation and context.

  • Highlighting: Scim directs readers’ attention to key passages through highlights that extend into paragraph content they might otherwise overlook.Readers can follow Scim’s highlights alongside their usual title, abstract, and piecemeal review strategy.
  • Highlighting: Highlights are faceted into four color-coded categories—Objective, Novelty, Method, and Result—to support readers’ differing skimming goals.The four-facet taxonomy combines a coarse-grained abstract-classification schema [14] with the NOV_ADV category from Argumentative Zoning.
  • Highlighting: Scim uses minimally distracting rectangular highlights and postprocesses predictions to distribute them approximately evenly throughout the paper.Consistent facet colors and a legend are intended to foster learned associations, while distribution addresses readers’ concern about passages without highlights.
  • Controls: Paper-level facet sliders let readers change highlight density or disable content types, while paragraph-level controls provide rapid access to more or fewer highlights.Slider changes update highlights, scrollbar markers, and highlight counts; paragraph controls support closer skimming of lengthy passages such as results sections.
  • Navigation: Scrollbar annotations reveal where to skim and suggest paper structure, while a dynamically updating sidebar lists highlighted passages by section and links each passage back to its context.Colored bars indicate facets in the passage list, and clicking a passage scrolls the paper to its location through context linking.

5 IMPLEMENTATION

Scim combines PDF parsing, facet classification, weak supervision, and heuristic highlight selection in an interactive web application. Its sentence classifier achieved an F1 score of 0.533 against an annotator-annotator F1 score of 0.725 on 20 NLP papers.

  • Facet classification: The classifier adapts Cohan et al.’s [14] sequential model by replacing BERT with pretrained MiniLM and using up to 512 words or 10 sentences of context.It was fine-tuned initially on the manually labeled CSAbstruct dataset [14].
  • Document processing and classification: Scim parses PDFs into sentences with bounding boxes and metadata, then uses a fine-tuned language model to classify sentences into facets.MMDA extracts tokens, mathematical symbols, section headers, and metadata; sentence labels retain section and paragraph information for later prioritization.
  • Highlight selection and interface: Scim selects highlights by combining facet predictions and probabilities with section-consistency and distribution heuristics that spread highlights across papers.The interface also preserves existing PDF markup and implements highlights, sidebars, and controls as interactive React components atop pdf.js.
  • Weak supervision: Weak supervision expanded training to 606,400 unlabeled sentences from 3,051 papers using heuristic and keyword labeling functions.Negative examples were added using sentence similarity to abstracts, with sentences below an empirically chosen cosine-similarity threshold of 0.25 treated as irrelevant.
  • Preliminary evaluation: The classifier achieved an F1 score of 0.533, compared with an annotator-annotator F1 score of 0.725, on a test set from 20 NLP papers.Three NLP-experienced annotators labeled each paper, and sentences selected by at least two formed the ground truth.

6 STUDY 1: IN-LAB USABILITY STUDY

The in-lab usability study compared Scim with a standard document reader using information-seeking tasks. Scim reduced answering time without significantly changing accuracy or perceived difficulty, especially when answers appeared in highlights.

  • Method: The within-subjects study used three tasks comparing Scim and a standard reader, measuring time, accuracy, and perceived difficulty across 19 NLP-paper readers.Task 2 questions had answers within Scim highlights, whereas Task 3 questions did not; interface and paper order were counterbalanced.
  • Results: When answers appeared in Scim highlights, participants took 93.8s versus 127.3s with the standard reader, a significant difference.This Task 2 difference was significant (F(1, 54) = 4.84, p < .05).
  • Results: Scim significantly reduced question-answering time compared with the standard reader, from 117.7s to 94.3s overall.The overall difference was significant (F(1, 126) = 4.17, p < .05); Figure 5 reports the same overall comparison.
  • Results: Scim did not significantly change accuracy or perceived difficulty relative to the standard reader.Accuracy was 0.80 with Scim versus 0.76 with the standard reader (p = .64), while perceived difficulty was 2.32 with Scim (p = .92).

7 STUDY 2: LONGITUDINAL DIARY STUDY

A two-week longitudinal diary study examined how researchers use Scim during realistic, self-directed paper skimming, including its value, usefulness, limitations, and implications for future tools.

  • 7 STUDY 2: LONGITUDINAL DIARY STUDY: The study let participants use Scim for two weeks while choosing papers relevant to their disciplines and deciding when to read.This design aimed to reflect typical skimming motivations and allow acclimation to the novel reading interface.
  • 7 STUDY 2: LONGITUDINAL DIARY STUDY: The study investigated the value, use, usefulness, and limitations of intelligent highlight-based skimming aids for researchers.It also asked what features future intelligent skimming tools should provide.

7.1 Study Design

The study combined a two-week diary study of Scim with an exit interview and thematic analysis, using NLP researchers who skimmed conference papers. Participants first used a standard reader for comparison, then Scim during nine subsequent sessions.

  • Participants: The diary study recruited 12 experienced research-paper readers, equally split by gender, including 2 master’s and 10 PhD students, with preference for NLP experience.Participants were recruited through university-affiliated mailing lists, Slack channels, and public posts; PhD students ranged from first- to fifth-year study.
  • Materials: The study used preprocessed NAACL 2022 papers because Scim was primarily fine-tuned on NLP papers and preprocessing reduced loading time during the diary study.The selected proceedings were considered exciting for participants and came from a recent, widely read NLP conference.
  • Diary procedure: Nearly all participants completed the 10-day diary protocol: 11 of 12 finished all entries, while one completed 7 of 10.Researchers sent light email reminders when participants fell behind.
  • Diary procedure: Participants skimmed at least one paper for 5–10 minutes daily across 10 days, using a standard reader on day one and Scim on the next nine days.They recorded structured diary reflections about attention, missed content, helpfulness, and possible improvements after each session.
  • Analysis: Researchers analyzed diary entries and exit-interview transcripts thematically, with one author developing themes iteratively and a second validating them against excerpts and proposing revisions.Exit interviews elicited detailed reflections on Scim’s support for skimming and opportunities for improvement.

7.2 Results

Scim supported skimming by helping readers focus on salient content, understand papers at a high level, and navigate dense or unfamiliar material. However, highlights sometimes lacked context or conflicted with authors’ visual cues, creating back-and-forth and unpredictability.

  • Skimming process: Scim reduced the effort of skimming by directing attention to highlighted content while readers skimmed the rest, sometimes alleviating a second pass through relevant passages.The highlight browser also supported navigation and rapid understanding, with readers opening it an average of 9.3 times.
  • Interaction: Readers primarily used highlights alongside existing skimming strategies, while global and local controls let them configure highlight density and the browser provided an extractive summary.Nearly all readers used both control levels, typically setting an acceptable global density and making local adjustments as needed.
  • Adoption: Some readers needed time to trust and learn Scim, but their interactions evolved as they recognized useful highlight types and adapted the tool to their goals.One reader reported quickly learning when to use the side panel and focusing immediately on helpful results highlights.
  • Usefulness: 74 of 105 diary responses (70.4%) reported that Scim’s highlights helped readers skim, especially dense, text-heavy, or unfamiliar papers.Highlights helped readers identify key concepts, main ideas, results, and details they might otherwise skip.
  • Limitations: Highlights sometimes lacked sufficient context, forcing disruptive back-and-forth between highlighted and surrounding text, including across sections.Readers also wanted tighter integration with author-emphasized text and highlights covering complete contribution lists, mathematical notation, tables, and figures.
  • Future directions: Readers suggested future tools augment dense-paper comprehension with abstractive summaries and improve navigation by linking an abstract or introduction to related highlights.These extensions were proposed to lessen the effort required to understand dense sections of papers.

8 DISCUSSION · 8.1 Skimming versus Scanning · 8.2 Supporting Experts and Novices

Scim augments scientific-paper skimming with automatic faceted highlighting, reducing task completion time in a lab study without changing reported difficulty. The discussion qualifies these findings by distinguishing scanning from skimming and limiting Scim’s intended audience to experienced skimmers.

  • 8 DISCUSSION: Scim uses automatic faceted highlighting to augment scientific-paper skimming, and the studies examined its support for readers.The lab study found faster completion of short information-seeking tasks without a significant difference in self-reported task difficulty; a subsequent diary study observed researchers’ use.
  • 8 DISCUSSION: Scim reduced completion time for short scientific-paper information-seeking tasks without significantly changing readers’ self-reported task difficulty.The lab tasks were intended to measure speed and accuracy while skimming.
  • 8.1 Skimming versus Scanning: Because lab participants often searched for keywords rather than building high-level understanding, the study sometimes measured scanning instead of skimming.Participants used strategies such as “Control+F,” although the task wording was designed to undermine keyword-based scanning.
  • 8.1 Skimming versus Scanning: For participants who scanned rather than skimmed, the results may say less about whether Scim supports the skimming process.Their behavior focused on locating exact answers instead of understanding the paper broadly.
  • 8.2 Supporting Experts and Novices: Scim was designed to help experienced skimmers get more from skimming, not to develop inexperienced readers’ skimming proficiency.Teaching-oriented skimming assistants may therefore share only some characteristics with Scim.
  • 8.2 Supporting Experts and Novices: Future AI skimming assistants could identify salient content and coach readers to find it, while ensuring productive learning experiences across backgrounds.This creates a design challenge around supporting readers with different prior skimming knowledge.

8.3 Risks to Attention · 8.4 Limitations of Highlights

Scim may unintentionally burden attention or encourage shallow reading, while its highlights provide only one pathway through a paper and do not address broader sensemaking. Skimming aids should therefore be accurate, reliable, attentive to deep reading, and supported by richer controls and contextual affordances.

  • 8.3 Risks to Attention: Scim may deplete readers’ limited attention by imposing cognitive burden, especially when highlights appear in unexpected ways.Skimming already requires substantial attention to understand papers’ idiosyncrasies and nuances.
  • 8.3 Risks to Attention: Skimming aids should be accurate and reliable while their effects on readers’ engagement with texts are studied.The paper calls for deployment alongside studies that examine these effects.
  • 8.3 Risks to Attention: Scim should be developed alongside tools that support deep reading and encourage good reading practices in the research community.This recommendation aims to preserve the value of deeply reading while supporting skimming.
  • 8.4 Limitations of Highlights: Without sophisticated controls for goal-driven or personalized skimming, highlights present only a single pathway through a paper.The limitation motivates more flexible skimming support.
  • 8.4 Limitations of Highlights: Highlighting cues attention and supports information foraging, but it does not address other sensemaking aspects of skimming.Highlights function as an attentional aid rather than holistic support for the full skimming process.
  • 8.4 Limitations of Highlights: Readers’ diary-study feedback suggests that skimming aids could provide more holistic support by adding context for highlighted passages.Additional context is presented as one direction for extending highlights beyond cueing.

8.5 Future Work

Future work should improve Scim’s highlighting accuracy and PDF processing, while exploring social annotations and personalized, proactive reading support to address reader distrust and cognitive overhead.

  • Algorithmic and processing improvements: Future algorithms could use long-form summarization or other generative models, together with paper structure, author cues, and visual content, to improve highlight accuracy.Scim’s usefulness is limited by the accuracy of its underlying AI models.
  • Algorithmic and processing improvements: Improved PDF processing is needed because errors concatenated footnotes, headers, tables, or figures with sentences, producing poorly classified passages and seemingly disparate highlights.
  • Trust and social support: Social annotations could produce more trusted highlights, since preliminary studies found readers hesitant to adopt Scim and some preferred highlights created by fellow researchers over AI-generated ones.
  • Personalized reading support: AI-infused reading systems could learn from repeated interactions to provide personalized, proactive support tailored to readers’ experience, information needs, and reading goals.This direction aims to mitigate undesirable cognitive overhead introduced by augmented reading interfaces.

9 CONCLUSION

The paper presents Scim, an intelligent skimming interface implementing seven design goals through faceted, evenly distributed, minimally intrusive, configurable highlights. A lab study found that participants located information more quickly with Scim than with a standard document reader.

  • 9 CONCLUSION: Scim instantiates seven formative-research goals for scientific-paper skimming through faceted, evenly distributed, minimally intrusive, configurable highlights.
  • 9 CONCLUSION: Participants located information in papers more quickly with Scim than with a standard document reader in a lab usability study.
Loading 2205.04561v3…