PaperlayerView exact revision on arXiv

Source-linked AI summary

PaperRobot: Incremental Draft Generation of Scientific Ideas

Qingyun Wang, Lifu Huang, Zhiying Jiang, Kevin Knight, Heng Ji, Mohit Bansal, Yi Luan

arXiv:1905.07870v4cs.CLcs.AIcs.LG

TL;DR

PaperRobot addresses the challenge of keeping up with scientific papers and developing incremental ideas by building knowledge graphs, predicting new links, and drafting paper elements. In biomedical Turing tests, its generated outputs are sometimes chosen over human-written ones.

  • Problem

    Scientists struggle to keep up with the volume of papers, while creating new entities often requires laboratory experiments.

    Source: §1 · p. 1

  • Method

    PaperRobot builds background knowledge graphs, predicts links by combining graph structure with contextual text, and incrementally drafts paper elements.

    Source: §1 · pp. 1–2, §2.4 · p. 4

  • Results

    Biomedical Turing tests show that PaperRobot-generated outputs are sometimes chosen over human-written strings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PaperRobot is an assistant for speeding scientific discovery and production, with experiments and verification of predicted links remaining beyond its scope.

    Source: §5 · p. 9

    The generation model lacks knowledge reasoning and can produce incorrect abbreviations, numbers, and pronouns.

    Source: §3.6 · p. 7

Abstract

from arXiv · show

We present a PaperRobot who performs as an automatic research assistant by (1) conducting deep understanding of a large collection of human-written papers in a target domain and constructing comprehensive background knowledge graphs (KGs); (2) creating new ideas by predicting links from the background KGs, by combining graph attention and contextual text attention; (3) incrementally writing some key elements of a new paper based on memory-attention networks: from the input title along with predicted related entities to generate a paper abstract, from the abstract to generate conclusion and future work, and finally from future work to generate a title for a follow-on paper. Turing Tests, where a biomedical domain expert is asked to compare a system output and a human-authored string, show PaperRobot generated abstracts, conclusion and future work sections, and new titles are chosen over human-written ones up to 30%, 24% and 12% of the time, respectively.

1 Introduction

PaperRobot addresses the difficulty of keeping up with scientific literature by building background knowledge graphs, predicting new links, and incrementally drafting paper elements. Biomedical Turing tests report that its outputs are sometimes chosen over human-written strings.

1 Introduction

  • More than 500K biomedical papers are published annually, while scientists' reading ability remains nearly unchanged across years.A 2012 survey estimated that US scientists read an average of 264 papers per year.Source: §1 · p. 1
  • PaperRobot builds background knowledge graphs from existing papers and proposes automating incremental idea creation by predicting new links.The introduction notes that creating new entities usually requires laboratory experiments, whereas predicting new edges is easier to automate.Source: §1 · p. 1
  • Its link-prediction representation combines knowledge-graph structure with unstructured contextual text.Source: §1 · p. 2
  • A memory-attention network generates an abstract from a title and predicted related entities, then generates conclusions, future work, and a follow-on title.Source: §1 · p. 2
  • PaperRobot-generated outputs are sometimes chosen over human-written strings in biomedical Turing tests.The tested outputs include abstracts, conclusions and future work, and new titles.Source: §1 · p. 2

2 Approach

PaperRobot builds biomedical knowledge graphs from existing papers, enriches them through link prediction, and uses retrieved related entities to incrementally generate paper sections.

2.2 Background Knowledge Extraction

  • The system extracts Disease, Chemical, and Gene mentions, links them to MeSH and CTD, and records 133 relation subtypes in biomedical knowledge graphs.Relations include Marker/Mechanism, Therapeutic, and Increase Expression.Source: §2.2 · p. 2

2.3 Link Prediction

  • Link prediction combines graph structure and contextual text to identify similar entities and propagate one entity’s neighbors to another.The example predicts CD14 molecule and neuropilin 2 as new Calcium neighbors based on Calcium’s similarity to Zinc.Source: §2.3 · pp. 2–3

2.4 New Paper Writing

  • For title-to-abstract generation, PaperRobot retrieves up to 10 related entities and combines vocabulary generation with copying from the title or entities.A multi-hop memory attention mechanism filters related entities to initialize the decoder; the same architecture supports abstract-to-conclusion and future-work generation and follow-on title prediction.Source: §2.4 · pp. 4–5
  • Coverage loss and test-time masking address repetition by discouraging repeated attention and preventing reuse of already generated non-stop words and punctuation.Generation uses beam search with beam size 4.Source: §2.4 · p. 6

3 Experiment

PaperRobot is evaluated through automatic metrics, human judgments, post-editing, ablations, and an NLP-domain case study. Results show human preference for some generated outputs, while analyses identify limits in topic control and domain transfer.

3.1 Data

  • The graphs cover 1,687,060 papers, 30,483 entities, and 875,698 relations.The paper assumes a cited paper's title is generated from another paper's conclusion and future work to construct title-prediction ground truth.Source: §3.1 · p. 6

3.2 Automatic Evaluation

  • The framework outperforms all previous approaches under METEOR topic-relevance and language-model perplexity evaluation.Perplexity uses a language model learned from other PubMed papers not used for training or testing.Source: §3.2 · p. 6

3.3 Turing Test

  • Experts chose PaperRobot outputs over human-written ones up to 30% for abstracts, 24% for conclusions and future work, and 12% for titles.Judges focused on different aspects: the expert on content and the non-expert on language.Source: §3.3 · p. 7, Table 4 · p. 7

3.4 Human Post-Editing

  • An expert edited 50 generated abstracts in about 40 minutes, and most edits were stylistic.The paper compares abstracts before and after editing using BLEU, ROUGE, and TER.Source: §3.4 · p. 7, Table 5 · p. 7

3.5 Analysis and Discussions

  • Without memory networks, generated abstracts tend to diverge from the main topic, with irrelevant entities appearing in later Methods and Results sections.The example includes “imipramine,” used to treat depression, in an abstract about human prostate cancer.Source: §3.5 · p. 7, Table 6 · p. 8
  • Link prediction introduces related entities such as RT-PCR and western blot, while beam-search masking removes repeated phrases and the plagiarism check indicates outputs are not simply copied.The predicted entities are often genes or proteins associated with the title's disease, or diseases from the same family.Source: §3.5 · p. 7, Table 9 · p. 8

3.6 Remaining Challenges

  • Generation remains limited by weak knowledge reasoning, incorrect abbreviations, numbers, and pronouns, and declarative titles that are less engaging and detailed than human titles.Human titles may use questions, concrete ideas, and newly created entity abbreviations.Source: §3.6 · pp. 7–8

3.7 Requirements to Make PaperRobot Work: Case Study on NLP Domain

  • In NLP, limited data impairs out-of-vocabulary copying, while coarse entity and relation types impair related-entity prediction, yielding generic text and inaccurate entities.The case study uses 23,594 NLP papers.Source: §3.7 · pp. 8–9

4 Related Work

Prior work informs PaperRobot’s combination of graph-based link prediction and text generation, while the paper brings incremental idea creation into automatic biomedical paper writing.

Link Prediction

  • Multi-head graph attention and contextual sentences enrich link-prediction representations, and the paper incorporates link-based idea creation into automatic paper writing for the first time.Earlier link-prediction approaches used translation-based methods.Source: §4 · p. 9

Knowledge-driven Generation

  • Knowledge-to-language work covers structured knowledge, biographies, and image/video captions; PaperRobot uses pointer-generator and copy mechanisms to handle unknown words.The cited methods include Duma and Klein (2013), Lebret et al. (2016), and Gu et al. (2016).Source: §4 · p. 9
  • Prior systems generated abstracts from titles, posters, or science-news titles; this paper is the first to automatically write biomedical conclusions, future work, and follow-on titles.The cited examples include Wang et al. (2018b), Qiang et al. (2016), and Vadapalli et al. (2018).Source: §4 · p. 9

5 Conclusions and Future Work

PaperRobot predicts related entities and drafts selected paper elements, but remains an assistant requiring human intervention; future work would broaden its extraction and writing capabilities.

Conclusions and Future Work

  • PaperRobot predicts related entities for an input title, drafts an abstract, conclusion, and future work, and predicts a follow-on title.Automatic evaluations and human Turing tests demonstrate promising performance.Source: §5 · p. 9
  • PaperRobot cannot conduct experiments, and its knowledge graphs, predicted links, and drafts still require human verification or editing.The authors note that constructed graphs cannot cover all technical details.Source: §5 · p. 9
  • Planned extensions include finer-grained entity extraction and generating related work, authors, affiliations, and publication venues.Source: §5 · p. 9

A Hyperparameters

The appendix identifies a table of model hyperparameters.

Hyperparameters

B Language model details

The appendix reports language-model perplexity scores for titles, abstracts, and conclusions.

Language model details

C Good Examples that Passed Turing Test

This section presents two tables of examples labeled as having passed the Turing Test.

C Good Examples that Passed Turing Test

  • Two tables present examples labeled as having passed the Turing Test.The tables are numbered 11 and 12.Source: Table 11 · p. 11, Table 12 · p. 12
  • Table 11 is labeled as a collection of good examples that passed the Turing Test.Source: Table 11 · p. 11
  • Table 12 is labeled as a collection of good examples that passed the Turing Test.Source: Table 12 · p. 12
Loading 1905.07870v4…