PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursively Summarizing Books with Human Feedback

Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, Paul Christiano

arXiv:2109.10862v2cs.CLcs.AIcs.LG

TL;DR

The paper asks how humans can provide training signals for tasks too difficult or time-consuming to evaluate directly. It recursively decomposes book summarization and combines human demonstrations with preference-based reinforcement learning. The resulting models generate plausible whole-book summaries, achieve state-of-the-art results on BookSum, and support competitive zero-shot NarrativeQA performance.

  • Problem

    Humans struggle to provide accurate training signals for tasks requiring substantial time or specialized expertise, while easier proxy objectives may be misaligned with actual goals.

  • Method

    The method recursively decomposes books into summarization subtasks, trains models on smaller sections, and uses behavioral cloning plus reinforcement learning from human preferences.

  • Results

    The model generates plausible summaries of entire books, achieves state-of-the-art BookSum results, and yields competitive zero-shot NarrativeQA performance from those summaries.

  • Takeaways & Limitations

    Recursive task decomposition with human feedback is presented as a practical approach to scalable oversight for difficult long-document NLP tasks.

  • Takeaways & Limitations

    Book-level summaries often lack coherence, reading more like lists of events, and task decomposition can miss information whose importance depends on later context.

Abstract

from arXiv · show

A major challenge for scaling machine learning is training models to perform tasks that are very difficult or time-consuming for humans to evaluate. We present progress on this problem on the task of abstractive summarization of entire fiction novels. Our method combines learning from human feedback with recursive task decomposition: we use models trained on smaller parts of the task to assist humans in giving feedback on the broader task. We collect a large volume of demonstrations and comparisons from human labelers, and fine-tune GPT-3 using behavioral cloning and reward modeling to do summarization recursively. At inference time, the model first summarizes small sections of the book and then recursively summarizes these summaries to produce a summary of the entire book. Our human labelers are able to supervise and evaluate the models quickly, despite not having read the entire books themselves. Our resulting model generates sensible summaries of entire books, even matching the quality of human-written summaries in a few cases ($\sim5\%$ of books). We achieve state-of-the-art results on the recent BookSum dataset for book-length summarization. A zero-shot question-answering model using these summaries achieves state-of-the-art results on the challenging NarrativeQA benchmark for answering questions about books and movie scripts. We release datasets of samples from our model.

1 Introduction

The paper addresses scalable oversight for difficult-to-evaluate tasks by applying recursive decomposition and human feedback to abstractive book summarization. The resulting system produces plausible whole-book summaries while reducing the burden of human supervision.

  • Motivation: Scalable oversight is needed because humans cannot efficiently provide accurate training signals for tasks requiring extensive time or specialized expertise.Proxy objectives may be easier to measure but are less aligned with actual goals and can have unintended consequences.
  • Approach: The paper applies task decomposition and learning from human feedback to abstractive summarization of entire fiction books.Existing book-summarization methods are typically extractive or focused on shorter stories.
  • Approach: The model recursively summarizes book chunks and then composes those summaries into summaries of progressively larger sections and entire books.The procedure uses behavioral cloning and reward modeling, with the same policy operating at all levels.
  • Conclusion: Recursive summarization makes human supervision of book-length tasks more scalable than non-recursive data collection.A non-recursive dataset would require humans to read entire books and is reported as over 50x more expensive per data point.
  • Results: The model produces summaries containing important events and sometimes abstractively synthesized details, although broader context and important details are often missed.Quantitative evaluation shows significant improvement over the behavioral-cloning baseline, while only a small number of summaries approach human-level quality.
  • Results: The approach achieves state-of-the-art BookSum results and competitive zero-shot NarrativeQA performance using the generated summaries.These evaluations cover book-length summarization and question answering about books and movie scripts.

2 Approach

The approach recursively decomposes book summarization into simpler tasks, trains one model with demonstrations and preference-based reinforcement learning, and composes outputs up the resulting task tree. Previous summaries provide context, while training prioritizes earlier nodes to address distribution shift.

  • 2.1 Task decomposition: Task decomposition recursively breaks a parent task into simpler subtasks whose responses help a human supervise the parent.Leaf tasks are designed to be simple enough for humans to demonstrate or evaluate.
  • 2.2 Decomposition for book summarization: For summarization, short text is summarized directly, whereas longer text is chunked and summarized recursively into a tree.Only leaf tasks operate on original book passages; higher-level composition tasks consume concatenated summaries.
  • 2.2 Decomposition for book summarization: Previous summaries from the same depth are concatenated as context for later tasks, helping summaries flow across book sections.The authors found this context improved behavioral-cloning log loss, while previous unsummarized text performed worse under limited context length.
  • 2.3 Training: Training combines human demonstrations for behavioral cloning with human comparisons, reward modeling, and reinforcement learning.The process can be iterated with newer models, alternative node-sampling strategies, and different data types.
  • 2.3 Training: Auto-induced distributional shift becomes more severe later and higher in the tree because each model receives inputs produced by another model.The authors therefore prioritize training on earlier, lower nodes before later computation stages.
  • Implications: The procedure lets humans supervise parts of a task and generalizes to books of unbounded length at test time.The paper states that humans can use lower-level summaries to summarize quickly, while recursion supports longer books than those seen during training.

3 Task details

The experiments focus on primarily fictional books averaging over 100K words, with evaluation centered on abstractive narrative summarization. Human raters judge quality while conditioning on summary length and considering only each subtask’s direct input.

  • Dataset: The training corpus uses primarily fiction books averaging over 100K words and excludes non-narrative books.The books are drawn from a subset of books in GPT-3’s training data.
  • Dataset: Narrative fiction is selected because its low-level sentence descriptions make abstractive summarization particularly challenging for extractive methods.The authors report later qualitative support for this motivation and model gains over an extractive oracle on BERTScore.
  • Evaluation: The target is to trace narrative arcs and larger themes rather than list events, with overall quality as the primary evaluation metric.Labelers use a 1–7 Likert scale on held-out books and also assess accuracy, coverage, coherence, and abstraction.
  • Evaluation: Each subtask generally compresses its input by 5–10x, with length limits of 128–384 tokens depending on task height.Labelers evaluate quality conditioned on the summary’s length to avoid simply preferring the longest allowed output.
  • Evaluation: Labelers judge each subtask against its direct model input rather than the corresponding subset of the full book.This evaluation condition separates local task quality from the true end-to-end summarization target.

4 Results

Full-book evaluations show that recursive models produce plausible summaries, with reinforcement learning outperforming behavioral cloning in several settings, while remaining below human quality on average. On BookSum, 175B models outperform non-oracle baselines, and comparisons become more human-time efficient than demonstrations.

  • Full-book evaluations: RL models significantly outperform BC models at larger model sizes, but model performance remains far from human performance.Figure 2 reports full-book evaluations as functions of model size and number of labels.
  • Full-book evaluations: Over 5% of summaries from the best 175B model scored 6 out of 7, while over 15% scored 5 out of 7.These scores were sometimes also assigned to human-written summaries, although average model quality remained significantly lower than human-written summaries.
  • Full-book evaluations: Training on the first subtree performs comparably to training on the full tree and generalizes to full-book summarization.The authors also report that 175B RL policies outperform the 175B BC baseline, with a smaller improvement for 6B models.
  • BookSum results: 175B models beat all non-oracle BookSum baselines on ROUGE by 3-4 points and significantly outperform all baselines on BERTScore, including the extractive oracle.The 6B models are comparable to baselines on ROUGE while outperforming all baselines on BERTScore, including an 11B T5 model fine-tuned on BookSum.
  • Human label efficiency: RL comparisons become more efficient than BC demonstrations after 10k-20k demonstrations, and comparisons take 3x less time to collect.After 5k-10k demonstrations, RL comparisons were about as effective as BC demonstrations; the larger efficiency advantage appeared on the margin.

5 Related work

The paper builds on human-feedback reinforcement learning and task decomposition, applying these ideas to long-form fiction summarization. It relates recursive assistance for human labeling to broader decomposition and iterated-amplification approaches.

  • Human feedback and decomposition: The approach is framed as a fixed-decomposition form of iterated amplification that begins training from leaf tasks rather than the entire tree.It is also described as related to recursive reward modeling.
  • Human-feedback learning: Human-feedback reinforcement learning has been applied across dialogue, translation, semantic parsing, story generation, review generation, evidence extraction, and simulated agents.These applications provide precedent for using human feedback beyond summarization.
  • Long-form summarization: Prior novel-summarization work is sparse and has primarily used graph-based or extractive methods, often targeting chapters or shorter texts.The paper positions abstractive book-length fiction summarization as a less explored setting.
  • Long-form summarization: Related long-document summarization research includes scientific papers, patents, and multi-document collections.This literature addresses other long-document domains rather than focusing specifically on novels.
  • Task decomposition: Task decomposition has been used in NLP to generate stories through staged prompts and to use lower-level models to assist human labeling of higher-level tasks.These examples connect hierarchical generation and human-time savings to the paper’s recursive strategy.

6 Discussion

The paper argues that recursive task decomposition and human feedback make difficult long-document summarization more scalable, while identifying important limits in coherence, context, training, and generality.

  • Motivation: Human feedback on difficult tasks is motivated by scalable oversight: producing effective training signals becomes hard when supervision requires substantial time or expertise.The paper frames this challenge as central to optimizing models for tasks humans care about.
  • Conclusions: The approach is feasible for abstractive book summarization and may generalize beyond the fixed decomposition strategy used here.The authors present this as progress toward human-judged summarization performance and broader scalable oversight.
  • Limitations: A single model mapping an entire book directly to a summary is not addressed because it would require a very large context window and add complexity.The authors also note that most compute occurs at leaf tasks, so distillation would not save much test-time compute.
  • Limitations: Book-level summaries often contain important information but lack coherence, reading more like event lists than human-written summaries.More top-level reinforcement learning might help in theory, but the authors found higher-level RL challenging in practice.
  • Limitations: Local task inputs can cause inaccurate summaries when later revelations or broader context determine the meaning of earlier details.The paper gives a Pride and Prejudice example where “their hand” is summarized as a marriage proposal rather than a request to dance.
  • Limitations: Task decomposition may miss themes distributed across many individually minor details, and the kinds of tasks suitable for decomposition remain an open problem.The authors suggest learned decompositions may alleviate some issues but may not resolve all of them.
  • Limitations: Errors can compound across composition levels, while curriculum and node-sampling choices were selected ad hoc and full-tree training did not improve performance.The paper identifies auto-induced distributional shift and curriculum choice as possible contributors.
  • Broader impacts: The models make many summarization mistakes and should not be deployed where high accuracy is necessary; preserved book intent may also retain harmful or biased content.These constraints define a practical boundary for automatic book summarization.

A.4 General task decomposition pseudocode

The pseudocode implements recursive decomposition by repeatedly obtaining subtasks and responses, then synthesizing them into an answer; book summarization recursively processes chunks while preserving prior context.

  • General decomposition: A task receives the primary task and subtask-response pairs, which are assumed to come from earlier recursive invocations.The subtask responses assist the model in answering the primary task.
  • General decomposition: The decomposition loop recursively solves a returned subtask, appends its response, and repeats until the task can be answered directly.The base path returns an answer synthesized from the task and accumulated subtask pairs.
  • General decomposition: decompose_if_needed either signals Respond when direct synthesis is possible or returns Decompose(subtask) when further help is required.Both decomposition and direct answering could be learned, although fixed decomposition implements the former programmatically.
  • General decomposition: Decompose returns one subtask at a time, allowing later child subtasks to depend on earlier results.This creates an ordered dependency among child subtasks rather than solving an independent list in parallel.
  • Context handling: Previous context stores same-depth summaries written earlier, such as summaries for pages 1–4 when summarizing pages 5–6.This context is tracked by depth and concatenated in order.
  • Book decomposition: For book summarization, small text is summarized directly, whereas long text is split into chunks whose summaries are recursively generated and then concatenated for final synthesis.The recursive process repeats across all chunks before producing the parent summary.
  • Book decomposition: The final concatenated chunk summaries are summarized while conditioning on previous context, so the output flows from summaries preceding the current text.At higher levels, the input consists of child summaries rather than the original text alone.
  • Human supervision: Labeler training used hourly pay, onboarding materials, and quality-control feedback from additional labelers.The process also included ongoing communication with labelers.

B.3 Task interface

The task interface defines summary quality through coverage, accuracy, coherence, and abstraction, while requiring summaries to connect naturally to prior context and controlling evaluation for length.

  • Interface: The interface uses customized renderers for demonstrations, comparisons, and final evaluations.Figure 5 is identified as an example of a demonstrations renderer.
  • Quality criteria: Coverage requires including important information without omitting more important content or adding gratuitous minor details.Both omission and unnecessary inclusion are penalized.
  • Quality criteria: Accuracy requires every summary claim to faithfully reflect the original passage, while coherence requires the summary to remain clear and logically consistent without the passage.These criteria evaluate fidelity and standalone readability.
  • Quality criteria: Abstraction favors describing larger arcs and themes rather than merely listing events, especially at higher composition levels.This criterion is primarily applicable at higher height.
  • Interface guidelines: Summaries should flow from the end of the previous context, and labelers are given style guidelines for pronouns, tense, uncertainty, scene changes, and preamble removal.These guidelines apply to demonstrations and comparisons.
  • Length control: Labelers compare summaries overall while controlling for their different lengths by judging how well each uses its own token budget.The study reports that length had an insignificant effect on summary quality under this procedure.
  • Length control: Demonstrations use varied suggested target lengths within the acceptable range, with approximately 20% headroom in either direction.Suggested targets typically range from roughly 100 to 200 BPE tokens.

C.2.2 First leaves “contamination”

The first-leaves contamination setup combines comparison collection, diagnostic ratings, supplementary annotations, and model-training procedures while tuning supervised baselines across temperatures.

  • Contamination: Labelers sometimes completed first-leaf tasks consecutively, exposing them to more previous context than the model saw for the same task.This creates a mismatch between human and model context during data collection.
  • Contamination: During contaminated comparisons, labelers generally saw matched previous contexts, but the reward model temporarily received summaries with different contexts.The mismatch affected reward-model inputs for some collected data.
  • Data collection: Comparison collection could present up to three pairs among three summaries for one input, increasing efficiency at the cost of correlated data.Overlapping pairs were allowed.
  • Diagnostics: Diagnostic collection included Likert ratings, input-text coherence ratings, inaccurate or incoherent span annotations, questions, and free-form task notes.These measurements were collected at various points in time.
  • Diagnostics: Supplementary diagnostic datasets did not affect the primary demonstration or comparison task and were intended for future experimentation.They were separate from the main training-signal collection.
  • Training: The implementation follows standard BC and reward-model training choices, including one epoch, cosine decay, Adam, reward normalization, and tuned RL hyperparameters.RL runs used up to 200,000 episodes, with 150,000 used for most of the project.
  • Temperature tuning: For leaf-task baselines, the 6B supervised model performed best around T=0.6, whereas the 175B supervised model performed best around T=0.3.The temperatures were selected after labelers evaluated models across a sweep.
  • Temperature tuning: Better temperatures for individual tasks later predicted performance on full-book summarization tasks, although higher-level estimates were noisier.The same overall pattern was observed at higher levels.

E.1 First leaves

The section examines human-data collection and training choices for leaf summarization tasks, emphasizing the efficiency of comparisons and unresolved implementation issues.

  • Comparisons were about 40% faster than demonstrations when reading time was included.For leaf tasks, reading took 2.5 minutes, while written demonstrations took 4 minutes and comparisons 1.5 minutes.
  • Amortizing read time made three comparisons nearly 3x faster than one demonstration.Estimated collection times were 2.3 minutes versus 6.5 minutes, while empirical timing was 1.8 minutes.
  • RL’s advantage became more apparent when performance was plotted against estimated human time.The comparison favored RL despite each comparison yielding less information than a demonstration.
  • Reading and writing a full-book summary took over 50 times longer than one decomposed summarization task.A labeler needed over 12 hours to read a book and over 1 hour to write its summary.
  • The authors identified unresolved issues including data contamination, reward-model initialization, RL initialization, and limited exploratory feedback training.They also reported that postamble filtering was unnecessary and Likert-score reward modeling matched binary comparison models in accuracy.

G Difficulty and mysteries of full tree training

Full-tree training produced mixed results, with several plausible explanations involving tuning, degraded higher-level supervision, node sampling, and distribution shift; NarrativeQA evaluations also exposed strong QA-model effects.

  • Full-tree training did not improve performance, and the authors lacked a definitive explanation.They could not rule out a training-code bug or randomness across RL runs.
  • The authors did not tune the 175B models extensively because of compute costs.This was listed as one possible reason full-tree training failed to improve performance.
  • Higher-level training can degrade supervision when input summaries are incoherent and labeler agreement falls.The authors caution that training higher-level tasks too early may weaken the training signal.
  • The RL node-sampling strategy may have underrepresented height-0 tasks despite their importance during evaluation.The best full-tree model sacrificed lower-height performance for higher-height performance, while another later model declined at all heights.
  • Pretrained GPT-3 achieved extremely strong NarrativeQA performance without summaries, including state-of-the-art results across metrics except ROUGE-L for the 175B model.This result contrasts with the summary-based QA ablations and indicates a strong effect of the QA model itself.
  • Question augmentation improved correct answerability from 10 to 29 questions out of 100, with partial or inferable answers rising from 12 to 8.The passage also reports that many NarrativeQA examples appeared to contain incorrect book-question pairings.
  • Recursive summaries enabled abstractively inferred answers and reduced repeated full-book processing, but questions unanswered by the summary remained inaccessible.The approach also retained source traceability through the decomposition tree.
  • Controlling for summary length with linear regression did not significantly change BERTScore.The model’s BERTScores remained consistently higher than average across overlapping summary lengths.

J.1 Limitations observed by labelers/researchers

Labelers found the summaries often preserved events but struggled with character attribution, narrative essence, abstraction, chronology, and language-specific or pretrained-knowledge effects.

  • The model frequently confused characters, misattributed actions, and represented interpersonal relationships incorrectly.Pronoun resolution, long dialogues, and summary concatenation exacerbated these errors.
  • Many summaries omitted the story’s important information or failed to integrate essential world-building details.Science fiction and fantasy books were particularly difficult when their speculative elements were important.
  • The model often failed to abstract from specific events into mental states, authorial intent, or coherent long-range developments.This limited its ability to capture higher-level narrative meaning.
  • Summaries tended to emphasize earlier material and handle scene switches or flashbacks poorly.Non-chronological narratives and changes between universes were cited as difficult cases.
  • The model leveraged pretrained knowledge in ways that could help or introduce false information.Examples included inferring Leia from Anakin Skywalker’s daughter and hallucinating characters from related books.
  • The model sometimes partially translated bilingual text instead of preserving its original meaning.One example converted a Spanish phrase into an English paraphrase in the summary.
  • Fiction books received lower Likert ratings than the six nonfiction books in the full-book evaluation.The only judged non-narrative book had the second-highest average rating, though the authors call this weak evidence.

K.2 Book samples

The appendix presents held-out book-summary examples spanning multiple quality levels and model policies, alongside evaluation-book metadata and additional sample tables.

  • The sample appendix shows summaries selected at Overall Likert scores from 2 to 6.Examples compare a human-written summary with outputs from the 175B RL and behavioral-cloning policies.
  • The evaluation books were published in 2020 and therefore were absent from the model’s pretraining data.Table 5 also records genre labels derived from Goodreads user classifications.
  • The samples cover “In an Instant,” “Transcendent Kingdom,” “The Vanishing Half,” “Big Summer,” and “Dear Edward.”Each table selects a book where the RL policy achieved the stated target score.
Loading 2109.10862v2…