PaperlayerView exact revision on arXiv

Source-linked AI summary

LongZero: Enhancing Long-Context Reasoning in LLMs from Zero Data

Zhennan Wan, Jianfei Chen

arXiv:2610.30132v1cs.CLcs.LG

TL;DR

Long-context RL needs long, challenging, verifiable training tasks, but existing sources are costly, scarce, or external, and prior zero-data methods target short contexts. LongZero has the policy specify questions, evidence, and answers before expanding contexts, then trains on self-generated reasoning and retrieval tasks. Across two base models and five benchmarks, it generalizes to real tasks and outperforms external-corpus baselines.

  • Problem

    Long-context RL requires long, diverse, challenging, verifiable data, but human annotation is expensive, naturally occurring QA is scarce, and existing synthesis depends on external resources.

    Source: §1 · pp. 1–2

  • Method

    LongZero has the policy write a structured question, evidence, and answer before expanding evidence into long contexts, and combines self-generated reasoning and retrieval tasks.

    Source: §1 · p. 2

  • Results

    Across two base models and five long-context benchmarks, LongZero generalizes to real tasks and outperforms both external-corpus synthetic-data baselines.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    LongZero demonstrates a path to improving long-context reasoning using tasks generated by the model itself, without external data.

    Source: §5 · p. 8

Abstract

from arXiv · show

Reinforcement learning (RL) is an effective way to strengthen the long-context reasoning of large language models (LLMs), but it needs training data that are long, diverse, challenging, and verifiable. Human annotation of such data is expensive, naturally occurring data are too scarce to sustain RL, and existing synthesis methods depend on external corpora or stronger teacher models, which become harder to obtain as the trained models get stronger. Recently, RL from zero data, in which a model learns from tasks it proposes itself, has achieved excellent performance, but existing methods target short-context tasks. Therefore, we propose $\textbf{LongZero}$, a method that lets an LLM improve its own long-context reasoning without any external data. Instead of writing questions and answers over existing documents, LongZero first has the model synthesize the question and answer with its supporting evidence in a compact structured form, and only then separately expands the evidence into a long multi-document context. LongZero combines these self-written multi-hop reasoning tasks with self-written multi-round retrieval tasks and trains the model with GRPO. We evaluate LongZero on two LLMs across five long-context reasoning benchmarks and compare it with RL on data synthesized from external corpora. Experimental results show that LongZero can generalize to real tasks, and outperforms both baselines, suggesting a viable and promising path to improving long-context reasoning of LLMs.

1 Introduction

LongZero addresses the challenge of obtaining long, difficult, verifiable training tasks without relying on external data by having the policy specify tasks before expanding them into long contexts. It combines self-generated reasoning and retrieval tasks, and its evaluations report generalization and gains over external-corpus baselines.

1 Introduction

  • Long-context RL data must be long, retrieval-intensive, challenging, and verifiable, yet human annotation is expensive and naturally occurring QA data are scarce.Existing synthesis pipelines also depend on external data or models, while real-world facts can introduce parametric-knowledge and evaluation-overlap confounds.Source: §1 · pp. 1–2
  • LongZero reverses task generation: the model first writes a structured question, evidence, and answer, then expands the evidence into a long multi-document context.This fixes the answer before the long text exists, leaving expansion as the stated remaining risk of error.Source: §1 · p. 2
  • LongZero synthesizes multi-hop reasoning tasks over eight fictional-world documents and multi-round retrieval tasks that recover an earlier response from a long conversation.Both task types are self-written by the model being trained.Source: §1 · p. 2
  • Questions answered both correctly and incorrectly across repeated attempts are retained because those cases provide GRPO a non-zero learning signal.The passage describes this filtering criterion for the reasoning questions.Source: §1 · p. 2
  • LongZero generalizes to long-context reasoning tasks and outperforms both external-corpus synthetic-data baselines.The evaluation covers two base models and five long-context reasoning benchmarks.Source: §1 · p. 2

2 Related Work

Prior zero-data RL methods let models propose tasks, but target short-context problems; long-context reasoning additionally requires synthesizing a consistent context. Long-context RL studies commonly draw tasks or contexts from external resources, while evaluation has shifted toward realistic reasoning benchmarks.

2 Related Work

3 LONGZERO

LONGZERO uses the model being trained to synthesize verifiable long-context tasks and then trains it on them with GRPO. Its data combine multi-hop reasoning over fictional documents with retrieval from long, self-written conversations.

3 LONGZERO

  • The model synthesizes its own reasoning and retrieval data, then optimizes on both with GRPO.The answer is determined by construction rather than by judging a long text.Source: §3 · p. 3, §3.1 · p. 3

3.1 Data Synthesis

  • Reasoning samples combine facts scattered across long documents, distractors, and questions requiring multiple hops.Question difficulty is guided by context length, hop count, and distractor count; the relation is used as a design guide, not a fitted model.Source: §3.1.1 · p. 3

3.1.1 Reasoning Data Synthesis

  • For reasoning data, the model fixes questions, supporting facts, and answers in a compact specification before expanding facts into long documents.Facts are assigned across documents, distractors are placed away from the facts they imitate, and questions are retained only when the answer or an equivalent alias appears in the expanded text.Source: §3.1.1 · pp. 3–4

3.1.2 Retrieval Data Synthesis

  • Retrieval samples ask the model to reproduce a selected response, prefixed by a random string, from a long conversation of its own responses.A program determines the target needle and its order, making the reference answer correct by construction; the task requires distinguishing near-duplicates and tracking order.Source: §3.1.2 · p. 4

3.1.3 Difficulty Filtering

  • Difficulty filtering keeps questions the model answers both correctly and incorrectly, discarding questions whose first W answers agree.This targets questions that can contribute a GRPO gradient; consistently failed questions were also often defective. The final training mixture contains 2,304 reasoning questions and 1,152 retrieval conversations.Source: §3.1.3 · p. 5

3.2 Reinforcement Learning

  • GRPO trains on grouped outputs using group-normalized advantages, a token-level policy-gradient loss, and a per-token KL penalty.Reasoning rewards are binary answer matches, while retrieval rewards require the target prefix and then score similarity to the selected needle, giving partial credit for near-correct reproductions.Source: §3.2 · pp. 5–6

4 Experiments

The experiments evaluate LONGZERO on two base models and five long-context benchmarks against external-data RL baselines. LONGZERO achieves the highest average scores, with gains across tasks, models, and context lengths, while its two synthesized data types work best in combination.

4.1 Settings

  • The training setup uses Qwen3-4B-Thinking-2507 and Qwen3-8B, evaluates five long-context benchmarks, and compares against two baselines that differ in reasoning-data source.All methods share the RL configuration; the reported results use each run’s final checkpoint, and benchmark scores are means of at least two runs.Source: §4.1 · pp. 6–7

4.2 Results and Analysis

  • LONGZERO achieves the highest average score on both base models and is best on four of five benchmarks for each, counting a tie on AA-LCR for Qwen3-8B.All three RL methods improve over the base models on average, but LONGZERO gains the most; the teacher-synthesized baseline does not surpass it.Source: §4.2 · p. 7
  • LONGZERO transfers to general long-context benchmarks, including real-entity and much-longer-input tasks, despite training on fictional worlds and shorter contexts.For Qwen3-4B-Thinking-2507, it improves on every benchmark; the gains span extractive, multiple-choice, and model-judged free-form answers.Source: §4.2 · pp. 7–8
  • LONGZERO improves both base models, but its gains are clearly larger on Qwen3-4B-Thinking-2507; Qwen3-8B does not improve on LBv2.All methods score near zero on AA-LCR for Qwen3-8B, where 120 of LONGZERO’s 200 responses contain no answer because the model often runs out of output tokens.Source: §4.2 · pp. 7–8
  • The 2:1 mixture of reasoning and retrieval data is better on average than either alone and improves over the base model on all five benchmarks.The mixture is best on four benchmarks and avoids losses that either data type causes on its own; reasoning teaches cross-document fact chaining, while retrieval teaches locating and reproducing an exact item.Source: §4.2 · p. 8, Table 3 · p. 8

5 Conclusion and Future Work

LONGZERO trains a model on its own structured question-evidence-answer specifications, expanded into long documents and combined with self-synthesized retrieval tasks. Across two models and five benchmarks, it achieves the highest average score and outperforms RL baselines built from external data; the paper also outlines extensions to the method.

5 Conclusion and Future Work

  • LONGZERO has the trained model write a structured question, evidence, and answer before expanding them into long documents, then combines reasoning tasks with self-synthesized retrieval tasks.It keeps questions the model answers inconsistently, with answers determined by construction and no external corpus, human annotation, or teacher model.Source: §5 · p. 8
  • Across two base models and five long-context reasoning benchmarks, LONGZERO achieves the highest average score and outperforms RL on external-corpus data and questions written by a much larger teacher model.The conclusion reports transfer from fictional to real-world documents and from short training contexts to much longer inputs, with the two data types complementary.Source: §5 · p. 8
  • Future work includes iterating self-regeneration and refiltering, synthesizing longer contexts, adding counting, comparison, or arithmetic questions, and improving data-correctness checks.Source: §5 · p. 8

A.1 Experiment Details

The appendix details reasoning-data generation and filtering, retrieval-conversation construction, shared RL training, baselines, and benchmark evaluation. It also reports compute requirements and points to the prompt templates.

A.1 Experiment Details

  • World specifications set 8–14 entities, four question chains of 3–5 hops, and 2–3 distractor facts per chain; invalid worlds and unsupported questions are filtered.Plans distribute facts across eight documents, which are expanded into long prose; worlds exceeding 27.5K estimated context tokens are discarded.Source: §A.1 · p. 12
  • Retrieval conversations combine pooled model-written responses with distractor turns and eight needle turns, targeting 8,193–16,384 tokens and asking for a randomly selected earlier response.For Qwen3-4B-Thinking-2507, median length is 11.4K tokens; for Qwen3-8B, it is 11.1K.Source: §A.1 · p. 13
  • LONGZERO and both baselines share RL hyperparameters; Qwen3-4B-Thinking-2507 runs take 72 steps, while Qwen3-8B runs take 108; no hyperparameter was tuned on evaluation benchmarks for the final experiments.The held-out validation set is evaluated every 6 steps with greedy decoding, and checkpoints are saved every 18 steps.Source: §A.1 · p. 13
  • The baselines use distractor-augmented HotpotQA and teacher-synthesized QA, with model-based difficulty filtering and separate document-disjoint training and validation samples for the latter.The teacher baseline uses 2,304 training and 256 validation questions per model after filtering and deduplication.Source: §A.1 · p. 13
  • Evaluation covers LB1, LBv2, FRAMES, AA-LCR, and LongReason with benchmark-specific prompts, context limits, answer rules, or judging procedures.Each benchmark is run at least twice, with the mean of two runs reported; evaluation of one checkpoint across all five benchmarks takes about 12 hours.Source: §A.1 · p. 14

A.2 Prompt Templates

  • Appendix A.2 lists the prompts used for LongZero.The appendix also specifies seed-list choices for reasoning worlds, including 12 domains and eight document genres.Source: §A.1 · p. 12, §A.2 · p. 14

LONGZERO: reasoning data

The reasoning-data prompts construct fictional worlds with linked multi-hop questions, distribute supporting facts across documents, and expand those facts into long documents. A separate prompt asks for evidence chunks and a short answer.

LONGZERO: reasoning data

  • The specification prompt requires 8–14 named entities, four 3–5-hop question chains, and 2–3 distractor facts per chain, with chains sharing bridge entities.Questions refer to intermediate entities descriptively, answers use concrete names or attributes from facts, and at most one answer may be a bare year.Source: LONGZERO: reasoning data · p. 14
  • The planning prompt distributes every fact into one of eight documents and places facts supporting the same question across different documents where possible.Distractors are assigned to documents different from the facts they could confuse with, where possible.Source: LONGZERO: reasoning data · p. 15
  • Expansion prompts request long, genre-specific documents that state each assigned fact exactly once while avoiding forbidden facts and references to other documents.They also require in-world prose without meta-commentary about the instructions or embedding strategy.Source: LONGZERO: reasoning data · p. 15
  • The filtering prompt asks the model to identify evidence chunks and provide a short answer using only the supplied documents.It requires a response with the useful chunk identifiers and answer enclosed in specified tags.Source: LONGZERO: reasoning data · p. 15
  • The RL prompt requests step-by-step reasoning grounded in the passages and a precise, succinct final answer.Its final line must use the format “The answer is <answer>.”Source: LONGZERO: reasoning data · p. 15

LONGZERO: retrieval data

The retrieval-data prompts build long multi-turn conversations around repeated requests for the same format and noun. The final request asks for a selected earlier response prefixed by a random string.

LONGZERO: retrieval data

  • The response-pool prompt asks for a single-turn piece in a selected format about a selected noun.Source: LONGZERO: retrieval data · p. 15
  • Conversation examples demonstrate repeated requests for the same format and noun, with a later instruction to prepend a string to an earlier response.Source: LONGZERO: retrieval data · p. 15
  • The conversation template ends by asking for the k-th response among repeated requests, with k expressed as an ordinal and a random 10-character alphanumeric string as the prefix.The prompt instructs the model to return no other text.Source: LONGZERO: retrieval data · p. 15

Baselines and evaluation

The appendix notes which prompts are reused for baselines and benchmarks, and identifies a separate article-heading format for FRAMES.

Baselines and evaluation

B Additional Results

Additional results break down LB1 and LBv2 scores across models, datasets, difficulty, input length, and ablation variants. The case studies show how a fully correct multi-document question is supported by separate facts and remains nontrivial for the model.

B Additional Results

C Case Studies

  • The case studies present a correct training question, a defective question removed by filtering, and a defective question that passes filtering.Source: C Case Studies · p. 17

C.1 A Fully Correct Question

  • The fully correct question asks who used Vitae Extract developed in 1885, combining supporting facts from Documents 1 and 8 to identify Sofia Bianchi.Its distractors assign Bianchi other roles at different times without contradicting the supporting facts.Source: §C.1 · p. 17
  • The correct question remains nontrivial because its two filtering answers disagree, placing it in the unstable band and keeping it for training.One incorrect answer names Marco Rossi, introduced in the same document as the first supporting fact.Source: §C.1 · p. 17

C.2 A Defective Question Removed by Filtering

The discarded example passes the faithfulness gate but asks for a first-use year contradicted by its documents, so all filtering answers disagree with the reference answer. A separate example shows that a contradictory distractor can nevertheless pass difficulty filtering and enter training.

C.2 A Defective Question Removed by Filtering

  • The discarded question’s answer, 1897, appears in the text, but two distractors give earlier uses by the same person, so the documents contradict the question’s “first use” premise.All four filtering answers disagree with the reference answer, and the case illustrates that questions the model never answers correctly may be defective rather than hard.Source: §C.2 · p. 17

C.3 A Defective Question That Passes Filtering

  • A distractor assigns the same founder role to a different person, making both names document-supported; the question enters training because the model gives each answer once.Difficulty filtering cannot detect this contradiction when the reference answer is sometimes reproduced; checking specifications before expansion would remove such cases.Source: §C.3 · p. 17
Loading 2610.30132v1…