Learning to Learn from Context: Synthetic Training from Perturbed Public Documents

• Context learning [11] requires applying a document’s rules, facts, and procedures rather than merely retrieving facts, yet frontier models solve fewer than 30% of CL-bench tasks [11].CL-bench covers 1,899 long-document tasks across 18 domains, and plausible-sounding answers often fail fine-grained checking.Source: §1 · p. 2
• Pretraining exposure to long documents does not teach document-conditioned answering, while short-context instruction data offers little relevant supervision and expert annotation is exceptionally costly.Annotators must design lengthy documents, verify answers against them, and specify what correct answers must contain.Source: §1 · p. 2
• Real-document supervision risks memorization from pretraining [28], while synthetic contexts tend to be simpler and more homogeneous, and synthetic-haystack fine-tuning underperforms real-document training [54].Source: §1 · p. 2
• The pipeline perturbs public documents through entity renaming and numeric changes, generates questions and rubrics, then filters for samples with higher rubric pass rates when the document is provided.It produces about 10k samples from 3.5k documents without human annotators; rewriting a known standard caused a model’s unaided answers to fall from perfect to nearly zero.Source: §1 · p. 2
• SFT raises Qwen3.6-35B-A3B from 13.7% to 22.8% on CL-bench, and rubric-reward RL reaches 24.6%, comparable with HY3 (23.5%) and Qwen3.8-2.4T (23.9%).Source: §1 · pp. 2–3
• AALCR improves from 62.6 to 69.8, IFBench from 57.7 to 71.7, and ARC-AGI-1 from 47.4 to 63.8, while SimpleQA declines from 20.9 to 18.7.The transfer spans long-context understanding, instruction following, and reasoning, while knowledge benchmarks decline slightly.Source: §1 · p. 3

2 The Synthesis Pipeline

• Each sample pairs a document, question, rubric list, and teacher answer, with deep rewriting and a gap check enforcing context dependence and verbatim reasoning traces preserving native reasoning.Rubrics are retained as metadata for reuse as rubric-based RL prompts.Source: §2 · p. 3

2.1 Source acquisition and deep rewriting

• The corpus contains 3,515 public documents across 16 of CL-bench’s 18 subcategories, with recent sources preferred and each document deeply rewritten to reduce memorization risk.Sources include IETF RFCs, SEC filings, court opinions, arXiv papers, handbooks, wikis, and encyclopedias; rewriting perturbs names, numbers, section numbering, and list order.Source: §2.1 · pp. 3–4

2.2 Question generation

• For each rewritten document, one batched generator call produces up to three questions, each paired with at least seven rubrics, including at least five tied to specific document values.Question generation varies across 9 persona archetypes, 7 question types, and 4 rubric types.Source: §2.2 · p. 4

2.3 Answer generation

• The teacher answers using the rewritten document, and the training target preserves its verbatim reasoning-content followed by the answer, without post-hoc synthesis or revision.The authors report that native reasoning traces are more effective than polished or cleaned versions, even if they may not satisfy some rubrics.Source: §2.3 · p. 4

2.4 The gap check

• Samples pass the gap check only when document-context and memory-only rubric pass rates meet p+ ≥0.60, p−≤0.30, and p+ −p−≥0.40.The teacher answers both with and without the rewritten document; 32.5% of generated samples fail, and rejected samples may be retried with regenerated questions and rubrics.Source: §2.4 · p. 4

3.2.2 Reinforcement learning

• 24.6 CL-bench and 13.8 CL-bench Life are the best scores in the series, up from 22.8 and 12.6 with SFT.The RL model starts from the best SFT model; applying RL directly to the base model raises CL-bench only from 13.7 to 16.9, consistent with SFT providing an initialization that yields more informative rubric rewards.Source: Table 1 · p. 5, §3.2.2 · p. 5

3.2.1 Supervised fine-tuning

• 22.8 CL-bench and 12.6 CL-bench Life show that synthetic-data SFT substantially improves the student over its baseline.Results depend on the answer-generating teacher: CL-bench scores range from 19.7 to 22.8; Kimi-K3, the strongest teacher at 27.0, yields the weakest student, while Qwen3.8-2.4T, mid-ranked at 23.9, yields the best.Source: Table 1 · p. 5, §3.2.1 · p. 5

3.1 Setup

• The experiments evaluate task accuracy on CL-bench and CL-bench Life with GPT-5.1 judging, and assess broader capabilities in long-context understanding, instruction following, reasoning, code generation, and knowledge.The student is Qwen3.6-35B-A3B; Qwen3.8-2.4T is the default SFT data-generation teacher.Source: §3.1 · pp. 4–5

3.2.2 Reinforcement learning

• 24.5 versus 24.6 CL-bench after RL shows that models initialized from GLM-5.2 and Qwen3.8-2.4T SFT converge to similar scores despite their 19.8-versus-22.8 SFT gap.CL-bench Life remains teacher-dependent after RL: the GLM-5.2-initialized model scores 9.1 versus 13.8 for Qwen3.8-2.4T, retaining an advantage when evaluation departs from the training distribution.Source: §3.2.2 · p. 5

3.2.3 General capability improvements

• +14.0 IFBench, +16.4 ARC-AGI-1, and +2.9 LongBench v2 are examples of SFT gains in instruction following, reasoning, and long-context understanding.SFT also improves AALCR (+5.4), MRCR (+2.1), AdvancedIF (+3.3), ARC-AGI-2 (+11.8), GPQA (+5.1), and HMMT-2025 (+4.1); code generation is essentially unchanged, while knowledge is mostly flat with small declines.Source: Table 2 · p. 6, §3.2.3 · p. 6
• RL improves AALCR, GPQA-Diamond, and HMMT-2025 over SFT, but declines on ARC-AGI-1 (-12.0) and ARC-AGI-2 (-9.4).Thus, its additional benefit is concentrated on the target capability rather than transferring uniformly across the general-capability board.Source: Table 2 · p. 6, §3.2.3 · p. 6
• AALCR reaches 69.8 for the 35B student versus 67.0 for HY3-preview and 71.2 for Qwen3.5-397B, while SuperGPQA trails 66.1 to 71.0 and SimpleQA 20.4 to 52.4.The pattern is consistent with SFT data focused on long-context learning rather than teaching general knowledge.Source: §3.2.3 · p. 6

4.1 The Gap Check

• Standard-dataset training achieves 19.17% accuracy versus 17.75% for unfiltered training, supporting the gap check’s effectiveness.The gap check filters out 32.5% of generated samples before retrying.Source: §4.1 · pp. 6–7, Table 3 · p. 7

4.2 Data scaling

• Performance rises monotonically from 13.7 for the untrained base to 22.8 with the full 9,625-sample pool, with no saturation within the tested range.At 1,000 samples, performance grows from 16.0 to 20.2; the authors note that the model’s learning capacity may not be exhausted by the current pool.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 The RL Reward

• A length penalty drops CL-bench accuracy from 24.6 to 20.91; adding overall task pass rate lengthens reasoning traces but does not improve accuracy over SFT.The authors suggest the sparse task-level reward may provide insufficient feedback; they also report that the RL model’s longer traces accompany an about 2% failure rate on rubrics requiring hallucination avoidance.Source: §4.3 · pp. 7–8, Table 4 · p. 8

4.4 The Performance on Fictional Contexts

• Fictional-context accuracy rises from 17.5% baseline to 31.1% with SFT and 32.9% with SFT+RL, versus public-document accuracy of 9.0%, 12.2%, and 14.1%, respectively.The authors infer that the training data teaches a general reasoning mode that transfers to fictional contexts. Their rough audit classified 184 of 500 contexts and 1,063 of 1,899 tasks as fictional.Source: §4.4 · p. 8

Synthetic supervision over long documents

• Real-document supervision uses contexts assembled from multiple passages [5] [47] [16] or single long documents in specific domains [24] [28] [51], which may have appeared in pretraining [28].Source: §5 · p. 8
• Programmatic [53] and prompt-based [38] document synthesis scales freely but produces logically simpler, more homogeneous contexts.Source: §5 · p. 9
• Perturbing real documents aims to prevent answers from parametric memory and elicit teacher reasoning over document content.Source: §5 · p. 9

Context–memory conflict resolution

• Entity substitution [21] and rule-based reward tasks [36] target context–memory conflict resolution, whereas this pipeline uses perturbation to elicit document-grounded reasoning traces from the teacher.Source: §5 · p. 9

6 Conclusion

• 10k samples from 3.5k public documents train a 35B student to perform strongly on CL-bench and CL-bench Life, with gains across long-context understanding, instruction following, and reasoning.The pipeline uses document perturbations and document-grounded gap checks, without human annotation.Source: §6 · p. 9

AI use statement

• Authors used generative AI for literature search, experiment-log aggregation, figures, and manuscript polishing, then reviewed and verified all AI-assisted content.The authors state that their own training and evaluation pipelines produced the experimental results and that they take responsibility for the final work.Source: AI use statement · p. 9

Reproducibility statement

• Replacing full-width brackets in the CL-bench judge prompt may affect judge behavior and reproducibility, and identical results to the official prompt are not guaranteed.The change was made because the official prompt reproducibly triggered OpenAI content moderation in bulk with HTTP 400 errors; the rest of the prompt remained verbatim identical.Source: Reproducibility statement · p. 9

A Data Sources by Subcategory

• 3,515 documents cover 16 of 18 CL-bench subcategories; Workflow Orchestration and Operational Procedures were omitted because public sources rarely provide their agent-style conversation task traces.Sources were selected by matching document genre to each subcategory topic, and Table 6 lists the primary sources.Source: A Data Sources by Subcategory · p. 14, Table 6 · p. 15
• Candidate documents had to exceed 4,000 characters and pass global URL deduplication across all subcategories.GitHub documents also received an LLM-based relevance and quality check, while arXiv papers were collected in multiple formats to diversify document formats.Source: A Data Sources by Subcategory · p. 14

B Full Evaluation Board

• The general-capability board covers long-context understanding, instruction following, reasoning, coding, and knowledge, using native scoring for objective tasks and gpt-oss-120b [1] to judge free-form outputs.Its benchmark families include LongBench v2 [4], IFEval [56], ARC-AGI-1 [7], LiveCodeBench [20], and MMLU-Pro [43].Source: B Full Evaluation Board · p. 15
• The board compares three reference models with the Qwen3.6-35B-A3B student and SFT/RL variants, and omits some results because of evaluation errors.The references are Qwen3.8-2.4T, HY3-preview, and Qwen3.5-397B-A17B; the SFT variants include one trained with a GLM-5.2 teacher.Source: B Full Evaluation Board · p. 15
• IFBench 55.7, InverseIFEval 61.1, and AdvancedIF 50.6 leave the GLM-5.2-teacher student below baseline on instruction following; adding diverse samples further lowers its CL-bench performance.The authors hypothesize that the student's general ability depends strongly on the teacher model, leaving detailed analysis for future work.Source: B Full Evaluation Board · p. 15

• Table 7 presents a full evaluation board judged by gpt-oss-120b; SFT and SFT+RL use the Qwen3.8-2.4T teacher unless stated otherwise, and experiments use no tool calls.Source: Table 7 · p. 16

C.1 SFT is Strongly Teacher-Dependent

• CL-bench scores range from 19.7 to 22.8 across students trained on different teachers; Kimi-K3 scores 27.0 but yields the weakest student at 19.7.A mid-ranked teacher yields the best student, though the passage does not name that teacher.Source: §C.1 · p. 16
• Students inherit teacher reasoning style: Kimi-K3’s 3,764-token reasoning yields a student with 4,109-token median think length, while the GLM-5.2-trained student has 2,936 tokens.Kimi-K3 is more token-efficient than other teachers, but its student does not inherit that efficiency; the Kimi-trained student’s performance is close to the GLM-5.2-trained student’s.Source: §C.1 · p. 16, Table 8 · p. 17
• On a persona task, the 155-token Kimi-K3-trained trace misses the personal-discourse rule, whereas the 408-token Qwen-trained trace checks and covers it.Both teachers pass the task, but their students diverge: the Kimi-trained student answers only the practical question and fails the persona rubric.Source: §C.1 · pp. 16–17, Table 9 · p. 17

C.2 The Effect of Rewriting on the Teacher’s Reasoning

• Without rewriting, the median reasoning trace length drops from 4.0K to 2.6K tokens.The comparison used 320 randomly sampled documents; questions could differ between original and rewritten versions, but both groups used the same prompt.Source: §C.2 · p. 17
• The rewritten-RFC trace cites document sections 39 times, while the original-RFC trace is 584 tokens and recites title-page metadata from memory.The original-question answer was drafted before its verification step, and answering without the document scored 100% on content rubrics.Source: §C.2 · pp. 17–18, Table 10 · p. 18
• The paired traces illustrate that rewriting prompts the teacher to reveal its reasoning and cite document evidence rather than rely on memorized knowledge.The questions were not exactly the same, though both were fact-extraction questions in the same domain and used the same teacher model.Source: §C.2 · pp. 17–18