PaperlayerView exact revision on arXiv

Source-linked AI summary

From Final Artifacts to Trajectories: Retrospective Process Supervision for Evidence-Grounded Long-Form Generation

Junjie Huang, Jiarui Qin, Di Yin, Weiwen Liu, Yong Yu, Xing Sun, Weinan Zhang

arXiv:2608.30461v1cs.CL

TL;DR

Open-ended evidence-grounded tasks lack singular ground truth and make trajectory supervision costly to annotate or verify. RETROGEN reconstructs candidate trajectories from expert artifacts, verifies them against the artifact and evidence, and trains models on successful reconstructions; experiments report improved evidence-grounded generation across domains and benchmarks.

  • Problem

    Open-ended evidence-grounded tasks lack singular ground truth, making trajectory supervision difficult and costly to annotate or verify.

  • Method

    RETROGEN reconstructs candidate evidence-seeking trajectories from expert artifacts, verifies artifact fidelity, evidence faithfulness, and procedural plausibility, and trains agents on retained reconstructions.

  • Results

    RETROGEN improves evidence-grounded long-form generation and domain-specific reasoning across scientific writing, financial analysis, and legal judgment drafting while preserving general capabilities.

  • Takeaways & Limitations

    Expert-curated artifacts can serve as scalable sources of process supervision for self-improving open-ended agents.

  • Takeaways & Limitations

    RETROGEN depends on high-quality artifacts containing recoverable evidence and structural signals, and may be less reliable when outputs are underspecified, stylistically diverse, or weakly grounded.

Abstract

from arXiv · show

Trajectory data is getting more vital for training large language models for boosting the agentic abilities. Unlike the verifiable domains such as coding or mathematics, scaling trajectory data for open-ended tasks is much more difficult because these tasks lack singular ground truth and are costly to annotate or verify. In this paper, we propose RetroGen, a self-improving framework of retrospective process supervision. Our key observation is that although expert trajectories are scarce, high-quality final artifacts such as literature reviews, analyst reports and legal judgments, are abundant in pre-training data and can be viewed as compressed traces of the evidence-seeking processes that produced them. RetroGen reconstructs candidate latent trajectories from expert artifacts, verifies them against both the artifact and supporting evidence, and trains models on their own successful reconstruction data, without requiring trajectory data from stronger models. Experiments show that RetroGen improves grounding, faithful synthesis, and long-form evidence-seeking agent tasks.

1 Introduction

Open-ended evidence-grounded tasks make trajectory supervision difficult because they lack singular ground truth and are costly to annotate or verify. RETROGEN addresses this bottleneck by reconstructing and verifying trajectories from abundant expert artifacts, then training on successful reconstructions.

  • Open-ended evidence-grounded tasks lack singular ground truth for validating trajectories and make evidence sufficiency, claim grounding, and rigorous reasoning difficult to assess automatically.
  • Existing trajectory supervision relies on stronger teacher models or human experts, but both sources are inherently unscalable for evidence-intensive workflows.Forward trajectory generation can also produce wandering search behavior because models lack holistic foresight for complex synthesis.
  • High-quality expert artifacts are abundant relative to expert trajectories and act as lossy but informative compressions of latent evidence-seeking processes.Artifacts implicitly encode searching, filtering, comparison, and synthesis, while expert vetting provides a quality prior for reconstruction.
  • RETROGEN reconstructs plausible evidence-seeking trajectories that arrive at known expert artifacts instead of requiring external teachers to demonstrate complete forward trajectories.
  • RETROGEN retrieves and organizes evidence, reconstructs traces, scores them across artifact fidelity, evidence faithfulness, and procedural plausibility, and filters them through weighted thresholding.The retained reconstructions form an iterative artifact-to-trajectory-to-agent training loop.
  • RETROGEN significantly improves evidence-grounded long-text generation on static and agentic benchmarks while preserving general capabilities.A verification ablation finds that combining scoring dimensions outperforms any single filter.

2 Problem Formulation

The problem formulation treats an expert-written artifact as the observed result of an unobserved task, evidence set, and expert workflow. RETROGEN reconstructs a plausible executable surrogate trajectory and evaluates it against artifact fidelity, evidence faithfulness, and procedural plausibility.

  • The task is to reconstruct a plausible evidence-seeking trajectory from an expert-written artifact whose generative process is unobserved.
  • Latent Expert Process: The latent expert process consists of an underlying task specification, supporting evidence, and a process transforming them into the final artifact.
  • Latent Expert Process: The artifact is assumed to be a lossy but highly informative compression of the unobserved expert process.
  • Retrospective Reconstruction: RETROGEN reconstructs a task, recovered evidence set, and reverse-engineered trace describing how an agent could derive an output consistent with the artifact.The trace is an executable surrogate trajectory rather than a reconstruction of the expert’s historical trial-and-error process.
  • Desiderata: A high-quality reconstructed trace should preserve the artifact’s essential content and structural organization, ground reasoning and claims in recovered evidence, and follow a logical domain workflow.

3 Method

RETROGEN reconstructs evidence-seeking trajectories from expert artifacts, verifies candidates against artifact-specific criteria and recovered evidence, and trains on retained traces. The framework adds procedural supervision beyond final-artifact imitation by teaching decomposition, evidence organization, and grounded synthesis.

  • Artifact-Anchored Initialization: RETROGEN reconstructs candidate trajectories by inferring a task specification, inducing an artifact-specific rubric, retrieving evidence, and planning tool-mediated actions.The shared action vocabulary includes Search, Open, Extract, Compare, Outline, and Draft.
  • Trajectory Reconstruction: Recovered observations come from actual tool executions, while intermediate synthesis notes and controlled procedural diversity make traces empirically grounded and varied.Diversity is injected across query formulation, plan realization, outline granularity, reflection frequency, and surface formatting.
  • Evidence-Constrained Verification: Candidate traces are scored on rubric satisfaction, holistic quality, evidence grounding, and workflow consistency, then failed or below-threshold traces are discarded.Trace consistency checks whether comparisons reference opened documents, outlines match final structure, and claims are licensed by earlier observations.
  • Retrospective Process Supervision: RETROGEN retains verified prefixes paired with their synthesized drafts, preserving causal consistency between intermediate reasoning and final generation.Each sample includes the inferred task, recovered evidence, tool interactions, intermediate notes, and verified synthetic artifact.
  • Retrospective Process Supervision: Compared with final-output-only supervision, the objective teaches both which artifact to produce and how to decompose tasks, gather evidence, and synthesize grounded long-form outputs.The training samples are serialized as single autoregressive sequences and optimized with standard language modeling.

4 Experiments

RETROGEN is evaluated across evidence-grounded generation, domain-specific reasoning, agentic research, and general capability retention. Across these settings, it improves grounded performance and process behavior while avoiding systematic degradation of general capabilities.

  • Experimental Setup: RETROGEN is evaluated on scientific writing, financial analysis, and legal judgment drafting using expert artifacts as targets.These domains stress-test citation-heavy scientific synthesis, document-grounded financial reasoning, and statute- or case-grounded legal argumentation.
  • Experimental Setup: The experiments compare RETROGEN with Initial, ForwardGen, Artifact-Only, and Public-Baseline under matched training conditions.Artifact-Only removes intermediate tool interactions, while Public-Baseline substitutes public retrieve-then-write corpora.
  • Main Results: +4.8 points is RETROGEN’s average evidence-grounded generation gain over ForwardGen across ALCE, ScholarQA, and QASPER.The gains are +4.3 on ALCE, +6.7 on ScholarQA, and +3.5 on QASPER.
  • Main Results: +2.7 points is RETROGEN’s average improvement over Artifact-Only on domain-specific reasoning benchmarks.The gains are +3.4 on LAiW and +2.0 on LegalBench, indicating added value from intermediate tool-interaction traces.
  • Main Results: RETROGEN outperforms Public-Baseline most on tasks requiring multi-step evidence aggregation, indicating a benefit from explicit tool-use trajectories.Public-Baseline uses the same token budget for public web-grounded long-form QA in retrieve-then-write form.
  • Dynamic Agentic Evaluation: On AstaBench, RETROGEN doubles LitQA2 accuracy from 0.20 to 0.40 and raises SQA’s global average from 0.47 to 0.59.LitQA2 precision rises from 0.33 to 0.67 while coverage remains 0.60; SQA citation precision rises from 0.47 to 0.67 and citation recall from 0.33 to 0.57.
  • Dynamic Agentic Evaluation: RETROGEN reduces search-to-search transitions from 0.54 to 0.39 and increases search-to-read transitions from 0.15 to 0.23.The transition analysis collapses tool calls into SEARCH, READ, and END actions.
  • General Capability Retention: Across four backbones, RETROGEN shows no systematic degradation on GSM8K, MMLU-Pro, IFEval, and HumanEval.Most RETROGEN-trained checkpoints are comparable to or slightly above their initial counterparts when mixed with modest general SFT data.

5 Related Work

Related work spans evidence-grounded long-form generation, tool-use agents, process supervision, and self-improvement. RETROGEN differs by reconstructing and verifying trajectories from expert artifacts rather than depending on stronger teachers or human demonstrations.

  • Evidence-Grounded Long-Form Generation: Evidence-grounded generation uses retrieval to improve factuality and verifiability, extending to web- and citation-grounded long-form synthesis.Scientific and legal domains impose especially demanding evidence-grounding requirements.
  • Tool-Use Agents and Process Supervision: Tool-use agent research studies models that interleave reasoning with search, retrieval, API calls, and environment interaction.Reliable-agent training remains difficult because high-quality process trajectories are scarce.
  • RETROGEN’s Position: RETROGEN reconstructs candidate trajectories retrospectively and filters them with evidence-constrained verification.This connects process supervision to expert artifacts as sources of executable, verifiable training traces.
  • Self-Improvement Without Stronger Teachers: Prior self-improvement and retrieval-enhanced systems commonly rely on stronger teacher models, human demonstrations, preference labels, or distilled retrieve-then-write data.RETROGEN instead targets self-improvement without stronger teachers.

6 Conclusion

RETROGEN trains evidence-seeking agents by reconstructing candidate tool-use trajectories from expert-curated artifacts and filtering them through evidence-constrained verification. Across scientific writing, financial analysis, and legal judgment drafting, it improves evidence-grounded generation and domain-specific reasoning while preserving general capabilities.

  • RETROGEN reconstructs candidate tool-use trajectories from high-quality expert artifacts and filters them for artifact fidelity, evidence faithfulness, and procedural quality.
  • Across scientific writing, financial analysis, and legal judgment drafting, RETROGEN improves evidence-grounded long-form generation and domain-specific reasoning over various baselines.
  • Ablations show that the four verification signals are complementary.
  • Analyses show more structured agentic behavior on dynamic research tasks while preserving general capabilities.

Limitations

RETROGEN depends on high-quality expert artifacts whose evidence and structure support reliable retrospective reconstruction. Its evaluation also focuses on retrieval-oriented, evidence-grounded long-form generation rather than richer interactive or non-textual environments.

  • RETROGEN relies on high-quality expert artifacts containing enough recoverable evidence and structural signals for retrospective reconstruction.Artifacts that are underspecified, stylistically diverse, or weakly grounded in explicit evidence may yield less reliable reconstructed trajectories.
  • The experiments focus on evidence-grounded long-form generation with retrieval-oriented tools.Further evaluation is needed for richer interactive environments, longer-horizon planning, and non-textual tools.

A Implementation Details

The implementation uses publicly available expert artifacts from three domains as retrospective anchors while withholding the target artifact during trajectory execution. Trajectories are diversified through randomized prompts, plans, thought templates, and tool-call formatting.

  • Each domain uses an expert artifact drawn from a publicly available source as the anchor for retrospective reconstruction.
  • Scientific writing: Scientific-writing instances pair author-written related-work sections with paper abstracts and cited reference lists.The original related-work text is not exposed during executable trajectory generation.
  • Financial analysis: Financial-analysis instances use Item 7 MD&A sections from U.S. SEC 10-K filings, with remaining filing content available through filing-reader calls.
  • Legal judgment drafting: Legal-judgment instances use pseudonymized Chinese first-instance civil and administrative judgments, while statutes and prior cases must be retrieved during execution.
  • The target artifact serves as a retrospective anchor for task induction, evidence recovery, and verification, but is unavailable as an observation during execution.
  • Trajectory sampling randomizes query phrasing, system-prompt style, plan flow, thought templates, and tool-call formatting, with fixed seeds within each backbone run.

A.3 SFT Mixture and Training

Fine-tuned variants share the same SFT recipe, token budget, sequence-length cap, and optimization configuration, while baselines differ in their agentic training data. RETROGEN uses a domain-balanced trajectory mixture plus general data.

  • The Base baseline is the original checkpoint without SFT, while all fine-tuned variants use the same SFT recipe.
  • Token-budgeted mixture: RETROGEN trains on a 5 × 10^7-token corpus split into 40% scientific, 20% financial, 20% legal, and 20% general data.Samples longer than 32,768 tokens are dropped, and corresponding baselines use the same budget and mixture proportions.
  • Baseline construction: ForwardGen generates trajectories from the inferred task specification without using the expert artifact as a target anchor.
  • Baseline construction: Artifact-Only removes intermediate tool calls, observations, and synthesis notes, while Public-Baseline replaces the agentic slice with WebGLM-QA and WebCPM-WK examples.All fine-tuned variants share the same sequence-length cap and optimization configuration.
  • Optimization: Training uses full-parameter supervised fine-tuning with AdamW, cosine scheduling, one epoch, and an effective batch size of approximately 512K tokens per optimizer step.The optimization configuration is shared across fine-tuned baselines.

A.4 Evaluation Details

Evaluation uses a vLLM backend with domain-dependent sequence lengths and a fixed random seed. Agentic evaluation uses AstaBench suites, while transition analysis reduces tool calls to SEARCH, READ, and END.

  • Evaluation configuration: Evaluations use a vLLM backend with max_length=32,768 for evidence-grounded and legal tasks and 4,096 otherwise.The random seed is fixed to 1234 unless otherwise specified.
  • Dynamic agentic evaluation: AstaBench’s literature-understanding suite evaluates dynamic agents with LitQA2-Validation and SQA using task-specific accuracy, grounding, citation, and recall metrics.LitQA2 measures accuracy, precision, and coverage; SQA reports ingredient recall, answer precision, citation precision, citation recall, and their global average.
  • Transition analysis: For Figure 3, raw tool calls are collapsed into SEARCH, READ, and END, and transition probabilities are computed from consecutive action pairs.SEARCH covers retrieval calls, READ covers opening or inspecting retrieved content, and END denotes stopping tool use to answer.
  • Trajectory format: Assistant turns may contain tool calls followed by an optional freeform note, whereas final-answer turns emit no tool call.The default interaction schema returns tool results in the next user turn, and a ReAct-style variant is also sampled.

B Verification Protocol

The verification protocol scores reconstructed trajectories against domain-specific criteria and filters low-quality traces. It combines rubric, holistic, grounding, and consistency judgments, then retains verified data for training while diagnosing common rejection patterns.

  • Verification scoring: Verification uses the generating backbone itself under a rubric-guided protocol rather than introducing a stronger external judge.Each retained trace receives rubric satisfaction, holistic quality, evidence grounding, and trace consistency scores.
  • Verification scoring: The overall verification score averages four dimensions with uniform weights λ = 0.25: rubric satisfaction, holistic quality, evidence grounding, and trace consistency.Each component score lies in [0, 1].
  • Filtering: Domain-specific thresholds of 0.52, 0.48, and 0.63 for scientific, financial, and legal traces yield approximately 25K verified trajectories.Thresholds remove each domain’s low-score tail while retaining enough data for the 50M-token SFT mixture.
  • Failure modes: Rejected traces commonly miss evidence themes, show weak grounding, provide shallow comparisons, or produce incomplete final artifacts.A fluent scientific trajectory can still be rejected when it discusses related concepts but omits required rubric themes.
  • Failure modes: Accepted traces search and browse noisy retrieval spaces, retain relevant sources, discard weak results, and synthesize artifacts covering the induced rubric.The protocol therefore distinguishes plausible-sounding workflows from traces recovering the correct evidence lineage.
  • Rubric construction: Rubrics classify reconstructed-trajectory requirements into coverage, structure, synthesis, and citation, with domain-specific criteria for scientific, financial, and legal tasks.Financial and legal criteria include coverage of filings or statutes, analytical depth, and citation of controlling authority.
  • Task induction: Task induction specifies professional intent and topical scope without revealing target papers, authors, citations, conclusions, or required evidence.The original artifact is used offline as a retrospective anchor while the agent must search, inspect, select, and synthesize evidence.
Loading 2608.30461v1…