PaperlayerView exact revision on arXiv

Source-linked AI summary

PROOF-Gen: From Optimized Data to Better Distillation

Anh Ta, Junjie Zhu, Shahin Shayandeh

arXiv:2608.23911v1cs.AIcs.LG

TL;DR

Generate-and-filter distillation discards hard teacher failures, leaving a coverage gap for tool-calling students. PROOF-Gen recovers those failures with per-scenario reflective prompt optimization, removes the guidance, and trains on the resulting clean trajectories. It recovers 93% of sampled telecom failures, improves benchmark student performance, and transfers gains to production evaluations across locales.

  • Problem

    Generate-and-filter keeps passing teacher trajectories but discards failures, including near-misses, so recurring distillation cycles leave hard scenarios without training signal.

  • Method

    PROOF-Gen uses reflector-written, per-scenario cheatsheets to steer the teacher from failed traces toward passing trajectories, then strips the cheatsheets before training.

  • Results

    PROOF-Gen recovers 93% of sampled telecom failures and is the only training condition improving task completion across all four model–benchmark pairs, with positive production transfer in every locale.

  • Takeaways & Limitations

    Recovered trajectories expand task coverage while preserving clean demonstrations, improving small-model tool-calling performance on benchmarks and production-agent evaluations.

  • Takeaways & Limitations

    The method assumes the executor can follow long, structured cheatsheets, and benchmark task volume limits end-to-end validation mainly to τ2-bench telecom.

Abstract

from arXiv · show

Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models. Post-training pipelines that drive shipped tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle, yet the mechanism is generate-and-filter (keep the teacher's passing trajectories, discard the rest) and each cycle leaves behind the same hard scenarios because failures supply no signal. On τ2-bench, 57% of teacher trials fail, two-thirds of them near-misses (most tool calls correct, undone by one decisive error). We introduce PROOF-Gen (Per-scenario Reflective Optimization to Overcome FailedGeneration), which recovers golden trajectories from these failures via per-scenario prompt optimization. For each failed task, a reflector analyzes the execution trace and evaluation feedback, then writes corrective guidance that steers the teacher to a passing trajectory. The guidance is stripped before training, so the student learns from clean demonstrations with no task-specific scaffold. On τ2-bench, per-scenario optimization recovers 93% of failed scenarios. Fine-tuned on the combined data, Qwen3-4B-Instruct-2507 improves from Pass^1=0.132 to 0.529 and Gemma 4 E4B-it gains +7.2pp on BFCL v4 multi-turn. In a deployed pipeline, the method lifts trajectory quality by +6.3pp goal completion and transfers to a deployed on-device model (+1.5pp goal completion; +1.7 to +5.0pp across response-quality metrics), with positive transfer in every locale (non-English average +1.48pp).

1 Introduction

PROOF-Gen addresses the coverage gap created when generate-and-filter discards hard teacher failures, especially near-misses that lack only a decisive action. It recovers clean demonstrations through per-scenario prompt optimization while preserving the student’s training setup and, potentially, product voice.

  • PROOF-Gen: The cheatsheet is removed before training, so students learn from clean recovered trajectories rather than task-specific scaffolding.The fine-tuning procedure otherwise remains unchanged, and the executor can retain its established interaction style.
  • The data-generation gap: SFT pipelines repeatedly pay frontier-teacher costs while generate-and-filter discards failed trajectories and leaves hard scenarios without training signal.The standard mechanism keeps passing trajectories and discards failures during recurring post-training cycles.
  • The data-generation gap: 67% of failed GPT-4o trajectories execute over half of the required tool calls correctly, yet one missed action can cause full task failure and trajectory discard.These near-misses show that failures often contain substantial correct behavior but lack a complete path through the decisive step.
  • The data-generation gap: Training on partial-credit signals lifts tool-call accuracy by 25pp but does not improve end-to-end completion in any model–benchmark pair.Because tool-call accuracy is already high in failures, rewarding intermediate actions does not supply the missing decisive trajectory.
  • PROOF-Gen: PROOF-Gen optimizes a separate cheatsheet for each failed scenario, using execution traces and evaluator feedback to steer the teacher toward a passing trajectory.The per-scenario loop inverts conventional prompt optimization by overfitting guidance to the individual task rather than seeking one general prompt.
  • Evaluation questions: The study tests recovery, downstream student performance, and deployment-scale transfer across τ2-bench, BFCL v4 multi-turn, and a production tool-calling system.The authors use two small instruction-tuned students and report results across benchmark and production settings.

2 Method

PROOF-Gen extends generate-and-filter by recovering each failed task with disposable, per-scenario guidance, then training on the filtered and recovered trajectories together. The guidance is removed so the student receives only the demonstrated behavior.

  • Pipeline: PROOF-Gen adds recovery after filtering: it optimizes a disposable prompt for each failed task, strips that prompt, and adds the recovered trajectory to training.The student is fine-tuned on the union of filtered and recovered trajectories.
  • Per-scenario recovery: A reflector reads the failed trace and verifier feedback, writes a scenario-specific cheatsheet, and iteratively revises it as the teacher re-executes from scratch.The loop runs for up to K=10 iterations and retains a trajectory only when every evaluator passes.
  • Per-scenario recovery: GEPA supplies the prompt-optimization loop, although the method is designed to work with any prompt optimizer.The optimizer accumulates diagnoses from successive traces and feedback into the cheatsheet.
  • Scaffold removal: The cheatsheet is deleted before training, making recovered trajectories indistinguishable from filtered ones to the student.Any post-training gain is attributed to what the trajectories demonstrate rather than to visible per-scenario scaffolding.
  • Configurable objective: The verifier-driven loop can target multiple evaluable quality dimensions, including groundedness, brevity, and formatting, in addition to task completion.This configurable objective is used in the production setting.

3 Experimental Setup

The experiments evaluate PROOF-Gen on stochastic τ2-bench and multi-turn BFCL using GPT-4o-generated trajectories, two small students, and filtered-versus-combined training data. The setup also includes a production-scale deployment evaluation.

  • Benchmarks: τ2-bench uses an LLM user simulator, whereas BFCL uses pre-scripted user turns, providing stochastic and scripted benchmark settings.Both benchmarks use execution-based evaluators and teacher-generated candidate trajectories.
  • Benchmarks: τ2-bench telecom contains 2,285 tasks across approximately 50 tools, with 114 held-out base tasks and 2,171 training tasks.Telecom was selected for its large task space and low 7.3% teacher pass rate; airline and retail were not used for the primary end-to-end experiments.
  • Models: The study fine-tunes Gemma 4 E4B-it and Qwen3-4B-Instruct-2507, which differ in architecture and tool-call format.Improvements on both models are used to assess whether the approach is model-specific.
  • Data generation: GPT-4o generates one trajectory per task at temperature 0, while GPT-5.1 and GPT-5.4 serve as primary reflectors for τ2-bench and BFCL.The reflectors operate at high reasoning effort and were the strongest available when each benchmark was run.
  • Data generation: 93.0% of attempted telecom failures were recovered, compared with 33.7% on BFCL, where the teacher’s initial pass rate was 52.9%.Telecom recovery used a fixed random sample of 300 failures from 2,013, while all 264 BFCL failures were attempted.
  • Training and evaluation: Training uses QLoRA for 3 epochs, and evaluation runs three temperature-0 trials while reporting Pass^1 as the fraction of tasks passing at least once.Sampling robustness is separately tested at temperature 1 on post-hoc flip tasks.

4 Results

PROOF-Gen recovers failed teacher trajectories through per-scenario optimization and improves downstream task completion across benchmarks and production evaluation. Recovery depends strongly on reflector capability, while the method preserves the executor’s interaction voice and scales through parallel orchestration.

  • RQ1: Recovery effectiveness: 93% of sampled telecom failures are recovered at production scale, including 279 of 300 scenarios.The optimized cheatsheet drives 264 recoveries, while 15 arise from stochastic user-simulator replay.
  • RQ2: Downstream impact: Filtered-only training improves action accuracy by +25pp but does not improve task completion in any of four model–benchmark pairs.Combined training is the only condition producing consistent end-to-end gains.
  • RQ1: Recovery effectiveness: 97.6% versus 24.4% recovery in the cross-domain ablation shows that reflector capability strongly affects recovery.GPT-5.1 recovers 80/82 scenarios, compared with 20/82 for GPT-4o self-reflection, under fixed executor and optimizer settings.
  • RQ2: Downstream impact: Qwen3-4B-Instruct-2507 improves from Passˆ1=0.132 to 0.529 on τ2-bench, while Gemma 4 E4B-it gains +7.2pp on BFCL.These gains come from combined training on filtered and recovered trajectories.
  • Capability vs. voice: PROOF-Gen preserves the executor’s interaction voice while importing stronger reflector problem-solving without inheriting its voice or latency profile.On matched telecom tasks, GPT-4o remains terse and incremental, whereas GPT-5.1 uses a longer numbered protocol.
  • RQ3: Production scaling: +1.5pp goal completion transfers to the deployed on-device model, with +1.7 to +5.0pp across response-quality metrics and positive transfer in every locale.The production evaluation reports no regression on any dimension.
  • Efficiency and orchestration: Per-scenario optimization runs independently across scenarios and uses the expensive reflector mainly for diagnosis and cheatsheet revision.This allows parallel fan-out while the cheaper executor handles rollouts.

5 Related Work

The paper positions PROOF-Gen as recovering failures upstream of training, unlike prior approaches that filter teacher outputs, reuse failures with an existing successful trajectory, or retain optimized prompts at inference.

  • SFT for tool-calling agents: Established tool-calling SFT pipelines uniformly sample teacher trajectories, keep passing outputs, and discard failures.PROOF-Gen differs by recovering the discarded failures.
  • Trajectory recovery and self-improvement: Prior trajectory-reuse methods generally assume that a successful trajectory already exists from an expert, stronger model, or passing rollout.PROOF-Gen instead targets upstream recovery when such a paired success is unavailable.
  • Prompt optimization as a data-generation instrument: Automatic prompt optimization usually seeks one prompt that generalizes across inputs at inference time.This contrasts with PROOF-Gen’s per-scenario, data-generation use of prompts.
  • Prompt optimization as a data-generation instrument: Unlike ACE’s persistent inference-time playbooks, PROOF-Gen discards each scenario-specific cheatsheet before training and retains only the passing trajectory.The paper presents this use of prompt optimization as a data-generation engine.

6 Conclusion

PROOF-Gen treats failed teacher trajectories as recoverable training data by steering the teacher with disposable per-scenario prompts. The method improves benchmark task completion and production transfer across locales.

  • Conclusion: PROOF-Gen recovers 93% of sampled telecom failures and improves downstream task completion for two small instruction-tuned students.It is the only training condition improving all four benchmark model–pairs and also improves a production agent system.
  • Conclusion: The disposable scaffold is stripped before training, so students learn from clean passing demonstrations rather than task-specific guidance.This addresses the coverage gap created by generate-and-filter distillation.

Limitations

The study’s evidence is bounded by its reflector, teacher, instruction-following, and benchmark settings. Several proposed extensions and category-level findings remain untested or exploratory.

  • Reflector generalization is limited because the study covers four models from one provider; other providers and open-source models remain future work.
  • The teacher scope is limited to GPT-4o, leaving self-distillation and preservation of native response behavior open questions.
  • The method assumes executors can follow 7–9K-character structured cheatsheets, but the recoverable fraction for a 4B student has not been measured.
  • Benchmark scale limits end-to-end validation: τ2-bench airline and retail are too small, while BFCL category-level effects are directional rather than definitive.

Ethics Statement

The paper reports use of public benchmarks and models alongside an internal synthetic production evaluation set. The production scenarios were human-reviewed, contained no personally identifiable information, and involved no human subjects or private user data.

  • The study uses openly available τ2-bench and BFCL benchmarks and cites the teacher, reflector, and student models.
  • The production evaluation uses an internal held-out set of synthetic scenarios simulating production use cases.
  • The production scenarios were authored and human-reviewed, were not sampled from real user traffic, and contained no personally identifiable information.
  • No human subjects or private user data were used during data generation, training, or evaluation.

A.2 Data Generation

Data generation combines per-scenario recovery with filtered teacher trajectories, using reflection to recover failures and then evaluating fine-tuning across benchmarks and sampling conditions. Recovery depends strongly on reflector capability and usually converges within a few iterations.

  • Recovery effectiveness: 97.6% of failed scenarios were recovered with GPT-5.1 reflection versus 24.4% with GPT-4o self-reflection.
  • Recovery effectiveness: 279 of 300 failed telecom tasks were recovered, including 264 cheatsheet-driven recoveries and 15 stochastic replays.
  • Convergence: 58.7% of cheatsheet-driven recoveries resolved on the first iteration, 76.5% by the second, and 84.1% by the third.
  • Frontier-model comparison: GPT-5.1 high-reasoning direct generation recovered 83.3%, while per-scenario optimization with GPT-4o execution and GPT-5.1 reflection recovered 93.0%.
  • Frontier-model comparison: A single reflection step raised GPT-5.1 high-reasoning recovery to 98.3%, adding 45 recoveries beyond direct high-reasoning generation.
  • Post-training results: Combined training improved Qwen3-4B-Instruct-2507 from 0.132 to Pass^1 = 0.529 on τ2-bench telecom and yielded Gemma 4 E4B-it’s +7.2pp gain on BFCL v4 multi-turn.
  • Stochastic evaluation: On temperature-1 flip-task evaluation, Qwen averaged 54.1% and Gemma 39.0%, versus near-zero baseline rates of 8.3% and 2.9%.

B.1 Data Quality

The data-quality analysis tests whether cheatsheet-guided trajectories preserve genuine tool-based discovery and whether recovered data improves end-state correctness. It also separates task coverage from response style.

  • Cheatsheet integrity: The integrity audit checks that tool arguments use user-provided, previously returned, or schema-default values rather than cheatsheet leakage.
  • Cheatsheet integrity: Both judges agreed that all 90 audited trajectories preserved value discovery through tool calls, with 97.8% inter-judge agreement.
  • Cheatsheet integrity: Cheatsheets teach troubleshooting processes, tool ordering, and recovery patterns rather than answers, although shorter GPT-4o cheatsheets sometimes include task-specific hints.
  • Reward breakdown: Filtered-only training improved action accuracy by +25pp but left environment and database accuracy flat; combined training improved environment assertions by +31pp and database accuracy by +6pp.
  • Response style: On matched tasks, GPT-5.1 responses were 3.8× longer than GPT-4o responses despite comparable tool-call counts.
  • Response style: Because GPT-4o executes recovered trajectories, cheatsheet guidance changes tool use while preserving its incremental communication style.

C.2 Sonnet False Positive Analysis

The analysis finds that Sonnet’s shortcut detector produced false positives for trajectories whose tool-call arguments were supported by the task context or prior tool returns. The review protocol therefore distinguishes genuine discovery bypasses from legitimate execution-style and reasoning differences.

  • False-positive findings: 2 of 90 samples were flagged as HAS SHORTCUT, and both were false positives on manual inspection.The flagged examples involved a plausible phone-number match and truncated trajectory display, respectively.
  • False-positive findings: The telecom example used line ID L1002 because the customer phone number matched its suffix, and the tool return confirmed the match.The agent then acted on the confirmed match rather than proceeding without verification.
  • False-positive findings: The BFCL example appeared to use credentials from the cheatsheet only because the displayed user message was truncated; the full deterministic message contained both values.The truncation was a trajectory-display artifact, not missing information in the conversation processed by the agent.
  • Cheatsheet styles: GPT-5.1 produces detailed, parameterized procedures, whereas GPT-4o produces shorter guidance that may include task-specific hints.The comparison spans reflection-model style, not a claim that verbosity itself determines transferability.
  • Review criterion: Cheatsheet guidance is stripped before SFT, so the review asks whether the resulting trajectory teaches a transferable tool-calling pattern rather than whether the scaffold was prescriptive.Parallel calls, action ordering, missing confirmation, and policy violations are explicitly excluded from shortcut detection.
  • Review criterion: The reviewer flags HAS_SHORTCUT only when a tool-call argument was not discoverable from the user request or prior tool returns.Checks also cover hardcoded values, unjustified numbers, and information used before it became available.
  • Downstream relevance: The miss func category is the hardest tested category: GPT-4o passes 43%, Qwen3-4B-Instruct-2507 10.5%, and Gemma 4 E4B-it 0.5% over 200 tasks.On the 60-task test split, the corresponding baselines are 11.7% for Qwen and 0.0% for Gemma.
  • Downstream relevance: Fine-tuning raises Gemma’s miss func accuracy to 21.5% and reduces text-only responses from 85.5% to 22.0% of turns.The combined-data model learned to respond with tool calls on holdout turns.

L BFCL Per-Category Results

BFCL per-category results show the clearest and most stable gains in the miss func category, where both models begin with their lowest accuracy. The evaluation is strict and small per category, so other category-level effects remain exploratory.

  • Miss func: The miss func category starts lowest for both models and shows the clearest improvement: Gemma rises from 0% to 15.6%, while Qwen rises from 11.7% to 31.7%.These tasks require recognizing that a requested function is unavailable.
  • Evaluation caveat: BFCL requires every turn to pass, and a turn fails immediately when no decodable tool call is produced, regardless of accompanying text.This makes partial progress harder to observe than in τ2-bench’s end-state evaluation.
  • Stability: The miss func improvement is stable across three evaluation runs: Gemma moves from 0.0% in every baseline run to 15.0–16.7% combined, while Qwen moves from 11.7% to 31.7% in every run.Table 13 identifies this as the most stable category-level signal.
  • Cross-benchmark stability: Qwen3-4B-Instruct-2507’s combined τ2-bench result remains stable across independent runs, with Pass^1 ranging from 0.518 to 0.538.The spread is 0.020, supporting stability beyond a favorable training or evaluation seed.
  • Evaluation caveat: Other per-category results are mixed at n=60 and should be treated as exploratory.The table reports 60 test tasks per category averaged across three runs.
Loading 2608.23911v1…