Source-linked AI summary
Train What You Deploy:Token-Faithful Post-Training of a Production Coding
Cheng Li, Jiexiong Liu, Yixuan Chen, Chi Hong
TL;DR
Existing coding-agent post-training pipelines can mismatch production scaffolds and distort token provenance. This paper introduces a fidelity-aware coupling framework and C-DPPO, which improves matched TMax-100 scores by 3.0 points over Standard DPPO at both Baize5B and Baize10B scales, while auditing certificate operation.
Problem
Coding-agent post-training must address mismatches between simplified training environments and production scaffolds, plus distorted token provenance from offline log reconstruction.
Method
The paper couples production agent scaffolds to trainer-side sampling through a negotiated protocol, verifiable token-span masking, and C-DPPO's TV certificates, adaptive-K decisions, sequence budgets, and robust policy masking.
Results
3.0 points: C-DPPO exceeds Standard DPPO on final TMax-100 scores at both Baize5B and Baize10B scales under matched protocols.
Takeaways & Limitations
The matched study reports a direction-consistent improvement and exercised certificate paths, while positioning fidelity-aware coupling as a deployable training approach.
Takeaways & Limitations
The single-run, single-node comparison does not establish statistical significance, scaling, or broad cross-benchmark superiority, and its 3.0-point difference could reflect seed or sampling noise.
Abstract
from arXiv · showhide
Existing post-training pipelines for coding and terminal agents suffer severe token and control fidelity errors: simplified training environments mismatch production deployments, and offline token reconstruction from agent logs distorts original prompts and conflates policy calls with background model operations. We present a fidelity-aware training coupling framework that retains trainer-side sampling over original prompts, eliminates spurious model calls via a negotiated training protocol, and restricts loss computation to verifiable token spans with closed-failure guarantees. We further propose Certified Divergence Proximal Policy Optimization (C-DPPO), which establishes tight two-sided TV certification bounds, adaptive-K rules, budget-aware sequence guarantees, and error-robust policy masking atop standard DPPO. Evaluated on matched Baize5B and Baize10B models with identical training and test protocols on TMax-100, C-DPPO yields a consistent +3.0-point performance gain over standard DPPO across model scales. Certificate audits validate the reliability and full operational coverage of our certified training pipeline.
1 Introduction
The paper addresses fidelity failures caused by simplified training scaffolds and passive transcript reconstruction, then introduces a negotiated coupling and C-DPPO to preserve token provenance and certify divergence. In matched experiments, C-DPPO improves TMax-100 performance by 3.0 points across model scales while audits assess operational coverage.
- Production agents depend on scaffold behavior, so faithful post-training requires trajectories generated by the same scaffold and a measured deployment gap.
- Simplified training loops optimize a policy for a scaffold it will not encounter in production, creating a conditional-distribution mismatch.
- Passive model proxies preserve the production scaffold but distort prompts, tokenization, hidden reasoning, and attribution between policy calls and background operations.
- The coupling resamples assistant turns from exact rendered prompts, eliminates non-policy model calls, and verifies provenance before assigning loss.
- C-DPPO adds two-sided TV certification, adaptive K, sequence-level divergence budgets, and error-robust masking to standard DPPO.
- 3.0 points separate C-DPPO from Standard DPPO on TMax-100 across Baize5B and Baize10B, with complete records, resolved certificates, low mask disagreement, and feasible budgets audited.
2 Related Work
Related work spans outcome-driven terminal-agent reinforcement learning, divergence-proximal optimization, production agent scaffolds, and integrated training-serving systems. The paper distinguishes its approach by making the production scaffold an audited participant through a negotiated interface.
- Terminal-agent reinforcement learning uses executable environments and verifiable outcomes to train coding policies.
- DPPO uses Binary or Top-K approximations to distributional divergence, while this paper adds the missing observable upper side for TV decisions.
- Prior agent training either simplifies the agent loop or replays transcripts, and neither preserves the deployment conditioning distribution.
- The proposed black-box coupling requires a hardened mode and two request headers, allowing agents with configurable provider transport to participate.
- The coupling contract combines session-as-key routing, provenance headers, zero-background-request checks, and a production binary shared by training and deployment.
3 System Overview
The system separates training, rollout, and coupling responsibilities while preserving the production KCoder binary and connecting it to trainer-owned sampling. Sessions are sandboxed, audited, rewarded, linearized, and synchronized into the next training step.
- Setting: The setting trains a terminal-agent policy in KCoder using executable TMax tasks, hidden final-state checks, and SGLang-served weights updated each step.
- Three planes: Three planes divide responsibilities: training holds the actor, rollout runs KCoder sessions and rewards, and coupling mediates model requests with provenance.
- Coupling plane: The coupling adapter terminates agent requests, resamples them with the current policy, records provenance, and returns the expected wire format.
- Harness lifecycle: Harness admission probes required capabilities, rejects unsupported binaries, isolates configuration, restricts tools, and fixes provider, permissions, and session budgets.
- One training step: Each session uses an identifier as its adapter credential; after completion, hidden verification produces rewards, message trees become masked samples, and updated weights reach all engines.
- Coupling overview: The coupling records exact tokens and matched distribution payloads, then applies Standard DPPO or C-DPPO after weight synchronization.
4 Method
The method negotiates sampling ownership, suppresses background calls, declares protocol deviations, and conservatively linearizes session trees into attributable training spans. It also governs rewards and corpus quality so invalid or unverifiable data is excluded before optimization.
- Negotiated coupling: The protocol assigns sampling to the trainer, requires agent-declared side effects, and rejects unattributable data.
- Sampling and recording: The adapter uses session-specific routing and exact token IDs, records turns only after responses are delivered, and preserves per-token rollout probabilities.
- Training mode: The hardened training mode disables secondary model calls and verifies exactly two provider requests across a tool round-trip.
- Negotiated deviations: Depth and replay-reasoning headers declare sub-agent nesting and prior reasoning content, allowing alignment-preserving reconstruction or fail-closed rejection.
- Drift-aware linearization: Session trees become root-to-leaf samples with prompt masks, generated-span masks, rollout probabilities, shared-prefix deduplication, and explicit dropping of incomplete sessions.
- Drift diagnostics: Conservative masks sacrifice gradient on uncertain spans rather than misl labeling them, while drift counts expose agent-side changes that threaten conditioning fidelity.
- Corpus governance: The reward pipeline removes invalid environments and verifier exposure, while manifest-pinned splits make reward quality and train/evaluation contamination auditable.
5 C-DPPO: Certification Theory for DPPO
C-DPPO extends DPPO with certified two-sided TV bounds, adaptive refinement, trajectory-level divergence budgeting, and robustness to log-probability error. These mechanisms turn approximate divergence estimates into sound update decisions while preserving explicit unresolved and fail-closed outcomes.
- Framework: C-DPPO adds a computable upper error certificate, adaptive K, sequence-level divergence budgeting, and robust masking to DPPO.The framework extends DPPO in three directions: upper certification with adaptive K, adaptive per-position trajectory thresholds, and uncertainty-aware policy decisions.
- Certified Binary/Top-K approximation: b_DS ≤ D_TV(µ, π) ≤ b_DS + τ_S provides a two-sided observable certificate from tracked probabilities and tail masses.The upper error τ_S is tight when the tail contains at least two tokens, so no uniformly smaller bound follows from the same reported information.
- Certified Binary/Top-K approximation: Adaptive refinement stops when the certificate is inside or outside the trust region, and otherwise fails closed as unresolved at K_max.The rule begins with the sampled token and adds behavior-policy Top-K tokens; full refinement resolves the decision.
- Long-trajectory composition and adaptive δ_t: A sequence-level divergence budget allocates adaptive thresholds δ_t across response positions and supports additive or product-form guarantees.Path feasibility requires U_t ≤ δ_t at each position, while the sequence-distribution theorem additionally requires an almost-sure or prefix-uniform premise.
- Long-trajectory composition and adaptive δ_t: Token masks do not themselves certify sequence divergence; C-DPPO attaches sequence guarantees only when the stated budget premise holds.A directionally safe update can remain outside the current trust region, so masking and certification are separate claims.
- Robust C-DPPO certificates under log-probability uncertainty: Robust masking accounts for bounded train/serve log-probability errors and rejects updates that could be both outward and outside the certified trust region.The rule reasons over every policy pair consistent with the declared probability intervals.
6 Experiments
The experiments compare matched Standard-DPPO and C-DPPO runs at Baize5B and Baize10B under identical 350-step training and final TMax-100 evaluation protocols. The study also retains interface-level measurements and certificate diagnostics, while limiting conclusions to descriptive cross-scale evidence.
- Experimental setup: Matched Standard-DPPO and C-DPPO pairs at Baize5B and Baize10B start from the same checkpoints, task order, and interaction budget for 350 training steps.Only the update rule differs within each pair; final step-350 checkpoints are evaluated on the same frozen TMax-100 set.
- Experimental setup: Both arms collect identical token, probability, tail-mass, and provenance records, while C-DPPO additionally applies certificate decisions and remaining sequence budgets.Missing certificate information fails closed, and settings are frozen before runs.
- Measurements: The retained measurements include average reward curves at both model scales, while interface checks assess whether the production observation path trains and produces nontrivial signals.These interface-level checks are not treated as C-DPPO performance results.
- Results: +3.0 points: C-DPPO scores 37/100 versus 34/100 at Baize5B and 44/100 versus 41/100 at Baize10B on final TMax-100 evaluation.The reward curves show comparable early plateaus, followed by stronger late degradation for Standard-DPPO than C-DPPO at both scales.
- Scope: The matched pairs support only a direction-consistent descriptive statement across the two scales, conditional on passing the observable-record audit.The study does not establish statistical significance, scaling, or broad benchmark superiority.
7 Limitations
The empirical evidence is narrow, while the coupling protocol and certification results depend on explicit transport, theorem, and error assumptions. The reported improvement is descriptive rather than statistically or broadly conclusive.
- Evidence scope: The empirical comparison uses one matched run per method and scale, 350 training steps, one production scaffold, and one frozen TMax-100 evaluation.This design leaves seed- and schedule-level variance unidentified.
- Evidence scope: +3.0 points is below the metric’s approximately ±4.8-point binomial sampling error and could plausibly reflect seed or sampling noise.The comparison is also confounded because C-DPPO masks tokens that Standard DPPO trains on.
- Evidence scope: The fixed 350-step horizon and absence of checkpoint-level confidence intervals make the measured gap horizon-dependent.An earlier or later stopping point could narrow or reverse the observed difference.
- Coupling assumptions: The coupling requires preserved session metadata, exact token identities, and paired policy probabilities; unsupported or missing fields fail closed.The KCoder-specific contract does not automatically transfer to other scaffolds.
- Certificate assumptions: C-DPPO certificates remain conditional on theorem-specific inputs, sampled-path budget coverage, declared log-probability error bounds, and the TV-specific guarantee.KL requires an additional tail likelihood-ratio condition.
8 Conclusion
The paper addresses fidelity failures when post-training occurs inside a production terminal-coding scaffold by negotiating and verifying sampling, side effects, and provenance. C-DPPO adds certified divergence controls, while the matched study reports a descriptive +3.0-point difference across scales without establishing broad superiority.
- Problem: Production context management, memory, orchestration, and retries can break the token and control fidelity required by reinforcement-learning training.The paper frames fidelity as a problem of post-training the model inside the scaffold it will serve in.
- Coupling: The coupling gives the trainer sampling ownership, gives the agent side-effect ownership through a checked request-count contract, and makes residual deviations explicit and fail-closed.A drift-classifying message-tree linearizer produces sequences whose loss-carrying tokens are continuations of policy-sampled tokens.
- C-DPPO: C-DPPO combines two-sided Binary/Top-K TV certification, adaptive-K decisions, long-trajectory divergence budgets, and robust masking for bounded log-probability errors.The framework also identifies a KL boundary: tail mass alone cannot provide the same unconditional guarantee.
- Interpretation: The single-run, single-node study is descriptive and does not establish statistical significance, scaling, or broad cross-benchmark superiority.The audit nevertheless reports exercised certificate paths at both scales.
A Proofs for the C-DPPO certificates
The proofs establish tight coarsening-error bounds, adaptive certification decisions, sequence-level divergence guarantees, robust error-aware masking, and a boundary for KL certification. They also verify budget feasibility and recover expected special cases.
- TV coarsening: The coarsening gap is bounded by the smaller aggregated positive or negative tail mass, and this bound is tight when at least two tail tokens are available.The gap vanishes when signed tail changes have no cancellation.
- Adaptive certification: Adaptive certification is sound because a lower endpoint above the threshold proves divergence is outside, while an upper endpoint at or below it proves divergence is inside.With the full vocabulary tracked, the tail vanishes and one threshold decision always holds.
- KL boundary: Tail masses alone cannot finitely upper-bound the KL remainder because two tail distributions can preserve those masses while making the likelihood-ratio contribution diverge.The proof fixes positive tail masses and sends one conditional tail probability toward zero.
- Sequence guarantees: The sequential proof composes per-token TV bounds into additive and product-form sequence guarantees through a certified divergence budget.The product bound follows by sequential maximal coupling, while the additive bound follows from monotonicity of the certified budget.
- Budget feasibility: Adaptive thresholds preserve the remaining budget because each accepted certified divergence is bounded by its allocated allowance.Fixing the schedule before certification prevents the current token from choosing its own allowance retroactively.
- Robust certification: Bounded log-probability errors yield robust lower and upper TV bounds by combining tracked-token uncertainty with an inflated tail term.The construction clips the upper bound at one and uses the tail calculation to bound the residual uncertainty.
- Policy masking: The robust mask retains updates only when every consistent policy pair is inside the trust region or certified inward, preventing possibly-outside and possibly-outward updates.The positive- and negative-advantage cases use opposite probability inequalities to establish the directional rule.
- Boundary cases: The formulas recover Binary-TV, full-vocabulary, zero-error, one-step, and zero-budget boundary cases.At full vocabulary, uncertainty and coarsening terms vanish; with zero budget, every retained position requires zero divergence.