LLM Post-Training as Brownfield Maintenance: An Industrial Perspective on Dataware Engineering

Gopi Krishnan Rajbahadur; Amir M. Ebrahimi; Boyuan Chen; Ahmed E. Hassan

Source-linked AI summary

LLM Post-Training as Brownfield Maintenance: An Industrial Perspective on Dataware Engineering

Gopi Krishnan Rajbahadur, Amir M. Ebrahimi, Boyuan Chen, Ahmed E. Hassan

arXiv:2608.31102v1Aug 31, 2026cs.SEcs.AIcs.LG

TL;DR

The paper asks how industrial post-training can improve a deployed checkpoint under fixed mixture and compute budgets without regressing existing capabilities. It frames the task as brownfield dataware maintenance and combines stable-failure mining, structured synthesis, and yield engineering. The resulting patch increased accepted supervision 2.84× and improved target coding benchmarks while staying within the regression gate.

Problem
Industrial post-training must improve a specific capability on an inherited checkpoint under fixed budgets without degrading the rest.
Method
The paper treats post-training as bounded dataware patching and combines failure-driven synthesis, structured candidate filtering, and yield-engineering interventions.
Results
The yield-engineered patch improved CodeForces pass@1 by +2.59 points and LiveCodeBench v6 pass@1 by +6.11, while accepted supervision increased 2.84× under the same four-attempt cap.
Takeaways & Limitations
The case study supports constrained end-to-end yield engineering as a practical way to improve a brownfield post-training loop.
Takeaways & Limitations
Evidence is strongest in code settings with objective execution feedback and is not claimed to transfer universally across models, domains, or mixture designs.

Abstract

from arXiv · show

Industrial post-training is a brownfield regime. Teams inherit a deployed checkpoint and must land targeted improvements under fixed compute and mixture budgets without regressing the rest. The maintained artifact is increasingly dataware: behavior governed by a curated post-training mixture, updated via bounded mixture patches rather than clean-slate retraining. From an industrial code-generation improvement effort, we offer a maintainer's perspective on why this work is hard in practice, distilling three recurring challenges, zero-sum mixture design, yield as the binding metric, and end-to-end integration under uncertainty, and arguing that progress depends less on one-off recipes than on an engineering discipline for programming dataware. In our case study, interventions that raised the conversion of teacher distillation into usable training data increased accepted supervision by 2.84 times while using the same solution teacher and four solution attempts per candidate problem. In our primary evaluation, the yield-engineered patch improved CodeForces pass@1 by +2.59 points (+3.11 pass@3) and held-out LiveCodeBench v6 pass@1 by +6.11 (+8.05 pass@3), all statistically significant across 16 stochastic evaluations of each benchmark from one fixed checkpoint per condition, with internal AIME and MATH regression suites within tolerance.

1 Introduction

Industrial post-training is framed as brownfield maintenance: teams improve a specific capability on an inherited checkpoint under fixed budgets while protecting existing behavior. The maintained artifact is dataware, updated through bounded mixture patches and governed by operational data-as-code contracts.

Industrial post-training starts from an inherited checkpoint whose capabilities and failure modes cannot be erased by starting fresh.
Fixed mixture budgets make new data zero-sum because it must displace existing data under limited compute windows and shared cluster time.
Scarce high-quality human data makes teacher distillation attractive, but its practical bottleneck is how much output remains usable after verification.
Post-training shifts from greenfield creation to maintenance, where teams cannot simply expand the mixture after deployment.
Each training example acts as an instruction, the mixture as the program, and the checkpoint as the compiled binary, making the maintained artifact dataware.
The paper documents three coupled challenges and proposes five reviewable contracts for programming dataware, validated through repeated fixed-checkpoint benchmark evaluations.

2 The Challenges of Data Programming

The paper identifies three coupled challenges in data programming: fixed budgets force zero-sum allocation, yield limits usable supervision, and stochastic integration makes regressions difficult to diagnose safely. These constraints turn post-training into end-to-end maintenance rather than isolated component optimization.

Challenge C1: Zero-Sum Mixture Design: Fixed token, compute, and cluster budgets make mixture allocation zero-sum: new examples must displace or reweight existing content.
Challenge C2: Yield as the Binding Metric: Yield is the binding metric because distillation effort matters only insofar as it produces valid, trainable solutions after verification.
Challenge C2: Yield as the Binding Metric: 3,412 syntactically valid solutions emerged from 4,064 hard synthetic problems, with a median of 0 valid solutions per problem in the baseline.
Challenge C3: End-to-End System Integration: A single benchmark evaluation produces one stochastic generation per task, so apparent gains can depend on decoding noise and require repeated runs for statistical judgment.
Challenge C3: End-to-End System Integration: 35 = 243 configurations arise from only five three-option decisions, making exhaustive ablation impractical under industrial budgets.
Challenge C3: End-to-End System Integration: Components that perform well in isolation can fail when combined under the same budget and regression constraints because mixture edits create non-local side effects.

3 A Maintenance Industrial Example

The case study patches a deployed checkpoint by mining stable failures, structuring synthesis, and engineering distillation yield within a bounded mixture budget. It integrates these controls with fixed-attempt generation, deterministic admission checks, and regression-gated evaluation.

Failure-Driven Synthesis: Failure-Driven Synthesis mines stable model failures, extracts underlying concepts, and synthesizes targeted training data.
3.1 Setting: a bounded patch on a live system (C1, C3): The study targets competitive-programming performance on TuringThinker-7B within a fixed budget and a patch capped at 2% of the code allocation.
3.1 Setting: a bounded patch on a live system (C1, C3): The production patch contains 9,697 examples, about 1.5% of the roughly 650K-example code allocation, while controlled evaluation uses a 3,412-example subset.
3.1 Setting: a bounded patch on a live system (C1, C3): The patch must improve CodeForces and LiveCodeBench v6 while AIME and MATH decline by no more than one point.
Stability first: deciding what is worth fixing (C3): Stable failures are defined by repeated evaluation, retaining tasks with no passing sample across R = 3 mining runs using N = 3 samples.
Structure next: keeping patches coherent (C3): A concept taxonomy, concept-conditioned synthesis, semantic deduplication, and solvability judging are used to keep candidate problems coherent before distillation.
Yield finally: two interventions to escape the yield trap (C2): Testcase rectification regenerates representative I/O, while constraint injection focuses teacher reasoning by prepending extracted constraints.
Keeping the loop affordable (C1, C2, C3): Both settings use at most four teacher queries per problem and deterministic Tree-sitter admission of responses that parse as valid Python.

4 Experimental Setup and Results

The study compares additive and capacity-constrained mixture-replacement patches using matched distillation budgets and repeated stochastic benchmark evaluation. Yield-engineered supervision is more useful than standard distillation across both setups, with gains transferring to held-out LiveCodeBench v6 while regression suites remain within tolerance.

Experimental Setup: Five continued-training conditions use the same TuringThinker-7B checkpoint across Base, additive, and mixture-replacement setups.The primary brownfield setting keeps total mixture size fixed by displacing 3,412 rehearsal examples.
Experimental Setup: FDS-3K-Cov matches Distill-3K at 3,412 added examples but uses coverage-first examples retained across 2,699 FDS task IDs.The coverage-first patch fills the remaining 713 slots uniformly at random and does not inspect benchmark outcomes.
Evaluation: 16 stochastic generations per task and paired 95% bootstrap intervals test whether benchmark differences are statistically significant.Each condition uses one fixed checkpoint per benchmark, with confidence intervals bootstrapped over benchmark tasks.
Results: +1.05 and +1.60 points on CodeForces pass@1 and pass@3, and +6.75 and +8.88 on LCBv6 pass@1 and pass@3, are achieved by FDS-3K-Cov additively.Standard Distill-3K remains close to Base, with changes from −0.14 to +1.13 points and no statistically significant improvement.
Results: +2.59 points on CodeForces pass@1 and +3.11 pass@3 accompany +6.11 and +8.05 on LCBv6 under mixture replacement.FDS-3K-Cov is higher than Distill-3K on three of four target metrics, while both patches significantly outperform Base.
Transfer: Mining only CodeForces failures still produces substantially larger gains on held-out LCBv6, indicating transfer beyond the failures driving synthesis.The paper treats increasing concept-level improvement with targeted synthetic volume as diagnostic rather than causal because mixture effects remain coupled.
Efficiency and Safety: FDS-3K-Cov raises accepted supervision from 3,412 to 9,697 solutions, or 2.84×, while reducing mean teacher tokens per attempt by 28%.Median trace length also falls by 46.5%, and regression-suite changes remain within the one-point tolerance.

5 Toward a Discipline of Programming Dataware

The paper proposes explicit engineering contracts for programming dataware because maintenance requires measurable, reviewable, and safely integrable mixture changes. These contracts target valuation, statistical regression control, efficient experimentation, displacement accounting, and operational reporting.

Engineering Contracts: Per-LOD valuation estimates a patch’s marginal value within a mixture so teams can justify budget shares and negotiate trade-offs explicitly.This contract addresses zero-sum mixture allocation.
Engineering Contracts: Probabilistic regression frameworks replace single-evaluation deltas with variance-aware acceptance criteria calibrated to stochastic evaluation regimes.The experiments evaluate each fixed checkpoint 16 times per benchmark and report task-bootstrap intervals.
Engineering Contracts: Information-efficient experiment design uses sequential planning, fractional-factorial thinking, and proxy gates to move validation earlier when exhaustive ablation is unaffordable.The contract addresses difficult interactions between programming-data interventions.
Engineering Contracts: Integration expectations require each mixture component to specify its target, consumed budget, displacement, and regression risks.Patch sizing, displacement baselines, and conservative merges are proposed as shared practice.
Reporting: Operational signatures such as yield per teacher token, cost per accepted sample, gate-survival rates, and return on distillation effort determine feasibility.The paper argues that downstream scores alone hide these determinants.
Toward a Discipline: Programming dataware should be governed by shared contracts and tooling that make mixture changes reviewable, testable, and safely integrable.The paper frames this shift as moving from ad-hoc craft toward engineering practice.

6 Conclusion

Post-training under fixed budgets is framed as brownfield software maintenance: deployed models receive budgeted, regression-safe patches rather than clean-slate rebuilds.

Post-training under fixed budgets is brownfield software maintenance, not clean-slate model rebuilding.

Limitations

The paper’s evidence is constrained to a narrow industrial case study, with limited disclosure and incomplete causal attribution. Its verification and transfer claims are also bounded by syntax-based filtering and objective code-evaluation settings.

The study covers one internal TuringThinker-7B checkpoint family, competitive programming, and two benchmarks of 65 and 175 tasks.
Exact mixture composition, teacher identity, token cap, and other internal measurements cannot be fully disclosed.
Distill-3K and FDS-3K-Cov match added example count at 3,412 each but differ in content, task distribution, and token count.The FDS comparison evaluates a complete yield-engineering package, so it does not isolate individual pipeline effects.
The syntax-only acceptance gate can admit semantically incorrect programs, making yield an operational proxy for usable supervision rather than direct correctness.Execution-verified checks are reserved for CodeForces and LiveCodeBench v6 evaluation.
The strongest evidence comes from objectively verifiable code tasks and may not transfer to repository-level workflows or less-verifiable domains.The paper does not claim universal superiority across models, domains, or mixture designs.

Ethical Considerations

The paper follows ACL ethics guidance while focusing on competitive-programming code improvement without user or personal data. It addresses licensing, misuse, synthetic-data, judge-bias, and environmental considerations.

The work focuses on improving a competitive-programming code model via failure-driven synthetic data generation and uses no user or personal data.
Evaluation artifacts are treated as internal, with hidden tests and proprietary logs withheld.
Dual-use risk is mitigated by focusing on contest-style tasks and pipeline reliability rather than deployment guidance.
Synthetic-data artifacts are addressed with judge-based filtering and a deterministic Tree-sitter syntax gate.
The interventions aim to reduce token and compute cost while maintaining evaluation integrity.

Appendix A: Responsible NLP Checklist (summary)

The checklist summarizes data practices, evaluation-artifact restrictions, decontamination, human-subject status, risk mitigations, and compute-cost considerations.

The study uses no user or personal data and instead uses competitive-programming problems and model-generated synthetic problems.
Proprietary evaluation artifacts, including hidden tests and internal logs, are not released.
Deduplication and decontamination checks use n-gram overlap and embedding similarity against evaluation benchmarks before fine-tuning.
The study recruited, surveyed, or compensated no human subjects.
46.5% reduction in median teacher reasoning-trace length, with mean −28%, is reported under constraint injection with rectified testcases.The paper presents this as evidence of reduced distillation cost.
Dual-use risks are discussed and mitigated by focusing on contest problems rather than exploit development.
LLM-judge bias and preference leakage are addressed by anchoring acceptance in Tree-sitter syntax validity.
An AI writing assistant was used only for language editing; authors wrote and verified the technical content, claims, and conclusions.

Appendix B: Reproducibility: Pipelines, Experiments, and Prompts

Appendix B specifies the standard and FDS pipelines, reports their derived artifacts, and documents supporting plots and verbatim prompts for reproducibility.

B.1 Pipeline definitions (operational order): The standard pipeline collects failures, extracts concepts, generates and filters synthetic problems, then distills solutions with four teacher queries per problem.It uses deduplication, quality filtering, a Tree-sitter syntax gate, and no execution-based filtering.
B.1 Pipeline definitions (operational order): The FDS pipeline adds testcase rectification and constraint injection before distillation while retaining the same four-query attempt cap.Steps (1)–(6) match the standard pipeline, with a Tree-sitter syntax gate applied after the interventions.
B.3 Key derived artifacts and counts: Adding rectification and constraint injection increases successful distillations from 3,412 to 9,697 solutions, or 0.84 to 2.39 solutions per problem.The comparison is shown as a representative end-to-end pipeline run.
B.4 Additional plots (space-saving): Appendix plots distinguish full-patch production analysis from the example-count-matched FDS-3K-Cov evaluation used in Table 2.Base denotes rehearsal-only TuringThinker-7B; Distill-3K and FDS-9K denote corresponding fine-tuned models.
B.4 Additional plots (space-saving): Figure 3 visualizes the teacher reasoning-length distribution, including its median, interquartile range, and outliers.FDS interventions reduce median trace length by 46.5% and mean length by 28%.
B.4 Additional plots (space-saving): Figure 4 relates targeted synthetic volume per concept to downstream CodeForces improvement as a diagnostic for iteration planning.The reported positive association is Spearman ρ = 0.469, with p = 0.0007 across n = 49 concepts.

B.2 Benchmarks and Evaluation Protocol

The appendix evaluates matched training conditions on CodeForces and held-out LiveCodeBench v6 using repeated execution-verified sampling and task-level pass@k estimates.

B.2 Benchmarks and Evaluation Protocol: The evaluation uses fixed decoding settings and one fixed checkpoint per condition, with 16 stochastic generations per task on each benchmark.Confidence intervals use nonparametric bootstrap over benchmark tasks, with paired resampling for comparisons against Base.
B.2 Benchmarks and Evaluation Protocol: Pass@1 measures single-generation reliability, while pass@3 estimates whether at least one of three independently sampled solutions succeeds.The estimator avoids partitioning the 16 observed generations into arbitrary triplets.
B.2 Benchmarks and Evaluation Protocol: CodeForces contains 65 competitive-programming tasks, including 43 test-failing tasks used as the failure-mining pool and 22 test-passing tasks.Correctness requires compilation and passing all hidden tests in the offline execution harness.
B.2 Benchmarks and Evaluation Protocol: LiveCodeBench v6 contains 175 longer-context tasks and is held out from failure mining for transfer evaluation.Its pass@1 and pass@3 scores use the same execution-based estimator as CodeForces.
B.2 Benchmarks and Evaluation Protocol: Table 2 compares Base with additive and mixture-replacement variants of two 3,412-example evaluation patches.Base is trained only on the 269,198-example rehearsal mixture, while FDS-3K-Cov is a 3,412-example coverage-first subset.
B.2 Benchmarks and Evaluation Protocol: The final shipped evaluation is omitted from the controlled table so FDS and standard-distillation patches remain matched by added example count.This preserves the controlled comparison between the two patch types.