Source-linked AI summary
Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Agents
Prabal Vashisht, Zheng Hu, Ying Yang
TL;DR
Production agents face an evaluation problem because live data changes make replayed sessions incomparable and benchmarks can go stale. The paper builds an offline harness with reproducible state and regenerable benchmarks, then uses it as a correctness-gated prompt-optimization signal. On a production merchant-facing agent, compression reduced prompt size without regressing held-out correctness across three subject-model families, and correctness measurably improved for one.
Problem
Changing backend state makes replayed evaluations incomparable and can leave benchmarks out of date with the data they describe.
Source: §1 · p. 1Method
The paper generates and evaluates benchmark data against live production state using tool-specific reproducibility mechanisms, then uses the harness as a GEPA fitness function for correctness-gated prompt compression.
Source: §1 · p. 2Results
Takeaways & Limitations
The results show that, for this production merchant-facing agent, prompt size and held-out task correctness need not trade off against each other.
Source: §4 · p. 4The benchmark is a frozen snapshot, and the paper has not measured how a compressed candidate’s advantage decays as production data drifts.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Evaluating conversational agents that act on live, mutable enterprise data is hard: replayed sessions drift as backend state changes, and a benchmark built today can silently go stale as the data it describes moves on. We present an offline evaluation harness for a production conversational agent that stays reproducible despite the live data it operates on changing, and that can be regenerated on demand once it goes stale. We then use this harness as the fitness function for GEPA-based prompt optimization under a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, correctness-gated compression shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness, consistently across three independent subject-model families, and for one, correctness measurably improves.
1 Introduction
The paper addresses reproducible evaluation for production agents whose live data changes, then uses that evaluation signal to optimize already spec-complete prompts. Its correctness-gated compression reduces prompt size without regressing held-out task correctness across three subject-model families.
1 Introduction
- The paper targets a spec-complete production agent, keeping its product specification and tool contracts fixed rather than deriving behavior from a blank prompt.Its optimization goal is to make evaluation trustworthy and the prompt leaner without re-deriving the agent’s behavior.Source: §1 · p. 1
- Because live ground truth changes and write actions alter state, live session replay produces incomparable results, while benchmarks can stale as their referent data moves.For write tools, the relevant outcome is the resulting state rather than the tool’s text.Source: §1 · p. 1
- Existing benchmarks and optimizers address pieces of the problem, but the paper distinguishes live-data staleness from corpus freezing, environment simulation, saturation, contamination, and evaluation-signal reproducibility.It argues that moving referent data calls for regeneration rather than continual sourcing.Source: §1 · p. 2
- The harness matches reproducibility mechanisms to tool data shapes, using staged clocks, frozen observed payloads, and seed-then-read-back for writes.It also generates archetype-stratified tasks and scores correctness and policy compliance with deterministic and LLM-rubric judges.Source: §1 · p. 2
- 3.5–7.6% smaller full prompt surfaces without regressing held-out task correctness across three subject-model families, with measurable correctness improvement for one family.Compression uses GEPA with a correctness gate.Source: §1 · p. 2
- The harness makes stale benchmarks regenerable from current backend state, and its stable scalar score serves as GEPA’s fitness function.The prompt objective prioritizes correctness and only uses token count to break exact performance ties, with a hard gate against rule violations.Source: §1 · p. 2
2 Methodology
The method builds a reproducible production-state benchmark from scoped tasks and tool traces, then uses it to optimize the agent’s full textual prompt surface under a hard correctness gate. Datapoints are reviewed and can be regenerated when their underlying production state changes.
2.1 Preliminaries
- The agent is modeled as a compound system with a core prompt assembled from skill instructions; the contributions target the skill-independent base prompt and jointly rephrase it with skills and tool descriptions.A benchmark table reports paired winner-minus-seed differences on Dtest (n = 196), averaged over five rollouts per datapoint with a bootstrap 95% confidence interval.Source: §2.1 · p. 2, Table 1 · p. 4
2.2 Sandbox: Reproducible Production State
- Each benchmark datapoint pairs a task and ground truth with the production-state slice needed to reproduce it, using staged clocks, frozen payloads, or seed-then-read-back according to data mutability.Writes are evaluated on sandboxed end state; calls outside the staged scope can hit live, unstaged state and go unflagged.Source: §2.2 · pp. 2–3
2.3 Generation: Grounding Datapoints in Achievable Tool-Call Traces
- Tasks are generated from end-to-end tool-call traces under the same reproducibility mechanism, then human-reviewed for well-posedness, intended capability, and alignment with the product specification.The repeatable pipeline can be rerun against later production state to produce a fresh dataset when the original becomes stale.Source: §2.3 · p. 3
2.4 Correctness-Gated Prompt Compression
- GEPA jointly edits the base prompt, skill instructions, and tool descriptions, assigning zero to candidates that violate mandatory rules and using token count only to break exact correctness ties.The search samples parents from a hybrid Pareto frontier, selects the candidate by validation fitness, and reports held-out test performance.Source: §2.4 · p. 3, Table 1 · p. 4
3 Results
Across three subject-model families, correctness-gated compression reduced prompt size without a held-out correctness regression; Anthropic’s result was a statistically distinguishable improvement. The experiment jointly optimized the production agent’s prompt components and evaluated winners on a separate test set.
3 Results
- The dataset used 302 training datapoints, 99 validation datapoints, and a disjoint 196-point held-out test set, stratified by skill–archetype pair across ten skills.Seed and winner were each scored on the test set using the mean of five rollouts per datapoint.Source: §3 · p. 4
- The experiment jointly optimized the base system prompt, all ten skill bodies, and shared tool descriptions across Anthropic, Gemini, and OpenAI subject-model families.Each family used the same 36,336-token seed and an 800-rollout budget.Source: §3 · p. 4
- 7.6% smaller, Anthropic’s winner showed a statistically distinguishable improvement over its seed; Gemini’s and OpenAI’s winners were smaller with no detectable quality change.Taken together, compression did not cost correctness for any family.Source: §3 · p. 4
4 Conclusion
The harness and correctness-gated compression reduced a production merchant-facing agent’s prompt size without regressing held-out correctness across three subject-model families. The authors note that gains vary by model and that decay as production data drifts remains unmeasured.
4 Conclusion
- 3.5–7.6% prompt reduction without regressing held-out correctness across three subject-model families; correctness measurably improved for one family.Evaluation used a confirmatory held-out set.Source: §4 · p. 4
- The compression gain’s size varies by subject model, suggesting its benefit depends on the model optimized, not solely on the harness.Source: §4 · p. 4
- Because the benchmark is a frozen snapshot, the authors have not measured how a compressed candidate’s advantage decays as production data drifts.The harness is designed to regenerate benchmarks on demand.Source: §4 · p. 4
A Example Datapoint
The example datapoint shows how a generated sales evaluation task records a merchant question, expected outcome, and production-call trace. Its store identifier and generation call arguments and output are redacted.
A Example Datapoint
- The sales datapoint’s store identifier and generation tool-call arguments and output are redacted.The example is described as a generated sales-skill datapoint.Source: A Example Datapoint · p. 7
- The task combines the merchant’s natural-language question and context, while its expected outcome contains the evaluator’s expected response and required figures.Source: A Example Datapoint · p. 7
- The existence witness is a production query_sales_data trace captured at a specific time; render_chart is expected but omitted because it does not fetch state backing the graded answer.The example’s expected response reports $352.28 across 18 orders, with 352.28 as a required figure.Source: A Example Datapoint · p. 7
B Reward-Hacking Near-Miss
A candidate deletion exposed gaps in batch-dependent no-harm checks and duplicate-blind text coverage. The critical-rule gate addresses these gaps by checking every candidate’s actual rule compliance deterministically.
B Reward-Hacking Near-Miss
- A candidate deleted a frozen store_id_handling block, and the no-harm check did not catch a later candidate in the same lineage.The no-harm check is computed per iteration against the batch sampled that round, so an unexercised rule can slip through.Source: B Reward-Hacking Near-Miss · p. 7
- The text-coverage judge passed the deletion because it found the instruction duplicated elsewhere, without checking the specific copy that had been removed.Source: B Reward-Hacking Near-Miss · p. 7
- The critical-rule gate checks every candidate’s actual rule compliance deterministically, independent of sampled batches or duplicate text elsewhere.Source: B Reward-Hacking Near-Miss · p. 8
C Sample Compression
The financials skill excerpt illustrates prompt compression through concise rephrasing and removal of redundant wording. The changes retain the distinct constraints in the original text.
C Sample Compression
- The two-bullet Overview becomes one sentence, while preserving its coverage of refunds and adjustments, payouts and fees.The original describes returned money, deposit history, schedules, commission rates, and fee structure.Source: C Sample Compression · p. 8
- The Display Rules merge overlapping bans on text lists and markdown tables into one rule without dropping a distinct constraint.Both versions require frontend tools and direct table data to the table renderer.Source: C Sample Compression · p. 8
- This example presents lossless rewording as the kind of compression the harness is designed to reward.Source: C Sample Compression · p. 8