Anchor: Mitigating Artifact Drift in Agent Benchmark Generation

Motivation

• Artifact drift arises when separately authored instructions, environments, oracle solutions, and verifiers describe subtly different tasks.Such inconsistencies can make tasks unsolvable, reward-hackable, or inconsistent with their intended requirements.Source: §1 · p. 1

Anchor

• Anchor compiles the instruction, environment, oracle solution, and verifier from one solved constraint-program specification.A single specification provides a common source for all four task artifacts.Source: §1 · p. 1, Figure 2 · p. 2

Evaluation

• 70.5% to 22.3%, 46.5% to 7.7%, and 56.0% to 9.5%: Pass@5 falls across easy-to-hard tiers for coding, browser, and computer-use harnesses, respectively.The evaluation covers five frontier models across 18,000 trials, with zero reward-hacking instances.Source: §1 · p. 2

Benchmark

• ERP-Bench contains 300 procurement and manufacturing tasks spanning 29 workflow patterns in Odoo 19.The benchmark targets long-horizon enterprise workflows in a production-grade ERP system.Source: §1 · pp. 1–2

Contributions

• Anchor’s contributions are a single-source task pipeline, ERP-Bench’s verifiable 300-task benchmark, and a controlled cross-harness model evaluation.The benchmark provides controlled difficulty and certified optima for long-horizon procurement and manufacturing workflows.Source: §1 · p. 2

Workflow formalization

• Enterprise workflows with structured data, explicit rules, and measurable objectives can be expressed as parametric constraint satisfiability and optimization programs.A domain expert and engineer define decision variables, business-rule constraints, and an objective function.Source: §2 · pp. 2–3

Solved specification

• For each parameter setting, the solver rejects infeasible samples or certifies an optimal solution, producing a solved specification S_θ.The solved specification contains the parameters, constraints, objective, and certified solution.Source: §2 · p. 3

Artifact compilation

• Four deterministic translation layers compile S_θ into natural-language instructions, environment records, an oracle terminal state, and a terminal-state verifier.Because these artifacts are projections of the same solved specification, the pipeline mitigates cross-artifact inconsistency by construction.Source: §2 · p. 3

Verification

• Solver-certified optimality keeps verification tractable without contriving tasks to a single action path.Agents can receive credit for multiple valid terminal states that satisfy the program and achieve its objective.Source: §2 · p. 3

Residual validation

• Anchor does not eliminate incomplete business logic or renderer mistranslation, so five end-to-end checks detect residual defects.Checks include no-op scoring, oracle replay, artifact cross-reading, reward-hacking canaries, and expert spot checks.Source: §2 · p. 3

Benchmark scope

• ERP-Bench contains 300 long-horizon procurement and manufacturing tasks across 29 patterns in Odoo 19.Tasks run in fresh containers seeded with the records required by each scenario and expose persistent ERP state through API or web-client interactions.Source: §3 · p. 3

Verification

• The verifier weights constraint satisfaction, optimality, and traceability at 25/60/15%, with constraint failures gating the other dimensions.Structural prerequisites can produce hard-zero scores, while optimality compares realized objectives with certified optima.Source: §3 · p. 3

Difficulty control

• Difficulty recipes scale demand, tighten supply, expand sourcing complexity, and enrich objectives from feasibility toward consolidation, capacity preservation, and plan repair.Sourcing layers add manufacturing stages, workcenter capacity, and broken initial states that agents must diagnose and repair.Source: §3 · p. 3

Difficulty intervention

• 70.5% to 22.3% in coding, 46.5% to 7.7% in browser, and 56.0% to 9.5% in computer-use pass@5 from easy to hard tiers.The decline is monotonic across all evaluated harnesses, with 95% Wilson confidence intervals reported in Figure 3.Source: §4 · p. 3, Figure 3 · p. 4

Feasibility and optimality

• 26.1% of trials satisfy every explicit constraint, but only 17.4% reach fully optimal solutions, producing a 23.4-point gap for GPT-5.5 in coding.Figure 5 summarizes the overall gap; Figure 7 attributes difficulty-related score loss to broken constraints rather than poorer trade-offs among feasible plans.Source: §4 · p. 4, Figure 5 · p. 4

Construction validation

• The construction checks pass end-to-end: no-op agents score zero, oracle replay receives full credit, and no rollout beats the certified optimum while remaining feasible.These checks support consistency between generated tasks, the verifier, and solver-certified solutions.Source: §4 · p. 4

Harness comparison

• Coding harnesses resolve 16–56 percentage points more tasks than GUI harnesses while costing 3.1–14.3× more per attempted task for the proprietary models.The comparison uses identical task instances and includes browser and computer-use settings.Source: §4 · p. 4

5 Limitations

• Task validity remains dependent on expert specifications, renderer fidelity, and ERP defaults that may omit or obscure intended business logic.The authors identify incomplete constraint programs, unclear renderings, and unintended ERP defaults as residual construction risks.Source: §5 · p. 4
• Workflows relying on tacit managerial judgment, negotiation, free-text persuasion, or outcomes outside the system of record do not fit this framework well.ERP-Bench instead prioritizes feasible, reproducible, and exactly gradable terminal-state tasks with explicit constraints and objectives.Source: §5 · p. 4

6 Conclusion

• Anchor and ERP-Bench show that constraint-program solvers can generate realistic, well-specified, solvable tasks for enterprise knowledge work.The conclusion states that the constructed tasks are solvable by both humans and agents.Source: §6 · p. 4
• Extending the framework to sales qualification, workforce management, patient intake, CRM, HRIS, and EHR systems is identified as a natural next step.Source: §6 · p. 4

A Glossary

• The glossary collects recurring ERP-Bench domain terms used throughout the paper and appendix.Source: A Glossary · p. 6

B Task Taxonomy

• Each task requires deciding what to buy, make, retain, or repair so the final ERP state covers demand, obeys business rules, and meets the stated objective.Source: B Task Taxonomy · p. 6
• ERP-Bench contains 300 tasks across 29 seeded workflow patterns covering buying, disruption recovery, and combined purchasing and manufacturing.Source: B Task Taxonomy · p. 6
• 183 tasks minimize new-supply spend, while 34 consolidate vendors, 31 preserve workcenter capacity, 28 minimize repair changes, and 24 test feasibility only.Several non-spend objectives use spend as a secondary tie-breaker.Source: B Task Taxonomy · p. 6
• Tables 2 and 3 organize the taxonomy’s buy-and-intake and disruption-recovery patterns.Source: Table 2 · p. 6, Table 3 · p. 6
• Seeded state scale increases from easy to hard tasks, including customers, vendors, products, BOMs, and average workcenters, while unrelated records add realism.Agents must identify relevant records and leave adjacent records untouched.Source: B Task Taxonomy · p. 6
• The corpus spans 12 industrial product categories, with 82 buy-only tasks, 207 allowing buying and manufacturing, and 11 requiring manufacturing only.Source: B Task Taxonomy · p. 6

C Task Technical Specification

• ERP-Bench represents procurement and manufacturing workflows through Harbor directories containing instructions, Odoo environments, solver-certified solutions, and terminal-state tests.Evaluation reads the final Odoo database, and the same verifier grades agent runs, oracle replay, and no-op replay.Source: C Task Technical Specification · pp. 6–7

D.1 Workflow Formalization

• CP-SAT models enforce demand coverage, minimum-order and capacity tiers, bill-of-material feasibility, workcenter capacity, and deadline compliance while optimizing pattern-specific objectives.The portfolio solver supplies native primitives for channeling, BOM disjunctions, and cumulative capacity, and returns optimality certificates used downstream by the verifier.Source: §D.1 · p. 7

D.2 Difficulty Recipes

• Difficulty recipes vary demand, deadlines, stock, vendor capacity, manufacturing depth, workcenter capacity, screening, invoicing, and repair disruptions across easy, medium, and hard tasks.Sampling these parameters changes vendor subsets, stock-versus-purchase decisions, make-or-buy choices, and feasible supply paths, producing qualitatively different solutions from the same workflow pattern.Source: §D.2 · p. 7, Table 5 · p. 8

D.3 Sampling, Rejection, and Determinism

• 300 accepted tasks were generated after rejecting 732 sampled parameter sets, using multi-phase optimization and deterministic tie-breaking to align each task’s artifacts with one stable certified plan.The generator completed in 656.76 seconds; 59 UNKNOWN statuses occurred only in tie-break phases and did not affect the certified primary solution.Source: §D.3 · p. 8

E.1 Verifier Protocol and Reward Formula

• The verifier grades terminal states through constraint, optimality, and traceability checks, with hard-zero gates preventing inactive screened or unrepaired states from receiving reward.The reward combines 0.25 constraint satisfaction, 0.60 optimality, and 0.15 traceability when all constraints pass; constraint failure gates the other dimensions.Source: §E.1 · p. 8, §E.4 · p. 9

E.5 Reward-Hacking Defense

• Reward-hacking defenses recompute objectives from authoritative ERP data and enforce tier-price compliance, while harnesses share task instances and verifiers but expose different tool surfaces.Coding uses shell, filesystem, and API tools; browser and computer-use harnesses restrict agents to their respective UI interfaces.Source: §E.5 · p. 9, F Harness Details · p. 9, Table 11 · p. 9

G.1 Headline Metrics by Model and Harness

• Coding-to-UI transitions reduce pass@5 by 49–56 percentage points for GPT-5.5 and GLM-5.1, while Claude Opus 4.7 has the smallest GUI penalty at 16 and 22 points.The comparison is reported across coding-to-browser and coding-to-computer-use transitions where available.Source: §G.1 · pp. 9–10

G.2 Task Parameters Predict Action Burden

• Task-scale parameters raise turns to resolve with Spearman correlations of +0.6 to +0.7 across every harness, linking generated difficulty to action burden as well as success probability.The correlated parameters include purchase orders, distinct vendors, manufacturing orders, components, and BOM lines.Source: §G.2 · p. 10, Figure 6 · p. 10

G.3 Feasibility and Optimality Split

• Across models, harnesses, and difficulty tiers, score loss with increasing difficulty comes from broken explicit constraints rather than worse trade-offs among feasible plans.Mean optimality among constraint-clean trials stays above 86 wherever at least one clean trial exists.Source: §G.3 · p. 10

G.4 Cost and Step Efficiency

• Coding points form the Figure 8 Pareto frontier, while every UI point and the Claude coding point fall below it; coding uses 24 turns per trial versus 171 for computer-use and 219 for browser.Tokens per turn remain within approximately 28k–30k across harnesses, so turn count drives the cost gap through action batching and UI limits.Source: §G.4 · p. 10, Figure 8 · p. 10, Figure 9 · p. 10

G.5 Reliability Across Repeated Trials

• Aggregate pass^5 is 9.8% in coding and 3.6% in browser, roughly an order of magnitude below corresponding pass@5 rates, indicating insufficient repeated-trial reliability for unattended deployment.pass@5 requires at least one success in five trials, whereas pass^5 requires all five to succeed.Source: §G.5 · p. 10, Table 12 · p. 11

H Failure Analysis

• Failure families span demand, timing, sourcing, manufacturing, and hygiene/state checks, with coding failures emphasizing plan execution and UI failures emphasizing ambiguous or partially committed form edits.The families are non-exclusive, and optimality is evaluated only after terminal states clear constraints and hygiene checks.Source: H Failure Analysis · p. 11, Table 13 · p. 11

H.1 Failure Examples

• A coding agent scheduled manufacturing before component receipts, then recreated invalid orders without rebuilding dependent purchases, triggering timing, feasibility, and origin-traceability failures.The trajectory illustrates how an initially invalid schedule propagates into later lineage-repair errors.Source: §H.1 · p. 11
• A browser agent continued after a sales-order autocomplete failed to commit, leaving required fields unverified and later entering a vendor it had already identified as late.The verifier flagged demand, deadline, sourcing, confirmation, pricing, and traceability rules.Source: §H.1 · p. 11
• A computer-use agent placed a purchase order despite stock covering the accepted demand, creating avoidable timing, consolidation, quantity, and price-tier obligations.The trace also records a displayed vendor price of 800.54 against a tier price of 865.41.Source: §H.1 · p. 11

• Single-source generation still permits incomplete business logic and renderer mistranslation, so validity depends on the specification and translation layers.Source: I Validity Checks · p. 12
• 300 of 300 tasks scored zero for a no-op agent, indicating seeded environments did not already satisfy their verifiers.Source: I Validity Checks · p. 12
• 300 of 300 oracle replays received full reward, showing agreement among setup, solver-certified plans, and verifiers.Source: I Validity Checks · p. 12
• LLM review across 300 tasks and 12 development iterations found generator-level defects in stock initialization, record-lineage checks, and policy-clause verification.Fixing these defects at the generator level propagated corrections to affected instances.Source: I Validity Checks · p. 12
• Zero of 16,159 completed trials triggered the reward-hacking canary for terminal states exceeding certified optima.Source: I Validity Checks · p. 12
• Experts achieved a mean reward of 90.48 across 30 task trials but encountered a broken explicit constraint in 3 trials.Their mean completion time was 55 minutes per task, comparable to the agents’ 60-minute trial budget.Source: I Validity Checks · p. 12