PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling

Zizhe Wang

arXiv:2608.23653v1cs.SEcs.AI

TL;DR

Physical system models can compile and simulate while violating engineering requirements, and iterative agents can lose requirements or reuse evidence from outdated candidates. The paper presents Pufibara and a source-grounded 232-task benchmark with independent evaluation, finding higher pass counts and lower reported resource use than Claude Code under matched backends.

  • Problem

    Modelica agent evaluation must account for physical consistency and scenario-dependent behavior, not only plausible code or successful execution across revisions.

  • Method

    Pufibara maintains persistent engineering state, binds execution and simulation evidence to candidate identity, and makes submission explicit, while the benchmark uses realistic independently evaluable tasks.

  • Results

    Across 232 tasks and two matched LLM backends, Pufibara passes more tasks than Claude Code and records lower reported logical-token totals and runtime.

  • Takeaways & Limitations

    Complete agent harnesses can differ substantially in task success and resource use for physical system modeling under matched LLM backends.

  • Takeaways & Limitations

    Each task is run once per harness-backend combination, and the complete-system comparison does not isolate individual harness mechanisms.

Abstract

from arXiv · show

AI agents are increasingly used for simulation-driven engineering. Physical system modeling presents different requirements from general-purpose code generation in software engineering, because correctness depends not only on syntax and executability but also on physical consistency and scenario-dependent behavior. We study this challenge in Modelica, an equation-based modeling language in which a model may compile and simulate while still violating its intended physics or engineering requirements. Across successive revisions, an agent may lose track of requirements or rely on simulation evidence produced by an outdated candidate. To address this challenge, we present Pufibara, an agent harness that maintains persistent engineering state across revisions, associates execution and simulation evidence with the candidate that produced it, and makes submission an explicit agent action. To evaluate end-to-end Modelica agent workflows, we also propose a source-grounded method for constructing realistic and independently evaluable tasks. We use this method to build the 232-task Modelica Agent Workflow Benchmark, spanning Model Repair, Model Generation, and Model Tuning. Each submitted candidate is scored by a benchmark-owned evaluator outside the agent loop. We compare Pufibara with Claude Code as complete harnesses under two matched large language model (LLM) backends. With DeepSeek v4 Flash, Pufibara passes 202 tasks, compared with 185 for Claude Code. With Claude Sonnet 5, Pufibara passes 202 tasks, compared with 187 for Claude Code. Under the repository-reported token accounting, Pufibara records 76.4%-82.5% lower logical-token totals. Its sequential runtime is 6.1%-58.4% lower. These findings show that, even under matched LLM backends, complete agent harnesses can differ substantially in both task success and resource use for physical system modeling.

1 Introduction

Physical system modeling requires agents to preserve engineering requirements and interpret simulation behavior across revisions, because compilation and execution alone do not establish physical correctness. The paper addresses this gap with Pufibara, a source-grounded benchmark, and matched-harness evaluation.

  • Motivation: Executable Modelica code can still violate task-specific physical constraints and behavioral requirements after iterative revisions.Agents may lose track of requirements or reuse simulation results from outdated candidates.
  • Benchmark: 232 end-to-end tasks are evaluated from engineering-brief interpretation through tool use, simulation-driven revision, and explicit submission.The benchmark spans complete Modelica agent workflows rather than only generated models.
  • Contributions: Pufibara preserves engineering requirements across revisions and binds execution and simulation results to the candidates that produced them.This harness is designed specifically for end-to-end Modelica workflows.
  • Contributions: The benchmark uses source-grounded task construction to balance realistic physical structure with independently evaluable criteria.Public models may overlap with pretraining data, while arbitrary synthetic tasks may lack realistic structure or trustworthy evaluation.
  • Evaluation: Pufibara achieves higher pass counts with lower reported logical-token use and runtime than Claude Code under two matched LLM backends.The comparison treats the harnesses as complete agent systems across the benchmark.

2 Related Work

Prior agent and Modelica research provides tools, datasets, and domain-specific workflows, but end-to-end evaluation of a common agent workflow for constructing and revising equation-based physical models remains underexplored. This paper positions its harness and benchmark as addressing that gap.

  • General Agent Workflows: Coding-agent benchmarks primarily use repository state and executable tests as feedback and acceptance criteria for patch revision.Physical modeling additionally requires evaluating continuous, scenario-dependent trajectories against physical constraints.
  • Simulation-Driven Engineering: Simulation-driven agent systems span finite element analysis, multibody simulation, Simulink, process systems engineering, and power-system analysis.These efforts operate in complementary domain settings rather than one common end-to-end Modelica workflow.
  • Modelica Research: Recent Modelica work emphasizes code generation, benchmark datasets, model evolution, repair, and agent-facing editing or simulation tools.These efforts do not by themselves define a complete agent evaluation workflow.
  • Gap: Prior work had not introduced an agent harness specifically designed for end-to-end Modelica workflows or a benchmark evaluating those workflows.The paper identifies this as the remaining gap after existing artifact, dataset, grading, and tooling efforts.

3 Engineering Invariants and Candidate-Bound Evidence

The paper represents engineering requirements as persistent, candidate-relative obligations linked to observables, scenarios, and adjudicated simulation evidence. This state guides revision and readiness while leaving official acceptance to an independent evaluator.

  • Persistent Obligations: An engineering invariant is a task-level obligation that remains in force across modeling iterations and must be addressed before submission.It may express physical relationships, engineering constraints, or scenario-specific behavioral requirements.
  • Persistent Obligations: Each invariant specification links an engineering proposition to relevant observables, simulation scenarios, and an evidence-judgment condition.The condition may be quantitative, such as a tolerance, or qualitative, such as an expected response pattern.
  • Evidence Binding: Candidate-bound evidence records the addressed invariant, candidate, scenario, inspected observables, resulting trajectories or measurements, and agent adjudication.Evidence from an earlier candidate remains in audit history but cannot support readiness for a revised candidate.
  • Evidence Binding: Successful simulation alone is insufficient because observations must be interpreted against the engineering proposition that motivated the simulation.Adjudication classifies evidence as supporting, violating, or inconclusive.
  • Revision and Readiness: Candidate modification requires relevant evidence to be refreshed, while open, violated, or inconclusive obligations trigger targeted collection, revision, or further simulation.The workflow uses the ledger to decide whether to gather evidence, revise the candidate, or submit.
  • Revision and Readiness: Readiness requires coverage and support for each required scenario using evidence bound to the current candidate, but official task outcome remains independently evaluated.The readiness rule is specific to both the candidate and the engineering brief.

4 The Pufibara Agent Harness

Pufibara structures Modelica agent runs around persistent engineering state, transparent execution, candidate provenance, and explicit submission. Its internal readiness process supports agent decisions, while a benchmark-owned evaluator determines official acceptance after submission.

  • Run Protocol: Each run begins with a task package and ends with explicit submission or termination without submission.Pufibara mediates access to engineering state, workspace tools, model checking, and simulation.
  • Architecture: The harness combines an Agent Runtime, Persistent Engineering State, and Transparent Execution Plane.These components mediate workflow semantics, preserve candidate-bound records, and perform workspace operations, checking, and simulation.
  • Workflow Profiles: Workflow profiles define task representation, permitted actions, candidate representation, and submission semantics for Model Repair, Generation, and Tuning.The profiles function as execution protocols rather than merely prompt templates.
  • Persistent State: Persistent state records task and profile context, candidate identity, workflow-specific engineering state, execution and simulation records, transitions, prior actions, and submission.Candidate identity changes whenever evaluation-relevant submitted content changes.
  • Execution and Provenance: Transparent execution returns and records the requested action, target candidate, resulting observation, and produced artifacts without silently modifying candidates.Model checking results and simulations remain associated with the candidate that generated them.
  • Execution and Provenance: Agents must adjudicate simulation results against engineering obligations because Pufibara does not determine whether their physical interpretation is correct.The harness preserves the adjudication and its candidate binding.
  • Submission Boundary: Submission freezes the exact selected candidate, and a benchmark-owned evaluator later applies a task-specific acceptance contract to that submission.The harness neither promotes the last runnable model nor infers submission from tool success.

5 Modelica Agent Workflow Benchmark

The benchmark uses source-grounded synthesis to create realistic, novel, independently evaluable Modelica tasks and evaluates complete iterative agent workflows through explicit submission and frozen contracts.

  • 5.1 Source-Grounded Task Construction: Source-grounded synthesis starts from executable clean references and derives new faults, briefs, tuning targets, or evaluation contracts rather than reusing the reference as the answer.This preserves realistic physical structure while reducing direct memorization of public source models.
  • 5.3 Independent Evaluation Protocol: Each run begins with an agent-visible task package and ends at explicit submission or an execution limit, while a benchmark-owned evaluator scores only the exact submitted output.Runs without submission are recorded as failures to submit.
  • 5.2 Benchmark Scope and Workflow Families: Each task fixes its top-level model, dependency context, and simulation configuration, preserving the package and library structure used by the agent.The evaluator reconstructs the corresponding package workspace around the submitted candidate.
  • 5.3 Independent Evaluation Protocol: Task validation requires faulty Repair inputs to fail at intended stages and valid Generation or Tuning answers to pass while executable off-target variants fail behavioral checks.The private contract is reviewed against the agent-visible brief and checks are repeated when evaluators or dependencies change.
  • 5.2 Benchmark Scope and Workflow Families: The benchmark contains 232 tasks: 132 Model Repair, 50 Model Generation, and 50 Model Tuning tasks.Difficulty is assigned using empirical agent performance and workflow complexity rather than source-code length alone.
  • 5.3 Independent Evaluation Protocol: Evaluator independence does not guarantee correctness, because incorrect observables, scenarios, tolerances, grids, or sign conventions can cause false acceptance or rejection.A PASS establishes satisfaction of the frozen, scenario-covered engineering contract, not formal verification or universal physical correctness.

6 Evaluation

The evaluation compares Pufibara and Claude Code as complete harnesses under matched LLM backends on the 232-task benchmark, measuring success, resource use, and correctness beyond executability. Pufibara records higher observed pass counts, lower token use and runtime, and fewer executable submissions that fail behavioral requirements under the reported conditions.

  • Evaluation Scope and Design: The evaluation compares complete Pufibara and Claude Code harnesses across 232 tasks under two matched LLM backends.Both harnesses use the same underlying LLM, task set, Modelica environment, and benchmark-owned evaluator within each backend and workflow.
  • Evaluation Scope and Design: Task success uses pass counts, resource use uses logical tokens and sequential wall-clock runtime, and correctness beyond executability uses the behavioral contract.A pass requires every applicable benchmark gate to pass; logical tokens include uncached input, cache creation, cache reads, and output.
  • Benchmark Results: 202 tasks pass for Pufibara under both DeepSeek v4 Flash and Claude Sonnet 5, versus 185 and 187 for Claude Code, respectively.Pufibara has the higher observed pass count in all six backend-workflow combinations.
  • Benchmark Results: Pufibara’s pass-rate advantage is largest in Model Generation, reaching 16 percentage points with DeepSeek v4 Flash and 12 with Claude Sonnet 5.The corresponding gains are 4.5 points in Repair and six points in Tuning under each backend.
  • Benchmark Results: 76.4%-82.5% lower logical-token totals and 6.1%-58.4% lower sequential runtime are reported for Pufibara across comparisons.All reductions use Claude Code as the baseline within the same backend and workflow; the smallest runtime difference occurs in Generation.
  • Beyond Executability: Among 38 hard Model Generation tasks under Claude Sonnet 5, executable Claude Code submissions failed the behavioral contract on 21 tasks, versus four for Pufibara.These submissions produced valid simulation trajectories but did not satisfy required behavior in the evaluated scenarios, which execution-only evaluation would miss.

7 Discussion

The evaluation distinguishes executable Modelica artifacts from models that satisfy engineering requirements, showing that harness organization is associated with task success and resource use. Pufibara makes requirements, candidate-bound evidence, and submission decisions explicit, while the benchmark separates official evaluation from the agent loop.

  • Evaluation implications: Pufibara has higher pass counts and lower reported resource use than Claude Code across two matched LLM backends and three workflow families.The consistent direction does not establish universal superiority, but indicates that workflow organization remains consequential under matched conditions.
  • Workflow differences: Model Generation shows the largest pass-count difference because agents must determine structure, observables, simulation strategy, behavioral adequacy, and submission.Repair receives comparatively direct feedback and operates near ceiling, while Tuning restricts actions to parameters of a frozen model.
  • Beyond executability: Among 38 hard Generation tasks under Claude Sonnet 5, executable but behaviorally incorrect submissions occur on 21 Claude Code tasks and four Pufibara tasks.Model checking and successful simulation alone do not establish that trajectories satisfy intended engineering behavior.
  • Harness design: Pufibara’s architectural contribution is explicit linkage among engineering obligations, observables, scenarios, candidate identity, evidence, and submission decisions.The harness records revision and submission as auditable actions rather than treating a runnable artifact as an implicit endpoint.
  • Benchmark design: The benchmark evaluates the exact submitted artifact with frozen contracts outside the agent loop, exposing executable-but-behaviorally-incorrect submissions.Source-grounded task synthesis uses executable reference models for realistic structure while introducing new faults, briefs, tuning targets, and private contracts.
  • Limitations: The complete-system comparison uses one run per harness-backend combination and does not isolate individual harness mechanisms.Repeated trials and controlled ablations are identified as ways to address this limitation.
  • Limitations: PASS results are limited to fixed scenarios and acceptance contracts, while proprietary industrial workload coverage remains untested.The evaluation is also limited to Modelica, Claude Code as the comparison harness, and two LLM backends.

8 Conclusion

The paper presents Pufibara and a source-grounded benchmark for independently evaluating end-to-end Modelica agent workflows. Across 232 tasks and two matched LLM backends, Pufibara achieves higher pass counts and lower reported resource use than Claude Code, while the conclusion remains bounded by the evaluated scenarios and contracts.

  • Contributions: Pufibara preserves engineering requirements, candidate-bound evidence, and explicit submission decisions for physical system modeling agents.The paper also introduces a source-grounded method and the Modelica Agent Workflow Benchmark for independent end-to-end evaluation.
  • Results: Across 232 tasks and two matched LLM backends, Pufibara achieves higher pass counts than Claude Code for every workflow.It passes 202 tasks versus 185 with DeepSeek v4 Flash and 202 versus 187 with Claude Sonnet 5; logical-token totals are 76.4%–82.5% lower and runtime is 6.1%–58.4% lower.
  • Scope: The findings provide system-level evidence for the reported Modelica tasks and scenario-bounded contracts but do not isolate individual harness mechanisms or establish universal superiority.Future work includes repeated trials, controlled ablations, broader contract validation, and additional modeling languages and simulation environments.

Artifact Availability

The paper releases documentation, summary results, and selected supporting files, while retaining most implementation and complete benchmark evaluation files privately to reduce contamination risk.

  • Availability: Most of the Pufibara implementation and complete benchmark tasks and evaluation files remain private to reduce future benchmark contamination.Controlled research access or a hosted submission service may later support independent evaluation while keeping hidden tasks and contracts confidential.

A Agent Configurations

The reported evaluation fixes matched-backend comparisons and execution accounting across the two harnesses. Configuration records define task limits, usage-field mappings, failure handling, and the evaluator’s treatment of exact submissions.

  • Model configurations: Pufibara uses temperature 0.1 with DeepSeek v4 Flash, while Claude Code retains native sampling semantics; Claude Sonnet 5 runs without an explicit temperature setting.The two harnesses retain their native request semantics within each matched backend condition.
  • Execution limits: Table 5 records the per-task execution limits used in the reported evaluation.Verification grace extends wall time only for final verification already in progress after explicit submission and does not add agent turns.
  • Failure handling: Token limits function as nonbinding safety guards, while infrastructure-invalid attempts are excluded when they fail before a valid capability measurement.Any protocol-authorized replacement uses a fresh execution identity with unchanged task configuration.
  • Evaluation protocol: Both harnesses receive the same agent-visible task package and frozen evaluator contract, which runs only on the exact explicit submission after the loop ends.Pufibara exposes persistent engineering state, whereas Claude Code retains its native prompt, context, and tool-use semantics.
  • Accounting: Reported runtime sums valid sequential task wall times in the specified OpenModelica Docker environment.Harness-native usage fields are mapped to logical-token categories, and unreported cache categories are not reconstructed.
Loading 2608.23653v1…