Source-linked AI summary
SimVerity: When Does Simulated Agent Success Survive Physical Deployment?
Zhonghao Zhan, Yefan Zhang, Krinos Li, Hamed Haddadi
TL;DR
SimVerity addresses the lack of systematic evidence about whether simulated agent passes justify physical deployment. It replays matched scenarios on target smart homes and cross-validates execution with qualified physical witnesses. Deployment success split across process-level verdicts, a frozen risk profile beat a property-blind baseline in all eleven held-out sessions, and a second simulator added no independent cross-check.
Problem
Whether a simulator’s pass justifies physical deployment remains insufficiently quantified, despite simulated evaluation being widely used for agent benchmarking.
Method
SimVerity replays declared scenarios on target deployments, compares matched traces against declared properties using qualified witnesses, and issues risk, coverage, and abstain-or-escalate dispositions.
Results
A frozen risk profile beat a property-blind baseline in all eleven held-out sessions across two cohorts, while a second simulator never disagreed on overlapping cases and physical measurement exposed shared blind spots.
Takeaways & Limitations
Verdict transfer becomes an explicit deployment decision—clear, abstain, or escalate—rather than an assumption that simulation answers generalize physically.
Takeaways & Limitations
The primary predictor covers one held-out physical path–rung pairing, and only whitelisted lights and isolated proxies were tested.
Abstract
from arXiv · showhide
Simulated evaluation is widely used to benchmark AI agents, yet how much evidence a simulated pass provides about physical deployment has not been systematically quantified. We present SimVerity, a verdict-transfer assurance framework: it replays matched scenarios on target smart home deployments and cross-validates agent execution against independently qualified physical witnesses. Our evaluation highlights that deployment success is a real-world process, not a static property in simulation: completion, reported state, observable effect, and settled outcome diverged within the same execution. Although an advanced simulator cleared all 240 light trials, a camera caught 42 sub-second failures invisible to settled-state checks. False clearance was predictable: a risk profile learned from measured trials and locked before evaluation predicted failures on a path it never physically measured, beating a property-blind baseline in all eleven held-out sessions across two cohorts. Agent auditability was also measurable: switching one agent loop's model-client/serving configuration raised its scenario-matching share from 52-88% to 100%. Finally, a second qualified simulator added no independent cross-check: it never disagreed on any overlapping case, and only physical measurement exposed their shared blind spots. SimVerity turns verdict transfer into an explicit decision: clear, abstain, or escalate before deployment.
Introduction
SimVerity frames simulated clearance as a verdict-transfer question: whether a simulator’s pass survives on a physical deployment. It addresses an abstraction mismatch by measuring process-level outcomes with qualified witnesses and explicit risk and abstention decisions.
- Simulation treats success as a static property, whereas physical deployment unfolds as a process whose verdicts can diverge.
- The simulator passed all five held-out off trials, yet a qualified camera saw the light on in 5/5 at the 100-ms boundary.At the settled boundary, all 30 trials passed, including those five.
- SimVerity replays declared scenarios on target deployments, compares matched traces against the same property, and abstains when physical evidence is missing or unqualified.It produces risk, coverage, and clear-or-abstain-or-escalate dispositions indexed by property, read boundary, path, and deployment rung.
- The framework measures verdict fidelity and false-clearance risk rather than choosing actions, updating policies, or serving as a model leaderboard.
- A frozen risk profile beat a property-blind baseline in all eleven held-out sessions across two cohorts, while a second simulator never disagreed on overlapping cases.Physical measurement remained the independent check that exposed shared simulator blind spots.
Methods
SimVerity aligns source and target traces, grades declared properties with pass, fail, or abstain, and converts matched evidence into explicit deployment dispositions. Calibration-only rows freeze structured risk profiles, while qualified witnesses and complete accounting prevent missing evidence from becoming agreement.
- A source rung Q issues a clearance for scenario x, while target rung T measures whether the declared property ϕ survives after semantic alignment.
- Abstention is assigned for unqualified witnesses, missing trace stages, unsupported capabilities, or unavailable boundary evidence, and excluded from binary estimands.
- Trace adapters pair trials bijectively and map executions onto six semantic stages without relabeling ingestion time as physical event or effect time.
- Property monitors evaluate completion, reported reads, observable effects, settled outcomes, effect order, and fanout using declared observation channels.
- Qualified witnesses must distinguish reference outcomes before testifying, while sessions additionally require cross-response separation, trace completeness, and settled-control agreement.
- Calibration-only risk profiles estimate deployment-failure probabilities by property, stage or direction, and path, using Jeffreys smoothing and frozen backoff ladders.
- Every audited cell reports clearances, blocks, abstentions, session and path totals, exact Clopper–Pearson intervals, and a verdict-transfer disposition.
Experimental Design
The experiments use a multi-rung smart-home testbed, preregistered calibration and held-out splits, frozen predictors, and live agents under executable model–client configurations. The design separates verdict-transfer tests from dynamic-agent auditability and records blocked configurations explicitly.
- The testbed combines unmodified SimuHome with Home Assistant at software and physical rungs, where real devices are observed by an independently qualified camera witness.
- Calibration and held-out data are separated by session and cell, with repetitions between device resets kept within a split.
- The main campaign used nine valid calibration sessions, 586 trials, 1,070 eligible source-cleared pairs, and at least 3 qualified physical sessions.
- Six frozen predictors form a structural ladder from trusting source verdicts to property × stage/direction × path profiles.
- Live-agent tests used a ReAct tool agent and a planner–executor agent at temperature zero, measuring matched-action and natural-distribution verdict metrics under an 80% eligibility gate.
- Executable configuration rows index model–client/serving combinations, while simulation and physical runs use fixed trial orders, budgets, manifests, and temperature-zero execution.
- Blocked rows remain explicit when temperature enforcement or model identity cannot be established, and physical evidence is not rerun.
Evidence and Calibration
Physical evidence shows that deployment verdicts depend on the property, boundary, observation channel, and environmental conditions. The pilot and replication results motivate calibrated prediction while defining where witness limitations require abstention.
- Pilot: 40 calibration and 30 held-out pilot trials produced five observable false clearances at an unseen 0.1-s boundary, while completion failed every trial and settled control failed none.
- Main campaign: Across two devices and boundaries down to 0.05 s, settled effect was uniformly valid, observable failures were off-only through 0.25 s, reported state had a sub-50 ms boundary, and completion was uniformly invalid.
- Environmental validity: Ambient lighting invalidated two early calibration sessions because optical separation failed at dusk and on–off contrast failed at midday.
- Main campaign: Figure 3 organizes false clearances by property and boundary, frozen Brier scores by cohort, and off-direction counts by boundary.
- Replication: A registered replication found stale television reported state after external power-on, intermittent cold turn_on failure, and event multiplication in physical door-contact traces.
- Site replications: Additional deployments abstained on short-boundary optical cells because their cameras could not certify sub-second timing, while their scope did not include risk profiles or agents.
Results
Physical evaluation shows that verdict transfer depends on the property, observation channel, deployment path, and executable agent configuration. Frozen risk prediction beat a path-only baseline across both cohorts, while a second simulator added no independent disagreement.
- Property-Selective Fidelity: 1,090 calibration rows exposed property-selective validity: settled effect remained valid, observable and reported verdicts had boundaries, and completion was invalid at its declared read.The same executions produced different conclusions across properties and timing boundaries.
- Predicting False Clearance: 8/8 confirmatory sessions favored the frozen profile over path-only, yielding wins in all 11 held-out sessions across two cohorts.The secondary comparison favored strong lookup in 7/8 controlled sessions, indicating condition-dependent performance.
- Auditability Under Live Agents: 32/32 planner trials failed completion because its immediate return read pre-update state, while ReAct’s closing turn occurred after the update.The live-agent pattern remained measurable under dynamic tool selection.
- Auditability Under Live Agents: Switching the custom ReAct loop and task grid to the registered frontier configuration restored every action match, making auditability a property of executable configuration.The intervention could not isolate model weights from provider, endpoint, and serving behavior.
- Cross-Simulator Transfer and Clearance Policies: Zero eligible disagreements on 160 anchors meant strict consensus reduced conditional FCR by 1.25 points only by surrendering 25 coverage points.The second simulator’s agreement reflected shared blind spots rather than independent detection.
- Cross-Simulator Transfer and Clearance Policies: Deployment anchors alone calibrated risk: the second simulator never disagreed on eligible cases, and simulator agreement could not replace deployment-grounded assurance.The audit therefore preserves clear, abstain, and escalate dispositions rather than treating agreement as sufficient.
Related Work
Prior work formalizes simulation validity, sim-to-real predictivity, sandbox evaluation, and evaluation-pipeline auditing, but does not directly test whether a simulator verdict survives witnessed deployment.
- Simulation validity and verdict preservation: Validity and conformance frameworks formalize model trust, while SimVerity instantiates verdict transfer through per-property agreement, FCR, abstention, and held-out prediction.The paper positions verdict transfer as a distinct evaluation target.
- Sim–real predictivity and physical testing: Sim-to-real and driving studies compare virtual and physical failures or rankings, whereas SimVerity evaluates whether each specific scenario pass survives deployment.The per-scenario test-track comparison is identified as the closest precedent.
- Agent benchmarks and sandbox validity: Smart-home and tool-agent benchmarks commonly use saved-state or sandbox verdicts, while FCR asks whether a simulation pass becomes an observed deployment failure.This frames SimVerity’s question at the verdict level.
- Evaluation validity as an alignment problem: Alignment-track studies audit contamination, steerability, evaluator bias, and success–safety separation, but not verdict survival on witnessed deployment.Benchmark-artifact and replay-determinism audits likewise address different failure modes.
- Evaluation validity as an alignment problem: Deployment simulation forecasts prevalence and Composable Assurance propagates assertions through MLOps; SimVerity instead calibrates deployment admissibility of simulator verdicts.The distinction is between forecasting or assertion propagation and physical verdict preservation.
Limitations, Ethics, and Reproducibility
The study’s evidence is bounded to a narrow physical scope and supports risk estimation rather than certification, with configuration and environmental limitations affecting interpretation.
- Scope: The primary predictor covers one held-out physical path–rung pairing across 11 sessions in two cohorts and estimates risk, not certification.The strong-lookup comparison favors lookup under stationary control, while unpooled site cohorts test portability alone.
- Scope and ethics: Only whitelisted lights and isolated proxies were tested; frames were reduced to scalars and discarded, and no human data were collected.These choices constrain generalization to broader home devices, observations, and human-centered settings.
- Configuration and reproducibility: The model–client/serving intervention cannot isolate weights from serving, and ecosystems without first-class determinism resist frozen evaluation.Invalid ReAct sessions mark a configuration-level auditability boundary.
- Reproducibility: Versioned manifests, sanitized ledgers, hashes, replay code, and frozen arrays support regeneration of reported aggregates with abstentions and invalid sessions retained.The materials are provided for one-shot metric recomputation.
Conclusion
SimVerity finds that simulation passes are promises about deployment, not deployment facts: physical measurement revealed property, configuration, and shared simulator blind spots. The supported operational conclusion is to audit deployment survival before shipping.
- Conclusion: Physical executions split simulated passes into four verdicts, while camera observation exposed short-lived light failures missed by settled-state checks.The conclusion treats deployment success as a process rather than a static simulation property.
- Conclusion: A risk profile frozen before testing beat a property-blind baseline in all 11 held-out sessions across two cohorts, despite never measuring the held-out path physically.This supports prediction of false clearance beyond the measured path.
- Conclusion: Auditability followed executable configuration, and a second qualified simulator never disagreed with the first on overlapping cases.Only physical measurement exposed their shared blind spots.
- Conclusion: The paper’s practical conclusion is to ship the audit rather than treat simulator answers as deployment-local facts.The recommendation is bounded to the measured verdict-transfer setting.