PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy as Code: A Coroutine-Bridge Harness for Fast-Reasoning Reliability on CAR-bench

Ivan Matveev

arXiv:2609.29251v1cs.AIcs.CL

TL;DR

Dependent tool chains in CAR-bench force conventional agents to make a model call after each evaluator result, while repeated-trial consistency is central to the benchmark. The paper uses a coroutine bridge to let Python programs span tool exchanges and encode deterministic policies in tool logic. The harness achieved 60.0% Pass3, 4.5× the organizer baseline, and won Track 2; the unchanged harness reproduced 60.0% Pass3 on GPT-5.5.

  • Problem

    CAR-bench externalizes tool execution, so dependent calls require repeated agent round-trips, and its Pass3 metric rewards tasks that pass all three trials.

    Source: §1 · p. 1

  • Method

    A coroutine bridge suspends a Python program at tool calls and resumes it with evaluator results, while deterministic policies are enforced in tool-layer logic.

    Source: §1 · p. 1

  • Results

    60.0% Pass3, 4.5× the organizer baseline; the unchanged harness also won Track 2 and reproduced 60.0% Pass3 on GPT-5.5.

    Source: §5 · p. 3, §6 · p. 4

  • Takeaways & Limitations

    The hidden-set comparison reports that the harness’s consistency advantage helps tasks solved once become more likely to pass all three trials.

    Source: §5 · p. 3

    40% of hidden tasks still fail at least one trial, with residual failures attributed to simulator and judge variance and rare valid-looking but wrong model actions.

    Source: §6 · p. 4

Abstract

from arXiv · show

CAR-bench evaluates whether tool-using agents stay reliable under real-world uncertainty, executing every tool inside the evaluator so that each tool-result exchange is a separate agent round-trip. A conventional next-action agent can batch parallel tool calls, but a chain of dependent calls costs it one model call per round of results. We present a coroutine-bridge harness in which the model's only action is to emit a Python program that blocks and resumes in place across evaluator tool exchanges. This decouples model invocation from tool round-trips: on the public test split the agent uses a median of two model calls against seven agent turns per task, resolving a full multi-turn task in a median of 1.8 s of model latency on Cerebras gpt-oss-120b. Because the action surface is executable code, deterministic CAR-bench policies are encoded directly as logic in the tool layer rather than as prompt rules, enforcing compliance at zero reasoning cost. On the official hidden evaluation the harness won Track 2 with 60.0% Pass^3, 4.5x the organizer baseline, at the lowest estimated cost and the fastest median task latency (3.14 s) of any entry scoring above that baseline; the same unchanged harness reproduced an identical 60.0% Pass^3 on GPT-5.5 in the Open track, matching frontier-model agents. A single static prompt, appended with per-task state at the tail, stays byte-identical across calls and across tasks: the frozen submission prompt served 78% of input tokens from cache (86.6% across its warm tail), against 73% over a three-week development corpus in which prompt edits repeatedly reset the cache. This compounds the few-call design into a small fraction of nominal input compute.

1 Introduction

CAR-bench tests whether in-car tool agents remain consistent and limit-aware across repeated trials. The paper introduces a coroutine-bridge harness to reduce the model-call cost of dependent tool chains while keeping policy compliance out of the model’s reasoning burden.

1 Introduction

  • CAR-bench evaluates consistency and limit-awareness across 58 tools, 19 policies, and 254 public tasks, using Pass3 to credit tasks that pass all three trials.The benchmark spans base, hallucination, and disambiguation categories.Source: §1 · p. 1
  • Dependent tool chains require a conventional next-action agent to make a model call after each round of results, unlike independent calls that can be batched.The benchmark externalizes tool execution to the evaluator, which returns tool results in a fresh message.Source: §1 · p. 1
  • The harness emits one executable Python program that suspends at tool calls and resumes the same stack frame when results arrive, avoiding a new model call.The bridge keeps tool calls visible in the evaluator’s trajectory while reducing model invocations between round-trips.Source: §1 · p. 1

2 The Coroutine Bridge

CAR-bench’s evaluator-executed tools make each dependent exchange a separate agent round-trip. The coroutine bridge lets a single Python program span those exchanges, while batching also shortens the path for independent calls.

2.1 Externalized Tools Force Round-Trips

  • An n-step dependent tool chain requires n separate agent round-trips because the evaluator executes tools and returns their results to the agent.Externalized execution also enables trajectory scoring and environment perturbations.Source: §2.1 · pp. 1–2

2.2 Decoupling Model Calls from Tool Round-Trips

  • A median of two model calls carries a median of seven agent turns per task because the bridge resumes the blocked Python frame with tool results without another model invocation.The tool calls remain in the official evaluator trajectory; only the number of model invocations between round-trips changes.Source: §2.2 · p. 2

2.3 Parallel Batching

  • Independent reads are sent together in one parallel tool-call message, while dependent calls remain sequential Python operations.Batching reduces independent tool round-trips in addition to model-call savings.Source: §2.3 · p. 2

3 Policy as Code

The harness places deterministic policy in tool-layer logic rather than relying on the model to recall prompt rules. This divides responsibility between runtime-enforced facts and policy and the model’s interpretation of user intent.

3 Policy as Code

  • Deterministic policies grounded in facts, such as AC confirmation for window opening above 25%, can be enforced as tool-layer logic rather than prompt prose.The stated test is whether a policy is deterministic given grounded facts; genuine interpretation remains in the prompt.Source: §3 · pp. 2–3
  • The per-user-turn call-depth audit caps sequential model calls at five, with a measured median of two calls per task and a p90 of four.Parallel tool batches resume the blocked Python frame without a model call and do not count toward the sequential-call limit.Source: Figure 1 · p. 2
  • The runtime owns facts and strict policy, while the model interprets intent; guarded wrappers enforce policy-critical behavior across public and helper tool names.A confirmation state machine stores the grounded pending action and resumes it on the user’s next turn.Source: §3 · p. 2

4 Reliability Mechanisms

Three mechanisms target CAR-bench failure classes: response obligations, unknown-value sentinels, and turn-scoped outcome tracking.

4 Reliability Mechanisms

  • Response obligations append policy-mandated disclosures only when the model’s response omits them, preserving flexibility for compound requests.Examples include toll-road mentions and cross-zone temperature warnings above 3°C.Source: §4 · p. 3
  • Unknown-value sentinels turn attempts to use blanked fields in meaningful operations into a missing-response acknowledgement rather than silent fabrication.The string "unknown" is normalized to a storage- and serialization-safe sentinel.Source: §4 · p. 3
  • Turn-scoped outcome tracking preserves mutation failures by tool and target arguments across Python blocks, preventing success claims from outrunning failed side effects.Source: §4 · p. 3

5 Results

The harness ranked first in Track 2, converting the same number of successful trials as the runner-up into more three-trial passes. It also reduced model latency and maintained a cacheable prompt structure.

5 Results

  • 1.8 s was the median model latency for a complete multi-turn task, while agent LLM time accounted for 899 of 5,351 s of the public evaluation’s wall time.The remaining wall time was attributed to the evaluator’s simulator and judge.Source: §5 · p. 3
  • 60.0% Pass3 beat the runner-up’s 50.0% despite identical 57/90 successful trials; Table 2 records first place among 13 Track 2 submissions.The caption attributes the gap to consistency: 92.5 versus 73.9.Source: Table 2 · p. 3, §5 · p. 3
  • 4.5× Pass3 (60.0 vs 13.3) accompanied 38% lower median latency and 20% fewer mean tokens per trial versus the same-model organizer baseline.Source: §5 · p. 3
  • The unchanged harness scored 60.0% Pass3 on GPT-5.5 in the Open track, with 14.1 s median task latency and a different coverage-consistency decomposition.The GPT-5.5 model had higher coverage but lower success consistency than the Cerebras entry.Source: §5 · p. 3
  • 73.2% of 918M input tokens were cached across three weeks of development; frequent prompt edits reset the cache, unlike the frozen deployed prompt.The stable-prompt test day reached 86.6% cache use over its final half hour.Source: §5 · pp. 3–4

6 Limitations and Conclusion

The coroutine bridge pairs fast inference with branching programs and deterministic policy compiled into tools. The harness won Track 2 and reproduced its score on a frontier model without architectural changes.

  • 40% of hidden tasks still fail at least one trial, while Pass@3 (66.7%) trails the runner-up (76.7%).The residual gap includes simulator and judge variance, rare valid-looking but wrong actions, and foreclosed lucky recoveries from a more exploratory agent.Source: §6 · p. 4
  • The design combines fast calls that execute entire branching programs with deterministic policy compiled into the tools those programs invoke.Source: §6 · p. 4
  • The harness won Track 2 at the lowest cost and fastest latency among entries scoring above baseline, and reproduced its score on a frontier model without architectural changes.Source: §6 · p. 4
Loading 2609.29251v1…