Source-linked AI summary
CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty
Johannes Kirmayr, Lukas Stappen, Elisabeth André
TL;DR
Existing LLM-agent benchmarks often overlook reliable behavior under incomplete information and ambiguous requests in user-facing settings. CAR-bench evaluates multi-turn, policy-constrained agents in an automotive environment using new Hallucination and Disambiguation tasks. Results show substantial gaps between occasional and consistent success, with premature actions, policy violations, and fabricated responses remaining common.
Problem
Existing benchmarks provide limited evidence about reliable agent behavior when user requests are incomplete or ambiguous and capabilities or information are missing.
Method
CAR-bench evaluates multi-turn, policy-constrained LLM agents with a simulated user, 58 interconnected tools, domain policies, and Hallucination and Disambiguation tasks.
Results
Frontier reasoning models achieve less than 50% consistent success on Disambiguation tasks, while evaluations also reveal premature actions, policy violations, and fabricated responses.
Takeaways & Limitations
Reliable deployment requires evaluating consistency, uncertainty management, accurate tool use, policy adherence, and honest communication of system limitations.
Takeaways & Limitations
The LLM-simulated user can introduce model-specific errors, noise, or bias into benchmark outcomes and may misrepresent natural interaction patterns.
Abstract
from arXiv · showhide
Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. In domains, such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. The environment features an LLM-simulated user, domain policies, and 58 interconnected tools spanning navigation, productivity, charging, and vehicle control. Beyond standard task completion, CAR-bench introduces Hallucination tasks that test agents' limit-awareness under missing tools or information, and Disambiguation tasks that require resolving uncertainty through clarification or internal information gathering. Baseline results reveal large gaps between occasional and consistent success on all task types. Even frontier reasoning LLMs achieve less than 50% consistent pass rate on Disambiguation tasks due to premature actions, and frequently violate policies or fabricate information to satisfy user requests in Hallucination tasks, underscoring the need for more reliable and self-aware LLM agents in real-world settings.
1 Introduction
CAR-bench addresses reliability gaps in user-facing LLM agents by evaluating policy-constrained, multi-turn interaction under missing capabilities and ambiguity. It introduces new task types and reports substantial consistency failures, including premature actions, policy violations, and fabrication.
- Research gap: Existing benchmarks largely evaluate tool use or task completion with complete information and idealized interaction histories.These settings underrepresent unsatisfiable requests, incomplete environment data, and underspecified user goals.
- CAR-bench: Hallucination tasks remove required tools, parameters, or results, while Disambiguation tasks require agents to resolve uncertainty through internal information gathering or user clarification.These tasks test whether agents acknowledge missing capabilities and avoid acting before ambiguity is resolved.
- Research gap: In-car assistants make uncertainty handling safety-critical because users issue ambiguous speech-like requests amid heterogeneous APIs and strict distraction constraints.The domain combines non-expert users, vehicle-specific tools, and safety-sensitive actions.
- CAR-bench: CAR-bench evaluates multi-turn, policy-constrained agents with an LLM-simulated user, 19 domain policies, interconnected tools, and dynamic environment components.The framework includes mutable states, fixed context variables, and contextual databases.
- Results: 54% average Pass^3 and a GPT-5 Disambiguation decline from 68% Pass@3 to 36% Pass^3 reveal large gaps between potential and consistent success.The results also identify fabrication, premature actions, and stochastic policy violations as recurring failures.
- Contributions: The benchmark contributes 58 interconnected tools, 19 policies, new Hallucination and Disambiguation tasks, and an error taxonomy comparing reasoning and non-reasoning models.These components support systematic evaluation of limit-awareness and uncertainty resolution.
2 Related Work
Prior LLM-agent benchmarks cover diverse environments and tool-use settings, but many assume single-turn interaction, complete tool coverage, or idealized histories. Recent work increasingly addresses multi-turn dialogue, hallucination, disambiguation, reasoning, and domain-specific policy adherence.
- Agent benchmarks: Existing benchmarks evaluate web navigation, embodied tasks, code execution, and single, parallel, or sequential API calls.Tool-use evaluations typically assume full tool coverage and limited conversational uncertainty.
- Multi-turn dialogue: Many multi-turn benchmarks rely on pre-collected trajectories that condition agents on continuing idealized interaction histories.This differs from testing policy development during iterative, under-specified interactions.
- Hallucination and disambiguation: LLMs often generate plausible incorrect information instead of acknowledging uncertainty, while disambiguation requires detecting missing information and selecting clarification strategies.Reasoning-enhanced models are discussed as a related response to these challenges.
- Domain-specific policies: Policy-guided benchmarks extend evaluation beyond general knowledge, with automotive applications adding safety-critical APIs and driver-distraction constraints.The car domain makes hallucination avoidance and efficient disambiguation especially consequential.
3 CAR-bench
CAR-bench builds a dynamic automotive assistant environment with simulated multi-turn users, policy-guided tool-using agents, interconnected state and context, and three task types. Its evaluation distinguishes ordinary completion from limit-awareness and ambiguity resolution, aggregating repeated binary outcomes with consistency and potential metrics.
- Benchmark components: The benchmark uses an LLM-simulated user with personas, task instructions, multi-turn text interaction, and control words for termination and correctness evaluation.Personas vary by age, conversation style, and technical proficiency.
- Benchmark components: Agents use 58 tools across six domains, including get tools for information retrieval and set tools for environment modification, while following 19 domain-specific policies.Tools support sequential or parallel autonomous interaction with the automotive environment.
- Benchmark components: The environment separates mutable state variables from fixed context variables and connects them to realistic automotive databases.State variables can be modified by set tools, whereas context remains fixed within each task and varies across the dataset.
- Dataset and task types: The dataset contains 100 Base, 90 Hallucination, and 50 Disambiguation tasks.Base tasks target validated end states; Hallucination tasks remove required capabilities or information; Disambiguation tasks introduce controlled ambiguity.
- Dataset and task types: Disambiguation requires detecting ambiguity and choosing an informative resolution action, with failure caused by premature incorrect actions or unnecessary user queries.The system prompt directs agents to exhaust internal resolution before querying the user.
- Evaluation: Tasks are solved only when all relevant binary reward metrics equal 1, with Base, Hallucination, and Disambiguation using different metric subsets.Hallucination evaluation includes acknowledgment of missing capability, while Disambiguation includes the Base metrics and conversation outcome.
- Aggregated metrics: Pass^k measures success in all k trials, whereas Pass@k measures success in at least one trial.Repeated binary task outcomes are aggregated and averaged across tasks.
4 Experiments
CAR-bench evaluates state-of-the-art models across Base, Hallucination, and Disambiguation tasks using consistency-focused metrics. Results show substantial consistency gaps, increasing advantages for thinking models with task complexity, and distinct weaknesses across task types and models.
- Experimental setup: Pass^3 is reported as the primary metric, averaging performance across task types to prioritize consistency.The benchmark sets k=3 to discriminate reliability while avoiding saturation at higher k.
- Key findings: GPT-5 drops from 68% Pass@5 to 36% Pass^3 on Disambiguation, revealing a large gap between task-solving potential and reliable reproduction.The result shows that consistent ambiguity resolution remains difficult even with reflective decisions.
- Key findings: Thinking-enabled models outperform non-thinking models across task types, with the performance gap widening as action count and task complexity increase.Claude-Opus-4.5 achieves the highest Pass@1 on Base at 77% and Disambiguation at 56%, while GPT-5/5.2 reaches 74% on Hallucination.
- Key findings: Base tasks are easiest, while Disambiguation is hardest: 32% of Base tasks were solved by all models, but no model exceeded 50% Pass^3 on Disambiguation.Hallucination tasks particularly exposed non-thinking models’ difficulty acknowledging limitations.
- Key findings: Claude-Opus-4.5 and GPT-5 have comparable Base performance with ∆.01 but complementary weaknesses on Hallucination and Disambiguation.Claude trails GPT-5 by .14 Pass@1 on Hallucination, while GPT-5 trails Claude by .10 Pass@1 on Disambiguation.
- Caveat: User simulation errors affected benchmark outcomes, with sole-source errors reducing Pass^5 by 6% on Base, 9% on Hallucination, and 8% on Disambiguation.The inspected GPT-5 trials contained 12 Base, 22 Hallucination, and 7 Disambiguation user errors.
5 Discussion
CAR-bench reveals that reliable deployment requires more than occasional task completion: agents must consistently manage policies, uncertainty, tool use, and practical deployment constraints. Reasoning improves some errors, but premature actions and completion-oriented behavior remain major reliability barriers.
- Failure Analysis: Five failure categories span premature actions, policy violations, logical errors, execution errors, and fabrication.Fabrication includes concealing missing information or hallucinating nonexistent capabilities.
- Failure Analysis: ∼80% of persistent GPT-5 failures in Base tasks involve premature actions, alongside policy violations such as omitting required route alternatives or weather checks.The same model may follow a policy in some trials but not others, indicating unstable constraint adherence.
- Reasoning as Partial Mitigation: Reasoning reduces logical, execution, and severe policy errors on Base tasks and lowers active fabrication in Hallucination tasks, but does not resolve premature actions.Premature actions account for ∼90% of GPT-5 Disambiguation failures.
- Reasoning as Partial Mitigation: 68% Pass@3 versus 36% Pass^5 on Disambiguation quantifies GPT-5’s gap between identifying correct strategies and applying them consistently.GPT-5 often asks the user or executes a best-guess action before gathering complete environment information.
- Practical Implications: Latency and cost constrain deployment because delays compound across agent steps and input-token pricing dominates costs.Tool definitions contribute ∼10K tokens and the agent policy ∼3K tokens, while reasoning adds only 100–500 tokens per completion.
- Practical Implications: GPT-5 reaches high performance but takes 22 seconds per step, while Claude-Sonnet-4 costs over 10× more than Gemini-2.5-Flash without caching.Latency values are approximate because they depend on server location, hardware, and current model traffic.
6 Conclusion
CAR-bench combines realistic interaction, policies, tools, and dynamic environment data to evaluate reliable LLM-agent behavior. It provides baselines and error taxonomies for tracking and improving tool use, instruction following, uncertainty management, and honest communication of limitations.
- 6 Conclusion: CAR-bench combines multi-turn dialogues, domain policies, interconnected tools, and dynamic environment data to evaluate reliable LLM-agent capabilities.The benchmark targets accurate tool use, instruction following, uncertainty management, and honest communication of system limitations.
- 6 Conclusion: Its baselines and error taxonomies provide a grounded framework for systematically tracking progress in agent reliability.The framework is intended to support advances in architecture, prompting, and training.
7 Limitations
The benchmark’s conclusions are bounded by simulated-user noise, incomplete coverage of real-world contexts, an agent-centered safety design, limited dataset scale, and proprietary-model baselines.
- Simulated Users: The LLM-based simulated user can introduce hallucination, inconsistent intent revelation, noise, or bias into benchmark outcomes.Different user LLMs may change user errors and agent success, while evolving user familiarity may shift interaction styles and expectations.
- Scope: The environment cannot fully capture real-world diversity, leaving multi-user interactions, long-horizon planning, and multimodal cues out of scope.Examples include car-interior scene understanding, graphical user interfaces, and external conditions such as lightning.
- Safety Design: CAR-bench assigns policy compliance and safety checks to the agent, whereas production systems may use redundant rule-based safety layers.The optimal division between agent reasoning and external safeguards remains dependent on application-specific risk tolerance and regulatory requirements.
- Dataset Scale: The manually validated dataset provides reliable benchmark values but is currently too small for large-scale fine-tuning.Expanding it requires costly manual validation and deep domain expertise; synthetic augmentation is proposed as a scaling pathway.
- Model Coverage: Current baselines use state-of-the-art proprietary models, leaving open-weight and smaller-model evaluation for future work.The benchmark’s verifiable trajectories and structured evaluation framework may support supervised fine-tuning, preference optimization, and reinforcement learning.
8 Ethical Considerations
LLM-generated portions of the dataset may inherit and amplify harmful social biases, potentially affecting scenario design, naming conventions, and assumptions about user behavior.
- 8 Ethical Considerations: LLM-generated POI names, meeting topics, and user instructions may encode harmful biases related to gender, race, geography, socioeconomic status, and other protected attributes.These biases could appear in scenario design, naming conventions, or implicit assumptions about user behavior and preferences.
A Benchmark Components
CAR-bench combines domain policies, interconnected vehicle-control tools, and simulated user instructions to evaluate agents under ambiguity and missing information. Its protocols require policy-aware action, active information gathering, clarification when needed, and faithful interaction control.
- 58 interconnected tools support tasks across navigation, productivity, charging, and vehicle control.
- Vehicle-control policies encode interdependencies, confirmations, weather checks, and constraints such as blocking high beams when fog lights are on.
- Disambiguation Protocol: Disambiguation requires checking policies, user requests, preferences, heuristics, context, and car state before selecting among multiple valid options.
- Disambiguation Protocol: The priority order places strict policies first, followed by explicit requests, preferences, heuristics, contextual information, and user clarification.
- User simulation requires natural, incremental conversations, multiple intents when applicable, and no invented information beyond scenario instructions.
- End-interaction instructions distinguish completed tasks, continuing actions or confirmations, and scenarios that are out of scope.
B Experiments
The experiments evaluate multiple LLM agents under controlled inference settings and examine representative failures in uncertainty handling and policy adherence. Examples include premature action, policy violation, and incorrect route selection.
- Reasoning-enabled models used a medium 2048-token reasoning budget, while qwen3 had no reasoning-budget control.
- Temperature was fixed at 0.0 when configurable, with provider defaults retained for gpt-5 thinking and claude-sonnet-4.
- A premature-action error occurred when GPT-5 selected fan level 1 after finding no preference instead of requesting clarification.
- A policy-violation error occurred when GPT-5 automatically selected the fastest route instead of presenting multiple route options.
C.3 E3: Logical Errors
Logical errors arise when an agent has the necessary information but reaches an incorrect conclusion. One example shows GPT-5 narrowing airflow incorrectly during window-defrost activation.
- A logical error occurs when the agent holds all necessary information but draws an incorrect conclusion.
- GPT-5 restricted airflow to only WINDSHIELD even though the observed setting included windshield, head, and feet.
C.5 E5: Fabrication
Fabrication errors involve inventing or concealing information missing from the environment. The examples distinguish implicit fabrication from active fabrication when tools or state information are unavailable.
- Fabrication includes inventing or concealing information missing or inconsistent with the environment state.
- E5a: Implicit Fabrication: In the implicit-fabrication example, the agent closed the driver window while concealing that the rear-passenger window position was unknown.
- E5b: Active Fabrication: In the active-fabrication example, the agent used open_sunroof after open_sunshade was removed and falsely reported both actions as successful.