Source-linked AI summary
Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants
Vaishnav Negi, Lev Sorokin, Soroosh Tayebi Arasteh, Andrea Stocco
TL;DR
Multi-turn ICA reliability is difficult to evaluate because requests evolve across turns, valid responses may differ, and safety constraints matter. The paper proposes black-box closed-loop testing with guided simulation and a two-tier LLM judge; strategy guidance finds more failures, and the judge agrees substantially with human annotators.
Problem
ICA evaluation must capture evolving constraints and context, multiple valid answers, and safety-sensitive behavior that single-turn or reference-based measures do not adequately cover.
Source: §I · p. 1Method
A black-box closed-loop framework combines strategy-guided user simulation, adversarial failure-mode strategies, and an LLM judge for turn-level failures and conversation-level quality.
Source: §I · p. 1, A. Framework Components · p. 2Results
2.96× more unique failure types per conversation were found with strategy guidance; it yielded 2.21× more unique failing conversations than the baseline, while the automated judge achieved Cohen’s κ = 0.68 and F1 = 0.94.
Source: §I · p. 1, B. RQ1: Effectiveness · p. 5, C. RQ2: Failure Diversity · p. 5Takeaways & Limitations
In the industrial ICA evaluation, strategy guidance broadened failure discovery compared with unguided simulation, and the automated judge showed substantial agreement with human annotators.
Source: §I · p. 1, §VII · p. 6Voice and multimodal evaluation remain future work.
Source: §VII · p. 6
Abstract
from arXiv · showhide
In-car conversational assistants (ICAs) are increasingly integrated into vehicles to support route planning, vehicle control, and information access. Ensuring their reliability is challenging due to multi-turn interactions, the absence of explicit ground truth, and strict safety constraints. Existing evaluation techniques fall short, as they target single-turn settings and fail to capture constraint handling, context retention, and safety-critical behavior across turns. We propose an automated framework for testing the multi-turn conversational capabilities of ICAs. The system is treated as a black box and evaluated via closed-loop simulation with a strategy-guided user simulator, an adversarial strategy manager, and a two-tier LLM judge assessing turn-level failures and conversation-level quality. We evaluate the approach on an industrial ICA with six LLM backends and twelve human annotators. The automated judge shows substantial agreement with humans, and strategy guidance uncovers 2.96 times more unique failure types per conversation and more than doubles the number of unique failing conversations compared to unguided simulation.
I. INTRODUCTION
Multi-turn ICA evaluation must account for evolving constraints and context, multiple valid answers, and safety-sensitive behavior. The proposed black-box framework uses guided simulation and an LLM evaluator; evaluation reports substantial human agreement and increased failure discovery with strategy guidance.
I. INTRODUCTION
- Reliability evaluation is difficult because errors may emerge only across turns, multiple answers can be valid, and constraint or safety failures may compromise driving safety.Quality must be judged by task completion, constraint satisfaction, and safety rather than benchmark similarity.Source: §I · p. 1
- Existing metrics, human review, multi-turn benchmarks, and automated tests each leave gaps in scalable, automotive-specific evaluation under safety constraints.Reference-based metrics require references, human evaluation is expensive to scale, and prior approaches do not target automotive safety and black-box deployment constraints.Source: §I · p. 1
- The framework treats the assistant as a black box and combines strategy-guided user simulation, adversarial failure-mode probing, and a two-tier LLM evaluator.It assesses turn-level failures and conversation-level quality across four dimensions, including constraint adherence and safety compliance.Source: §I · p. 1
- 2.96× more unique failure types per conversation were uncovered with strategy-guided probing, while detected conversation failures rose from 19.2% to 82.7%.The framework was evaluated on an industrial ICA with six LLM backends.Source: §I · p. 1
A. Multi-Turn Testing Problem
The paper formalizes multi-turn testing as generating user utterances that expose failures in a black-box assistant. Its motivating example shows how incremental constraints and requests can reveal failures that a single-turn test misses.
A. Multi-Turn Testing Problem
- A multi-turn test combines the assistant, a driver-goal seed, contextual information, adversarial strategies, and an evaluator that assigns failure labels and quality scores.The seed may specify a destination and constraints; context contains location, date, and time.Source: A. Multi-Turn Testing Problem · p. 1
- The testing objective is to generate a sequence of user utterances that maximizes detected failures in the resulting conversation.The conversation consists of user–assistant exchange pairs.Source: A. Multi-Turn Testing Problem · p. 1
B. Motivating Example
- Incremental route constraints expose constraint violations, context forgetting, and request omission that may remain hidden in a single-turn test.In the example, after eight turns the assistant completes only one of nine targets, including failing to integrate stops and constraints into a route.Source: B. Motivating Example · p. 2
III. APPROACH
The approach tests ICAs through text-based black-box interaction to isolate dialogue quality from speech recognition. Its components generate user turns, target failure modes, and evaluate both individual exchanges and complete conversations.
III. APPROACH
- The framework accesses the ICA as a black box through text input and output, isolating dialogue quality from speech-recognition errors.The paper notes that production ICAs use voice interaction, while this evaluation focuses on text.Source: §III · p. 2
A. Framework Components
- The user simulator derives targets from a seed and context, while personas shape utterance style and target statuses track whether tasks are dropped, incomplete, or completed.The context vector contains location, date, and time.Source: A. Framework Components · p. 2
- The strategy manager embeds failure-mode-specific adversarial strategies in user utterances and removes a strategy after its target failure occurs.The LLM judge labels turn-level failures and scores full conversations across four quality dimensions.Source: A. Framework Components · p. 2
B. Conversation Loop
The conversation loop initializes targets and strategies, repeatedly queries and evaluates the assistant, and then scores the completed dialogue. It can stop early when targets are resolved after at least half the turn budget.
B. Conversation Loop
- Initialization extracts targets from the seed, generates the first user utterance from the seed and context, and activates the strategy set.The algorithm takes a maximum turn count and strategies as inputs.Source: B. Conversation Loop · pp. 2–3
- At each turn, the system sends a user utterance to the assistant, evaluates the exchange, updates target statuses, removes triggered strategies, and appends the exchange to history.It generates another utterance unless all targets are resolved and at least half the turn budget has been spent.Source: B. Conversation Loop · pp. 2–3
- Final evaluation assigns conversation-level dimension scores and computes a success score from those scores and final target statuses.The algorithm returns the conversation history, dimension scores, and success score.Source: B. Conversation Loop · pp. 2–3
C. Adversarial Strategies
Adversarial strategies target specific failure modes and are removed once their associated failure is detected, redirecting later turns toward unexplored modes. Strategy-use limits also constrain consecutive repetition.
C. Adversarial Strategies
- A strategy is removed from the active pool once its mapped failure mode is detected, redirecting subsequent turns to unexplored failure modes.The mapping function links each strategy to a target failure mode.Source: C. Adversarial Strategies · p. 3
- Consecutive-use limits vary by strategy: safety probes are limited to one consecutive use, while constraint-stacking probes may be used twice.The limits aim to avoid unrealistic repetition while allowing accumulation where useful.Source: C. Adversarial Strategies · p. 3
D. Evaluation Rubric
The rubric distinguishes target-compliance failures from response-level failures, then scores conversation quality across four dimensions. A conversation passes when its success score meets the threshold.
D. Evaluation Rubric
- Turn-level failures divide into per-target compliance failures and response-level failures independent of a particular target.Compliance categories are fallback response, irrelevant response, request omitted, and constraint missed; response categories include forgotten context, unresolved ambiguity, poor planning, and safety violation.Source: D. Evaluation Rubric · p. 3
- The rubric filters trivial issues: ambiguity qualifies when requests contain multiple named entities, conflicting interpretations, or safety-relevant uncertainty.A context-forgotten failure requires information the user explicitly established and the assistant later ignores or contradicts.Source: D. Evaluation Rubric · p. 3
- Conversation-level scores cover Instruction & Constraint Adherence, Context & Ambiguity Handling, Plan Coherence, and Safety Compliance.Each dimension receives a three-point score; the final success score combines normalized dimension scores with the target completion ratio, and passing requires C ≥ θ.Source: D. Evaluation Rubric · p. 3
IV. EVALUATION
The evaluation asks whether LLM judges match human assessments, whether strategy guidance improves failure discovery, and how diverse discovered failures are. Experiments use a black-box industrial assistant with multiple backends, scenarios, and a controlled unguided baseline.
A. Research Questions
- The study evaluates judge accuracy against humans, strategy-guided discovery against an unguided baseline, and the diversity of failures uncovered.These are the paper’s RQ0, RQ1, and RQ2, respectively.Source: A. Research Questions · p. 3
B. Experimental Setup
- The system under test is an industrial ICA prototype supporting route planning, POI recommendations, vehicle settings, and general information queries through a RAG architecture.It is treated as a black box and accessed only through text-based sessions.Source: B. Experimental Setup · pp. 3–4
- Six interchangeable LLM backends are tested, with each configuration run independently for six four-hour sessions.The backends comprise four cloud-hosted models and two locally hosted models.Source: B. Experimental Setup · p. 4
- Scenarios combine seven seed-phrase pools with two personas, 16 task combinations, two turn budgets, and four spatio-temporal context vectors.Each run draws from a different seed pool; the four-hour limit yields 55–134 framework conversations and 61–214 baseline conversations, depending on model latency.Source: B. Experimental Setup · p. 4
- The baseline holds the user simulator, judge, seeds, and rubric constant while removing adversarial strategies and evaluator feedback during the conversation loop.It uses one neutral persona and a fixed eight-turn budget to isolate strategy-guided probing.Source: B. Experimental Setup · p. 4
- Judge agreement is measured at dimension and conversation levels, while testing effectiveness uses pass/fail rates, success scores, and unique failure types.The evaluation also compares failure diversity through deduplicated failing conversations and cluster coverage; 37 framework-generated conversations were independently rated by 12 humans.Source: B. Experimental Setup · p. 4
A. RQ0: Judge Accuracy
GPT-5 and GPT-5-MINI meet or exceed human–human agreement baselines on ordinal ratings, and both reach or exceed the human baseline on conversation-level F1. GPT-5-MINI is selected for subsequent evaluation because it matches GPT-5’s ordinal agreement at lower cost.
A. RQ0: Judge Accuracy
- GPT-5 and GPT-5-MINI achieve κ = 0.68 and ρ = 0.70, matching or exceeding the human baselines κHH = 0.65 and ρHH = 0.70.On conversation-level pass/fail judgments, GPT-5 reaches F1 = 0.97 and GPT-5-MINI reaches F1 = 0.94, the human baseline.Source: A. RQ0: Judge Accuracy · p. 4, Table III · p. 5
- GPT-5-MINI leads or ties on three of four dimensions, including Safety Compliance (κ = 0.70) and Plan Coherence (κ = 0.73).GPT-5 leads on Instruction Adherence (κ = 0.66); both have κ coefficient of variation ≤0.08 across dimensions.Source: A. RQ0: Judge Accuracy · p. 5
- GPT-5-MINI was selected as automated judge because it matches GPT-5 on ordinal agreement (κ = 0.68, ρ = 0.70), meets the human F1 baseline (0.94), and achieves 93.8% within-one agreement, matching the human–human adjacent baseline.Source: A. RQ0: Judge Accuracy · p. 5
B. RQ1: Effectiveness
Strategy-guided simulation increased detected failures and failure diversity over unguided simulation, while both approaches achieved near-complete cluster coverage.
B. RQ1: Effectiveness
- The overall failure rate rose from 19.2% for the baseline to 82.7% for strategy-guided simulation.Across all six models, the reported mean success scores were C = 0.49 ± 0.23 and C = 0.87 ± 0.16, respectively.Source: B. RQ1: Effectiveness · p. 5
- Cumulative failures diverged early and widened steadily over the four-hour budget for every model, indicating sustained rather than front-loaded discovery.Source: B. RQ1: Effectiveness · p. 5
- Strategy-guided simulation identified 2.96× more unique failure types per conversation than the unguided baseline across six SUT models.Source: B. RQ1: Effectiveness · p. 5
C. RQ2: Failure Diversity
- Cluster coverage was 96.9% for strategy-guided simulation and 99.0% for the baseline; low optimal cluster counts and silhouette scores suggest homogeneous failures.The reported ranges were k = 2.2–4.4 and silhouette scores of 0.08–0.25.Source: C. RQ2: Failure Diversity · p. 5
- Strategy-guided simulation yielded 2,394 unique failing conversations versus 1,084 for the baseline (2.21×), with zero duplicates versus 0.5–8.3% baseline redundancy.Source: C. RQ2: Failure Diversity · pp. 5–6, Table IV · p. 6
VI. QUALITATIVE EVALUATION
The qualitative evaluation profiles which failures the framework uncovers and how they are distributed across conversations. The conclusion summarizes the framework’s evaluation and identifies future extensions.
VI. QUALITATIVE EVALUATION
- Planning poor appeared in 62.3% of strategy-guided conversations versus 0.6% of baseline conversations (24.1×), while safety violation increased 3.0×.Source: §VI · p. 6
- At least three distinct failure types appeared in 81.1% of framework-generated conversations versus 20.5% of baseline conversations; baseline failures mainly occurred early, while framework failures were more evenly distributed.Baseline failures mainly occurred in early turns (75.2%).Source: §VI · p. 6
VII. CONCLUSIONS AND FUTURE WORK
- The industrial-prototype evaluation used six LLM backends and 8,550 conversations; future work will extend testing to voice and multimodal settings and consider external fact-checking for navigation correctness.Source: §VII · p. 6