PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation-as-Search: Adaptive Discovery of Grounding Failures in Meeting Assistants

Sami Khairy, Yasaman Hosseinkashi, Vishak Gopal, Ross Cutler

arXiv:2608.20392v1cs.CLcs.AI

TL;DR

Meeting assistants need evaluation that exposes grounding failures tied to discourse structures and reasoning demands rather than relying only on static question sets. The paper introduces Evaluation-as-Search, which adaptively probes natural questions using evaluator feedback and constructs the MEETINGPROBE benchmark. EaS finds 2.5× more failures than random probing and reveals recurring failure categories, model capability differences, and universal failures.

  • Problem

    Static meeting-assistant benchmarks use fixed question sets and lack adaptive, multi-dimensional evaluation of grounding failures across discourse structures and reasoning demands.

  • Method

    Evaluation-as-Search adaptively searches natural meeting questions using evaluator feedback, UCB-guided coverage, and blind MARC scoring of faithfulness and completeness.

  • Results

    2.5× more failures are found than with random probing, while MEETINGPROBE contains 3,009 annotated pairs spanning 20 transcripts, three genres, and three assistants.

  • Takeaways & Limitations

    MEETINGPROBE provides a public resource for reproducible evaluation, with eight recurring grounding-error categories and a capability gradient across models.

  • Takeaways & Limitations

    MEETINGPROBE covers English-language QMSum transcripts and OpenAI GPT targets, leaving additional languages, meeting styles, and model families for future work.

Abstract

from arXiv · show

LLM-powered meeting assistants are deployed at scale, yet systematic evaluation of their grounding fidelity remains limited to static benchmarks that miss failure modes tied to specific discourse structures or reasoning demands. We propose Evaluation-as-Search (EaS), a feedback-driven methodology that frames quality evaluation as an adaptive search over the space of natural questions a meeting participant might ask. Rather than sampling uniformly, EaS learns from evaluator feedback across iterations to concentrate probing effort on cognitive demands where failures are most likely, guided by a UCB-scored coverage map and blind multi-dimensional quality evaluation. Using EaS, we construct MeetingProbe, a benchmark of over $3{,}000$ annotated question--answer pairs spanning 20 transcripts from three meeting genres and three LLM assistants. In ablations, adaptive search surfaces $2.5\times$ more failures than random probing ($7.1\%$ vs. $2.9\%$ finding rate), with the strategic planner contributing the largest individual effect. Across three models, we observe a clear capability gradient and identify eight recurring failure categories dominated by discourse-pragmatic challenges rather than factual recall errors. We further validate MeetingProbe across multiple model families and providers, finding a clean capability gradient and a curated subset of universal failures that no model handles. MeetingProbe is released publicly to support reproducible evaluation of meeting assistant grounding fidelity.

1 Introduction

Meeting-assistant evaluation must uncover grounding failures tied to discourse structure and reasoning demands, which static question sets can miss. The paper addresses this gap with adaptive search, a meeting-specific evaluator, and the MEETINGPROBE benchmark.

  • Motivation: Static benchmarks measure average-case performance but can miss failures involving multi-speaker disagreements or action items scattered across long transcripts.Their fixed item pools do not concentrate sampling on specific reasoning demands or discourse structures.
  • Research gap: Existing adaptive evaluation methods expose behaviors missed by static evaluation, but they generally target safety alignment or broad NLP capability rather than grounded meeting discourse.Meeting QA and faithfulness benchmarks likewise use fixed question sets without adapting their probing strategy to observed failures.
  • Contributions: EaS frames quality evaluation as feedback-driven adaptive search over natural questions, concentrating evaluation budget where failures are most likely.The method learns from evaluator feedback across iterations instead of sampling uniformly.
  • Contributions: MEETINGPROBE contains 3,009 annotated question–answer pairs, including 1,047 grounding failures and 1,962 hard negatives, from 20 transcripts, three genres, and three assistants.Hard negatives target demanding cognitive operations but receive correct responses.
  • Contributions: MARC provides meeting-specific multi-dimensional evaluation, while QMSumCal50 calibrates it using golden answers, controlled perturbations, and human-validated quality scores.Both resources are released publicly.
  • Results: EaS achieves a 2.5× higher finding rate than random probing, and the study reports eight grounding-error categories plus a clear capability gradient across models.Overconfident paraphrase, unjustified quantification, and speaker misattribution account for over 60% of failures.

2 Related Work

Prior work supplies meeting benchmarks, faithfulness metrics, adaptive testing, and feedback-based optimization, but EaS combines adaptive natural-question discovery with multi-dimensional meeting-grounding evaluation.

  • Meeting benchmarks: QMSum, MeetingBank, and ELITR-Bench provide meeting transcripts, summaries, or human-rated QA evaluation, while MEETINGPROBE adds adaptively generated probes and MARC.The paper uses QMSum transcripts as MEETINGPROBE’s source collection.
  • Faithfulness evaluation: HELM, FActScore, RAGAS, Chain-of-Verification, SummEdits, and BAMBOO establish multi-metric, verification-based, or long-context faithfulness evaluation.The cited benchmarks primarily evaluate fixed inputs, unlike EaS’s adaptive probing.
  • Dynamic evaluation: Dynamic evaluation methods use model-in-the-loop creation, adversarial procedures, or item-response theory to reduce static-test limitations and evaluation cost.One cited item-response approach reduces required evaluation items by roughly 90% while preserving accuracy.
  • Feedback-based optimization: Feedback-based optimization methods use language feedback to improve generated artifacts, whereas EaS uses feedback to discover questions exposing grounding failures.EaS produces a curated benchmark rather than an optimized artifact.
  • Automated search-based evaluation: Automated probing methods address diversity–effectiveness tradeoffs, but the paper distinguishes EaS through natural questions, multi-dimensional quality, and a curated benchmark output.Its coverage mechanism uses an open-ended cognitive-demand scoreboard with UCB exploration scores rather than a fixed behavioral grid.

3 The Evaluation-as-Search Framework

EaS adaptively searches natural meeting questions across topical, question-type, cognitive-demand, and transcript-position dimensions, then evaluates responses with a blind MARC evaluator. Its architecture separates strategic search from question generation and judgment to reduce evaluator bias.

  • Framework: EaS navigates natural questions by learning from evaluator feedback and concentrating probes on cognitive demands and transcript regions where failures are most likely.The resulting collection contains annotated question–answer pairs with MARC faithfulness and completeness scores.
  • Agent architecture: MARC receives only the transcript, question, and answer, scoring faithfulness and completeness from 1 to 5 without search context or generation rationale.This blindness is intended to mitigate confirmation bias from strategy-induced expectations.
  • Problem formulation: The search space combines transcript topics, question types, cognitive demands, and transcript positions, with all four dimensions left open-ended.Examples include attribution recall, temporal ordering, exhaustive enumeration, and boundary distinction.
  • Problem formulation: A probe is a finding when the lower of its faithfulness and completeness scores is at most 3 on MARC’s 1–5 scale.The fitness function measures grounding in the transcript and coverage of relevant information.
  • Agent architecture: The architecture separates the Planner, Generator, Self-Verify, target Assistant, and MARC evaluator through an orchestrator and information asymmetries.The Planner reasons from search history and coverage statistics without seeing the transcript, while the Generator grounds questions in transcript content.
  • Search operators and coverage: The Planner uses EXPLORATION, REFINEMENT, and MUTATION operators to balance underexplored demands, systematic retesting, and pivots from unproductive regions.A UCB scoreboard combines effective reward rates, probe counts, and exploration, while a transcript-position histogram supports spatial diversity.
  • Search operators and coverage: Search runs for 20 iterations, collecting findings below the threshold and hard negatives whose two scores are at least one point above it.The construction uses all probes from 20 transcripts and three target models.

4 MARC: Calibrated Quality Evaluation

MARC is a blind, reference-guided evaluator that supplies EaS with per-dimension grounding scores while keeping evaluation adaptable and efficient. It measures faithfulness and completeness under an information barrier and is validated on disjoint calibration and held-out meeting datasets.

  • Evaluator design: MARC produces per-dimension scores and natural-language justifications from a rubric, transcript, question, and candidate answer in one inference call.Its single-call design keeps evaluation cost proportional to probe count and supports rubric-based domain adaptation.
  • Scored dimensions: MARC scores faithfulness as transcript grounding without fabrication, distortion, or misattribution, and completeness as coverage without material omission.Both dimensions use a 1–5 Likert scale.
  • Blind evaluation: MARC receives only the transcript, question, and answer, preventing Planner strategy, Generator rationale, or search context from contaminating the fitness signal.This strict information barrier supports blind evaluation inside the search loop.
  • Validation: MARC is validated with QMSumCal50 for rubric refinement and a disjoint QMSumHeldOut set spanning six meetings and three genres.The held-out set contains 90 variants from meetings not used for calibration.

5 Experimental Setup

The experiments apply EaS to diverse, long meeting transcripts and evaluate findings with predefined rate, severity, efficiency, and transfer metrics. Runs cover three meeting genres and three target assistants under a bounded probing budget.

  • Protocol: Each run generates up to 200 probes by producing K=10 candidate questions per iteration for at most Imax=20 iterations.EaS agents and the MARC evaluator are powered by GPT-5.2 via the Azure OpenAI Responses API.
  • Data: Experiments use 20 QMSum transcripts spanning product design, research, and parliamentary meetings, each exceeding 100 dialogue turns.The collection includes six AMI, six ICSI, and eight parliamentary transcripts.
  • Metrics: The primary finding rate counts probes with faithfulness or completeness ≤3, alongside severity, tokens per finding, and compute time per run.Results use means, standard deviations, 95% confidence intervals, and paired t-tests across transcripts.

6 Results

EaS finds more grounding failures than random probing, with strategic demand selection and iterative feedback driving the largest gains. Across genres and models, findings reveal capability gradients, recurring discourse-pragmatic categories, and a transferable universal-failure subset.

  • 6.1 Search Configuration Analysis: 7.1% ± 1.9% is the full EaS finding rate, compared with 2.9% for random demand selection, a 2.5× difference.Removing the Planner costs −4.2 pp, the largest individual ablation effect.
  • 6.1 Search Configuration Analysis: 2.14M tokens per finding makes Full 1.4–2.5× cheaper than alternatives and places it on the Pareto frontier of rate, severity, and cost.The static bank is six times faster but produces milder findings, while PAIR-style serialization causes a 4.4× wall-clock slowdown without improving rate or severity.
  • 6.1 Search Configuration Analysis: 3.5× is the Planner’s cumulative-finding advantage by iteration 15, despite Full and −Planner starting with the same 0.3–0.7 findings at iteration 1.The widening gap is consistent with feedback-conditioned demand selection.
  • 6.2 Cross-Genre Generalization: 5.2% is GPT-5.2-chat’s mean finding rate across all 20 transcripts, with product-design, research, and parliamentary rates of 7.0%, 5.8%, and 3.4%.All 20 transcripts produced at least one finding; informal multi-party discussions with rapid topic shifts were associated with higher rates than structured parliamentary turn-taking.
  • 6.3 Cross-Model Analysis: 15.3% versus 5.2% marks the finding-rate increase from GPT-5.2-chat to GPT-4.1-mini, with severe-finding fractions of 27% and 39%, respectively.All pairwise rate differences are significant at p<0.0001, and parliamentary failures have the highest severity fraction across models.
  • 6.4 Transfer Analysis: 11.8% of 1,047 findings are universal failures across five models and four providers, forming a curated subset of transcript-level grounding challenges.Frontier-model findings transfer downward within the OpenAI family 51–54% of the time, whereas weaker-model findings transfer upward only 17–20%.
  • 6.5 Multi-Family Benchmark Validation: 44 pp is the finding-rate spread across five models from four providers, while architectural variety preserves the capability gradient.The universal-failure subset contains 124 of the 1,047 findings.
  • 6.6 Failure Taxonomy: Eight recurring failure categories are tied to discourse-pragmatic demands rather than random hallucinations, with overconfident paraphrase among the top categories at 27.3%.Speaker misattribution rises as model capability declines, while fabricated action items distinguish the frontier model’s failures.

7 Discussion

EaS improves failure discovery efficiency, while MeetingProbe supports practical re-evaluation through universal failures. The benchmark remains bounded by its language, transcript, model, and threat-model scope.

  • Search efficiency: 2.5× more failures per probe than random probing demonstrates the efficiency benefit of feedback-driven search.The Planner alone contributes a 4.2 pp improvement, and the full configuration is cheapest per finding among rate-comparable alternatives.
  • Practical use: 124 universal failures provide a model-independent subset for evaluating new assistants.Practitioners re-score released transcript–question pairs with MARC and compare results against Table 3.
  • Scope and limitations: The benchmark covers English-language QMSum transcripts and OpenAI GPT targets, with broader languages, meeting styles, and model families left for future work.The search agents and frontier target assistant are from the GPT-5.2 family, so MARC was calibrated against Claude Sonnet 4 judges to mitigate same-family evaluator bias.
  • Scope and limitations: The threat model is limited to insiders because targeted question construction requires access to ground-truth transcripts.Generated questions are natural and non-adversarial.

8 Conclusion

The paper introduces EaS for adaptive discovery of grounding failures and uses it to build MeetingProbe across diverse meeting data and models. EaS finds more failures than random probing while revealing capability differences and recurring failure categories.

  • Conclusion: EaS is a feedback-driven adaptive methodology for discovering grounding failures in LLM meeting assistants.It concentrates probing on cognitive demands where failures are most likely.
  • Conclusion: MeetingProbe contains 3,009 annotated question–answer pairs spanning 20 transcripts, three genres, and three models.The benchmark is constructed using EaS.
  • Conclusion: 2.5× more failures than random probing is surfaced by EaS through learning from evaluator feedback across iterations.The analysis also reports a capability gradient, asymmetric cross-model transfer, and eight recurring failure categories.

A Complete Evaluation Loop Pseudocode

The evaluation loop iteratively plans, generates, filters, answers, and blindly scores questions, updating history and confirmed findings. Its agents use coverage statistics, search strategies, novelty filtering, and transcript-grounded quality checks.

  • Initialization and planning: The algorithm iterates up to Imax=20 with K=10 candidates per iteration and tracks history, findings, reflections, deduplication, and embeddings.The loop initializes H, F, reflection, Hdedup, and E before repeated search iterations.
  • Initialization and planning: UCB-scored cognitive-demand coverage, strategy statistics, position histograms, and novelty trends guide the Planner each iteration.The Planner receives condensed history and outputs exploration, refinement, or mutation strategies.
  • Question generation: Novelty filtering retains K′≤K candidates before the target assistant answers each question using the transcript.Surviving question embeddings are added to the novelty store, while all generated questions are recorded for deduplication.
  • Evaluation and findings: MARC blindly scores faithfulness and completeness from the transcript, question, and answer on calibrated 1–5 scales.It receives no search context, Planner strategy, or Generator rationale.
  • Evaluation and findings: A probe becomes a finding when its minimum MARC dimension score is at most θ=3, after which the full history and confirmed findings are returned.Each evaluated tuple is added to history before thresholding.
  • Question generation: The Generator selects transcript passages and formulates natural questions from the Planner’s strategies and prior-question history.Each output includes question metadata such as topic, type, transcript region, estimated position, and cognitive demand.
  • Question validation: Self-verification checks answerability, factuality, meeting-level content, conversational naturalness, and transcript-grounded answerability.Failed candidates receive at most one topic-pivot replacement targeting another transcript region.

E Per-Transcript Detailed Results

The detailed results organize findings by transcript, genre, model transfer, and failure category. They document the dataset’s diverse meeting sources, shared search budget, asymmetric transfer, and taxonomy structure.

  • Dataset characteristics: 20 transcripts are sourced from QMSum and include AMI product design, ICSI research, and parliamentary meetings.The dataset-characteristics table covers the three source corpora and their meeting collections.
  • Search configurations: All six search configurations use the same 200-probe budget and MARC evaluator, differing only in probe-selection strategy.Configuration results are averaged across three independent trials.
  • Cross-model transfer: 51–54% downward transfer within the OpenAI family exceeds 17–20% upward transfer.The same finding questions are posed to other models and re-evaluated with MARC; weaker models additionally fail on easier questions.
  • Failure categories: Failure-category results report the percentage of each model’s findings assigned to each grounding-failure category.Table 10 uses 1,049 pre-deduplication findings, while the released benchmark retains 1,047.

J Perturbation Sensitivity Details

The paper validates MARC by testing its sensitivity to controlled, transcript-grounded perturbations and calibrating it against audited quality ratings. MARC detects most factual perturbations and distinguishes answer-quality tiers, while calibration and held-out procedures address evaluator reliability.

  • Perturbation Sensitivity: 239 controlled perturbations introduced one verified factual error into otherwise correct answers across 20 transcripts and three meeting genres.Errors included speaker swaps, number changes, claim reversals, fabrications, and temporal errors.
  • Perturbation Sensitivity: 91.6% of perturbations were detected overall, with temporal errors detected at 100% and speaker swaps at 97.9%.Claim reversals reached 94.8%, fabrications 83.3%, and number changes 78.6%.
  • Perturbation Sensitivity: 14 of 20 undetected perturbations were number changes, where a single altered digit remained embedded in an otherwise coherent answer.All undetected cases received a perturbed score of exactly 4, just above the finding threshold.
  • MARC Calibration: Calibration used 150 variants with independent judging and human review, followed by a cross-family blind audit that corrected 78 ratings.The MARC rubric was frozen after calibration and before held-out scoring.
  • MARC Calibration: MARC evaluates faithfulness and completeness because both dimensions require comparison with the source transcript and capture source-dependent grounding errors.Relevance and quality are described as largely source-independent dimensions.

K.3 Full Validation Metrics

Full validation shows that MARC reliably separates answer-quality tiers and maintains strong verdict-level performance across calibration and held-out data. Its complementary dimensions support different distinctions, while held-out recall declines for some subtle grounding failures.

  • Score-Level Agreement: Spearman correlations reached ρ ≥ 0.66 across all conditions, including held-out completeness at ρ = 0.73.Held-out faithfulness reached ρ = 0.66, reflecting difficulty ranking subtle unseen-meeting errors.
  • Within-Tier Discrimination: Golden–perturbed concordance exceeded 96% with d > 1.7, while weak–perturbed concordance exceeded 92% with d > 1.3.These results indicate discrimination between adjacent as well as extreme quality tiers.
  • Within-Tier Discrimination: Golden–weak faithfulness concordance was 79–81% with d ≈0, while completeness reached 90–95% with d = 1.0–1.5.Faithfulness separates perturbed from golden and weak answers; completeness separates golden from weak and perturbed answers.
  • Verdict-Level Validation: MARC achieved F1 = 0.91 on Cal50 and F1 = 0.83 on the held-out set at the verdict level.Cal50 precision and recall were 0.90 and 0.92; held-out precision and recall were 0.85 and 0.81.
  • Verdict-Level Validation: The held-out performance drop was concentrated in recall, while precision remained high for trustworthy flagged findings.The paper characterizes this precision–recall profile as suitable for the EaS search loop.
  • Benchmark Release: MEETINGPROBE provides transcript, model, question, answer, finding, MARC, cognitive-demand, failure-category, genre, and corpus fields in a public JSON release.The benchmark includes 1,047 post-deduplication findings, and 124 universal failures form a curated hardest subset.
Loading 2608.20392v1…