PaperlayerView exact revision on arXiv

Source-linked AI summary

Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting

Muneeb Khan, Frederic Kirstein, Terry Ruas, Bela Gipp

arXiv:2609.03923v1cs.AIcs.CL

TL;DR

Online meeting delegates struggle to recognize when an absent participant should speak because they lack structured representations of stances, coverage, and floor. CAPA maintains that state through a perceive–act–recalibrate architecture and evaluates interventions around participant-owned idea units. Across 137 AMI meetings, it reduces silence from 51.4% to 2.5%, doubles credited recovery from 26.1 to 52.2, and keeps hallucination at 0.6%.

  • Problem

    Prompt-based delegates miss intervention opportunities because they do not explicitly track participant stances, unresolved issues, prior coverage, or floor.

  • Method

    CAPA uses a perceive–act–recalibrate architecture with explicit meeting state, proposition-first action selection, generation, prediction, and judge-driven recalibration.

  • Results

    51.4% to 2.5% silence, 26.1 →52.2 credited recovery, and 0.6% hallucination are reported across 137 AMI meetings.

  • Takeaways & Limitations

    Explicit structured state closes the opportunity-recognition gap, whereas raw-context scaling alone does not resolve long-range recognition.

  • Takeaways & Limitations

    The primary evaluation focuses on AMI, while other corpora require adaptation or lack continuity or spontaneous floor-taking conditions.

Abstract

from arXiv · show

In online meeting delegation, LLM agents fail to recognize when to speak. With no structured way to track stances, coverage, and floor, they miss the moments where they should contribute. Prompt-only delegates stay silent on 51.4% of the absent participant's talking opportunities on the AMI corpus. We present CAPA (Collaborative Agent Predictive Architecture), an architecture for online meeting delegation. A Perceiver updates the meeting state from each observed turn. A Predictor forecasts how the conversation will continue. A Controller decides whether to speak and which proposition to surface. A Generator phrases the chosen contribution in the participant's style. Two judges score the forecast and the action against the next observed turn. A Recalibrator updates the meeting state from those verdicts for future decisions. To evaluate online delegation, we introduce an episode-level protocol that scores whether, when, and what a delegate contributes around the participant's actual idea units. The protocol's schema-constrained LLM judges align with human annotations at Cohen's kappa = 0.71. On 137 AMI meetings, CAPA reduces the silence rate from 51.4% to 2.5%, doubles credited recovery (26.1 --> 52.2), and keeps hallucination at 0.6%. The failure mode shifts from omission to selection, with each residual near-miss attributable to a specific module of the architecture. Mechanism ablations identify the meeting state as the lever that closes the recognition gap, where raw-context scaling alone does not.

1 Introduction

Online meeting delegates must decide both whether to intervene and what to contribute for an absent stakeholder, but prompt-based systems miss many opportunities because they lack structured state tracking. CAPA addresses this with continuous meeting-state maintenance, causal episode evaluation, and recalibration from forecast and action judgments.

  • Motivation: 51.4% of valid absent-participant talk opportunities remain silent under prompt-based delegation on AMI.Recurring failures include reliance on explicit name-calls and confusion between topical relevance and proposition-level coverage.
  • Approach: CAPA maintains a structured meeting state covering topics, decisions, open questions, stances, prior coverage, and floor.Its perceive–act–recalibrate loop updates state from each turn and feeds judge verdicts back into future decisions.
  • Evaluation: The evaluation protocol scores whether, when, and what delegates contribute relative to participant-owned idea units.Programmatic checks assess timing and intervention, while schema-constrained judges assess contribution content against human annotations.
  • Results: 2.5% silence, 52.2 credited recovery, and 0.6% hallucination are reported for CAPA across 137 AMI meetings.The comparison is from 51.4% silence and 26.1 credited recovery under the baseline.
  • Implications: Explicit state tracking shifts delegation failures from omission toward bounded, module-attributable selection errors.The contribution claim contrasts continuous state maintenance with raw-context scaling and prompt-only delegation.

2 Related Work

Prior meeting systems mainly support passive observation or post-hoc assistance, while live stakeholder representation requires continuous intervention decisions. CAPA is framed as sequential decision-making under partial observability and extends inference-time correction to silent failures.

  • Meeting agents: Meeting applications primarily summarize, facilitate, or answer questions after the fact rather than represent stakeholders continuously during live interaction.The related-work discussion identifies live proxy participation as a nascent computational challenge.
  • Sequential decision-making: Delegation couples each SPEAK or SILENT choice to the conversation and remaining opportunities while stances, coverage, and floor remain latent.The objective evaluates whether, when, and what the delegate contributes.
  • Inference-time correction: Post-action correction methods critique or rewrite generated utterances, but CAPA’s recalibration also addresses silent failures with no utterance to rewrite.Its correction channel updates the state conditioning subsequent decisions.

3 Methodology

CAPA implements delegation as a modular perceive–act–recalibrate loop over an explicit, inspectable meeting state. Shared memory, state perception, proposition-first action selection, generation, and judge-driven recalibration separate strategic decisions from surface realization.

  • Task formulation: CAPA treats delegation as sequential decision-making under partial observability for an absent participant acting through SILENT or SPEAK.On SPEAK, the delegate commits to a semantic proposition before realizing an utterance.
  • Design requirements: Six design constraints target silent abstention, redundancy, off-topic hallucination, and mistiming through task and architectural requirements.The requirements are derived from documented prompt-only failure modes.
  • Architecture: CAPA decomposes delegation into Shared Memory, a Perceiver, a Controller–Generator, and a Recalibrator.The modules maintain participant context, estimate state, select and realize actions, and integrate judge feedback.
  • Meeting state: The explicit state is an approximate belief over interaction variables that provides a persistent substrate for downstream action and correction.Centralized memory makes state fields available to action and recalibration modules.
  • Perceive: The Perceiver maps each observed turn and prior state to schema-validated fields for topic, decisions, questions, coverage, stances, and floor.The dedicated step supports auditing and ablation of state updates.
  • Act: The Controller chooses SILENT or SPEAK and commits to a proposition before the Generator phrases it in the participant’s style.Helpers assess briefing relevance, prior coverage, floor dynamics, and topical fit.
  • Recalibrate: A Predictor and two complementary judges distinguish state-reading errors from action-choice errors before recalibration updates future state.The withheld forecast is compared with the observed continuation alongside judgments of the delegate’s action.

4 Experimental Setup

The study evaluates CAPA on the full AMI scenario corpus using causal, episode-level tests centered on participant-owned idea units. It compares CAPA with raw-context baselines and validates outcome judgments against human annotations.

  • Dataset: 137 AMI scenario meetings provide the primary evaluation corpus, with matched 20-meeting subsets for ablations and robustness checks.AMI meetings involve four participant roles designing a remote control; default settings use N=20 preceding utterances and k=5.
  • Baselines: CAPA is compared with transcript-only and Reflexion-style raw-context delegates under fixed backbone, context, judges, and episode schedules.The Reflexion-style baseline adds procedural memory without structured state fields or state-directed recalibration.
  • Evaluation units: Participant-owned idea units are proposition-level claims, proposals, or decision-relevant facts introduced by the target participant.The first 60% of each meeting constructs the participant profile, while the final 40% forms the interaction testbed.
  • Episode construction: The causal replay protocol withholds future anchors and targets, allowing only context available before each delegate decision.State updates consume ground-truth turns only once they are causally observable.
  • Outcomes and metrics: Credited recovery requires proposition matching without hallucination or redundancy, distinguishing strict hits, loose hits, uncredited attempts, and non-attempts.Decision F1 measures floor-taking alignment, while output quality includes hallucination, redundancy, and offtopic indicators.
  • Validation: Cohen’s kappa = 0.71 measures agreement between schema-constrained LLM judges and human annotations.Human validation used balanced samples of extracted idea units and same-proposition decisions.

5 Results

Across 137 AMI meetings, CAPA substantially increases timely participation while preserving grounding, shifting the dominant failure from silent omission to diagnosable selection errors. Its gains come primarily from recognizing intervention opportunities through explicit meeting-state tracking rather than from better timing after speaking.

  • Engagement gain: CAPA reduces silent abstention and doubles credited recovery, with loose recall rising from 26.1 to 52.2 and strict recall from 10.7 to 25.1.Decision F1 also rises by 24.9 points, from 38.1 to 63.0.
  • Outcome redistribution: CAPA absorbs a 48.9% drop in not-attempted episodes through +26.1% credited and +22.8% uncredited attempts.The 27.1% strict–loose gap reflects credited interventions arriving off-anchor within the k=5 window.
  • Grounding preserved: Hallucination remains at 0.6% and redundancy at 0.0%, while offtopic rises by 5.2% per episode against the 48.9% reduction in not-attempted episodes.The results characterize this as a coverage–restraint trade-off for fixed-weight LLM agents.
  • Pre-anchor latency: Of CAPA’s 595 credited matches, 93.8% are anchor-aligned or pre-anchor, with a median lead time of 1.0 turn before the anchor.This timing precision accompanies the 24.9-point Decision F1 increase over the baseline.
  • The floor-taking bottleneck: CAPA and the baseline share timing distributions when they engage, but intervention rates differ by 48.9%: 97.5% for CAPA versus 48.6% for the baseline.This asymmetry identifies opportunity recognition, rather than subsequent timing mechanics, as the main prompt-only deficit.
  • Error composition: CAPA’s 504 uncredited episodes form four mutually exclusive categories: incomplete match, different proposition, late match, and safety/quality flag.The taxonomy links near-misses to candidate curation, timing, and output-filter components.
  • System observability: CAPA converts silent omissions into traceable uncredited attempts, enabling stepwise diagnosis and targeted refinement.The transcript-only delegate’s 51.4% silent-abstention rate produces no comparable decision trace.

6 Ablations

Ablations show that explicit meeting state governs whether and when CAPA engages, while recalibration refines contribution selection and the architecture transfers across model backbones and meeting corpora.

  • 6.1 Meeting State Carries the Threshold: State tracking governs the binary participation policy, while recalibration refines qualitative selection after the floor-taking decision.Removing state tracking reduces behavioral performance against a longer raw window; removing recalibration affects selection without changing the participation threshold.
  • 6.1 Meeting State Carries the Threshold: 22.4% loose recall drop, 14.5% strict recall reduction, and 24.7% Decision F1 reduction follow replacing meeting state and recalibration with a 50-turn raw window.Redundancy also increases by 10.7%, showing that longer raw context does not substitute for structured state.
  • 6.2 Recalibration Refines Selection: 3.9% loose recall drop and 4.6% increase in uncredited attempts follow removing recalibration, while Decision F1 and grounding remain statistically unchanged.The result separates feedback for updating meeting state from feedback for rewriting or narrowing an output.
  • 6.3 Backbone Model Portability: Decision F1 stays within 57.9 to 69.2 across GPT-4o, Gemini-2.5-Pro, Llama-3.3-70B, and Qwen3.6-27B, with backbone-specific coverage–precision trade-offs.Qwen abstains more but records 3.3% off-topic contributions and 0.0% hallucination, while Gemini gains recall at the cost of off-topic contributions.
  • 6.4 Cross-Corpus Robustness: On 10 ICSI meetings, silence remains near zero at 1.2% versus 2.5% on AMI, while loose recall is higher at 73.6% versus 52.2%.The robustness check adapts profile construction while retaining the identical evaluation protocol.
  • 6.5 Robustness Checks: Across raw windows of 10, 20, and 30 turns, loose recall remains between 50.5% and 56.7% and Decision F1 between 61.1% and 63.1%.With meeting state available, raw context is no longer load-bearing across the tested window lengths.

7 Final Considerations

CAPA addresses silent abstention in online meeting delegation by maintaining explicit state and separating floor-taking from generation. Its state-driven correction channel supports module-attributable selection errors and transfers across configurations.

  • 7 Final Considerations: CAPA uses a perceive–act–recalibrate loop over explicit state to address prompt-only delegates’ failure to recognize intervention opportunities.The state includes variables such as stance, coverage, and floor control, while evaluation scores whether, when, and what the delegate contributes.
  • 7 Final Considerations: CAPA reduces recognition errors to transparent, module-attributable selection errors and shows that output-rewriting correction is non-substitutable for silent-abstention failures.The architecture decouples floor-taking from surface generation, matching the correction mechanism to the observed failure regime.
  • 7 Final Considerations: Robustness checks confirm that the core architecture transfers across configurations.The paper frames this transfer as relevant to other domains with similar causal information boundaries.

Limitations

The evaluation is centered on AMI, with ICSI used only as an adapted robustness probe, and performance depends significantly on the underlying language model’s capabilities.

  • Dataset and Protocol Scope: The primary evaluation focuses on the AMI scenario portion because alternative corpora lack all protocol conditions without adaptation.ICSI is used as a robustness probe with only profile construction adapted while causal replay remains fixed.
  • Model Dependence: Performance depends significantly on the underlying language model, so models with different reasoning capabilities or smaller context windows may perform less accurately.Ablations nevertheless suggest that the framework can carry over to other model families.

Ethical Considerations

CAPA is evaluated offline and should not be treated as authorized live impersonation. Deployment requires recurring consent from all meeting participants because the system can surface participant-owned propositions.

  • Consent and Deployment: CAPA is evaluated offline on the publicly released AMI Meeting Corpus, which does not establish authorization for live impersonation.The paper distinguishes consent for offline transcript analysis from consent required in live meetings.
  • Consent and Deployment: Live deployment requires explicit, recurring consent from all participants because CAPA may surface a participant-owned proposition they did not state themselves.The paper identifies impersonation, speech laundering, and asymmetric advantage as misuse paths.
  • Release and Use: The released research infrastructure includes the code, evaluation protocol, prompt templates, judges, and analysis scripts under an MIT license after acceptance.The license supports research use but does not replace the paper’s participant-consent requirement for production deployment.

A.2 Datasets and Licensing

The evaluation uses AMI meeting data and a structured, schema-validated implementation with fixed decoding settings, latency accounting, and human-validated annotations.

  • Implementation: CAPA modules emit parseable structured outputs through schema validation, and Shared Memory is a typed dictionary indexed by participant, meeting, and turn.The implementation applies this validation across the Perceiver, Predictor, judges, Recalibrator, Controller, Generator, and helpers.
  • Models: GPT-4o is the default backbone, while Gemini-2.5-Pro, Llama-3.3-70B, and Qwen3.6-27B are evaluated for portability.Decoding settings remain fixed across the reported systems and conditions.
  • Evaluation settings: N=20 preceding utterances and k=5 evaluation turns are the default episode settings, with macro-averaged meeting-level metrics and 95% bootstrap confidence intervals.The context and evaluation windows are varied in robustness checks.
  • Runtime: 21.14 s is the observed average GPT-4o response-path latency, including a fixed three-second post-call throttle.Removing the throttle gives 11.12 s provider-only mean latency across 110 decisions.
  • Validation: κ = 0.71 is the aggregate agreement between human labels and the LLM judge across the evaluated dimensions.The inspected samples cover idea-unit extraction, proposition matching, hallucination, redundancy, and off-topic behavior.

D Mechanism Ablation Tables

The ablations separate the effects of explicit meeting state and post-action recalibration, with the meeting-state removal producing the broadest degradation.

  • Meeting-state ablation: Removing meeting state mt collapses coverage, Decision F1, and redundancy together.The state-ablated system also omits recalibration and receives a 50-turn raw window.
  • Recalibration ablation: Removing the Recalibrator narrows contribution selection while leaving the floor-taking threshold within bootstrap noise.Both recalibration variants retain mt, the Predictor, and the judges.
  • Meeting-state ablation: Figure 6 visualizes the meeting-state ablation as grouped bars corresponding to Table 8.The comparison concerns the joint effect of omitting mt and recalibration despite the longer raw context.
  • Recalibration ablation: Figure 7 visualizes the recalibration ablation as grouped bars corresponding to Table 9.The figure isolates recalibration while the meeting state, Predictor, and judges remain shared.

G Outcome Decomposition

CAPA changes episode outcomes from predominantly silent non-attempts to active engagement, while robustness checks examine roles, context windows, evaluation windows, and backbones.

  • Outcome decomposition: 51.4% to 2.5% is the drop in not-attempted episodes from the transcript-only delegate to CAPA.The systems’ confidence intervals do not overlap on any outcome category.
  • Per-role consistency: 43.5% to 61.8% loose recall and below 3.5% not-attempted rates across roles indicate broad per-role consistency.Decision F1 remains within a 62.6–66.7 range across the four roles.
  • Context-window robustness: 50.5% to 56.7% loose recall and 61.1% to 63.1% Decision F1 remain stable across N ∈{10, 20, 30}, with overlapping confidence intervals.The raw window supplies recent evidence, while longer-range decisions, issues, stances, and coverage remain in mt.
  • Backbone portability: 66.2% versus 65.6% Decision F1 is nearly unchanged between GPT-4o and Gemini, while loose recall differs at 53.7% versus 70.9%.The comparison shows backbone-dependent coverage–restraint operating points rather than a single fixed behavior.
  • Backbone portability: Backbone portability remains model-dependent because active models supply different intervention-threshold signals and also provide quality and safety judgments.The authors recommend calibrating suitability cutoffs and restraint rules on held-out episodes.

I.4 Evaluation Window k

The evaluation-window analysis treats k as a permissiveness–timing trade-off: wider windows credit more delayed matches but weaken floor-taking alignment.

  • Interpretation: A one-turn evaluation window can undercredit valid interventions that span a short action sequence beyond the adjacent turn.Longer windows admit more credited matches but add intervening speak/silent decisions and dilute anchor-aligned timing.
  • Window sensitivity: 43.9% to 62.7% loose recall rises as k grows from 3 to 9.Uncredited attempts fall from 54.1% to 35.9% over the same window sweep.
  • Window sensitivity: 75.3% to 54.5% ground-truth-aligned Decision F1 falls as the evaluation window expands from k=3 to k=9.Strict recall remains comparatively flat, so wider windows improve permissiveness without improving anchor synchrony.
  • Prompt terminology: The appendix identifies EIM as the development-time name for CAPA’s Perceiver.The two names refer to the same module in the system prompts.
Loading 2609.03923v1…