Source-linked AI summary
SpeechAnnotator: A Context-Aware Multi-Agent Framework and Benchmark for Multidimensional Speech Annotation
Qirui Zhan, Shuiyuan Wang, Jingbin Hu, Haoyu Zhang, Xiaming Ren, Jinrui Liang, Chaoren Yu, Bengu Wu, Yunxiang Chen, Houdun Liu, Su Feng, Liumeng Xue, Lei Xie
TL;DR
Existing speech data and benchmarks do not jointly cover the costs and contextual demands of multidimensional annotation across long, heterogeneous recordings. SpeechAnnotator uses context-aware agents and bounded field-level review, alongside SA-Bench and SA-Eval. On SA-Bench, it achieves the lowest reported timeline error and an 81.27% attribute macro average, with ablations reporting gains from planning, prior evidence, and review.
Problem
Existing pipelines and benchmarks address only parts of multidimensional speech annotation, while long heterogeneous recordings require evidence routing across speakers, neighboring segments, and recording context.
Source: §1 · p. 2Method
SpeechAnnotator coordinates planning, contract-guided labeling, and bounded evidence- and context-aware field revision through shared state.
Source: §1 · p. 3Results
SpeechAnnotator achieves the lowest timeline error with 12.42% CER and 33.69% tcpCER, and an 81.27% attribute macro average, ranking second overall and first on seven attributes.
Source: §5 · p. 14Takeaways & Limitations
Ablations report contributions of 1.48, 1.20, and 0.85 points to the average attribute score from context-aware planning, prior evidence, and the review loop, respectively.
Source: §5 · p. 14The cross-system comparison does not isolate the contributions of context-aware planning, prior evidence, or the review loop.
Source: §4.2 · p. 12
Abstract
from arXiv · showhide
Recent controllable speech generation requires training data with fine-grained annotations of speaker traits, prosody, emotion, paralinguistic cues, acoustic scenes, and context. Existing workflows often rely on manual correction, paid hosted multimodal services, or fixed processing chains, which limits large-scale data processing through annotation cost, external-service dependence, or weak cross-stage recovery. We introduce SpeechAnnotator, a locally deployable, context-aware multi-agent framework built entirely from open-source models and tools. Supporting frontend modules first obtain speaker-aware segments and final segment transcripts, while prior evidence extractors attach heterogeneous segment-level cues. Three specialist agents then collaborate through shared state: the Planning Agent converts local audio evidence, speaker history, neighboring segments, and recording-level context into field-specific contracts; the Labeling Agent performs contract-guided multimodal prediction for directly observable attributes; and the Review Agent runs a bounded review loop that checks evidence support and cross-segment consistency, triggering relabeling only for unsupported or inconsistent fields. To address the fragmentation of existing evaluation resources across isolated tasks and narrow-domain test sets, we introduce SpeechAnnotator-Bench (SA-Bench), containing 8.87 hours of human-annotated audio across nine source formats, together with SpeechAnnotator-Eval (SA-Eval), which separates Timeline-Eval for speaker-aware timeline recovery, Closed-Eval for finite-set attributes, and Open-Eval for open-ended attributes. Experiments and ablations show that SpeechAnnotator provides a locally deployable alternative to commercial audio-capable systems, while the bounded review loop improves multidimensional annotation through evidence- and context-aware field-level recovery.
1 Introduction
Controllable speech generation calls for segment-level annotations spanning speakers, delivery, scene, and context, while existing workflows and benchmarks cover only parts of this need. SpeechAnnotator combines context-aware annotation and a bounded review loop with SA-Bench and SA-Eval for unified evaluation.
1 Introduction
- Rich segment-level supervision must connect speaker traits, vocal realization, paralinguistic cues, acoustic scenes, and contextual meaning.The need extends beyond transcripts and coarse event tags.Source: §1 · p. 2
- Existing workflows face scale, cost, or context limits, while long heterogeneous recordings need evidence routing across segments, speakers, and recording context.Human-centered annotation can be costly at scale; hosted APIs may add recurring costs and throughput constraints, and event-centered annotation does not cover long-recording context.Source: §1 · p. 2
- SpeechAnnotator uses field contracts and bounded, field-level relabeling to check evidence and cross-segment consistency without restarting supporting stages.The Planning Agent prepares contracts, the Labeling Agent produces initial annotations, and the Review Agent requests targeted revisions.Source: §1 · p. 3
- SA-Bench and SA-Eval jointly assess speaker-aware timeline recovery, finite-set attributes, and open-ended descriptions under one schema.Open-Eval uses representation-based semantic scoring while keeping timeline, closed-set, and open-ended metrics separate.Source: §1 · p. 3
2 Context-Aware Multi-Agent Annotation Framework
SpeechAnnotator links segment-level artifacts through shared state, then routes evidence to specialist agents and selectively reviews predictions before deterministic output normalization.
2.1 Overview and Shared State
- A shared segment key connects timeline and transcript records, prior evidence, contracts, predictions, reviewed records, and final JSON across the workflow.The workflow combines frontend processing, prior extraction, multi-agent annotation, and deterministic output processing.Source: §2.1 · pp. 3–4, Figure 1 · p. 4
2.2 Agent Roles and Coordination Protocol
- The Planning Agent writes field-specific contracts from segment, transcript, evidence, recording, speaker-history, and neighboring-segment context.Contracts specify admissible evidence, priorities, context scope, and conflict handling; the Labeling Agent predicts directly observed attributes, while the Review Agent accepts supported predictions or requests targeted revisions.Source: §2.2 · p. 4, §2.3 · p. 5
2.3 Modules, Agents, and the bounded review loop
- Frontend modules enhance speech, estimate speaker-homogeneous segments and transcripts, refine transcripts, and reconcile canonical segment records.Prior extractors attach acoustic statistics, affective cues, and sound-event evidence; the fixed-timeline setting rebuilds the table from reference boundaries, speakers, and transcripts.Source: §2.3 · pp. 4–5
- The bounded review loop checks evidence support and cross-segment consistency, then relabels only affected fields while keeping frontend processing, accepted labels, and unrelated extraction fixed.The loop stops when reviewed fields are accepted or the revision budget is exhausted; shared state retains accepted predictions, supporting evidence, or reasons for relabeling.Source: §2.3 · p. 5
- Deterministic output processing normalizes reviewed records and serializes JSON, while semantic support and contextual consistency are handled by review.The figure summarizes the flow from segment records and evidence through contracts, labeling, review, and output.Source: Figure 1 · p. 4, §2.3 · pp. 5–6
3 SA-Bench and SA-Eval
SA-Bench and SA-Eval provide a shared-schema resource and protocol for speaker-aware timelines and multidimensional segment attributes. Evaluation separates timeline errors, finite-set matching, and open-ended semantic similarity.
3.1 SA-Bench Construction and Coverage
- SA-Bench contains 8.87 hours of Chinese-dominant audio across nine formats selected to expose different speaker-turn patterns, speaking styles, backgrounds, and discourse structures.Five trained annotators independently labeled the data, and two inspectors reviewed annotations and returned inconsistent or substandard records for revision.Source: §3.1 · pp. 6–7
3.2 Annotation Schema and Evaluation Protocol
- The 15 attributes span six evidence-oriented groups, while boundaries, speaker identifiers, and transcripts remain structural fields outside the attribute set.Perceived gender and perceived age are closed-set attributes; the remaining attributes are open-ended natural-language descriptions.Source: §3.2 · pp. 7–8, Figure 3 · p. 7
3.3 Timeline-Eval: Transcription and Speaker-Aware Timing
- Timeline-Eval separates lexical, speaker-attribution, and temporal-alignment errors using CER, cpCER, and tcpCER, with the latter two incorporating speaker mapping and timestamps.Because the benchmark is Chinese-dominant, the timeline metrics use character-level error rates.Source: §3.2 · p. 8, §3.3 · p. 8
3.4 Closed-Eval and Open-Eval
- Closed-Eval scores finite-set attributes by canonicalized exact matching under a fixed-timeline setting.Canonicalization normalizes punctuation, case, spacing, aliases, and empty-value markers.Source: §3.4 · p. 8, §3.2 · p. 8
- Open-Eval compares open-ended attributes through semantic similarity of hidden-state representations, allowing different wordings to receive comparable scores without a generative judge.Mean pooling and cosine similarity are applied consistently to SpeechAnnotator and all baselines.Source: §3.4 · p. 9
4 Experiments
The experiments compare SpeechAnnotator with commercial systems on speaker-aware timeline recovery and fixed-timeline attribute annotation, then test the contributions of review, prior evidence, and context through ablations. SpeechAnnotator is competitive overall, with results varying across tasks and annotation fields.
4.1 Evaluation Setup
- All systems used held-out SA-Bench recordings without training or adaptation; timeline recovery used original audio, while fixed-timeline evaluation supplied reference boundaries, speakers, and transcripts.Commercial systems were evaluated as black boxes; fixed-timeline attribute results were averaged across five runs.Source: §4.1 · pp. 9–10
4.2 Main Results
- 12.42% CER and 33.69% tcpCER improved on standalone MOSS-Transcribe-Diarize; tcpCER was 19.92 points below the strongest commercial baseline, Seed2.0 Lite [24].SpeechAnnotator had the lowest CER and tcpCER, while Qwen3.5-Omni-plus [25] had the lowest cpCER.Source: §4.2 · p. 10, Table 1 · p. 11
- 81.27% attribute macro average ranked second, 0.46 points below Gemini 2.5 Pro [5]; SpeechAnnotator led seven fields, including speed, texture, rhythm, and pronunciation.It ranked first in Prosody and Delivery and Voice Quality and Articulation, with category means of 86.75% and 86.48%, respectively.Source: §4.2 · p. 11, Table 2 · p. 11
4.3 Ablation
- Removing the Review Agent reduced the attribute macro average from 81.27% to 80.42% and increased its standard deviation from 0.34 to 0.39.The largest field drops were perceived age (2.59 points), perceived gender (1.91), caption (1.79), and speed (1.79).Source: §4.3 · p. 12, Table 3 · p. 13
- Removing prior evidence lowered the macro average to 80.07%, with the largest category drops in Acoustic Scene (2.03 points), Affective and Paralinguistic Cues (1.71), and Prosody and Delivery (1.51).Caption scored slightly higher without priors, while most acoustically grounded fields declined.Source: §4.3 · p. 12, Table 3 · p. 13
- Removing recording-level and cross-segment context reduced the macro average to 79.79%, with the largest category drops in Semantic Understanding (4.30 points) and Speaker Profile (2.41).Voice Quality and Articulation and Acoustic Scene changed little, while Prosody and Delivery fell by 0.76 points.Source: §4.3 · pp. 12–13, Table 3 · p. 13
5 Conclusion
SpeechAnnotator combines three specialist agents for segment-aligned multidimensional annotation and is introduced alongside SA-Bench and SA-Eval. On SA-Bench, it achieves the lowest character-level timeline error among evaluated systems and ranks second overall among attribute-evaluation systems.
5 Conclusion
- SpeechAnnotator combines Planning, Labeling, and Review Agents through shared segment-level state, alongside SA-Bench and SA-Eval.The agents organize field-relevant evidence and context, perform contract-guided multimodal annotation, and selectively revise inconsistent fields through a bounded review loop.Source: §5 · p. 13
- 12.42% CER and 33.69% tcpCER accompany an 81.27% attribute macro average; SpeechAnnotator ranks first on seven attributes and second overall among five systems.It achieves the lowest character-level timeline error among evaluated systems.Source: §5 · p. 14
- Context-aware planning, prior evidence, and the review loop contribute 1.48, 1.20, and 0.85 points, respectively, to average attribute score.Source: §5 · p. 14
- Future work will explore stronger field-specific evidence modeling, adaptive context selection, and more effective review strategies to improve annotation quality and generalization.Source: §5 · p. 14