OneVoice: An Intermediate Representation for Agentic Speech Pipelines

1. INTRODUCTION

• Speech evidence includes speaker identity, word- and phone-level timing, pronunciation, and behavioral events, often produced separately by specialized tools.A pipeline must preserve this evidence as it moves between agents, not only rely on the accuracy of individual components.Source: §1 · p. 1
• Tool-specific formats and implicit correspondence assumptions can detach timing, overwrite transcript variants, or confuse missing annotations with checked absences.Combining outputs such as alignment files and CHAT transcripts often requires custom parsing; each handoff can lose speech-specific structure.Source: §1 · p. 1
• OneVoice is a lightweight JSON-native interchange layer that uses stable identifiers, layered transcripts, explicit provenance, and validation to preserve speech evidence across handoffs.Its validator checks structural, temporal, and referential consistency before records propagate through the pipeline.Source: §1 · p. 1
• Across two workflows and three language models, OneVoice substantially improves handoff fidelity, reducing unstable aggregation and improving temporally grounded linking.The evaluation covers speech workflows that stress different forms of information preservation across agent handoffs.Source: §1 · pp. 1–2

2. RELATED WORK

• Speech formats such as TextGrid, alignment files, and CHAT capture temporal or interactional evidence, but differing structures and identifiers make cross-tool exchange difficult.These formats work within their intended workflows.Source: §2 · p. 2
• Annotation Graphs, LAF/GrAF, ELAN/EAF, NITE, and EXMARaLDA represent layered or temporally aligned annotations, primarily for corpus annotation, editing, or general-purpose interchange.OneVoice targets repeated exchange, validation, and combination of speech evidence during LLM-agent pipeline execution.Source: §2 · p. 2
• Dataset frameworks standardize access while retaining dataset-specific annotation structures, and agent protocols standardize tool or message exchange without defining speech-specific content.OneVoice targets that content layer with stable cross-artifact identifiers and explicit relationships among speech evidence.Source: §2 · p. 2

3. PROPOSED SCHEMA

• OneVoice is a lightweight JSON-native runtime interchange layer, not a replacement for existing annotation or dataset formats.A session contains speaker and metadata blocks alongside turns; each turn may contain transcript layers, optional alignments, and task-specific annotations.Source: §3 · p. 2, Figure 2 · p. 2
• Stable session, speaker, and turn identifiers plus hierarchical word and phone references let independently produced artifacts be joined without filenames or implicit ordering.Multiple transcript layers allow raw text, cleaned orthography, prompts, and phonetic or IPA representations to coexist without overwriting one another.Source: §3 · p. 2
• Turns integrate transcript layers, optional timing and word- or phone-level alignments, and task-specific annotations such as mispronunciations or behavioral events.Fields may remain empty when information is unavailable; provenance and coverage metadata distinguish unavailable or partial evidence from observed absence.Source: §3 · pp. 2–3
• Agents extend a shared logical session record and retrieve fields through stable references, preserving existing evidence as tools add complementary views.This avoids reconstructing correspondence from filenames, ordering, or free-form text.Source: §3 · p. 3
• A validator checks structural, temporal, and referential consistency, including interval containment and speaker references, before updated records are forwarded.Invalid records can be returned to their producing agent for repair, while namespaced optional annotations keep tool-specific information extensible.Source: §3 · p. 3

4.1. Experimental Setup

• The three-agent workflows have Agents A and B analyze different aspects independently, then Agent C combines their outputs into a fixed evaluation format.The downstream task stays unchanged across handoff conditions.Source: §4.1 · p. 3

4.2. Tasks

• UXSSD tests preservation of mispronunciation and behavioral-event counts across 15 child-speech bundles, while TIMIT tests event aggregation and phone-to-word temporal consistency across 40 speaker-level bundles.UXSSD gold annotations include 94 mispronunciation events and 112 behavioral events; TIMIT bundles contain five utterances each.Source: §4.2 · p. 3

4.3. Metrics

• Event-count precision, recall, F1, and MAE are reported for both tasks; TIMIT additionally uses word-link accuracy, invalid-link rate, and boundary error.TIMIT boundary error is the mean absolute deviation of predicted start and end times from gold boundaries, in milliseconds.Source: §4.3 · p. 3

4.4. Results

• On UXSSD, GPT-5 mini’s F1 rises from 0.265 with implicit handoff to 0.868 in C1 and 0.873 in C2; GPT-5.4 mini and Claude Haiku also improve.The UXSSD results illustrate more stable event aggregation with OneVoice.Source: §4.4 · pp. 3–4, Table 1 · p. 4
• On TIMIT, C2 raises GPT-5 mini word-link accuracy from 0.819 to 0.945 and reduces boundary error from 194.45 to 99.63 ms, while F1 rises from 0.851 to 0.936.Claude Haiku shows the same overall pattern; GPT-5.4 mini improves F1 and boundary error with C2, while C1 gives the highest word-link accuracy.Source: §4.4 · p. 4, Table 2 · p. 4

• OneVoice organizes speech artifacts into validated session records with stable identifiers, layered transcripts, explicit timing relationships, and provenance information.Across two multi-agent workflows, it improves handoff fidelity for event aggregation and temporal and hierarchical relationships.Source: §5 · p. 4