PaperlayerView exact revision on arXiv

Source-linked AI summary

Apriel Voice 27B Preview: An Open Model for Speech, Reasoning, and Tool Use

Aman Tiwari, Akshay Kalkunte, Akintunde Oladipo, Jash Shah, Tara Bogavelli, Hoang Nguyen, Nishanth Madhusudhan, Vishal Juneja, Sridhar Nemala, Srinivas Sunkara, Sathwik Tejaswi Madhusudhan, Shruthan Radhakrishna, Masoud Hashemi, Kelechi Ogueji, Khalil Slimi, Saloni Mittal, Pulkit Pattnaik, Shashank Maiya, Dhruv Jhamb, Esakkivel Esakkiraja, Katrina Stankiewicz, Gabrielle Gauthier Melançon, Fanny Riols, Amit Saha, Lakshmi Sirisha Chodisetty, Nitin Aggarwal, Anil Madamala, Sagar Davasam, Segan Subramanian, Vikas Yadav, Sai Rajeswar Mudumba

arXiv:2610.40738v1cs.SDcs.CLcs.LGeess.AS

TL;DR

Voice assistants need to reason over spoken requests, sustain conversations, and use tools, not merely recognize words. Apriel Voice adapts a shared text-and-image reasoning backbone for speech through staged training and speech-conditioned dialogues. In thinking mode, it reports results across speech reasoning, tool use, task completion, and interaction measures.

  • Problem

    A voice assistant must connect speech understanding with reasoning, conversation, and tool use because accurate transcription alone does not establish that it can complete a request.

    Source: §1 · p. 2

  • Method

    Apriel Voice extends a shared text-and-image reasoning backbone to speech using staged adaptation, supervised fine-tuning, text replay, and automatically constructed speech-conditioned dialogues.

    Source: §1 · p. 2, §5.1 · p. 6

  • Results

    In thinking mode, Apriel Voice scores 98.90 on Big Bench Audio, 81.13 on BFCL, and 65.51% mean task success on τ-Voice.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Apriel Voice demonstrates a shared model for spoken reasoning, instruction following, multilingual translation, service interactions, and complete workflows.

    Source: §11 · p. 18

    Multilingual capabilities and evaluations remain limited, with primary assessment through automatic speech translation and uneven multilingual robustness.

    Source: Limitations · p. 19

Abstract

from arXiv · show

A capable voice assistant must do more than recognize words; it must understand requests, reason over context, sustain conversations, and act through tools. We present Apriel Voice 27B Preview, an open speech-language model that unifies these capabilities within a shared reasoning backbone while retaining text and image input modalities. Built on the post-trained Gemma 4 26B-A4B checkpoint, Apriel Voice directly processes speech and generates both responses and tool calls without requiring a separate dialogue model. We introduce a two-stage training procedure to adapt the model to speech. The first stage establishes the speech interface, while the second jointly adapts the projector and backbone through supervised fine-tuning, text replay, and source-dependent cross-entropy and forward Kullback--Leibler divergence supervision. Automatically constructed speech-conditioned dialogues provide conversational training data, while local teacher serving, overlapping teacher and student computation, and fused full-vocabulary losses enable efficient online distillation. In thinking mode, Apriel Voice achieves a score of 98.90 on Big Bench Audio, 81.13 on Speech-BFCL, and a task success rate of 65.51% on $τ$-Voice, with a mean latency of 1.91 seconds. It also achieves 51.27% English EVA accuracy and 74.13% experience pass@1. Finally, we deploy Apriel Voice 27B Preview on a custom serving platform to support interactive voice interactions.

1 Introduction

Apriel Voice extends a shared text-and-image reasoning backbone to speech, with speech understanding, conversation, reasoning, and tool use evaluated as connected capabilities. Its staged adaptation and speech-conditioned dialogue training support this approach, while reported results span speech reasoning, instruction following, and task completion.

1 Introduction

  • Apriel Voice uses a shared reasoning backbone for spoken reasoning, conversation, and tool-call decisions while retaining text and image processing.The model is built on the post-trained Gemma 4 26B-A4B checkpoint, and speech synthesis can be replaced independently.Source: §1 · p. 2
  • A two-stage recipe first trains the speech interface, then adapts the projector and backbone with broad SFT, text replay, and source-dependent CE/FKL supervision.Automatically constructed speech-conditioned dialogues and online supervision from a frozen copy of the original backbone support adaptation and capability retention.Source: §1 · p. 2
  • In thinking mode, Apriel Voice scores 98.90 on Big Bench Audio, 81.13 on BFCL, and 65.51% mean task success on τ-Voice.English EVA accuracy is 51.27% and experience pass@1 is 74.13%.Source: §1 · p. 2

2.1 Connecting audio perception to language reasoning

  • Apriel Voice follows the encoder–projector approach but emphasizes evaluating the adapted model across reasoning, instruction following, complete interactions, and recognition.Prior work cited in the passage includes SLAM-ASR, Ultravox, and SALMONN.Source: §2.1 · pp. 2–3

2.2 Transferring behavior across modalities

  • Apriel Voice combines staged interface learning, broad second-stage SFT, selected speech distillation, and replay for a post-trained multimodal backbone.Its supervision aligns teacher and student response positions without requiring acoustic frames to match transcript tokens one to one.Source: §2.2 · p. 3

2.3 Speech generation and delegated voice systems

  • Apriel Voice places conversation, reasoning, and tool-call decisions in one speech-conditioned backbone, while application components execute tools and synthesize response text.The application returns tool results to the same backbone; turn detection and interruption control operate around the model.Source: §2.3 · p. 3

2.4 Evaluating and training capable voice agents

  • The evaluation includes benchmarks for service accuracy, interaction experience, task completion, text-agent interaction, and stateful enterprise workflows.Training combines student sharding with local online-teacher execution, a lookahead schedule, and full-vocabulary CE/FKL kernels.Source: §2.4 · p. 3

3 Model Architecture

Apriel Voice adds continuous speech input to an instruction-tuned text-and-image backbone, so the same model can reason and produce responses or tool calls from spoken requests. Acoustic features are projected into the decoder’s embedding space, and the adaptation recipe updates the speech interface and decoder while retaining the original visual components.

3.1 A shared backbone for perception, reasoning, and action

  • The instruction-tuned Gemma 4 26B-A4B backbone processes text, images, speech features, and tool results, then generates either a user response or a tool call.The application executes tools and returns their results to the conversation; response text is sent to a separate speech synthesizer.Source: §3.1 · p. 3

3.2 From acoustic frames to decoder embeddings

  • Eight adjacent encoder frames are stacked, then projected through linear and SwiGLU layers into the decoder’s 2816-dimensional embedding space.The final incomplete frame group is padded; the projector has approximately 47.7 M parameters, and projected features replace reserved audio positions while allowing multiple audio spans to interleave with text.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Adapting the decoder while preserving its interfaces

  • The response and tool-call formats remain shared across speech and text inputs, while stage two updates the projector, backbone, and router and keeps the original visual components frozen.The frozen teacher supplies training supervision and is removed at inference; because the decoder changes, visual retention is measured empirically.Source: §3.2 · p. 4, §3.3 · p. 4

4 Training Recipe

The recipe first teaches a frozen decoder to use speech, then broadens training to contextual tasks while preserving selected prior behavior. Synthetic speech-conditioned dialogues retain text targets and tool structure, and the data pipeline checks conversion fidelity.

4.1 Two stages with complementary goals

  • The two-stage recipe first learns speech input through multilingual transcription, translation, and continuation, then broadens task learning while updating the backbone and retaining earlier capabilities.Stage one freezes the backbone; stage two uses speech tasks, conversations, tool interactions, and text replay.Source: §4.1 · pp. 4–5, Figure 2 · p. 5

4.2 Stage one: learning the input interface

  • Stage one trains the speech projector while the speech encoder, language backbone, vision tower, and vision bridge remain frozen.The packed training mix spans two epochs and includes 32.2k hours of audio per epoch.Source: §4.2 · p. 5

4.3 Stage two: broad task learning with retention

  • Stage two updates the projector, language backbone, and router while keeping the speech encoder and original visual components frozen.Sources use CE, FKL, or both: CE supplies target behavior, while FKL transfers teacher behavior on selected speech tasks and constrains changes on replay contexts.Source: §4.3 · p. 5

4.4 Packing and exposure accounting

  • Stage two exposes the model to 52.6473 B total tokens across three epochs, including approximately 7.88574 B loss-bearing tokens.These are repeated training exposures, not unique data volume; text contributes tokens without an assigned audio duration.Source: §4.4 · p. 6, Table 2 · p. 6

5.1 Constructing speech-conditioned conversations

  • Custom task and conversation data begin as synthetic text interactions, then selected user content is rendered as audio while instructions, assistant targets, and tool messages remain text.This supports both whole-turn speech and mixed text/audio turns, including requests that refer to structured or case-sensitive material.Source: §5.1 · p. 6

5.2 Automatic span selection and consistent voices

  • The conversion pipeline selects user spans, normalizes written expressions for speech, preserves unselected text in place, and uses one reference voice across a conversation.The paired views give the student an acoustic task view and the teacher a corresponding text view.Source: §5.2 · p. 6, Figure 3 · p. 7

5.3 Conversion fidelity and paired export

  • Three fidelity checks validate source spans, spoken-form normalization, and synthesized speech, but do not establish the correctness of synthetic tasks or target answers.Export retains role order, assistant supervision, tool context, audio offsets, and the original text view.Source: §5.3 · p. 7

6 Task Learning and Capability Retention

The training objective combines desired-response targets with teacher predictive distributions, using source-specific settings to support new behavior and selected capability retention. Where both signals supervise the same positions, their combination can be interpreted as a scaled soft-target cross-entropy.

6.1 Learning from targets and predictive distributions

  • Task targets specify desired generations, while teacher distributions preserve the starting decoder’s relative preferences among possible next tokens.For speech tasks, the teacher receives the corresponding text view, and distribution matching applies to response positions rather than aligning acoustic frames with transcript tokens.Source: §6.1 · p. 7

6.2 Choosing the supervision for each source

  • Sources independently select CE, FKL, or combined supervision, separating the loss applied to an example from how often that source is sampled.Targets teach desired responses and behavior; teacher distributions transfer existing behavior to speech and preserve it on replay contexts.Source: §6.2 · p. 8

6.3 Interpreting the combined objective

  • When CE and FKL share supervised positions and normalization, their combined objective is a scaled cross-entropy with a soft target mixing the one-hot target and teacher distribution.The derivation’s mixture interpretation depends on common positions and normalization; with differing masks or normalizers, effective token weights determine the mixture, and retention on unseen inputs remains empirical.Source: §6.3 · p. 8

7 Efficient Training with an Online Teacher

Online teacher supervision is distributed locally and overlapped with student training, while fused loss computation bounds temporary memory. The implementation uses one-batch lookahead and coordinated checkpointing to manage the training schedule.

7 Efficient Training with an Online Teacher

  • Teacher inference adds a forward pass and distribution transfer, so the implementation distributes teacher work across local workers, overlaps it with student computation, and reduces temporary loss-evaluation storage.Student state is sharded, and teacher-student communication stays within each node.Source: §7 · p. 8

7.1 Local teacher workers and sharded students

  • Stage two uses eight nodes with two teacher and six student H100 GPUs per node, completing training in approximately three days.Teacher inference and student training use separate workers, with student state sharded across workers.Source: §7.1 · p. 8

7.2 Overlapping teacher and student computation

  • One-batch lookahead issues the next teacher request before current student computation, allowing teacher inference for batch k + 1 to overlap student work on batch k.Before checkpointing, workers drain pending teacher requests and then restart the lookahead pipeline.Source: §7.2 · p. 9, Figure 4 · p. 9

7.3 Fused CE and full-vocabulary forward KL

  • Fused reductions and vocabulary tiling reuse full-vocabulary normalizers to bound temporary probability storage, while full logits remain materialized.The teacher receives no gradient, and chunk losses are weighted by supervised-token counts.Source: §7.3 · pp. 9–10

7.4 Optimization settings

  • Stage two uses BF16 mixed precision, activation checkpointing, and fused multi-tensor AdamW, with a peak learning rate of 4 × 10^-6.The schedule warms up over 20% of training, decays cosine-wise to 10% of peak, and clips gradients at 1.0.Source: §7.4 · p. 10

8 Evaluation Design

The evaluation asks whether a unified speech-language backbone can perform reasoning, complete workflows, and retain text and image capabilities. It uses benchmark-specific implementations and settings to assess these dimensions across diverse system types.

8.1 Questions and coverage

  • The evaluation tests speech reasoning and instruction following, conversational task completion, and retention of text and image performance after adaptation.It covers foundational speech recognition and translation alongside audio reasoning, tool use, service interactions, and multimodal evaluations.Source: §8.1 · p. 10

8.2 Evaluation implementations

  • AU-Harness provides task definitions, prompts, preprocessing, output parsing, and scoring for BFCL, speech-recognition error rates, and applicable audio-to-text evaluations.EVA, τ-Voice, and the Audio Reasoning test suite use their official benchmark implementations.Source: §8.2 · p. 10

8.3 Model Inference Setup

  • Thinking and non-thinking results are reported separately when available, with thinking mode used for EVA, τ-Voice, and image/document evaluations.EVA uses Cartesia Sonic 3.5 synthesis, and τ-Voice also uses Cartesia; these results describe the complete voice system.Source: §8.3 · pp. 10–11

8.4 Metric Details

  • Pass@1 measures single-attempt success, while EVA-A and EVA-X are composite scores whose criteria include task completion, conversational experience, and turn-taking.τ-Voice latency measures time until the synthesized spoken response is available, including generation and synthesis; it is reported alongside task success.Source: §8.4 · p. 11
  • EnterpriseOps-Gym distinguishes strict workflow completion, which requires every check to pass, from partial progress and aggregate verifier success.Oracle mode supplies relevant tools, while additional-tool conditions also include distractors.Source: §8.4 · p. 11
  • Latency is mapped to a bounded 0–100 responsiveness score, with lower response latency receiving a higher score.The score uses 0.25 seconds as the 100-point threshold and 4.0 seconds as the 0-point threshold, with linear interpolation between them.Source: §8.4 · p. 11, Figure 5 · p. 12

8.5 Baseline Comparison

  • Baseline comparisons span proprietary, mixed, and open-source systems using cascaded ASR–LLM–TTS, speech-conditioned, and end-to-end speech-to-speech architectures.Table 3 summarizes baseline source, type, and access; Table 4 covers speech recognition and translation metrics across proprietary and open-source systems.Source: §8.5 · pp. 12–13, Table 3 · p. 12, Table 4 · p. 13

9 Results and Analysis

The evaluation covers foundational speech, reasoning and instruction following, conversational task completion, and retention of text and image capabilities. Results report strong performance across these dimensions, with some measures varying by task and domain.

9.1 Beyond foundational speech capabilities: Speech reasoning, instruction-following, and tool calls

  • Foundational speech results include 2.23 and 5.15 WER on LibriSpeech test-clean and test-other, and mean BLEU of 42.87 across German, French, and Spanish-to-English translation.The translation mean matches GPT-realtime-2.1 at 43.72; Spanish-to-English reaches 44.88 BLEU.Source: §9.1 · p. 13
  • In thinking mode, Apriel Voice scores 85.84 overall average, the highest among open-source 2-stage models, with 98.90 on Big Bench Audio and 81.13 on BFCL.It also scores 86.47 on audio-input IFEval and 86.19 on MT-Bench.Source: §9.1 · p. 13, Table 5 · p. 14

9.2 Conversational Capabilities and Task Completion

  • With Cartesia Sonic 3.5 synthesis, Apriel Voice reaches 51.27% EVA-A and 74.13% EVA-X, outperforming Parakeet + Gemma4-26B and approaching Gemini 3.8 Live Extended Thinking.Figure 6 places Apriel Voice at the highest EVA-X among compared models while maintaining competitive EVA-A accuracy.Source: §9.2 · p. 14, Table 6 · p. 14, Figure 6 · p. 15
  • Turn-taking reaches 86.52% versus 40.54% for Parakeet + Gemma4-26B, conciseness reaches 86.80% versus 84.99%, and speech fidelity is 98.89% versus faithfulness at 43.49%.EVA-X measures whether all experience criteria pass jointly.Source: §9.2 · p. 14, Table 7 · p. 15
  • Apriel Voice records 65.51% mean task success on τ-Voice at 1.91 seconds equal-domain mean latency, outperforming evaluated open-source 2-stage baselines on both measures.It has comparable task completion with substantially lower latency than the Gemma4-based 3-stage cascades.Source: Table 6 · p. 14, Figure 7 · p. 16, §9.2 · p. 16

9.3 Text and Image Capability after adaptation

  • On τ 2 with GPT-5.2 as user simulator, Apriel Voice scores 70.95% with thinking and 49.14% without, versus Gemma at 68.93% and 53.23%.The internal differences are +2.02 and −4.09 percentage points, respectively.Source: §9.3 · p. 17, Table 8 · p. 17
  • In EnterpriseOps-Gym with thinking, Apriel Voice achieves 23.0% task success and 57.6% verifier success, compared with Gemma at 22.7% and 57.8%.With 5, 10, and 15 additional distractor tools, thinking-mode task/verifier success is 24.7/61.1%, 23.6/59.9%, and 23.7/59.4%; task success varies non-monotonically with tool count.Source: §9.3 · p. 17, Table 9 · p. 17
  • Thinking-mode text scores are 84.42 on MMLU-Pro, 80.40 on GPQA Diamond, 92.00 on AIME 2024, 81.67 on AIME 2025, 92.88 on MATH-500, and 100.00 on AMC 2023.On images, MMMU-Pro changes from Gemma's 70.23 to 69.88, while MathVision MINI rises from 64.14 to 67.10; three of nine image scores are higher and six lower than Gemma's.Source: §9.3 · p. 17, Table 8 · p. 17

10 Discussion

The discussion presents speech reasoning and tool use as part of a broader capability profile, then describes the training and supervision mechanisms. It also notes that optimization losses need not move together and that component effects require controlled ablations.

10.1 Speech capability across tasks

  • Apriel Voice handles spoken reasoning, instruction following, tool calls, and task-oriented conversations while continuing to process text and images; reasoning-mode selection depends on the task.Thinking improves several reasoning and workflow scores, while some recognition and function-calling tasks score higher in non-thinking mode.Source: §10.1 · p. 17

10.2 Training and supervision

  • Training combines a frozen-backbone speech-projector interface stage with task SFT that adapts the projector and backbone, using source-dependent teacher supervision and text replay.Paired text/audio dialogues provide corresponding contexts; local teacher workers, lookahead, and fused losses support full-vocabulary online supervision, while component effects require controlled ablations.Source: §10.2 · p. 18

10.3 Optimization behavior

  • The combined stage-two objective decreases over training, but cross-entropy and forward KL need not move together because they measure target-response fit and divergence from the frozen teacher, respectively.Source composition and supervision weights also affect the aggregate; the plotted losses use separate scales and need not decrease monotonically under mixed supervision.Source: §10.3 · p. 18, Figure 8 · p. 18

11 Conclusion

Apriel Voice 27B Preview extends a text-and-image backbone to speech while using the same model for reasoning, conversation, and tool-call decisions. The conclusion reports evaluations across spoken reasoning, instruction following, multilingual translation, service interactions, and complete workflows, alongside an interactive deployment interface.

11 Conclusion

  • Apriel Voice 27B Preview extends a text-and-image backbone to speech, using the same model for reasoning, conversation, and tool-call decisions.Evaluations cover spoken reasoning, instruction following, multilingual translation, service interactions, and complete workflows.Source: §11 · p. 18
  • The training recipe combines interface learning, broad task SFT, synthetic speech-conditioned dialogues, and source-dependent CE/FKL supervision.Local teacher serving, lookahead, and fused losses support online distillation; a shared real-time API supports deployment and voice-benchmark integration through WebSocket communication.Source: §11 · p. 19

Limitations

The paper prioritizes English speech understanding and reasoning, while multilingual evaluation and robustness remain limited. Its reported comparisons also have limits in attribution, significance, and scope.

Multilinguality

  • English performance does not imply comparable quality across languages; multilingual speech understanding and conversational robustness remain priorities for future versions.The paper prioritizes English and assesses multilingual capabilities primarily through automatic speech translation.Source: Limitations · p. 19

Evaluation and comparison

  • Aggregate scores do not establish paired significance, non-inferiority to the starting backbone, or a controlled ranking against every frontier system.Internal and public results differ in prompting, language coverage, output modality, reasoning budgets, judging, and task limits.Source: Limitations · p. 19

Data and attribution

  • Synthetic data and distillation may propagate errors, fidelity checks do not establish coverage of natural speech variation, and transfer to other backbone families remains untested.The evaluations characterize the combined recipe and do not causally attribute outcomes to individual data or supervision choices.Source: Limitations · p. 19

A Synthetic Data Construction Details

The paper constructs speech-conditioned dialogues by converting selected user content while preserving dialogue structure and assistant targets, then describes the data mixture and training dynamics across two stages.

A Synthetic Data Construction Details

  • Selection strategies convert whole turns, instructions, questions, requested reading content, or multiple spans, with validation, retries, and rejection for malformed selections.The pipeline preserves dialogue structure and assistant targets, and each converted dialogue retains a text version.Source: A Synthetic Data Construction Details · p. 22
  • A reference utterance is reused across a conversation; spoken-form conversion and duration-limited sentence chunking preserve the placement of contiguous and disjoint speech spans.Speaker-aware scheduling and resumable processing preserve reference assignment during generation.Source: A Synthetic Data Construction Details · p. 22
  • Conversion fidelity is checked through normalized spoken-form comparison, recognition-based transcript comparison, and optional turn-level faithfulness checks.Source: A Synthetic Data Construction Details · p. 22

B Training Details and Loss Dynamics

  • Figure 9 tracks the combined objective and its cross-entropy and forward-KL components; measured and smoothed curves use separate scales.Component losses reflect both sampled tasks and their supervision weights.Source: Figure 9 · p. 22, B Training Details and Loss Dynamics · p. 23
  • Stage one uses 1,164 dataset configurations under 152 names, with 18.17 M examples and 99.2% token fill; Table 11 distinguishes audio duration from repeated exposure.The speech mixture includes Common Voice [1], GigaSpeech [6], People’s Speech [13], LibriSpeech, and CoVoST 2 [44]; text contributes tokens without assigned audio duration.Source: B Training Details and Loss Dynamics · pp. 22–23, Table 11 · p. 23
  • Each source independently selects its supervision; combined cross-entropy and forward KL divergence use separate positive coefficients, and forward KL also supports retention replay.Internally generated responses supply task targets, while the online teacher supplies full-vocabulary distributions for sources configured for distillation.Source: B Training Details and Loss Dynamics · p. 23, Table 12 · p. 23
  • Three stage-two epochs provide approximately 52.6 B total and 7.9 B loss-bearing token exposures, using seed 42, zero weight decay, gradient clipping, cosine decay, and fused AdamW.The mixture contains 17.56 M training examples and 1.56 M effective bins, including exposure weighting.Source: B Training Details and Loss Dynamics · p. 23

C Additional Modality Results

The additional-modality and workflow sections describe document, text-agent, enterprise-tool, EVA, and τ-Voice evaluations, including their benchmark scopes and comparison conditions.

C Additional Modality Results

  • Document-understanding results cover English pages and evaluate text, formulas, tables, and reading order; published Gemma text benchmarks and document and text-agent domain scores provide additional comparisons.The document table compares Gemma 4 26B-A4B and Apriel Voice in thinking mode.Source: Table 13 · p. 23, C Additional Modality Results · p. 23, Table 15 · p. 24
  • The internal τ 2-Bench uses GPT-5.2 as user simulator, reports task success across three equally weighted domains, and distinguishes thinking from non-thinking results.The Gemma baseline is Gemma 4 26B-A4B.Source: Table 14 · p. 24

D Enterprise Workflow Evaluation

E EVA and τ-Voice Domain Breakdown

  • EVA reports accuracy and experience across airline, ITSM, and healthcare HR, while τ-Voice reports pass@1 and latency across airline, retail, and telecom domains.The tables specify thinking-mode and Cartesia Sonic 3.5 conditions for Apriel Voice; EVA and τ-Voice captions highlight OSS baselines and Apriel Voice separately.Source: Table 18 · p. 25, Table 19 · p. 25, E EVA and τ-Voice Domain Breakdown · p. 25

Related papers

Loading 2610.40738v1…