NLE: Non-autoregressive LLM-based ASR by Transcript Editing

Avihu Dekel; Samuel Thomas; Takashi Fukada; George Saon

Source-linked AI summary

NLE: Non-autoregressive LLM-based ASR by Transcript Editing

Avihu Dekel, Samuel Thomas, Takashi Fukada, George Saon

arXiv:2603.08397v1Mar 9, 2026eess.AS

TL;DR

Autoregressive LLM-based ASR has strong accuracy but incurs sequential-decoding latency, motivating a parallel alternative. NLE edits an initial CTC hypothesis with a bidirectional LLM using interleaved insertion slots and CTC-based alignment. NLE++ reaches 5.67% average WER at 1630 RTFx, while NLE achieves a 27x single-utterance speedup over the autoregressive baseline.

Problem
Autoregressive LLM-based ASR limits parallelism and increases latency, while CTC-based drafts can contain locally correctable errors.
Method
NLE uses a pretrained speech encoder’s acoustic embeddings and CTC hypothesis as input to a bidirectional LLM editor with interleaved insertion slots and a CTC-style objective.
Results
5.67% average WER with 1630 RTFx is achieved by NLE++ on the Open ASR leaderboard, while NLE achieves 27x single-utterance speedup over the autoregressive baseline.
Takeaways & Limitations
NLE provides a non-autoregressive editing approach with competitive accuracy and faster inference, including up to 4x speedup in batched scenarios.
Takeaways & Limitations
NLE is less flexible when the desired output substantially diverges from the input hypothesis and incurs minor retokenization latency when tokenizers differ.

Abstract

from arXiv · show

While autoregressive (AR) LLM-based ASR systems achieve strong accuracy, their sequential decoding limits parallelism and incurs high latency. We propose NLE, a non-autoregressive (NAR) approach that formulates speech recognition as conditional transcript editing, enabling fully parallel prediction. NLE extracts acoustic embeddings and an initial hypothesis from a pretrained speech encoder, then refines the hypothesis using a bidirectional LLM editor trained with a latent alignment objective. An interleaved padding strategy exploits the identity mapping bias of Transformers, allowing the model to focus on corrections rather than full reconstruction. On the Open ASR leaderboard, NLE++ achieves 5.67% average WER with an RTFx (inverse real-time factor) of 1630. In single-utterance scenarios, NLE achieves 27x speedup over the AR baseline, making it suitable for real-time applications.

1. Introduction

NLE reframes LLM-based ASR as fully parallel transcript editing, addressing the latency and limited parallelism of autoregressive decoding while retaining pretrained LLM linguistic knowledge. NLE++ reaches 5.67% average WER at 1630 RTFx on the Open ASR leaderboard, and NLE achieves a 27x single-utterance speedup over an autoregressive baseline.

Autoregressive LLM-based ASR generates tokens sequentially, limiting parallelism and increasing latency, especially for real-time conversational use.
CTC provides fully parallel decoding but is limited by conditional independence, monotonic alignment assumptions, and weaker language modeling.
NLE edits a pretrained speech encoder’s hypothesis using acoustic context rather than autoregressively regenerating the transcript.
Interleaved insertion slots exploit Transformers’ identity mapping bias, letting the editor focus on local corrections instead of full reconstruction.
5.67% average WER and 1630 RTFx are achieved by NLE++ on the Open ASR leaderboard.
27x speedup over the autoregressive baseline is achieved by NLE in single-utterance inference.

2. Related Work

Prior ASR systems combine pretrained LLMs with speech encoders for linguistic knowledge, but most decode autoregressively or retain NAR limitations. NLE instead uses a pretrained LLM as a parallel editor for an initial CTC hypothesis.

LLM-based ASR systems use learned projection layers to condition pretrained language models on speech representations and improve transcription accuracy.
RNN-T and TDT support faster streaming inference but still generate tokens sequentially and lack pretrained LLM linguistic knowledge.
NAR ASR predicts tokens in parallel, with CTC providing efficient decoding but limited language modeling and conditional independence constraints.
Many NAR methods struggle with insertions and long-range consistency because fixed-length or masking strategies make insertions difficult or require multiple refinement iterations.
Post-processing research includes N-best and lattice rescoring, external LLM correction, and supervised error-correction approaches.
NAR translation and text-editing methods provide precedents for fertility-based generation, explicit edit operations, and CTC latent alignment.

3. Method

NLE freezes a pretrained CTC speech encoder, retokenizes its draft, inserts explicit blank slots, and applies a bidirectional LLM editor trained with CTC-based latent alignment. Parallel editing supports copying, replacement, deletion, and insertion, with optional iterative refinement.

3.1. Extracting Hypothesis and Embeddings: NLE processes audio with a pretrained speech encoder to obtain acoustic embeddings and a CTC transcript hypothesis for editing.
3.1. Extracting Hypothesis and Embeddings: The encoder is frozen during training, preserving its acoustic modeling capabilities while leaving joint fine-tuning for future work.
3.2. Retokenization and Interleaved Insertion Slots: The character-level CTC hypothesis is retokenized with the LLM’s subword tokenizer, splitting misspelled words when needed.
3.2. Retokenization and Interleaved Insertion Slots: The sequence ˜x = (ϵ, x1, ϵ, x2, . . . , ϵ, xN, ϵ) places insertion slots before every token and after the last token.
3.2. Retokenization and Interleaved Insertion Slots: An insertion of K tokens changes 2K−1 original tokens, while the remaining tokens stay unchanged; at most N + 1 tokens can be inserted.
3.3. Identity Mapping Bias: Residual connections and tied embeddings create an identity mapping bias that favors copying input tokens unchanged.
3.3. Acoustic and Token Representations: Projected acoustic embeddings and token embeddings are concatenated along the sequence dimension as input to the LLM.
3.4. Bidirectional LLM-based Editor: Bidirectional attention lets each position use past and future context, while LoRA adapts the pretrained LLM efficiently for editing.

4. Experiments

The experiments use a frozen pretrained CTC speech encoder and a lightweight trainable projector-plus-LoRA adaptation, evaluated across multilingual speech data with WER and RTFx under standardized inference settings.

Model Architecture: The model combines a 440M-parameter, 16-layer Conformer CTC encoder with a 1-layer Q-Former projector and a Granite 4.0 1B language model adapted using LoRA.The encoder processes 16 kHz audio; the projector downsamples 15-frame windows into 3 queries.
Model Architecture: Only 14M parameters are trainable in the base configuration, comprising the projector and LoRA adapters.The LoRA adapters adapt both attention and MLP layers while the language model remains lightweight.
Training Procedure: Training uses 3 epochs and 180K steps with AdamW, a 3e-5 peak learning rate, cosine decay, 5% warmup, and balanced sampling with α = 0.65.The training setup uses approximately 320 utterances per global batch across 8 H100 GPUs.
Data: The combined training corpus contains approximately 70K hours of speech across English, Spanish, French, German, and Portuguese.The experiments use AMI, VoxPopuli, YODAS, CommonVoice, MLS, Earnings22, Fisher, CallHome, and SwitchBoard data.
Evaluation Protocol: Evaluation focuses on the WER-RTFx trade-off using offline batched inference on a single H100 GPU with bf16 precision.Batch size is 96 for AR and NAR models, with an additional batch-size-1 evaluation for latency-critical settings.

5. Results

NLE reaches the Open ASR leaderboard Pareto frontier by combining competitive transcription accuracy with substantially faster inference, while NLE++ improves accuracy through increased training and model capacity.

5.1. Open ASR Leaderboard Comparison: NLE ranks 4th in average WER at 5.79% while achieving 1722 RTFx on the Open ASR leaderboard.Only Parakeet is faster among the compared top-six models, but its WER is higher at 6.05%.
5.1. Open ASR Leaderboard Comparison: NLE is 2-10x faster than several leading LLM-based systems while maintaining competitive accuracy.The comparison includes Canary-Qwen, Granite Speech, Qwen3-ASR, and Phi-4 Multimodal.
5.1. Open ASR Leaderboard Comparison: NLE is the only multilingual model on the leaderboard Pareto frontier, supporting English, Spanish, French, German, and Portuguese.Canary-Qwen and Parakeet are English-only models in the comparison.
5.1.1. NLE++ (Enhanced Training): 5.67% Open ASR WER is achieved by NLE++, improving 0.12% absolute over NLE while reducing RTFx from 1722 to 1630.NLE++ also improves the all-19 average from 6.54% to 6.44%.
5.1.1. NLE++ (Enhanced Training): NLE++ increases capacity and training budget through a larger projector, higher LoRA rank, higher learning rate, longer training, larger batches, and 120-second maximum audio.These changes increase trainable parameters from 14M in NLE to 280M in NLE++.
5.1.1. NLE++ (Enhanced Training): Scaling training compute and model capacity is associated with further NLE improvements, while NLE++ remains on the WER-RTFx Pareto frontier.The reported result suggests additional resources may yield further improvements.

5.2. Controlled Evaluation

In controlled evaluation across 19 datasets and 5 languages, NLE improves over CTC and closely matches AR accuracy while delivering much faster inference; ablations support its main design choices.

5.2. Controlled Evaluation: NLE reduces average WER from 7.40% to 6.54% versus the CTC encoder baseline and outperforms CTC on 17 of 19 test sets.The controlled evaluation covers 19 datasets across 5 languages.
5.2. Controlled Evaluation: NLE matches the controlled AR baseline in accuracy while achieving 4x speedup in batched inference and 27x speedup for single utterances.The reported RTFx values are 1722 versus 430 in batched settings and 322 versus 12 for single utterances.
5.2. Controlled Evaluation: NLE underperforms the AR baseline on CommonVoice subsets, with slightly worse multilingual performance attributed to weaker non-English hypotheses and an English-centric BPE vocabulary.The CTC encoder is trained predominantly on English data, limiting the quality of hypotheses that NLE refines.
5.3. Ablation Study: The full NLE model achieves the lowest validation loss among the ablation variants, and validation loss correlates with better test-set WER.The figure reports validation loss over training steps for the ablation study.
5.3. Ablation Study: Removing copying regularization worsens validation loss, indicating that the penalty supports training stability and final performance by encouraging identity mapping.The full model reaches lower loss despite including the non-negative regularization term.
5.3. Ablation Study: Bidirectional attention improves convergence and validation loss because editing benefits from future context despite the language model’s causal pretraining.Restricting the LLM to causal attention limits access to future context.
5.3. Ablation Study: Interleaved padding outperforms end-of-sequence padding because it preserves token locality during insertions instead of causing large token displacements.The result aligns with the locality bias of Transformer architectures.
5.3. Ablation Study: Removing acoustic embeddings, hypothesis conditioning, or LoRA adaptation degrades performance, supporting acoustic grounding, hypothesis refinement, and parameter adaptation.Without the hypothesis, the model struggles to predict the entire transcript from scratch.

5.4. Blank Density

Blank insertion density affects the accuracy-speed tradeoff: placing a slot between every token performs best, while sparser strategies degrade accuracy with little speed benefit.

5.4. Blank Density: Reducing blank density degrades accuracy while providing minimal speedup.The sequence length is dominated by acoustic tokens rather than text tokens.
5.4. Blank Density: Table 3 compares average WER and RTFx when insertion slots are added every K tokens.The comparison includes Every 1 and sparser insertion strategies.
5.4. Blank Density: Every 1 provides the best accuracy-speed tradeoff.It inserts a blank between every token.

5.5. Multi-Step Editing

Multi-step editing provides limited benefit: a second edit yields modest improvement at lower inference speed, while further editing can reduce performance.

5.5. Multi-Step Editing: A second editing step yields modest accuracy improvements at the cost of reduced inference speed.The editor output is decoded with CTC, re-interleaved with insertion slots, and fed back for another refinement step while acoustic embeddings remain fixed.
5.5. Multi-Step Editing: Multi-step editing repeatedly refines the transcript while keeping the acoustic embeddings fixed.Each iteration can correct errors introduced or missed in previous iterations.

5.6. Error Analysis

Error analysis shows different model behaviors across error types and datasets: AR produces more insertions, NLE more deletions, and MLS-PT exposes a limitation of editing when the initial hypothesis is strong.

5.6. Error Analysis: AR has the highest average insertion rate, while NLE has the highest deletion rate and lowest insertion rate.The AR insertion pattern is especially pronounced on AMI-SDM.
5.6. Error Analysis: Figure 4 decomposes WER into insertion, deletion, and substitution rates across all datasets, AMI-SDM, and MLS-PT.These conditions expose differences in model error behavior.
5.6. Error Analysis: MLS-PT shows that NLE and AR do not improve over CTC, with NLE exhibiting particularly high substitution errors.The small MLS-PT training set may cause overfitting or insufficient adaptation to Portuguese.

5.7. Qualitative Analysis

Qualitative examples show NLE correcting diverse transcription errors across languages, while inference profiling identifies the encoder as the dominant computational stage.

5.7. Qualitative Analysis: NLE corrects word-writing errors, substitutions, and severely corrupted input in representative examples.The examples also include successful German and French transcript corrections.
5.7. Qualitative Analysis: NLE successfully corrects German and French transcripts in multilingual examples.These examples illustrate the system’s multilingual editing behavior.
5.7. Qualitative Analysis: The encoder forward pass accounts for 66% of total inference time, roughly twice the LLM compute.The LLM processes a sequence five times shorter than the encoder output, while retokenization, projection, and output decoding together account for 4%.
5.7. Qualitative Analysis: Table 5 compares CTC outputs with NLE corrections, using colors to mark errors, corrections, and reference deletions.The table provides representative qualitative examples of the editing process.

6. Discussion

NLE reframes ASR as conditional transcript editing, using a bidirectional LLM editor and interleaved insertion slots to preserve efficient non-autoregressive inference. Its main boundary is that it is best suited to local corrections, while several extensions remain future work.

6. Discussion: NLE combines a pretrained CTC encoder with a bidirectional LLM editor and interleaved insertion slots for non-autoregressive transcript editing.The encoder supplies acoustic embeddings and an initial hypothesis; the editor refines that hypothesis.
6. Discussion: NLE achieves accuracy comparable to autoregressive baselines while providing up to 4x faster batched inference and greater speedup in single-utterance settings.The speed advantage is especially relevant to latency-critical conversational applications.
6. Discussion: Lightweight LoRA adapters and attention-mask modifications adapt pretrained LLMs for editing while retaining the Transformer identity-mapping bias.The interleaved insertion-slot scheme handles insertions efficiently.
6.1. Limitations: NLE is less flexible when the desired output substantially diverges from the input hypothesis, because it excels primarily at correcting local errors.Spoken question answering is identified as an example requiring major changes to the hypothesis.
Future Work: Future work includes text augmentation for multi-step editing, mask-predict combinations, separate audio-text processing with cross-attention, and extensions to streaming scenarios.Other directions include stronger CTC language-model decoding and joint encoder-editor fine-tuning.