Source-linked AI summary
Acoustic and Lexical Signals for Native Language Identification of Language Learners
Ahmet Yavuz Uluslu, Mengjie Qian, Kate Knill, Mark Gales
TL;DR
Spoken L1 identification has focused primarily on acoustic evidence, leaving transcript-based evidence and its proficiency-related patterns less studied. This paper compares WavLM audio and zero-shot GPT-4o transcript classifiers on learner English, finding that their relative strengths vary by proficiency and input length, while fusion performs best.
Problem
How audio- and transcript-based L1 identification compare across learner proficiency remains unclear, despite potential acoustic, lexical, and grammatical evidence in spoken learner English.
Source: §1 · p. 1Method
The study compares WavLM audio and zero-shot GPT-4o transcript classifiers on learner speech using a CEFR-balanced BULATS-L1 evaluation set.
Source: §1 · p. 1Results
Audio is stronger overall than unrestricted text, but text leads from A2 to C1; fusion reaches 94.81% accuracy versus 85.56% for audio and 82.59% for text.
Source: §6 · p. 7Takeaways & Limitations
Proficiency, input length, and modality all influence NLI performance, and combining acoustic and lexical evidence provides the strongest identification results.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Spoken native language (L1) identification for second language learners has traditionally relied on acoustic attributes of speech. Comparatively little attention has been paid to analysing speech transcripts for L1 traces. In this paper, we present a study of L1 identification using both speech and corresponding transcripts from a spoken learner English corpus. We compare a WavLM-based audio L1 classifier with a zero-shot GPT-4o classifier operating on transcripts and analyse their predictions across proficiency levels using a CEFR-balanced evaluation set. Our experiments show that the audio classifier outperforms the transcript-based approach overall when full transcripts are available, although their relative strengths vary with learner proficiency. Audio performs better at A1, whereas text performs better from A2 to C1 with full transcripts. Transcript-based accuracy generally increases with proficiency, while acoustic performance remains comparatively stable. Restricting transcript length substantially reduces text-based accuracy, whereas audio is less sensitive to input length and performs relatively well with limited input. These findings provide insights into L1 identification from acoustic and textual representations of learner speech and how performance varies with proficiency and the amount of available language material.
1 Introduction
Spoken learner speech offers acoustic and transcript-based evidence for identifying a learner’s first language, but how these sources perform across proficiency levels is uncertain. The paper compares WavLM audio classification with zero-shot transcript classification and examines proficiency and transcript-length effects.
1 Introduction
- Audio preserves phonetic and prosodic cues, while transcripts expose lexical and grammatical patterns as potentially complementary evidence for L1 identification.Earlier NLI work focused largely on learner writing and its lexical, grammatical, and spelling features.Source: §1 · p. 1
- The study addresses whether audio- and transcript-based L1 identification follow similar patterns across proficiency, where learner errors, pronunciation, and response length may change.Prior work describes changes in grammatical errors, pronunciation, and response length with proficiency, but their implications for the two modalities are unclear.Source: §1 · p. 1
- The study compares a WavLM audio classifier with zero-shot LLM classification of transcripts from the same learner English responses, using a CEFR-balanced evaluation set.BULATS-L1 is balanced jointly across six L1 backgrounds and five CEFR levels, A1–C1.Source: §1 · p. 1
- With full transcripts, text performs better from A2 to C1, while audio performs better at A1; restricting transcript length reduces text accuracy more than audio performance.The overall comparison and modality advantage vary with proficiency and available transcript length.Source: §1 · pp. 1–2
2 Related Work
Prior NLI research spans written and spoken learner language, while recent work explores prompted language models and pretrained speech representations. This study compares those approaches on the same learner responses across proficiency and transcript length.
2 Related Work
- The 2017 NLI Shared Task covered essays, spoken responses, and their combination; speech systems often combined transcript features with acoustic i-vectors, while transcript-only systems performed worse.Short spoken responses were suggested as a possible contributor to transcript-only systems’ lower performance, motivating investigation of longer transcripts.Source: §2 · p. 2
- Recent NLI work explores pretrained language models and zero-shot prompting, which infers L1 from task instructions without training a dedicated classifier on labelled NLI examples.Prior work suggests multilingual text and translation or grammatical-error-correction examples in pretraining may support cross-linguistic and nonstandard-language analysis.Source: §2 · p. 2
- Self-supervised speech representations support transferable audio models, including WavLM; this study compares WavLM with zero-shot transcript-based NLI on the same learner responses.The comparison examines proficiency patterns and transcript-length effects.Source: §2 · p. 2
3 Datasets
The study uses BULATS-L1 for its primary L1 identification experiments and Speak & Improve to assess transcript sensitivity. The corpora support a balanced evaluation and a separate comparison of manual and automatic transcripts.
3 Datasets
- BULATS-L1 is the primary spoken-NLI corpus, with self-reported L1 labels and a CEFR-balanced evaluation set; Speak & Improve supplies manual transcripts for assessing ASR sensitivity.BULATS-L1 lacks manual transcriptions, so its transcripts are generated automatically.Source: §3 · p. 2
3.1 BULATS-L1
- BULATS-L1 covers six self-reported L1 backgrounds, selected where submissions were sufficient to build a balanced evaluation set.The backgrounds are Arabic, Chinese, French, Italian, Japanese, and Spanish.Source: §3.1 · p. 2
- BULATS-L1 training uses 16,149 utterances from 769 speakers, but training data remain imbalanced across L1s despite downsampling higher-resource groups.Available training data range from 32 Arabic-speaking learners to 213 Spanish-speaking learners; the development set is used for model selection and hyperparameter tuning.Source: §3.1 · pp. 2–3
- The BULATS-L1 evaluation includes 270 submissions, balanced across six L1 backgrounds and five CEFR levels, with nine submissions per L1–level combination; C2 is excluded.The four non-read-aloud responses are concatenated, and proficiency levels are manually assigned by human experts.Source: Table 1 · p. 2, §3.1 · p. 3
- BULATS-L1 transcripts are generated with learner-adapted Whisper ASR, and mean submission length rises from 190 words at A1 to 555 at C1.Submission counts are balanced even though the amount of available text varies across proficiency levels.Source: §3.1 · p. 3
3.2 Speak & Improve 2025
- Speak & Improve contributes 300 evaluation-subset submissions with fluent manual transcripts that retain learner errors but omit disfluencies; without reference L1 labels, it supports consistency rather than accuracy evaluation.The manual transcripts are compared with automatic transcripts.Source: §3.2 · p. 3
4 Experimental Setup
The experimental setup pairs a WavLM-based speech classifier with zero-shot transcript classification and evaluates L1 identification on BULATS-L1. A separate Speak & Improve analysis measures prediction consistency across manual and automatic transcripts.
4.1 Speech-based NLI
- The speech pipeline uses a frozen WavLM-Large encoder and learns a softmax-weighted combination of its hidden-layer representations.Attention pooling produces a clip embedding for a classification head; only the layer weights and head are trained.Source: §4.1 · p. 3
- At inference, each submission is divided into non-overlapping 30-second clips, and their class-probability distributions are averaged to select a submission-level L1 prediction.Any final clip shorter than 30 seconds is retained.Source: §4.1 · p. 3
4.2 Transcript-based NLI
- Transcript-based NLI uses zero-shot GPT-4o multiple-choice classification, averages predictions across three randomized label orderings, and masks named entities and non-English words.Constrained decoding and normalized token-level log probabilities produce candidate-class distributions.Source: §4.2 · pp. 3–4
4.3 Evaluation
- On Speak & Improve, manual-versus-ASR transcript effects are assessed by predicted-label agreement and mean Jensen–Shannon divergence, not identification accuracy.Agreement tracks whether the predicted label changes; divergence measures differences between full predictive distributions.Source: §4.3 · p. 4
- BULATS-L1 evaluation compares submission-level predictions with self-reported L1 labels using accuracy and macro-F1, with equal weight assigned to each L1 class.Table 2 reports accuracy and macro-F1 by CEFR level for unrestricted text transcripts.Source: §4.3 · p. 4, Table 2 · p. 4
5 Results
Across the tested conditions, audio and transcript systems vary in their relative performance by proficiency and input length, while combining their predictions yields the strongest reported accuracy. Transcript accuracy rises with more text, whereas audio is less sensitive to input length; a consistency analysis also finds substantial agreement between modalities.
5.1 NLI across Proficiency Levels
- 82.59% transcript accuracy trails WavLM’s 85.56% overall, but audio leads only at A1, while text leads from A2 to C1.At A1, audio reaches 81.48% versus text at 48.15%; text peaks at 94.44% at B2, while audio performance varies over a narrower range. Macro-F1 follows a similar proficiency trend.Source: §5.1 · p. 4
5.2 Effect of Transcript Length
- 60.00% accuracy at 100 words rises to 78.15% at 200 words, 81.85% at 300 words, and 82.59% with full transcripts.Performance begins to saturate beyond 200 words, while proficiency differences remain under a common word cap; at 200 words, A1 accuracy is 51.85% versus 90.74% at B2.Source: §5.2 · p. 5, Table 3 · p. 5
5.3 Effect of Audio Length
- 85.56% audio accuracy with all four retained parts compares with 79.26% using only P1, showing less length sensitivity than transcript-based NLI.Text accuracy rises from 60.00% at 100 words to 82.59% with the full transcript; across CEFR levels, audio leads at A1, while text can overtake it when sufficient transcript material is available.Source: Table 4 · p. 5, §5.3 · p. 6
5.4 Fusion of Audio and Text Predictions
- 94.81% accuracy is reached by linear fusion at λ = 0.60, exceeding audio alone at 85.56% and text alone at 82.59%.Fusion improves over either individual system across a broad range of weights.Source: §5.4 · p. 6, Figure 2 · p. 6
5.5 Prediction Consistency on Speak & Improve
- 77.7% of Speak & Improve submissions receive the same top-ranked L1 from both models, with mean JSD of 0.221 bits between their six-class distributions.This indicates substantial label agreement alongside moderate differences in prediction probabilities.Source: §5.5 · p. 6
6 Conclusion
The audio classifier achieves higher overall accuracy than the transcript classifier, but their relative strengths vary with proficiency. Linear fusion performs best, reaching 94.81% accuracy compared with 85.56% for audio alone and 82.59% for text alone.
6 Conclusion
- Audio achieves higher overall accuracy than text, but audio is stronger at A1 while text is stronger at higher proficiency levels when sufficient transcript material is available.Audio is also less sensitive to shorter input than text.Source: §6 · p. 7
- 94.81% accuracy with linear fusion, compared with 85.56% for audio alone and 82.59% for text alone.The results indicate that the modalities provide complementary information.Source: §6 · p. 7
A L1 Classification Prompt
The L1 classification prompt asks GPT-4o to infer a learner’s native language from an English text, using lexical, syntactic, and grammatical clues, and choose among six candidate languages.
A L1 Classification Prompt
- The prompt casts the model as a forensic linguistics expert and asks it to choose the most likely L1 from six candidate languages using word choice, syntax, and grammatical errors.The response must contain only the letter corresponding to the selected language.Source: A L1 Classification Prompt · p. 8
B Audio Model Parameters
The audio classifier uses a frozen WavLM-Large encoder and trains only its layer-combination weights and classification head.
B Audio Model Parameters
- Training runs for 10 epochs with the WavLM-Large encoder frozen; only layer-combination weights and the classification head are updated.The best development-set accuracy checkpoint is retained for evaluation.Source: B Audio Model Parameters · p. 8
C Audio Model Per-part Performance
Table 5 reports audio NLI accuracy on per-part audios.
C Audio Model Per-part Performance
- Table 5 presents audio NLI accuracy for per-part audios.Source: Table 5 · p. 8