PaperlayerView exact revision on arXiv

Source-linked AI summary

Aslema at NADI 2026: Augmentation through Fewshot for SLU

Tajwaar Shafiq, Hunzalah Hassan Bhatti, Shammur Absar Chowdhury, Firoj Alam

arXiv:2608.18689v1cs.CLcs.AI

TL;DR

Dialectal Tunisian Derja remains challenging for audio-based spoken language understanding, motivating evaluation beyond standard Arabic resources. Aslema compares audio LLM supervision strategies and augments training with culturally grounded synthetic speech, achieving 86.8% intent accuracy and 34.7 WER on dev-test while ranking first in official-test slot filling.

  • Problem

    Audio LLM effectiveness for low-resource, code-switched Tunisian Derja spoken language understanding remains limited.

  • Method

    The system evaluates four audio LLMs under zero-shot and LoRA fine-tuning and augments training with few-shot LLM-generated, voice-cloned Tunisian Derja speech.

  • Results

    Fine-tuning and synthetic augmentation improve both tasks; the final system reaches 86.8% intent accuracy and 34.7 WER on dev-test.

  • Takeaways & Limitations

    Task-specific training with culturally grounded synthetic speech is effective for this Tunisian Derja SLU benchmark.

  • Takeaways & Limitations

    The official test covers 60 intent labels versus 23 in training and dev-test, while synthetic augmentation is evaluated only with Qwen3-Omni-30B.

Abstract

from arXiv · show

We present Aslema, our system for NADI 2026 Shared Task 5, which consists of two subtasks: intent recognition and slot filling. We evaluate four omni LLMs in a zero-shot setting and compare them with fine-tuned models. Our results show that fine-tuning consistently outperforms zero-shot inference. We further explore synthetic data augmentation by using an LLM to generate culturally grounded Tunisian Derja utterances, followed by voice cloning to generate synthetic speech. Incorporating this synthetic data improves performance on both tasks. Our final submitted system, based on Qwen3-Omni-30B and trained with a mixture of original and synthetic data, achieves 86.8% intent accuracy and 34.7 WER on the devtest split. On the official test set it ranks 1st in slot filling (59.5 CoER) and 4th among 8 teams in intent recognition (66.1% accuracy). We release our experimental scripts and will soon share the synthetic dataset to support further research in this area.

1 Introduction

The paper studies end-to-end spoken language understanding for Tunisian Derja, a low-resource and code-switched dialect for which audio LLM effectiveness remains limited. It evaluates audio LLMs and fine-tuned models across intent recognition and slot filling, while examining supervision levels, training-data size, and synthetic augmentation.

  • Tunisian Derja is low-resource, heavily code-switched with French and English, and substantially different from standard Arabic.
  • Shared Task 5 evaluates end-to-end SLU on SLURP-TN through intent recognition and slot filling.
  • The study compares four instruction-tuned audio LLMs spanning 3B–30B parameters with LoRA fine-tuning and a fully fine-tuned Whisper-small baseline.
  • The experiments examine audio LLM effectiveness under different supervision levels, training-data sizes, and synthetic augmentation using an LLM+TTS pipeline.

2 Related Work

The related work traces SLU’s shift from cascaded ASR–NLU systems toward end-to-end architectures and challenging benchmarks. It also motivates dialectal SLU augmentation through synthetic text and speech, drawing on VoxCPM’s multilingual, zero-shot voice-cloning capabilities.

  • Spoken language understanding: SLU historically used cascaded ASR transcription followed by text-based NLU, while recent systems increasingly adopt end-to-end architectures and challenging benchmarks.This progression is situated within established SLU research and newer benchmark releases.
  • Dialectal SLU augmentation: The study applies synthetic training samples to dialectal SLU to increase coverage of underrepresented intent classes in the original training set.This approach targets coverage gaps in dialectal SLU data.
  • Speech synthesis: VoxCPM is an open, tokenizer-free TTS model covering 30 languages, including Arabic, with zero-shot voice cloning from a small number of dialectal reference clips.Its cloning capability can represent Arabic dialects without dedicated TTS voices.

3 Task and Dataset

NADI 2026 Shared Task 5 evaluates end-to-end SLU for Tunisian Arabic speech using short spoken assistant commands, with separate intent-recognition and slot-filling subtasks. The relatively small, imbalanced SLURP-TN dataset motivates augmentation targeting underrepresented intents through few-shot generation and synthetic speech.

  • Intent recognition: 23 intent labels appear in the released training data, whereas the blind test set follows the full 60-label SLURP intent label set.Intent recognition is evaluated with accuracy, weighted-F1, and additionally macro-F1.
  • Slot filling: Slot filling requires transcriptions with inline slot annotations and is evaluated primarily with CoER and CVER, alongside WER and CER on lexical content.The lexical metrics exclude slot markup, while CoER and CVER measure errors in slot labels and associated values.
  • Dataset: ∼2.8 hours of training speech make SLURP-TN relatively small, with substantial class imbalance across its 23 released intent labels.Six intents have fewer than 10 training examples, three are absent from devtest, and the most frequent intent represents 18.8% of devtest utterances.
  • Data augmentation: The augmentation pipeline generates more examples for less frequent intents among the 23 training labels, using few-shot generation models selected for underrepresented intents.Generated speech is voice-cloned, filtered by duration, signal level, clipping, voiced-frame activity, and speaking rate, and totals 23,300 utterances.

4 System

The system evaluates four instruction-tuned audio LLMs alongside a Whisper-small baseline under a standardized inference setup. The submitted system fine-tunes Qwen3-Omni-30B-A3B for two epochs on combined real and synthetic data.

  • Models: Four instruction-tuned audio LLMs are evaluated: Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, Qwen3-Omni-30B-A3B-Instruct, and gemma-4-E4B-it.Whisper-small is additionally trained as a baseline.
  • Models: Inference uses ms-swift with a vLLM backend on a single H200 GPU and greedy decoding.Appendix D contains the full hyperparameters, model-merging, inference, and Whisper-baseline setup.
  • Final system: Qwen3-Omni-30B-A3B is selected for the submitted system and fine-tuned for two epochs on combined real and synthetic data.This configuration is identified as the final system.

5 Results

Fine-tuning substantially improves all four omni models on intent recognition and slot filling, while the final system ranks first in official-test slot filling and fourth in intent recognition. The final system achieves 66.1% intent accuracy and 59.5 CoER on the official test set, compared with 86.8% intent accuracy on dev-test.

  • Official test results: 59.5 CoER ranked the final system 1st in official-test slot filling, while 66.1% accuracy ranked it 4th in intent recognition.The system also achieved 94.2 CVER and 66.9 weighted F1-score on the official test set.
  • Effect of LoRA fine-tuning: Intent accuracy increased from 29.2-53.1% to 80.4-82.9% after LoRA fine-tuning, while slot CoER decreased from 97.5-150.1 to 47.7-57.0.Fine-tuning consistently improved all four omni models on both subtasks.
  • Effect of LoRA fine-tuning: 82.9% intent accuracy and 47.7 CoER made Qwen3-Omni-30B the best-performing model after fine-tuning.Fine-tuning also reduced the performance gap across model sizes.

6 Conclusions and Future Work

The study evaluates Tunisian Derja spoken-language understanding in both NADI 2026 Shared Task 5 subtasks and finds that fine-tuning and synthetic speech augmentation substantially outperform zero-shot audio LLM inference. The final system achieves 86.8% intent accuracy and 34.7 WER, ranking first in slot filling and fourth among eight teams.

  • The authors participated in both subtasks of NADI 2026 Shared Task 5.
  • 82.9% intent accuracy and 40.0 WER are achieved by fine-tuning on fewer than three hours of real speech.
  • 86.8% intent accuracy and 34.7 WER result when synthetic speech augments the training data.
  • Zero-shot audio LLM performance remains limited, and increasing model scale alone does not overcome this limitation.
  • The final system ranks 1st in slot filling and 4th of eight teams.

Limitations

The evaluation is centered on the SLURP-TN dev-test split, with only the final setup tested on the official blind set. Limited intent-label coverage and model-specific augmentation experiments constrain generalization and analysis.

  • Limitations: Evaluation primarily uses the SLURP-TN dev-test split, while only the final setup is evaluated on the official blind test set.
  • Limitations: Training and dev-test splits cover 23 intent labels, compared with 60 labels in the official test set, challenging open-intent generalization.
  • Limitations: Synthetic augmentation is evaluated only with Qwen3-Omni-30B-A3B due to computational constraints; smaller models and human speech validation remain future directions.

A Additional Results

Additional results show that fine-tuning improves all omni models across intent recognition and slot-filling metrics, with Qwen3-Omni-30B-A3B performing strongest overall. For this model, combining real and synthetic speech outperforms either source alone, while extending training from 2 to 2.5 epochs yields only marginal gains.

  • Detailed results: Fine-tuning improves all omni models across intent-recognition and slot-filling evaluation metrics.Intent recognition uses accuracy, macro-F1, and weighted-F1, while slot filling uses WER, CER, CoER, and CVER.
  • Detailed results: Qwen3-Omni-30B-A3B achieves the strongest overall performance across the evaluated models.The comparison covers zero-shot and fine-tuned settings across different data splits.
  • Synthetic augmentation: Combining real and synthetic speech performs better than using either source alone.Table 4 analyzes synthetic augmentation for Qwen3-Omni-30B-A3B, with Mix denoting combined real and synthetic training data and Synth denoting synthetic data only.
  • Training duration: Increasing training from 2 to 2.5 epochs provides only marginal gains.This training-duration analysis is conducted for Qwen3-Omni-30B-A3B.

B Error Analysis

The error analysis shows that zero-shot systems mainly struggle with slot-markup formatting and fine-grained intent distinctions, while fine-tuning and synthetic speech augmentation substantially reduce errors. Augmentation is especially beneficial for less frequent intents and slot types, improving coverage of underrepresented labels.

  • Zero-shot slot-filling errors: Fewer than 11% of zero-shot outputs use the expected <label> value > structure, with missing markup the dominant error.Fine-tuning brings markup rates close to the gold distribution and largely resolves these formatting errors.
  • Effect of data augmentation: 3.9 points of intent accuracy improvement and 10.8 points of CoER reduction result from augmentation on the dev-test set.Fine-tuning on real speech corrects many zero-shot errors, and synthetic speech adds further gains on both subtasks.
  • Effect of data augmentation: Less frequent intents and slot types benefit substantially more from augmentation than common ones.The pattern suggests synthetic data primarily improves coverage of underrepresented labels.

C Data Augmentation Pipeline

The augmentation pipeline filters generated utterances for validity, linguistic content, length, duplication, and speech quality, then applies LLM majority voting before retaining data. Augmentation also improves intent-label balance, raising macro-F1 from 80.3 to 85.8, especially for less frequent labels.

  • Deterministic filtering: Deterministic filters retain utterances with valid annotations, aligned text, valid labels, sufficient Arabic-script content, 2–25 tokens, and no near-duplicates.These filters reduce malformed, out-of-domain, and repetitive generations.
  • Speech-quality filtering: Generated speech can be rejected when it is unusually long for the utterance and fails the speaking-rate filter, even if the text is accepted.Accepted examples must pass text validation and acoustic filters for both synthesized renditions.
  • LLM roles: Gemini 3.6 Flash generates most data, Gemini 3.1 Pro targets lower-resource intents and reference ASR checks, and three models judge candidates independently.An utterance is retained when at least two judges accept it.
  • LLM roles: 12,138 of 13,876 candidates reaching LLM validation are retained by the 2-of-3 majority vote.The majority-vote stage follows generation and filtering.
  • Effect on label coverage: Macro-F1 improves from 80.3 to 85.8 across 13 intent labels with sufficient dev-test instances, with the largest gains among less frequent labels.Frequent intents such as weather_query and news_query change only slightly.

D Hyperparameters · E Prompts

The system uses parameter-efficient or full fine-tuning with fixed training and decoding settings, alongside task-specific prompts for intent recognition and slot filling. The official intent prompt expands to the full SLURP label inventory while preserving the model and adding guidance for unseen scenarios.

  • D Hyperparameters: 10−4 learning rate and effective batch size 8 are used while the audio encoder and audio-text aligner remain frozen.LoRA is applied only to attention projections of the language backbone.
  • D Hyperparameters: Two epochs are used for all fine-tuned systems, with Whisper-small fully fine-tuned and [INTENT] or [SLOT] prepended during decoding.After fine-tuning, the LoRA adapter is merged into the base model and decoding uses greedy decoding with the same vLLM serving setup.
  • E Prompts: Separate prompts are used for intent recognition and slot filling, with the intent prompt extended for the official intent-recognition test set’s broader label inventory.The official test-set change modifies only the prompt and keeps the model unchanged.
  • E.1 Intent Recognition: The intent prompt requires exactly one label from a fixed 23-label inventory and returns it in a single-line JSON object.It is used for all SLURP-TN training and dev-test experiments reported in Tables 3 and 4.
  • E.2 Slot Filling: Slot filling follows the organizers’ reference prompt, producing one exact transcription line with semantic slots marked inline as <label> slot value >.The format is shared across zero-shot and fine-tuned models and supports the same evaluation pipeline.
  • E.3 Official Test Set: The official intent prompt expands the inventory from 23 released training labels to the full 60-label SLURP inventory and permits the exact answer unknown.The prompt identifies the six training scenarios and discourages mapping unseen intents to familiar in-domain labels.
  • E.3 Official Test Set: The official test-set instructions emphasize classifying the full inventory across music, calendars, lists, IoT, transport, cooking, social media, question-answering, and audio-volume scenarios.They specifically prohibit using general_quirky as a catch-all and require valid single-line JSON output.
Loading 2608.18689v1…