PaperlayerView exact revision on arXiv

Source-linked AI summary

Indic-CLAP: Text--Audio Understanding in Indic Languages via Cross-Lingual Distillation

Sanat Kumar Agrawal, Srikanth Raj Chetupalli

arXiv:2610.08846v1eess.AS

TL;DR

CLAP’s English-only text encoder limits its use for audio–text tasks in Indic languages, where the authors report no public Indic-captioned audio datasets or existing models. Indic-CLAP distills a multilingual text encoder from frozen CLAP teachers using translated captions. The hybrid objective performs best on retrieval, and the authors relate the improvements to reduced modality gap.

  • Problem

    CLAP uses English captions and an English-only text encoder, while the authors report no public audio datasets with Indic captions or models for Indic audio–text tasks.

    Source: §1 · p. 1

  • Method

    Indic-CLAP trains a multilingual Indic text encoder by distilling from CLAP’s text and audio encoders using translated captions, while keeping the audio encoder frozen.

    Source: §1 · p. 1, §2.3 · p. 2

  • Results

    The hybrid objective gives the best cross-modal retrieval, while TT-MSE performs better for zero-shot classification; hybrid retrieval can exceed the teacher on in-domain audio.

    Source: §4.3 · pp. 3–4, §5 · p. 4

  • Takeaways & Limitations

    The authors attribute retrieval improvements to a reduced modality gap and report that cross-lingual teacher learning also extends to GLAP.

    Source: §5 · p. 4

    Translation quality is a key challenge: Helsinki-NMT translations yield poorer distillation fidelity, zero-shot classification, and retrieval than GPT-4o-mini translations.

    Source: §4.1 · p. 3

Abstract

from arXiv · show

Contrastive Language--Audio Pretraining (CLAP) learns a joint text--audio representation space, which enables zero-shot audio understanding from natural-language descriptions. However, its English-only text encoder restricts CLAP to tasks and datasets expressed in English. We present Indic-CLAP, a multilingual text--audio model that extends CLAP to nine Indic languages spoken by over a billion people. Exploiting CLAP's dual-encoder structure, we propose to train only an Indic text encoder by distillation while keeping the audio encoder frozen, using machine-translated AudioCaps captions. We compare distillation from CLAP's text encoder, audio encoder, and a novel hybrid objective combining both. Our evaluation shows that the Indic-CLAP encoder inherits the cross-modal alignment from the CLAP teacher, as evidenced by the performance on cross-modal retrieval and zero-shot classification tasks. Further, we study the cross-modal alignment using modality gap analysis, which explains the performance trends.

1. INTRODUCTION

CLAP supports zero-shot audio tasks through a shared text–audio space, but its English-only encoder and training data limit its language coverage. Indic-CLAP extends this approach to nine Indic languages, addressing a gap in Indic audio–text resources and models.

1. INTRODUCTION

  • CLAP maps audio and natural-language descriptions into a joint representation space used for zero-shot classification, tagging, captioning, retrieval, and generation.Its text branch uses English-only RoBERTa, and its training pairs contain English captions.Source: §1 · p. 1
  • No public audio datasets with Indic-language captions or models trained and evaluated for Indic audio–text tasks were identified by the authors.They describe audio–language tasks in Indic NLP as largely unexplored.Source: §1 · p. 1
  • Indic-CLAP extends CLAP to nine Indic languages by distilling into a multilingual text encoder, with joint training, hybrid distillation, translation quality, and GLAP extension also studied.The authors report that joint training improves downstream performance over language-specific models, while the hybrid objective gives the best cross-modal retrieval.Source: §1 · p. 1

2. METHODOLOGY

Indic-CLAP trains an Indic text encoder to place translated captions in CLAP’s shared embedding space while keeping the teacher encoders frozen. Its objectives use the CLAP text encoder, audio encoder, or both to guide the student.

2.1. Cross-Lingual Teacher Learning

  • Translated Indic captions train Tin to produce embeddings in CLAP’s shared space, where they are intended to align with the corresponding audio and English-caption embeddings.The CLAP embeddings are precomputed and cached, so audio is not processed during student training.Source: §2.1 · p. 1, Figure 1 · p. 2

2.2. Indic Text Encoder (Tin)

  • Tin uses pretrained XLM-RoBERTa-base with masked mean pooling, a linear projection, and L2 normalization, and is fine-tuned end-to-end.Training only the projection was insufficient in preliminary experiments.Source: §2.2 · p. 2

2.3. Training Objectives

  • TT-MSE trains Tin to mimic CLAP’s text encoder, while AT-MSE trains it to mimic the audio encoder; alignment with the other modality is expected by transitivity.The audio-teacher MSE objective distills across modalities, which the authors describe as a harder problem.Source: §2.3 · p. 2
  • AT-NCE uses symmetric cross-entropy against the frozen audio encoder, while the hybrid objective combines audio and text alignment terms with a weighting parameter λ.The text term uses cosine similarity because it is scale-commensurate with InfoNCE, unlike Euclidean distance.Source: §2.3 · p. 2

3. EXPERIMENTAL SETUP

The experiments use CLAP as the teacher, translated AudioCaps captions for training, and retrieval and zero-shot classification for evaluation. They also describe the test benchmarks and a Hindi translation-quality comparison.

3. EXPERIMENTAL SETUP

  • The teacher is CLAP pretrained on LAION-Audio-630K and AudioCaps and Clotho captions; its audio and text embeddings are precomputed for training.The training data are AudioCaps recordings with manually written English captions, translated for Indic-CLAP.Source: §3 · p. 2
  • Evaluation includes text-to-audio and audio-to-text retrieval on AudioCaps and Clotho, plus zero-shot classification on ESC-50 and UrbanSound8K.Retrieval reports R@1 and R@10; classification reports top-1 accuracy, using test sets unseen during training.Source: §3 · p. 2
  • For language-specific and multilingual Indic-CLAP models trained with TT-MSE, matched-setting CLAP teacher baselines are R@1 24.5% (A→T), 24.2% (T→A), and zero-shot classification accuracy 92.5%.Translated class labels were embedded in natural-language prompt templates for zero-shot classification.Source: Table 1 · p. 3, §3 · p. 3

4. RESULTS

Across the evaluations, hybrid distillation performs strongly on retrieval, while translation quality and training configuration shape Indic-CLAP’s results. Modality-gap analysis relates encoder alignment patterns to retrieval and classification performance.

4.1. Translation Quality

  • GPT-4o-mini translations outperform Helsinki-NMT across measured tasks; audio-context prompting preserves onomatopoeia and event structure, improving alignment with the CLAP teacher.Source: §4.1 · p. 3

4.2. Multilingual vs. Language-Specific Models

  • Multilingual TT-MSE reaches up to 75% retrieval accuracy; A→T exceeds CLAP, while T→A degrades relative to the teacher and ZSC remains about 17 points lower.Source: §4.2 · p. 3

4.3. Training Objectives

  • Hybrid distillation is markedly better on both retrieval tasks, while TT-MSE leads ZSC; audio-teacher-only objectives underperform text-teacher and hybrid objectives across tasks.Source: §4.3 · pp. 3–4, Table 3 · p. 4
  • GLAP has strong zero-shot retrieval but poor ZSC; fine-tuning with translated captions improves all metrics, with TT-MSE ZSC nearing Indic-CLAP and retrieval nearing the GLAP teacher.Source: §4.3 · p. 4
  • On UrbanSound8K and Clotho, hybrid performs best, with at most a 0.5% retrieval difference from TT-MSE; Clotho retrieval trails the teacher, attributed to training-data mismatch.Source: Table 4 · p. 4, §4.3 · p. 4

4.4. Modality Gap Analysis

  • TT-MSE aligns with CLAP’s text encoder but inherits poor audio alignment and favors classification; AT-MSE aligns with audio but degrades both tasks, while hybrid modestly improves alignment.Source: Table 5 · p. 4, §4.4 · p. 4

5. CONCLUSION

Indic-CLAP extends text–audio modeling to nine Indic languages through text-encoder distillation with CLAP’s audio encoder frozen. Hybrid distillation is effective, often surpassing CLAP on in-domain retrieval; the analysis attributes improvements to a reduced modality gap, while translation quality remains a key challenge.

5. CONCLUSION

  • Hybrid distillation using both CLAP encoders is effective and often exceeds the teacher’s performance on retrieval tasks for in-domain audio.The analysis attributes improvements to a reduced modality gap.Source: §5 · p. 4
  • Indic-CLAP is presented as the first text–audio model trained and evaluated for Indic languages, extending CLAP-based audio understanding to nine languages with machine-translated captions.Only the multilingual text encoder is trained; CLAP’s audio encoder remains frozen.Source: §5 · p. 4
  • Translation quality is a key challenge and limits student performance across every measured task.LLM-based translation approaches clearly outperform traditional NMT approaches.Source: §5 · p. 4

Compliance with Ethical Standards

The work used publicly available datasets under their respective licences and describes its label-verification and human-subject practices.

Compliance with Ethical Standards

Loading 2610.08846v1…