PaperlayerView exact revision on arXiv

Source-linked AI summary

SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

Paul-Ambroise Duquenne, Holger Schwenk, Benoît Sagot

arXiv:2308.11466v2cs.CL

TL;DR

SONAR addresses the challenge of learning fixed-size sentence representations that work across many languages and modalities while supporting translation. It combines multilingual text and speech encoders with a 200-language decoder, achieving strong similarity-search and translation results, including zero-shot speech-to-text translation.

  • Problem

    Existing sentence representation methods motivate learning a multilingual, multimodal fixed-size space that supports semantic comparison and translation across languages and modalities.

  • Method

    SONAR trains an encoder-decoder text space with translation, auto-encoding, denoising, and similarity objectives, then trains speech encoders with teacher-student learning.

  • Results

    SONAR provides a 200-language text encoder and decoder, speech encoders in the same space, and zero-shot text and speech translation capabilities.

  • Takeaways & Limitations

    A fixed-size bottleneck can support multilingual decoding and language-agnostic, cross-modal representations, including zero-shot speech-to-text translation.

  • Takeaways & Limitations

    Speech teacher-student training uses transcriptions as targets; using written translations is left for future work.

Abstract

from arXiv · show

We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space. Our single text encoder, covering 200 languages, substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. Our encoders outperform existing speech encoders on similarity search tasks. We also provide a text decoder for 200 languages, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations. Our text-to-text results are competitive compared to the state-of-the-art NLLB~1B model, despite the fixed-size bottleneck representation. Our zero-shot speech-to-text translation results compare favorably with strong supervised baselines such as Whisper.

1 Introduction

SONAR builds a multilingual, multimodal fixed-size sentence embedding framework by combining an encoder-decoder text model with teacher-student speech encoders. It targets cross-lingual and cross-modal similarity search and zero-shot translation across 200 languages.

  • 1 Introduction: SONAR is designed to address limitations of sentence representation methods that often lack sentence-level objectives for evaluating semantic similarity.Prior multilingual approaches include encoder-only, encoder-decoder, and teacher-student strategies.
  • 1 Introduction: The text embedding space is trained with an encoder-decoder approach combining translation, auto-encoding, denoising, and cross-lingual similarity objectives.The architecture is initialized from the NLLB 1B model, and the objectives are studied for their effects on embedding and decoding properties.
  • 1 Introduction: Speech encoders extend the multilingual text space through teacher-student training, using the text encoder as teacher and speech transcription data as targets.The resulting framework is evaluated for cross-modal similarity search and speech translation.
  • 1 Introduction: SONAR provides a 200-language text encoder, speech encoders trained for 37 languages, and a 200-language text decoder for multilingual and multimodal representation and translation.The framework supports text and speech embeddings in a shared space and enables zero-shot text and speech translation.
  • 1 Introduction: The project reports strong performance against existing sentence embedding approaches and makes its text and speech encoders and text decoders freely available.The contributions include analysis of cross-lingual and cross-modal similarity search and decoding capabilities.

2 Related work

Related work develops multilingual fixed-size sentence representations through encoder-only, encoder-decoder, and teacher-student approaches, alongside emerging multilingual speech-text embeddings. SONAR follows the encoder-decoder and teacher-student lines while extending them to a shared multimodal space.

  • 2 Related work: Multilingual sentence representations use encoder-only, encoder-decoder, or teacher-student architectures to learn fixed-size embeddings across languages.These approaches use objectives such as contrastive learning, translation, or regression to align translated sentences.
  • 2 Related work: LASER uses an encoder-decoder translation objective and originally covered 93 languages, while LASER3 extended coverage through teacher-student training and LaBSE covered 109 languages.Learned decoders for LASER representations also enabled zero-shot machine translation on unseen language directions.
  • 2 Related work: Prior work also explored multilingual and multimodal speech-text pre-training and sentence embeddings, including mSLAM and extensions of existing sentence spaces.These efforts place SONAR in a broader line of multilingual speech and text representation research.

3 Methodology

SONAR builds a multilingual text sentence space with a Transformer encoder-decoder bottleneck, then extends it to speech through teacher-student training. Its design combines translation, denoising, embedding-alignment, and decoder-finetuning strategies to balance semantic organization with decoding quality.

  • 3.1 Multilingual sentence representations for text: SONAR first learns a multilingual text embedding space with a Transformer encoder-decoder whose full-sentence representation passes through a single fixed-size vector bottleneck.Unlike standard sequence-to-sequence translation, the architecture does not use token-level cross-attention.
  • 3.1 Multilingual sentence representations for text: The text model investigates translation, auto-encoding, denoising auto-encoding, and MSE objectives to shape both sentence-space organization and decoding performance.The translation objective uses parallel data, while MSE explicitly brings translations closer in the embedding space.
  • 3.1 Multilingual sentence representations for text: Decoder finetuning improves decoding while preserving the learned embedding space, because the encoder remains frozen during task-specific decoder training.This preserves the representation space while allowing compatible decoders to improve translation and auto-encoding.
  • 3.1 Multilingual sentence representations for text: Random interpolation decoding freezes the encoder and trains the decoder on interpolations between source and target embeddings, combining translation and auto-encoding.Given bitext x, y, the method samples an embedding z between their frozen representations and learns to decode z into y.
  • 3.2 Multilingual sentence representations for speech: The speech extension trains student encoders to map audio into SONAR using transcription sentence embeddings as teacher targets and an MSE loss.Speech encoders are initialized from self-supervised w2v-bert models, while written translations are left for future work as targets.

4 Evaluations

SONAR is evaluated through complementary text and speech tasks that test cross-lingual similarity, semantic robustness, translation, and content preservation. Speech evaluations additionally test cross-modal retrieval and zero-shot decoding with text decoders.

  • Text evaluations: The evaluation suite measures cross-lingual similarity with xsim and the harder xsim++ task, alongside translation and same-language auto-encoding.xsim++ adds hard negatives involving causality, entities, and numbers, while auto-encoding tests content preservation.
  • Text evaluations: Text translation is evaluated with spBLEU and COMET, complementing similarity search by testing how much sentence information can be decoded into target languages.The paper cautions that decoder language-modeling capabilities may restore information not encoded in the sentence embedding.
  • Datasets: Text evaluations use the FLORES-200 devtest set, an N-way parallel corpus covering 200 languages.The speech evaluations use FLEURS, an N-way parallel speech dataset in 102 languages built on FLORES-101.
  • Speech evaluations: Speech xsim evaluates speech embeddings against English text translations, while speech xsim++ adds challenging modified English negatives.These tasks test cross-modal and cross-lingual retrieval rather than speech-only similarity.
  • Speech evaluations: Zero-shot speech-to-text translation combines speech encoders trained only on ASR data with text decoders that never saw speech embeddings during training.Speech recognition is also evaluated by decoding speech embeddings into the same language.

5 Experiments on text

Text experiments show that the fixed-size bottleneck retains strong translation and similarity-search performance, while objective choices create a trade-off between decoding and embedding organization. The final SONAR configuration and decoder finetuning improve this balance across multilingual evaluations.

  • 5.5 Training the SONAR embedding space: Approximately 45% relative xsim++ error-rate reduction over LaBSE and LASER3 is reported across the 200-language focus set.The paper also reports significantly better xsim and xsim++ results than these prior sentence-embedding systems.
  • 5.3 Experiments with auto-encoding objectives: +13 BLEU from denoising auto-encoding improves auto-encoding over the translation-only model, but significantly harms xsim and xsim++ scores.Scaling the denoising objective down mitigates translation losses while retaining much of the auto-encoding gain.
  • 5.5 Training the SONAR embedding space: The final SONAR setup limits denoising-loss degradation to 0.9% in xsim++ while improving translation over the MT-plus-MSE model and auto-encoding by more than 3.8 BLEU.The authors identify this variant as the best overall balance of sentence-space organization and decoding performance.
  • 5.5 Training the SONAR embedding space: +9.3 BLEU in auto-encoding follows decoder finetuning, while eng-X translation gains 0.9 BLEU and X-eng translation remains similar.Because the encoder is frozen, xsim and xsim++ do not change during this finetuning step.
  • 5.4 Translation evaluation: Less than 1 COMET point separates the finetuned SONAR model from NLLB 1B in both X-eng and eng-X translation directions.The comparison averages COMET over 89 languages supported by both COMET and NLLB 1B.
  • 5.5 Training the SONAR embedding space: SONAR supports encoding and decoding for 200 languages with a single encoder and decoder while improving both X-eng translation and xsim++ over the compared T-modules framework.The comparison uses FLORES-200 X-eng translation and xsim++ evaluations.

6 Experiments on speech

SONAR extends a multilingual text embedding space to speech through teacher-student training, enabling cross-modal similarity search and zero-shot speech translation. Across evaluations, it remains competitive with or outperforms strong speech baselines, including Whisper in several settings.

  • 6.1 Experiments on 5 languages: Attention-pooling achieves the best speech-embedding results among the tested mean-, max-, and attention-pooling methods.Attention-pooling uses a three-layer transformer decoder with cross-attention over speech encoder outputs to produce one vector.
  • 6.1 Experiments on 5 languages: 41% relative reduction in xsim++ error rate results from switching from LASER to SONAR as the teacher for speech encoders.The comparison uses FLEURS speech-to-English embeddings with hard negatives, because ordinary xsim scores saturate and provide limited insight.
  • 6.1 Experiments on 5 languages: Zero-shot speech-to-text results are close to supervised Whisper Large v1 for French and Spanish and significantly outperform Whisper models for Swahili.Russian-to-English favors Whisper, as expected from its greater training data and supervised setting.
  • 6.1 Experiments on 5 languages: SONAR decodes speech embeddings from English, French, Spanish, Swahili, and Russian into 200 text languages, with high BLEU scores beyond English.The results use the SONAR text decoder in a zero-shot speech-translation setting.
  • 6.1 Experiments on 5 languages: Zero-shot transcription trails Whisper on BLEU but narrows the gap under BERTScore because SONAR often paraphrases transcriptions.Both BLEU and BERTScore are reported for same-language speech decoding.
  • 6.2 Scaling to 37 languages: Across 37 languages, SONAR's average BLEU is slightly better than Whisper v2 large in zero-shot speech translation.SONAR is weaker for some high-resource languages but outperforms Whisper for Spanish, Dutch, Swahili, Uzbek, and several Indian languages.

7 Discussion

The discussion identifies objective-function trade-offs in SONAR's fixed-size multilingual representation and argues that teacher-student training transfers the space effectively to speech. Cross-modal compatibility is demonstrated by zero-shot decoding of speech into foreign text.

  • 7 Discussion: Translation objectives build language-agnostic representations while preserving information that can be decoded into another language.Adding MSE improves language alignment, while denoising auto-encoding can improve decoding but excessive denoising harms language-agnostic representations.
  • 7 Discussion: Fixed-size bottlenecks do not prevent strong auto-encoding because substantial information can be stored in a single vector.This conclusion follows from the observed ability to solve the auto-encoding task with a fixed-size encoder-decoder interface.
  • 7 Discussion: Speech embeddings can be decoded into foreign text without speech examples entering decoder training, highlighting compatibility between multilingual speech and text representations.The speech modality is added through teacher-student training, while the decoder operates on the shared embedding space.

8 Conclusion

SONAR is introduced as a multilingual and multimodal sentence embedding space with text and speech encoders linked to text decoders. The framework is evaluated for both similarity search and decoding across languages and modalities.

  • 8 Conclusion: SONAR combines a multilingual teacher-studied text sentence embedding space with a speech extension and text decoders.The acronym denotes Sentence-level multimOdal and laNguage-Agnostic Representations.
  • 8 Conclusion: The framework is evaluated on similarity-search and decoding tasks across text and speech modalities.The encoders and text decoders are made freely available by the authors.
Loading 2308.11466v2…