Source-linked AI summary
Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding
Gabriel Pirlogeanu, Dan Oneata, Horia Cucu, Herman Kamper
TL;DR
The paper addresses how to map written English keywords to spoken Hindi words when visually grounded speech lacks target-language transcriptions. It uses image captions for weak textual supervision and aligns self-supervised speech representations, finding that the alignment approach outperforms a neural baseline and benefits from negative evidence.
Problem
The study asks whether visually grounded speech can document a foreign language by linking English written words to spoken Hindi segments.
Method
The approach generates English image captions, partitions Hindi utterances into positive and negative sets, and aligns self-supervised speech representations to retrieve recurring target-word segments.
Results
The best continuous-feature setup outperforms a neural baseline by roughly 39% absolute for P@10 keyword localization and 44% for P@10 keyword spotting.
Takeaways & Limitations
Visually grounded speech can support cross-lingual vocabulary construction without textual resources in the target language.
Abstract
from arXiv · showhide
In many low-resource settings, even just eliciting speech for data collection is difficult. One promising approach has been to ask speakers to describe images. But how do we build models from such visually grounded speech data? Given a dataset of images with Hindi spoken captions, we consider how we can map a written English keyword to spoken realisations of that word in Hindi. Previous work trained end-to-end multimodal neural models. Instead, we explore a simpler alignment-based approach built on self-supervised speech representations. Written English tags are automatically obtained from images using off-the-shelf image captioning systems. Hindi utterances associated with the same keyword are then aligned (using self-supervised features), and alignment evidence is aggregated to identify recurring speech segments corresponding to the target word. Experiments evaluating keyword spotting and localization show that our alignment-based approach outperforms a previous attention-based neural model. We also show the benefit of incorporating negative examples during alignment. Our work demonstrates that cross-lingual word-to-speech mappings can be learned directly from visual grounding without transcriptions or explicit model training.
I. INTRODUCTION
The paper asks whether visually grounded speech can support cross-lingual documentation by linking written English words to spoken Hindi segments. It develops an alignment-based alternative to neural multimodal models and evaluates representation choices, captioning systems, and errors.
- Visually grounded speech collection offers a way to gather data for languages lacking writing systems or living in low-literacy communities.
- The study discovers Hindi audio segments corresponding to Hindi words and links them to English written words using image-associated captions.
- English vocabularies are generated from automatic image captions, which provide visual supervision for partitioning utterances into keyword-positive and keyword-negative sets.
- The alignment variants use continuous or discrete self-supervised speech representations, and the continuous approach achieves the best overall performance.
- The paper extends prior monolingual work to Hindi speech and English keywords, adds negative information, and provides broader analyses.
II. RELATED WORK
Prior research connects speech, images, and languages through shared visual grounding, while this work specifically combines automatic image captions with keyword localization across languages. Its distinction is using generated English text rather than an English spoken caption as the bridge.
- Connecting Speech to Images: Visually grounded speech models commonly learn shared audio–visual representations or align spoken segments with image objects and regions.
- Connecting Languages via Images: Vision has supported cross-lingual speech retrieval and word mapping because it provides a common grounding signal between languages.
- Connecting Languages via Images: Unlike prior cross-lingual speech associations based on two spoken captions, this work automatically generates an English text description from the image.
- Connecting Speech to Words via Images: Earlier approaches connected speech and written words through image tags or automatic captions, including keyword localization with image-based supervision.
- Connecting Speech to Words via Images: The paper combines caption-based speech–word mapping with keyword localization to guide cross-lingual localization of particular keywords.
B. Mining Utterances Using Visual Information
The method uses automatically generated image captions to mine utterances likely to contain a queried English word, while contrasting them with utterances lacking that word or containing co-occurring words.
- Positive and Negative Mining: For a query word, utterances are divided into positive and negative sets according to whether the associated image caption contains the word.
- Positive Mining: Positive mining focuses the search on audio utterances likely to contain the queried word through image-based supervision.
- Negative Mining: Negative mining supplies contrastive evidence to suppress frequently co-occurring words and common function words that could dominate the positive set.
- Negative Mining: Random negative samples are drawn from the nonmatching set with the same size as the positive set or at least 50 samples.
- Semantically Negative Mining: Semantic negative mining specifically targets utterances containing words that co-occur with the query, using the two most frequent co-occurring vocabulary words.
C. Aligning Utterances
The alignment stage searches for recurring audio subsequences across utterances using either discrete cluster codes or continuous self-supervised features. These alignments provide the evidence later aggregated for keyword retrieval.
- The method aligns positive utterances with other positive and negative utterances to identify recurring audio subsequences.
- For segment t in utterance a_i, the scoring function estimates how likely that segment is to appear anywhere in utterance a_j.
- Continuous alignment produces scores in [0, 1], whereas discrete alignment produces a binary signal from cluster codes.
- Discrete Features Alignment (DFA): Discrete feature alignment converts HuBERT representations into k-means cluster units, applies Smith–Waterman alignment, and produces binary match scores.
- Continuous Features Alignment (CFA): Continuous feature alignment compares HuBERT features with cosine similarity, smooths the resulting signal, and thresholds low values to reduce noise.
D. Ranking Segments by Aggregation
The method ranks candidate speech segments by aggregating alignments with positive examples while penalizing alignments with negative examples, then selects high-scoring regions as query-word instances.
- Positive-pair alignments increase each audio sample’s score, while alignments with negative samples decrease it.
- Negative mining suppresses frequent but non-informative segments, including co-occurrences and function words.
- The no-negatives variant uses only positive alignments and serves as an experimental baseline.
- After thresholding aggregated scores, contiguous speech regions are delimited by silence and scored using average frame-level values.
- High-scoring segments are selected as candidate instances of the query word in the foreign-language speech.
A. Data and Groundtruth
The evaluation uses Hindi image-caption speech, automatically constructed groundtruth, and a vocabulary derived from intersecting outputs of three image captioners.
- Data: The MIT Place Audio Captions Hindi dataset contains 100k images paired with spontaneously spoken Hindi descriptions.The experiments sample 20k speech–image pairs, split evenly between development and final evaluation; selected samples are at most seven seconds long.
- Groundtruth: Automatic Hindi transcriptions are obtained with NVIDIA NeMo’s Conformer Large CTC model to construct evaluation groundtruth.The cited model achieves a 9.4% word error rate on MUCS 2021.
- Vocabulary: The vocabulary is the intersection of words generated by Tag2Text, BLIP-2, and GIT for each image.Captions are processed by stopword removal and lemmatization before intersection; the most frequent 100 visually grounded words are retained.
- Evaluation: Keyword localization requires sufficient overlap with a Hindi translation, using intersection over union above 0.5.Keyword spotting instead checks whether any valid Hindi translation appears anywhere in the utterance.
- Evaluation: The two retrieval metrics report performance over the top 10 retrieved audio segments for each vocabulary word.Scores are reported as the number of correct predictions, P@10, averaged across vocabulary words.
D. Alternative Approach and Toplines
The study compares alignment variants, supervision settings, and mining strategies for cross-lingual keyword spotting and localization. Continuous alignment performs best, negative mining consistently helps, and weak visual supervision remains the main performance constraint.
- Experimental settings: Table II compares transcript toplines and visually grounded systems using CFA or DFA with positive, negative, or semantic-negative mining.The visually grounded systems use image-caption supervision, while toplines use transcripts.
- Continuous Features Perform Best: CFA consistently outperforms DFA, but aligning 250 audio clips takes 2m15s with CFA versus 24s with DFA.The paper attributes CFA’s performance advantage to retaining information discarded by discretization.
- Negative Mining Consistently Helps: Negative mining improves both alignment variants across topline and visually grounded settings, with unrestricted negatives slightly stronger than semantic negatives.In visually grounded localization, DFA rises from 16.8% to 34.6%, while CFA rises from 23.6% to 49.9%.
- Negative Mining Consistently Helps: Negative mining yields larger gains for localization than keyword spotting because it suppresses co-occurring words that mainly affect localization.CFA improves from 23.6% to 49.9% in localization and from 47.7% to 63.0% in spotting.
- Weak Supervision is the Primary Source of Performance Loss: Transcript supervision produces perfect spotting and nearly 90% CFA localization, indicating that weak supervision is the primary source of performance loss.Image captions may mismatch spoken content, including human-generated captions, with additional cultural and linguistic gaps in this setting.
B. The Cross-Lingual Gap
The cross-lingual setting introduces a performance gap partly because Hindi and English speakers describe the same images differently. These differences include broader or omitted category labels in Hindi captions, while imperfect image-caption supervision remains another source of error.
- Cross-linguality Partially Explains the Performance Gap: Cross-lingual performance is lower partly because Hindi and English speakers describe the same images differently.The paper compares Hindi–English with English–English settings to quantify this gap.
- Cross-linguality Partially Explains the Performance Gap: 75.4% visually grounded English–English localization approaches the 89.8% English–English transcript topline, compared with 49.9% cross-lingual localization.The remaining monolingual gap is attributed to imperfect image-caption supervision.
- Cross-linguality Partially Explains the Performance Gap: 89.8% cross-lingual transcript localization is close to 94.3% monolingual transcript localization, suggesting speech-representation language is not a crucial factor in the idealized setting.The idealized comparison removes image-captioning effects.
- English and Hindi Speakers Describe Images Differently: Hindi annotations align better with automatic captions than English annotations do, indicating that the mismatch is fundamental to human image descriptions rather than solely a captioning-system problem.Table IV measures precision and recall over 10k development images and 100 vocabulary words.
- English and Hindi Speakers Describe Images Differently: Hindi speakers often use broader terms or omit specific sports and scene labels that English speakers name explicitly.Examples include game instead of baseball, golf, or bowling; old building instead of castle; and table or chair instead of restaurant.
VI. FURTHER ANALYSES
The analyses examine how captioning-system choices and word-level factors affect localization, including the contrast between positive and negative mining.
- Word-Level Error Analysis: The further analyses vary captioning systems and inspect how sample counts, captioning precision, and co-occurrence affect word-level performance.Figure 5 reports these factors for CFA with negative mining, alongside regression fits and confidence intervals.
- Impact of Image Captioning Systems: Negative mining remains strong across captioning-system combinations, while positive mining is substantially more sensitive to that choice.Negative mining yields 46.7%–49.9% localization performance; positive mining improves when multiple captioners are combined.
B. Impact of Speech Representations
The paper evaluates speech representations and then analyzes word-level factors associated with localization performance. Representation choice matters, while caption quality and co-occurrence patterns explain substantial variation across words.
- Impact of Speech Representations: 49.9% P@10 is achieved by HuBERT Base with CFA and negative mining on Hindi localization.Larger HuBERT and WavLM variants degrade performance, while mHuBERT reaches 50.7% P@10 and Hindi wav2vec 2.0 reaches 44.4% P@10.
- Word-Level Error Analysis: Word-level localization correlates with caption count, although rare words can perform well with fewer than 50 captions.The relationship is positive but weak, especially at the low end.
- Word-Level Error Analysis: Localization has a strong positive relationship with captioning precision, while frequent co-occurring words reduce performance.Examples include confusion involving bus, bed, bridge, and the phrase “body of water.”
VII. CONCLUSIONS
The paper maps Hindi spoken captions to English written words through visually grounded alignment with positive and negative evidence. Continuous-feature alignment outperforms the neural baseline, while transcript supervision and cultural differences remain boundaries.
- Conclusions: Hindi spoken captions are mapped to English written words by aligning utterances selected into positive and negative sets from image-caption keywords.Positive alignments provide supporting evidence and negative alignments provide contrasting evidence for retrieving spoken segments.
- Conclusions: 39% absolute for P@10 in keyword localization and 44% for P@10 in keyword spotting are achieved over the neural baseline.Continuous-feature alignment performs best overall, while discrete features are more computationally efficient but worse.
- Limitations: A performance gap remains relative to transcript-supervised toplines because image descriptions vary and English–Hindi speakers describe images differently.The study uses images of interest to Westerners annotated by Hindi speakers.
- Conclusions: Visually grounded speech can support cross-lingual vocabulary construction without textual resources in the target language.The authors frame this as support for language technology and preservation.