Source-linked AI summary
SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languages
Puja Ahmad Habibi, Faiz Assabil Firdaus, Ashvanth S, Ekapol Chuangsuwanich, Pume Tuchinda, Peerat Limkonchotiwat
TL;DR
Southeast Asian languages remain poorly supported by compact multilingual text-vision models, amid regional data and computing constraints. SEA-CLIP-Tiny adapts distillation with Southeast Asian data and multilingual teacher guidance, achieving the strongest average retrieval performance among evaluated students across seven languages.
Problem
Compact multilingual text-vision models provide limited support for Southeast Asian languages, where linguistic diversity, low-resource paired data, and constrained computing are relevant challenges.
Source: §1 · p. 3Method
SEA-CLIP-Tiny adapts a CLIP-KD-style framework for Southeast Asia using regional training data, multilingual teacher guidance, and multi-objective distillation.
Source: §1 · p. 3Results
Across seven Southeast Asian languages, SEA-CLIP-Tiny leads evaluated students at 12.9% R@1, 31.5% R@5, and 42.2% R@10, improving average R@10 over MobileCLIP2 by 12.1 points.
Source: §1 · p. 3Takeaways & Limitations
The results support region-aware training, with general-domain and regional data shown to be complementary for compact multilingual vision-language models.
Source: §7 · p. 14SEA-CLIP-Tiny does not cover all regional languages, dialects, or cultural variations and remains behind large teacher models.
Source: Limitation, Broader Impact, and… · p. 14
Abstract
from arXiv · showhide
Multilingual text-vision embedding models are essential for cross-lingual image-text retrieval, but Southeast Asian languages remain poorly supported due to the region's linguistic diversity and limited data and computing resources. In this paper, we introduce SEA-CLIP-Tiny, a compact multilingual text-vision embedding model for Southeast Asia with fewer than 50M parameters. Our model adapts a CLIP-KD-style framework to Southeast Asian multilingual settings through regional data curation and multilingual teacher guidance. Experiments across seven Southeast Asian languages show that SEA-CLIP-Tiny achieves the strongest average retrieval performance among the evaluated student models, reaching 12.9%, 31.5%, and 42.2% at R@1, R@5, and R@10, respectively. Compared with MobileCLIP2, it improves average R@10 by 12.1 points while using 38.4% fewer parameters and lower measured CPU latency. These results highlight the importance of region-aware training for efficient multilingual text-vision models in Southeast Asia.
1 Introduction
Existing compact multilingual models provide limited support for Southeast Asian languages, despite the region’s linguistic and resource diversity. SEA-CLIP-Tiny adapts distillation to this setting and achieves the strongest average retrieval performance among the evaluated student models across seven languages.
1 Introduction
- Existing multilingual models show strong capabilities in high-resource languages, but their support for Southeast Asian languages is not demonstrated comparably.Source: §1 · p. 2
- Southeast Asia’s language-family and writing-system diversity coincides with low-resource text and paired-image data and constrained computing resources.Source: §1 · pp. 2–3
- SEA-CLIP-Tiny is a 46.11M-parameter model adapting CLIP-KD with multilingual teacher guidance, Southeast Asian data, and multi-objective distillation.Source: §1 · p. 3
- 12.9% R@1, 31.5% R@5, and 42.2% R@10 make SEA-CLIP-Tiny the strongest average-retrieval student across seven Southeast Asian languages.Source: §1 · p. 3
- Against MobileCLIP2, average R@10 improves by 12.1 points, with 38.4% fewer parameters and lower measured CPU latency.Source: §1 · p. 3
2 Related Works
Prior work advances multilingual CLIP transfer and a broad Southeast Asian AI landscape, but compact text-vision embedding specialized for the region remains underexplored.
2.1 Multilingual Text-vision Models
- Multilingual CLIP methods use multilingual encoders, translated captions, or distillation, while larger models often focus on high-resource languages, leaving compact SEA models underexplored.Source: §2.1 · p. 4
2.2 Southeast Asian Multimodal Research Landscape
- Southeast Asian research spans data, modeling, evaluation, multimodality, and safety, but text-vision embedding remains less explored than regional text-only embedding.Source: §2.2 · p. 4
3 Method
SEA-CLIP-Tiny’s method combines Southeast Asian data selection with a training framework that applies multilingual-teacher guidance and multiple distillation objectives alongside CLIP training.
3.1 Overview
- The overview separates SEA-CLIP-Tiny into data selection and a training framework, described in Sections 3.2 and 3.3.Source: Figure 3 · p. 5, §3.1 · p. 5
3.2 Training Data
- The SEA-focused mixture uses Cultural Ground Open-Ended (703K), WIT-HF (488K), BLOOM (22K), and SEA-Mammoth (540K), alongside general-domain CC12M.For SEA-Mammoth, the question and answer are concatenated as text and paired with the corresponding image; the paper reports SEA-focused data improves multilingual retrieval, while CC12M helps preserve broader visual recognition.Source: §3.2 · p. 5
3.3 Training Framework
- The multilingual teacher uses XLM-V-Base tokenization, while the compact student retains CLIP-BPE; the student averages 60.4 tokens per caption versus 20.0 for the teacher.The student tokenizer has high truncation rates for Vietnamese (38.9%), Thai (24.8%), and Myanmar (95.2%).Source: Tokenizer Settings · pp. 5–6
- Training combines standard CLIP alignment with MSE embedding matching, ICL student-anchor/teacher-target contrast, and CRD of teacher–student contrastive relations.Source: Training Loss · pp. 6–8
- The teacher is frozen, and training updates only the student encoders and projection layers while combining CLIP and three weighted distillation losses.Source: Training Loss · p. 8
4 Experimental Setup
SEA-CLIP-Tiny is trained with multilingual teacher guidance and evaluated on retrieval, zero-shot classification, and localized VQA benchmarks. Comparisons include controlled and publicly released compact-model baselines, with benchmark coverage and translated-data caveats noted.
4.1 Training Details
- Training uses frozen MetaCLIP-2 guidance, a ViT-T/16 student, a 12-layer text encoder, and MSE, ICL, and CRD objectives.The student encoders and projection layers are updated; training runs for 32 epochs on 40 NVIDIA A100 GPUs.Source: §4.1 · p. 8
4.2 Benchmarks
- Retrieval evaluation covers seven Southeast Asian languages, while Babel-IN measures zero-shot classification and the CVQA LOCAL split measures localized VQA.Because benchmark availability is limited and some datasets use translated data, results are reported separately by task family.Source: §4.2 · p. 9
4.3 Competitive methods
- Comparisons include CLIP-KD with the same student architecture but no Southeast Asian training mixture, plus TinyCLIP, MobileCLIP2, and the multilingual teacher.The publicly released compact baselines differ in training data and recipe, so they are comparisons with pretrained models rather than fully controlled experiments.Source: §4.3 · p. 9
5 Experimental Results
SEA-CLIP-Tiny leads the evaluated students on average retrieval across Southeast Asian languages and benchmarks, while its performance varies by language and task. Human judgments show that automatic-score differences do not always match perceived retrieval quality.
5.1 Main Results: Language Score
- 12.9%, 31.5%, and 42.2% at R@1, R@5, and R@10 are SEA-CLIP-Tiny’s best student averages, with 38.4% fewer parameters and 24.1 ms CPU latency versus MobileCLIP2’s 61.6 ms.It outperforms MobileCLIP2 at all ranks for Thai, Myanmar, and Vietnamese, but MobileCLIP2 remains stronger on Indonesian, Javanese, and Sundanese.Source: Table 2 · p. 10, §5.1 · p. 10
5.2 Main Results: Task Score
- 13.9% at R@1, 33.5% at R@5, and 44.5% at R@10 are SEA-CLIP-Tiny’s retrieval averages, versus MobileCLIP2’s 11.9%, 22.8%, and 28.4%.SEA-CLIP-Tiny also reaches 17.2%, 33.3%, and 41.1% on Babel-IN acc@1, acc@5, and acc@10, while MobileCLIP2 scores 33.5% on CVQA acc@1 versus 28.2%.Source: Table 3 · p. 11, §5.2 · p. 11
5.3 Human Preference Scores
- Human preference does not always match automatic retrieval scores: SEA-CLIP-Tiny wins 36% of Javanese examples versus MobileCLIP2’s 32.7%, while Thai wins are 28% versus 3.7%.The evaluation sampled 100 examples per language and asked native speakers to choose the more correct and natural output or mark a tie.Source: §5.3 · pp. 11–12, Figure 4 · p. 12
6 Ablation Studies
Controlled ablations vary the training-data mixture and distillation objective while holding the student architecture and hyperparameters constant. The full regional-plus-general data mixture and combined KD losses perform best on SEA retrieval.
Ablation Setup
- The ablations vary either the data mixture or training objective while keeping architecture, pipeline, and hyperparameters fixed.They report source-specific retrieval, ImageNet accuracy, and average retrieval R@1 over XM3600, Flickr30k-200, and XTD-200.Source: §6 · pp. 12–13
Dataset Ablation
- 13.9% R@1-Avg is achieved by the full CC12M, curated SEA, and Mammoth mixture, exceeding the tested partial and single-source mixtures.CC12M-only has the best ImageNet score at 40.0% but reaches 6.5% R@1-Avg; no individual SEA dataset suffices for overall SEA evaluation.Source: Table 4 · p. 13, §6 · p. 13
KD Loss Ablation
- Removing KD losses lowers R@1-Avg from 13.9% to 7.3%, and every tested single-loss variant remains below the full objective.Among single-loss variants, CLIP + ICL reaches 12.2% R@1-Avg, compared with 9.3% for CLIP + CRD and 7.2% for CLIP + MSE.Source: §6 · pp. 13–14
7 Conclusion
SEA-CLIP-Tiny is a compact multilingual text-vision embedding model for Southeast Asian languages. Its evaluation reports strong average student-model retrieval performance, resource advantages over MobileCLIP2, and limits in coverage and CVQA.
7 Conclusion
- SEA-CLIP-Tiny achieves the strongest average retrieval performance among evaluated student models, with large gains on under-supported languages.The conclusion also reports that MobileCLIP2 remains stronger on CVQA.Source: §7 · p. 14
- The model adapts a CLIP-KD-style framework using Southeast Asian data, multilingual teacher guidance, and multi-objective distillation.Source: §7 · p. 14
- SEA-CLIP-Tiny uses 38.4% fewer parameters and lower CPU latency than MobileCLIP2.Source: §7 · p. 14
- Ablations show general-domain and regional data are complementary, highlighting region-aware training for compact multilingual vision-language models.Source: §7 · p. 14
Limitation, Broader Impact, and Ethical Discussion
SEA-CLIP-Tiny improves compact multilingual retrieval for Southeast Asian languages but remains below large teacher models and does not cover all regional language and cultural variation. The paper also notes possible inherited biases and advises care in high-stakes or culturally sensitive uses.
Limitation, Broader Impact, and Ethical Discussion
- SEA-CLIP-Tiny still has a performance gap to large teacher models and does not cover all languages, dialects, or cultural variations in the region.Source: Limitation, Broader Impact, and… · p. 14
- The model may inherit biases from its training data and teacher models, so the authors advise care in high-stakes or culturally sensitive applications.Source: Limitation, Broader Impact, and… · p. 14
- Human evaluators were volunteer Southeast Asian undergraduate students who were native or fluent speakers of the evaluated languages.Participation was voluntary, and results were reported only in aggregate without personally identifiable information.Source: Limitation, Broader Impact, and… · p. 14
8 Supplementary
The supplementary results report consistent trends across image-to-text and text-to-image retrieval, with SEA-CLIP-Tiny competitive at higher ranks and for several under-supported languages. Human preferences vary by dataset, while the evaluation benchmarks and measurement protocols define important limits on interpretation.
8.1 I2T and T2I Results
- SEA-CLIP-Tiny (CC12M+SEA) substantially improves over CC12M-only and SEA-only variants, with gains for under-supported languages but some MobileCLIP2 advantages at R@1 and in high-resource languages.The discussion says the trend is consistent across retrieval directions and that MobileCLIP2 remains stronger in some high-resource languages.Source: §8.1 · p. 20, Table 5 · p. 20, Table 7 · p. 20
- Tables 5–8 organize per-language and per-task I2T and T2I performance, including retrieval averages, total averages, and CVQA accuracy; their captions provide metric scope but no result values.Source: Table 5 · p. 20, Table 6 · p. 20, Table 7 · p. 20
- Human preference is dataset-dependent: SEA-CLIP-Tiny is preferred for Thai on Flickr30k-200 and for Thai and Sundanese on XTD-200, while MobileCLIP2 is stronger for Sundanese on Flickr30k-200.On Flickr30k-200, many Thai, Indonesian, and Javanese examples are ties; on XTD-200 the models remain close for Javanese.Source: §8.2 · pp. 20–21, Figure 5 · p. 21, Figure 6 · p. 22
- Annotators compare anonymized top retrieved captions for the same image, judging target-language semantic correctness and naturalness, with ties allowed for equally acceptable or incorrect captions.Three native-speaking volunteer undergraduate annotators evaluate each language; they understand English to follow instructions.Source: §8.2 · p. 22, Figure 7 · p. 23
- Flickr30k-200 and XTD-200 use machine-translated rather than native-annotated text, limiting their evaluation of lower-resource languages.Filipino, Khmer, and Lao appear in training but are absent from the listed evaluation sets; Babel-ImageNet test-set sizes also vary by language.Source: Table 9 · p. 23
- Efficiency measurements distinguish single-sample A100 latency from batch-1 CPU latency, while throughput and peak memory use batch size 1,024.The captions specify warm-up and timed-run protocols and parameter counts that include projection heads and token-embedding tables.Source: Table 10 · p. 24, Table 11 · p. 24, Table 12 · p. 24
- Tokenizer and caption tables report script-vocabulary coverage, token lengths before truncation, and language-level truncation or fragmentation statistics.SEA-script totals exclude Latin entries, and caption statistics are macro-averaged across XM3600, Flickr30k-200, and XTD-200.Source: Table 13 · p. 24, Table 14 · p. 25, Table 15 · p. 26