PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Conditional Flow Matching for Balanced Multilingual Text Embedding Adaptation

Tirth Bhatt, Naren Kumar S, Mayank Singh

arXiv:2608.05785v1cs.CLcs.AI

TL;DR

Multilingual embedding adaptation often applies one objective across heterogeneous tasks, despite their differing optimization needs. TCFM routes Flow Matching to translation tasks and reports gains of 5.45% and 2.72% over two base models on Indic MTEB.

  • Problem

    A single objective across heterogeneous multilingual tasks can produce conflicting training signals and false negatives, limiting task-aligned embedding adaptation.

  • Method

    TCFM routes Flow Matching to translation-style pairs, uses other task objectives, and combines teacher guidance with a three-stage curriculum.

  • Results

    5.45% and 2.72% improvements on Indic MTEB over the Harrier-0.6B and Qwen3-Embedding-8B base models, respectively.

  • Takeaways & Limitations

    Task-aware optimization matching loss functions to task semantics effectively improves multilingual representation learning across multiple model architectures.

  • Takeaways & Limitations

    Because Flow Matching requires high-quality parallel translation pairs, TCFM may be difficult to apply in zero-resource languages lacking such corpora.

Abstract

from arXiv · show

Multilingual text embedding models are commonly adapted using a single training objective across diverse tasks, despite different tasks requiring fundamentally different optimization strategies. We introduce Task-Conditional Flow Matching (TCFM), a multilingual embedding adaptation framework that selectively applies Flow Matching to translation tasks while optimizing retrieval, classification, and pair-classification tasks with objectives better aligned to their learning dynamics. TCFM further combines teacher-guided representation preservation with a three-stage curriculum to enable stable adaptation. Evaluated on the Indic Massive Text Embedding Benchmark, TCFM establishes a new state-of-the-art, consistently improving embedding quality across a diverse set of multilingual tasks and generalizing across embedding model families. We will publicly release the codebase and datasets upon acceptance of the paper.

1 Introduction

TCFM addresses the limitations of uniform contrastive or transport objectives in heterogeneous multilingual embedding adaptation by routing tasks to better-matched objectives. It combines selective Flow Matching, teacher-guided representation preservation, and a three-stage curriculum, improving Indic multilingual embedding performance over two base models.

  • Motivation: Multilingual embedding adaptation is challenging because downstream tasks have substantially different characteristics despite shared representation requirements.The models support retrieval, bitext mining, semantic textual similarity, clustering, classification, and natural language inference.
  • Motivation: Contrastive learning can incorrectly push semantically similar sentences apart by treating all unpaired in-batch instances as strict negatives.This false-negative problem makes contrastive adaptation difficult in diverse multi-task or multilingual settings.
  • Method: TCFM applies Flow Matching only to translation-style sentence pairs while routing other task families to objectives better suited to their semantic characteristics.Flow Matching models a vector field between continuous representations, whereas retrieval, classification, and pair-classification capture different semantic relationships.
  • Method: TCFM combines task-aware objective routing, teacher-guided representation preservation, and a three-stage curriculum for multilingual embedding adaptation.The framework is introduced as a task-conditional alternative to applying Flow Matching uniformly across all tasks.
  • Results: 5.45% improvement on Indic MTEB over the Harrier-0.6B base model and 2.72% improvement over the Qwen3-Embedding-8B base model.The improvements are reported for Indic multilingual text embedding adaptation across multiple model scales.

2 Related Work

Prior work established sentence and multilingual embedding methods, contrastive objectives, and Flow Matching as foundations for representation learning. The proposed framework differs from unified contrastive approaches by selecting optimization objectives according to task-family semantics.

  • Sentence and multilingual embeddings: Sentence-BERT and SimCSE established siamese-network and contrastive-learning approaches for sentence embeddings, while LASER and LaBSE learned shared multilingual spaces from parallel corpora.The passage identifies these methods as foundational sentence and multilingual embedding models.
  • Contrastive learning and false negatives: InfoNCE-based contrastive learning is standard for sentence embeddings but can incorrectly separate multilingual or multi-task examples sharing labels, intents, or semantic neighborhoods.This limitation motivates objectives that accommodate multiple semantically related positives.
  • Flow Matching: Flow Matching learns a continuous velocity field that deterministically transports representations between probability distributions by regressing velocity along a predefined path.Unlike diffusion models, it learns transport dynamics directly rather than reversing stochastic noise.
  • Flow Matching: Because Flow Matching learns smooth transformations and parallel translations define semantically aligned pairs, it is attractive for multilingual embedding alignment.The passage notes that Flow Matching has primarily been studied for image and generative modeling tasks.
  • Task-conditional optimization: Unlike approaches using one contrastive objective for heterogeneous supervision, the proposed framework assigns optimization objectives according to each task family’s semantic characteristics.This task-conditional objective selection is the central distinction identified in the related work discussion.

3 The TCFM Framework

TCFM adapts multilingual embeddings with task-specific objectives, applying Flow Matching only to translation while using contrastive and conditional auxiliary losses for other task families. Teacher-guided preservation and self-consistency regularization stabilize adaptation and retain semantic structure.

  • Task organization: TCFM organizes training around translation, classification, pair classification, and retrieval/re-ranking task families with distinct semantic relationships.Translation provides cross-lingual correspondences, whereas retrieval is asymmetric and classification tasks need not contain paraphrases.
  • Task-specific objectives: Multi-positive InfoNCE is applied symmetrically to translation and asymmetrically from query to document for retrieval, with task-dependent handling of similar negatives.Highly similar non-positives are filtered during translation training but retained for pair classification as useful learning signals.
  • Conditional flow matching: Flow Matching is activated only for translation, where parallel pairs define meaningful continuous embedding-space transformations.The velocity predictor is conditioned on a task-instruction embedding from the frozen teacher encoder, and teacher-guided transport stabilizes optimization.
  • Representation preservation: Frozen-teacher pointwise and relational objectives regularize adaptation by preserving individual embedding directions and mini-batch similarity structure.These dual constraints retain the pretrained multilingual encoder’s global semantic relationships while allowing specialization to new task families.
  • Monolingual consistency: A SimCSE-style self-consistency objective preserves monolingual semantic quality by encoding each input twice under independent dropout masks.The objective regularizes local neighborhoods and improves monolingual robustness without additional supervision.

4 Experimental Setup

The experiments use a balanced four-family multilingual training mixture with task-specific supervision, evaluate multiple embedding architectures, and assess performance on the 25-language Indic MTEB benchmark. Training also follows a three-stage curriculum, with checkpoints evaluated against frozen base models.

  • Training Data: The training mixture spans four task families, converting datasets into objective-specific formats for translation, classification, NLI, and retrieval.Formats include symmetric parallel pairs, class-based semantic pairs, entailment-contradiction pairs, and asymmetric query-document pairs.
  • Training Data: The translation data adds local_wide_parallel, a self-curated corpus translated from English Wikipedia into 16 Indic languages.Stratified language subsampling and dataset-size caps limit gradient imbalance from massive parallel corpora overpowering low-resource tasks.
  • Models: Evaluation covers google/embeddinggemma-300m, microsoft/harrier-oss-v1-270m, microsoft/harrier-oss-v1-0.6B, and Qwen/Qwen3-Embedding-4B, while the best-performing model adapts Qwen/Qwen3-Embedding-8B with LoRA.Gemma, Harrier OSS, and Qwen models are compared across multiple architectures.
  • Implementation: Gemma models use mean pooling, whereas Harrier OSS and Qwen models use leftpadding safe EOS pooling, and all embeddings are L2-normalized.The Flow Matching velocity network is a two-layer MLP with SiLU activations, Layer Normalization, and sinusoidal timestep embeddings.
  • Benchmark: Indic MTEB evaluates Bitext Mining, Semantic Textual Similarity (STS), Classification, Clustering, Pair Classification, Retrieval, and Reranking across 25 linguistically diverse languages.The benchmark is used to assess multilingual embedding quality and generalization of the proposed training objectives.
  • Evaluation Protocol: The three-stage curriculum is analyzed by evaluating checkpoints after every stage alongside the final model, with improvements reported relative to the corresponding frozen base model.This comparison is used unless otherwise specified.

5 Results

TCFM improves Indic MTEB performance across diverse multilingual task families and generalizes from Harrier-0.6B to Qwen3-Embedding-8B. Gains are broad but not uniform, with a few tasks showing modest declines.

  • Harrier-0.6B results: TCFM improves microsoft/harrier-oss-v1-0.6b’s Indic MTEB Score by +3.59 points, with Clustering gaining more than 21 points.Performance also improves across the remaining task families.
  • Harrier-0.6B results: Cross-lingual alignment improves by +3.30 points on IN22ConvBitextMining and +2.24 points on IndicCrosslingualSTS.The evaluation spans Bitext Mining, Classification, Clustering, Pair Classification, Retrieval, Reranking, and STS.
  • Limitations: A small number of tasks decline, including SanskritShlokasClassification (-5.31), IN22GenBitextMining (-0.16), NepaliNewsClassification (-0.14), and XQuADRetrieval (-0.23).These non-uniform outcomes underscore the challenge of multilingual embedding adaptation as a multi-objective optimization problem.
  • Qwen3-Embedding-8B results: On Qwen/Qwen3-Embedding-8B with LoRA, TCFM yields an absolute Indic MTEB gain of +1.97 points and improves six of seven benchmark categories.Retrieval declines slightly by -0.28, while the framework is evaluated against the frozen base model.

6 Ablation Experiments

The ablations isolate TCFM’s component contributions and show complementary strengths between contrastive learning and Flow Matching. Curriculum and routing analyses further support staged, task-conditional optimization across embedding backbones and task families.

  • Component Ablations: Contrastive learning with teacher preservation improves Pair Classification (+6.27), Clustering (+21.32), and STS (+0.46), while Flow Matching with teacher preservation improves Retrieval (+1.06) and Clustering (+21.15).The results indicate complementary strengths across downstream task families.
  • Curriculum Ablation: Stage 1 translation supervision improves Harrier-0.6B clustering (+7.53), STS (+3.17), and pair classification (+2.61), while Qwen3-Embedding-8B gains in STS (+1.02) and bitext mining (+0.31).The stage establishes cross-lingual alignment before task-specific adaptation.
  • Curriculum Ablation: Stage 2 multi-task supervision gives Harrier-0.6B gains in classification (+2.90) and pair classification (+5.41), while Qwen3-Embedding-8B gains in classification (+1.36) and pair classification (+1.30).Translation replay prevents catastrophic forgetting, although strict contrastive boundaries create a temporary trade-off with spatial uniformity.
  • Curriculum Ablation: Stage 3 geometric regularization recovers Harrier-0.6B retrieval (+1.90 over Stage 2) to a final overall score of 69.46 and raises Qwen3-Embedding-8B clustering (+15.25) to 74.The final stage resolves multi-task optimization friction and spatial interference from Stage 2.
  • Task-Conditional Routing: +1.67 vs. +1.63 is the highest overall Indic MTEB score when Flow Matching is restricted to translation, also improving Bitext Mining (-0.20 vs. -0.43) and STS (-0.82 vs. -0.91).This compares task-conditional routing with uniform Flow Matching across all task families.

7 Conclusion

TCFM argues that a single contrastive objective is suboptimal for multilingual embedding adaptation. By applying Flow Matching only to translation and contrastive methods to retrieval and classification, it produces a more balanced representation space validated on Indic MTEB across multiple architectures.

  • TCFM shows that applying one contrastive objective across all training data is suboptimal for adapting multilingual embeddings.
  • Restricting Flow Matching to translation tasks and using standard contrastive methods for retrieval and classification creates a more balanced representation space.
  • Experiments on Indic MTEB validate TCFM across multiple model architectures.

8 Limitations

TCFM’s evaluation is limited to 22 scheduled Indian languages, and its generalization to other language families or global multilingual benchmarks remains untested. Further limitations concern the lack of formal geometric analysis, unexplored repulsive flow dynamics, computational overhead, and dependence on high-quality parallel translation data.

  • Evaluation scope: Evaluation covers only the 22 scheduled Indian languages in the Indic MTEB benchmark, leaving generalization to other language families and global multilingual benchmarks untested.The authors identify broader cross-lingual generalization as an open question.
  • Geometric analysis: Evidence that Flow Matching improves embedding-space uniformity is currently empirical, motivating formal analysis of its effects on representation alignment and anisotropy.Future work should quantitatively measure these geometric effects.
  • Flow design: Flow Matching currently transports and aligns parallel translation pairs, while adding an explicit repulsive force for dissimilar concepts or hard negatives remains unexplored.Such repulsive flow dynamics could improve discriminative boundaries without relying solely on the current alignment mechanism.
  • Computational cost: TCFM adds computational overhead through a frozen teacher, a velocity prediction network, and a multi-stage data curriculum compared with standard contrastive training.These components must be maintained or optimized during adaptation.
  • Data requirements: Because Flow Matching requires explicit source-target pairs, TCFM depends on high-quality parallel translation data and may be limited in zero-resource languages.The limitation arises where suitable parallel corpora are unavailable.

A Detailed Model Evaluations · B Implementation Details and Hyperparameters

TCFM generalizes across embedding architectures and scales, improving overall Indic MTEB performance and several task families. The paper also documents training configurations, parameterizations, and stage-wise hyperparameter schedules for reproducibility.

  • A Detailed Model Evaluations: TCFM extends beyond Harrier-0.6B and Qwen3-Embedding-8B to EmbeddingGemma-300M, Harrier-270M, and Qwen-Embedding-4B.These additional evaluations test generalizability across model scales and architectural families.
  • A Detailed Model Evaluations: Table 4 summarizes overall and task-family performance for TCFM across the evaluated model architectures on the Indic MTEB benchmark.The table provides detailed task-level results for the evaluated architectures.
  • A Detailed Model Evaluations: +2.34 overall score gain raises EmbeddingGemma-300M from 64.34 to 66.68, with Bitext Mining (+4.48) and STS (+2.53) improvements.GujaratiNewsClassification improves by +6.27, while MalayalamNewsClassification improves by +9.66.
  • A Detailed Model Evaluations: +1.67 overall score gain raises Qwen-Embedding-4B from 70.94 to 72.61 after LoRA adaptation.The 4-billion-parameter decoder-only model is used to examine scaling within that family.
  • A Detailed Model Evaluations: Qwen-Embedding-4B gains +11.88 in Clustering, +1.88 in Pair Classification, +1.44 in Classification, and +1.02 in Bitext Mining.The passage characterizes these gains as evidence that TCFM stabilizes auto-regressive backbones during task-specific fine-tuning.
  • B Implementation Details and Hyperparameters: The implementation section specifies training configurations, model parameterizations, and stage-wise hyperparameter schedules across all experimental runs.These details are provided to ensure complete reproducibility.

B.1 Hardware and Training Environment … C Detailed Stage-Wise Progression

Training uses a shared three-stage environment and model-specific architectural configurations, while the curriculum produces stable progression for the reported encoder models. Stage 1 establishes geometry, and Stage 2 adds discriminative supervision with a temporary Harrier retrieval regression.

  • B.1 Hardware and Training Environment: All models were trained on NVIDIA H200 GPUs with PyTorch, Hugging Face Accelerate, SDPA where applicable, AdamW, gradient clipping, warmup, and effective batch size 256.Weight decay was 10^-2, gradient clipping was capped at 1.0, and the learning-rate schedule used 1% linear warmup.
  • B.2 Architectural Configurations: EmbeddingGemma-300M used full fine-tuning, native mean pooling, maximum sequence length 512, and float32 precision across all three stages.Gradient checkpointing was disabled because of the model’s smaller memory footprint.
  • B.2 Architectural Configurations: Harrier-270M and Harrier-0.6B shared full fine-tuning, left-pad-safe end-of-sequence pooling, maximum sequence length 512, and enabled gradient checkpointing.The two Harrier scales used an identical training recipe.
  • B.2 Architectural Configurations: Qwen3-Embedding-4B and Qwen3-Embedding-8B shared LoRA hyperparameters, including rank r = 64, scaling factor α = 128, dropout 0.05, and bfloat16 mixed precision.LoRA was applied to all listed attention and feed-forward projection matrices, with left-pad eos pooling.
  • B.3 Stage-Wise Hyperparameter Schedule: Table 5 specifies the exact stage-wise hyperparameters and loss coefficients for all model families across the three-stage curriculum.The cited passage identifies Table 5 as the consolidated schedule for encoder and decoder families.
  • C Detailed Stage-Wise Progression: Table 6 reports incremental Indic MTEB performance for EmbeddingGemma-300M, Harrier-0.6B, and Qwen3-Embedding-8B as heterogeneous supervision is progressively introduced.The analysis compares how the three architectures respond to the curriculum stages.
  • C Detailed Stage-Wise Progression: Stage 1 establishes strong geometry with immediate Clustering gains, while Stage 2 sharply improves Classification but temporarily reduces Harrier-0.6B Retrieval by -0.87.EmbeddingGemma-300M and Harrier-0.6B otherwise show highly stable, monotonic overall growth under the curriculum.

C.1 Model-Specific Adaptation Dynamics … D.2 Stratified Subsampling and Multi-Task Balancing

The three-stage curriculum improves overall Indic MTEB performance across model architectures, while Stage 2 can cause task-specific interference that Stage 3 resolves. TCFM also uses objective-compatible supervision formatting and stratified sampling to balance multilingual multi-task optimization.

  • C.1 Model-Specific Adaptation Dynamics: Overall scores improve monotonically from Stage 1 through Stage 3 across all three models: EmbeddingGemma-300M rises from +0.98 to +2.34, Harrier-0.6B from +1.36 to +3.59, and Qwen3-Embedding-8B reaches +1.97.These results support curriculum effectiveness across architectures and scales.
  • C.1 Model-Specific Adaptation Dynamics: Stage 2 introduces task-specific Retrieval regressions when classification and NLI supervision is added without retrieval data.Harrier-0.6B falls 0.87 points below baseline before recovering to +1.03 in Stage 3, while EmbeddingGemma-300M follows +0.55 → +0.12 → +0.47.
  • C.1 Model-Specific Adaptation Dynamics: Qwen3-Embedding-8B experiences a 3.36-point-below-baseline Clustering regression at Stage 2, fully resolved by Stage 3 regularization to a +15.25 gain.The interference affects a task family not directly supervised during Stage 2.
  • C.1 Model-Specific Adaptation Dynamics: After Stage 3 regularization, Clustering shows the largest absolute improvement for every model: +5.72 for EmbeddingGemma-300M, +21.03 for Harrier-0.6B, and +15.25 for Qwen3-Embedding-8B.Harrier-0.6B and EmbeddingGemma-300M reach these gains through smooth, monotonic Clustering trajectories without below-baseline regression.
  • D Extended Training Data Details and Preprocessing: The training mixture requires detailed preprocessing and stratified sampling to construct task-routed supervision without manual annotations.These procedures provide the data-formatting and sampling infrastructure for TCFM training.
  • D.1 Supervision Formatting Pipelines: Raw datasets are converted into objective-compatible formats for task routing, including symmetric translation pairs, label-based semantic pairs, entailment triplets with contradiction hard negatives, and asymmetric query-document pairs.Translation uses both language directions, pair classification supports hard-negative margin repair, and retrieval preserves directional relevance semantics.
  • D.2 Stratified Subsampling and Multi-Task Balancing: Native dataset distributions can let massive translation corpora dominate optimization; Samanantar alone contains over 49.7 million parallel sentences, under-fitting lower-resource classification and retrieval tasks.The imbalance motivates explicit multi-task balancing.
  • D.2 Stratified Subsampling and Multi-Task Balancing: A two-level stratified subsampling scheme caps instances per dataset and enforces equal per-language representation across multilingual datasets for all 22 targeted Indic languages.The procedure prevents dominant corpora from monopolizing updates and gives low-resource language scripts proportional optimization weight throughout the curriculum.
Loading 2608.05785v1…