PaperlayerView exact revision on arXiv

Source-linked AI summary

COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Sampling

Noah Flynn

arXiv:2604.20720v1cs.LGcs.AIcs.CL

TL;DR

Multilingual adaptation is difficult because naive cross-lingual fine-tuning can suffer from interference and because target training data may not match intended usage. COMPASS selects semantically relevant auxiliary data with distribution-aware clustering and trains lightweight language-specific adapters, while COMPASS-ECDA updates them under production shifts. The paper reports consistent gains over linguistic-similarity baselines across architectures and multilingual benchmarks, while recognizing evaluation and proxy-distribution limitations.

  • Problem

    Multilingual fine-tuning can degrade target-language performance, while monolingual adaptation has high memory costs and loses cross-lingual transfer.

  • Method

    COMPASS uses semantic embeddings and clustering to select under-represented auxiliary data, trains lightweight language-specific adapters, and extends them with continual shift-triggered updates.

  • Results

    COMPASS consistently outperforms baseline methods across diverse model architectures and multilingual evaluation benchmarks.

  • Takeaways & Limitations

    The framework provides a parameter- and data-efficient approach for multilingual adaptation and ongoing maintenance in dynamic deployment settings.

  • Takeaways & Limitations

    Evaluation covers only part of global linguistic and cultural diversity, and initial adaptation assumes held-out evaluation data approximates live usage.

Abstract

from arXiv · show

Large language models (LLMs) often exhibit performance disparities across languages, with naive multilingual fine-tuning frequently degrading performance due to negative cross-lingual interference. To address this, we introduce COMPASS (COntinual Multilingual PEFT with Adaptive Semantic Sampling), a novel data-centric framework for adapting LLMs to target languages. COMPASS leverages parameter-efficient fine-tuning (PEFT) by training lightweight, language-specific adapters on a judiciously selected subset of auxiliary multilingual data. The core of our method is a distribution-aware sampling strategy that uses multilingual embeddings and clustering to identify semantic gaps between existing training data and a target usage distribution. By prioritizing auxiliary data from under-represented semantic clusters, COMPASS maximizes positive cross-lingual transfer while minimizing interference. We extend this into a continual learning framework, COMPASS-ECDA, which monitors for data distribution shifts in production and dynamically updates adapters to prevent model staleness, balancing adaptation to new data with the preservation of existing knowledge. Across three different model architectures (Phi-4-Mini, Llama-3.1-8B, and Qwen2.5-7B) and multiple challenging multilingual benchmarks (Global-MMLU, MMLU-ProX), including unseen long-context tasks (OneRuler), we demonstrate that COMPASS consistently outperforms baseline methods guided by linguistic similarity, providing an effective, efficient, and sustainable solution for developing and maintaining high-performing multilingual models in dynamic environments.

1 Introduction

COMPASS addresses multilingual adaptation by selecting auxiliary data according to target usage distributions rather than adding multilingual data indiscriminately. It combines lightweight language-specific adapters with continual updates for shifting production distributions.

  • Motivation: Multilingual fine-tuning can degrade target-language performance because training data may be distributionally misaligned with real-world usage.The mismatch reflects sampling biases, topical variation, and imbalanced language contributions.
  • Motivation: Using only monolingual data avoids interference but creates memory overhead and forfeits beneficial cross-lingual transfer.This trade-off is especially restrictive where high-quality human-curated data is scarce.
  • COMPASS: COMPASS selects semantically guided auxiliary data to improve target-distribution coverage while reducing task-irrelevant noise.The method is distribution-driven rather than based on linguistic structure or tokenization changes.
  • COMPASS: Parameter-efficient fine-tuning trains lightweight language-specific adapters on a shared base model, enabling multi-adapter deployment without a full model per language.The approach uses weight-decomposed low-rank adaptation (DoRA).
  • Continual learning: COMPASS-ECDA periodically updates adapters with fresh data after distribution shifts, recovering performance on new data while preserving performance on original data.This extends COMPASS from one-time adaptation toward long-term multilingual model maintenance.
  • Evaluation: The paper evaluates distribution-aware sampling across languages, architectures, benchmarks, data budgets, cross-task generalization, and component ablations.The contributions include both empirical validation and an extension for adapting to distribution shifts over time.

2 Related Works

Related work frames COMPASS as a data-centric alternative to optimization, architectural, and model-dynamics approaches for multilingual adaptation. Its distinctive signal is the distributional gap between training data and target usage, combined with modular PEFT.

  • Cross-lingual interference: The multilinguality curse reflects degraded performance in specific, especially lower-resource, languages despite broad multilingual pretraining.The related-work discussion attributes this partly to competition among languages for shared model capacity.
  • Cross-lingual interference: Prior interference-mitigation methods modify optimization or data selection, including gradient projection and cross-lingual gradient-based selection.PCGrad projects conflicting updates, while CONGRAD uses aggregated cross-lingual gradient directions for preference alignment.
  • Modular adaptation: Architectural alternatives specialize models through language clusters or modular adapters, reducing reliance on a single monolithic model.X-ELM trains smaller experts, while PEFT frameworks support separate language and task modules.
  • COMPASS positioning: COMPASS uses lightweight language-specific adapters on one shared base model to reduce the storage and training costs of full model ensembles.This positions it as a distribution-based, data-centric alternative to representation-alignment methods that often require parallel corpora.
  • Data selection: Existing PEFT architectures leave open how to select training data from heterogeneous multilingual pools, motivating COMPASS’s distribution-aware selection strategy.The paper argues that naively fine-tuning on all available data is suboptimal.
  • Data selection: COMPASS differs from dynamic selection methods by using unsupervised latent-topic clustering and the magnitude of distributional gaps as its primary selection signal.It combines under-represented-cluster diversity, prototypical-example relevance, and progressively harder examples as coverage improves.
  • Continual learning: Continual-learning work addresses evolving data while limiting catastrophic forgetting through rehearsal or regularization, whereas COMPASS targets domain-incremental multilingual adaptation.Task-incremental routing and task-specific modules are presented as unsuitable for this problem.

3 Distribution-Guided Sampling for Multilingual Adaptation

COMPASS formulates multilingual adaptation as selecting a budgeted auxiliary subset whose semantic distribution fills gaps between target training data and target usage. It uses multilingual embeddings, clustering, mismatch-weighted sampling, and adaptive within-cluster prioritization, with ECDA extending the process to distribution shifts.

  • Problem definition: COMPASS adapts a multilingual base model to a target language by combining all target data with a selected subset of auxiliary multilingual data through PEFT.The base parameters remain fixed while a small target-language adapter is trained.
  • Problem definition: The target training distribution may under-cover the intended usage distribution, while indiscriminate auxiliary-data inclusion can introduce irrelevant regions and negative transfer.Held-out evaluation inputs serve as a proxy for true usage in the initial formulation.
  • Semantic representation: A pretrained multilingual encoder maps target, auxiliary, and evaluation inputs into semantic space, where clustering defines regions of related content.The representation is intended to group content by topic or task more strongly than by language.
  • Distribution mismatch: COMPASS compares target-training and evaluation densities within each cluster to quantify distribution mismatch.Cluster counts separately track target, auxiliary, and evaluation examples.
  • Distribution mismatch: Clusters with ρk > 1 are under-represented relative to evaluation and receive heavier auxiliary sampling.The formal selection distribution Q moves effective training probabilities toward the evaluation distribution under an auxiliary-data budget.
  • Sampling procedure: The procedure allocates sampling weights first to clusters and then to individual auxiliary examples within clusters.Cluster weights are proportional to mismatch ratios, with smoothing used to avoid division by zero.
  • Sampling procedure: Within severely under-represented clusters, COMPASS prioritizes prototypical examples before progressively favoring boundary cases as coverage improves.This schedule combines semantic relevance with increasing robustness-oriented difficulty.
  • Continual adaptation: The ECDA extension uses distributional signals to detect shifts, resample data, and update adapters while integrating established continual-learning techniques.Its update mechanism is presented as a data-centric integration rather than a new continual-learning primitive.

4 Experiment Setup

The experiments evaluate COMPASS across multilingual datasets, three model architectures, embedding choices, adapter training procedures, and baselines spanning monolingual, full-data, random, and language-ranking selection.

  • Datasets: Aya supplies the primary fine-tuning pool, with 204K human-curated instruction-response examples across 65 languages and diverse tasks and domains.The dataset serves as the auxiliary-data pool Daux for COMPASS fine-tuning.
  • Datasets: COMPASS is evaluated on Global-MMLU, MMLU-ProX, OneRuler, and additional tests covering multilingual understanding, long-context reasoning, entailment, question answering, and mathematics.Global-MMLU covers 42 languages, MMLU-ProX covers 29, OneRuler covers 26 languages with contexts up to 128K tokens, while XNLI, XQuad, and MGSM8k provide narrower evaluations.
  • Models: The study uses Phi-4-Mini, LLaMA-3.1-8B, and Qwen2.5-7B to test COMPASS across different model sizes and multilingual pretraining backgrounds.Qwen2.5-7B includes pretraining across 29 languages, whereas Phi-4-Mini is treated as a smaller model with non-English regressions.
  • Sampling and training: Jina-Embeddings-v3-570M generates task-specific multilingual embeddings for sampling, supporting 100 languages and contexts up to 8192 tokens.Among the tested encoders, Jina provided the best reported cross-lingual alignment and language coverage.
  • Sampling and training: For each target language, the method fine-tunes a DoRA adapter using target-language Aya data plus COMPASS-sampled auxiliary data while keeping the shared base model unchanged.Adapters are trained for Global-MMLU and MMLU-ProX, with distribution approximation based on training and development data to avoid test leakage.
  • Baselines: Baselines compare target-only DoRA, full-model COMPASS fine-tuning, all-data multilingual DoRA, random auxiliary sampling, and LangRank-guided language selection.These comparisons test cross-lingual transfer, full-parameter adaptation, indiscriminate multilingual training, targeted selection, and linguistic-ranking strategies.

5 Results

COMPASS improves multilingual adaptation across models, languages, scripts, and evaluation settings while using targeted auxiliary data to limit interference. Its gains are strongest for lower-resource languages, with continual adaptation preserving prior performance while tracking new distributions.

  • COMPASS consistently improves performance over target-only and linguistically informed baselines across models and multilingual benchmarks.Permutation tests report p<0.05 versus Target across models and benchmarks, with medium-to-large effect sizes.
  • COMPASS gains extend across language scripts and genera, with the largest benefits generally appearing for lower-resource languages such as Swahili and Yoruba.Languages without related scripts or families show only marginal gains.
  • Optimal auxiliary budgets vary with language-specific access to high-affinity data and the base model’s initial capabilities.Languages with richer pools of related auxiliary data may tolerate larger budgets, whereas stronger initial support or limited high-quality data can cause earlier saturation.
  • COMPASS samples related source languages frequently, while some languages are broadly sampled without clear family-based relationships.Malagasy, Malay, Tamil, Telugu, and Sinhala are identified as broadly sampled sources; this pattern may reflect embedding-quality limitations.
  • On OneRuler, gains are strongest for lower- and medium-resource languages at shorter context lengths, while high-resource languages often regress as context length increases.Qwen2.5-7B shows the most robust short-to-long-context transfer, whereas positive transfer is largely absent for Llama3.1-8B.
  • COMPASS-ECDA achieves Pareto-optimal learning-forgetting performance by combining adaptation to new domains with retention of original capabilities.Its integrated strategy is evaluated after 5 adaptation epochs against retention on Global MMLU and adaptation to MMLU-ProX.

6 Discussion

COMPASS improves multilingual adaptation by selecting auxiliary data that fills semantic gaps, while COMPASS-ECDA extends this approach to shifting production distributions. The discussion also identifies boundaries from tokenization, data scarcity, evaluation coverage, proxy assumptions, encoder quality, and long-term clustering stability.

  • Discussion: COMPASS outperforms multilingual adaptation baselines by sampling under-represented semantic clusters to reduce training–usage distribution mismatch.The approach contrasts with linguistic-similarity heuristics and reactive gradient manipulation, aiming to increase positive transfer while limiting interference.
  • Discussion: COMPASS-ECDA supports dynamic deployment by updating adapters as user needs, topics, or demographics shift.Its semantic data-selection stage is decoupled from the PEFT training algorithm, allowing use with methods such as DoRA or LoRA.
  • Limitations: Tokenizer limitations remain a ceiling for low-resource-language adaptation because COMPASS does not change the base model’s vocabulary.Over-fragmented tokenization can increase sequence lengths and computational costs while impairing learning.
  • Limitations: COMPASS cannot compensate fully for scarce, representative pre-training data or limited linguistic and cultural coverage in current evaluation benchmarks.The paper frames fine-tuning as a partial intervention and calls for community-driven data curation and localized evaluation resources.
  • Limitations: The framework assumes held-out data can proxy live usage, while JS-triggered updates may miss performance-critical shifts and incremental clusters may become stale.The work leaves the threshold for full re-clustering over long horizons undefined.
  • Limitations and Future Work: Selection quality depends on multilingual encoder quality, with weak representations able to propagate or amplify sampling errors across languages.The paper reports indiscriminate sampling from under-tuned languages and proposes broader future extensions involving composable adapters, safety, fairness, and interpretability.

B Dataset Language Coverage

Table 4 covers language resources and translation data used in the dataset analysis.

  • Table 4 presents the paper’s language resources.
  • The table also covers translation data.
  • Its scope is dataset language coverage rather than model performance.

C Per-Language Performance on Global MMLU & MMLU-ProX

The paper reports per-language performance across Global-MMLU and MMLU-ProX for three model architectures, with results organized in Tables 5–10.

  • The paper directs readers to Tables 5–10 for these performance results.
  • Global-MMLU: Tables 5–7 report per-language Global-MMLU performance for Phi-4-Mini, Llama-3.1, and Qwen2.5 models.
  • MMLU-ProX: Tables 8–9 report per-language MMLU-ProX performance for Phi4-Mini and Llama-3.1 models.
  • MMLU-ProX: Table 10 reports per-language MMLU-ProX performance for Qwen2.5-7B-Instruct.

D.1 Subject Allocation for Learning-Forgetting Experiments

The controlled learning-forgetting experiments divide the 57 MMLU subjects into initial training subjects and held-out subjects used for an MMLU-ProX distribution shift.

  • The experiment partitions the 57 MMLU subjects into two groups.
  • Initial Training Subjects: Initial training uses 27 subjects covering broad basic knowledge while excluding advanced specialized topics.
  • Held-Out Subjects for MMLU-ProX Shift: The held-out shift contains 30 subjects representing advanced and specialized knowledge.

D.2 Temporal Distribution Shift Subject Allocation

The temporal-shift experiment simulates five periods of evolving subject distributions and evaluates the COMPASS-ECDA trigger across those transitions.

  • Five periods model sequential distribution shifts as user interests evolve.
  • T1 - Initial Deployment: T1 begins with adapters trained on 27 Global MMLU subjects.
  • T2–T4 Distribution Shifts: T2 adds advanced STEM subjects, T3 adds humanities and ethical domains, and T4 adds professional domains.
  • T5 - Cyclical Return: T5 returns to 10 subjects from the original T1 set to simulate seasonal usage.
  • Threshold Analysis: A threshold of θJS = 0.05 causes a 66.7% false positive rate during stable distributions.
  • Threshold Analysis: The threshold 0.15 produces zero false positives, identifies all four major transitions, and has an average delay of 2,180 samples.

D.3 Multi-Step Continual Learning Results

The multi-step continual-learning evaluation tracks five distribution-shift periods across three model architectures. COMPASS-ECDA shows a consistent temporal pattern across architectures, with results summarized at key checkpoints and across the full temporal span.

  • Five periods, T1–T5, represent successive distribution shifts in subject composition during continual-learning evaluation.
  • Figures 10 and 11 extend the main-text Qwen2.5-7B analysis to Phi-4-Mini and LLaMA-3.1.
  • Tables 12–14 report mean accuracy and standard deviation across three random seeds at T1, T5, and over the complete temporal span.

D.4 Memory-Performance Analysis

The memory-performance analysis examines rehearsal-buffer size and regularization choices for continual adaptation. It finds diminishing forgetting-mitigation returns beyond a 5% anchor buffer and selects λ = 2, β = 0.1 through Pareto optimization.

  • The memory trade-off is evaluated on Qwen2.5-7B using the Global MMLU-to-MMLU-ProX distribution shift.
  • Figures 10 and 11 provide complementary temporal performance evolution results for Phi-4-Mini and LLaMA-3.1.
  • 5% distribution anchor buffer size marks the point beyond which additional buffer memory yields diminishing returns for mitigating forgetting.The analysis spans 0% pure regularization without rehearsal to 100% full rehearsal of the original training data.
  • The buffer size reflects architectural susceptibility to catastrophic forgetting, which remains low because of PEFT.
  • λ = 2 and β = 0.1 provide the most effective Pareto-optimal trade-off between MMLU-ProX performance and minimal Global MMLU degradation.The pair was selected on a held-out validation set and then used for all COMPASS-ECDA experiments in the main paper.

E Clustering Methods

COMPASS evaluates clustering quality separately for each target language because sampling performance depends on semantic-clustering quality. K-means, HDBSCAN, hierarchical agglomerative clustering, and Taylor-Butina use metric- or density-based procedures to select configurations.

  • Clustering hyperparameters are tuned per target language, with practical recommendations to investigate broader value ranges than those tested here.The study uses clustering metrics rather than downstream performance for efficiency.
  • K-Means Clustering: K-means tests K = 10 to K = 120 and selects the value maximizing silhouette score; all 42 languages optimize at K ≥ 80, with 29 at K ≥ 100.
  • HDBSCAN Clustering: HDBSCAN selects configurations using DBCV over minimum cluster sizes [5, 10, 15, 20] and minimum samples [1, 5, 10].The majority of languages, 22/42, achieve optimal clustering with min_cluster_size = 20.
  • Hierarchical Agglomerative Clustering: Hierarchical agglomerative clustering uses Ward’s linkage, cuts the dendrogram with fcluster, and selects K by maximizing silhouette score over K = 80 to K = 120.Cluster centers are computed as the mean embeddings within each cluster.
  • Hierarchical Agglomerative Clustering: Hierarchical clustering produces optimal K values similarly to K-means, with most languages selecting K ≥ 110.
  • Taylor-Butina Clustering: Taylor-Butina adaptively searches a distance threshold in [0.70, 0.95] to maximize non-singleton clusters while assigning at least 95% of points.The threshold is adjusted by binary search until convergence.

F Computational Efficiency Analysis

COMPASS adds one-time embedding and clustering preprocessing, per-language adapter training, and small inference overhead, while providing a defined workflow for adding languages. Statistical evaluation uses cross-language permutation, effect-size, and sign tests.

  • Preprocessing Overhead: 204K Aya examples require 42.4 minutes for embedding and 2.2 hours for HDBSCAN clustering, amortizing to 4.15 minutes per language across 42 languages.
  • Per-Adapter Training Costs: Per-adapter COMPASS training takes 44.9, 104.7, and 86.3 minutes per language for Phi4-Mini-3.8B, Llama-3.1-8B, and Qwen2.5-7B, respectively.Using only target-language data takes 17.9, 49.7, and 32.8 minutes, respectively.
  • Inference Costs: GlotLID-v3 adds 6 milliseconds of sentence-level latency, adapter switching takes less than 1 millisecond, and dynamically loaded DoRA adapters add no more than 3% overhead.
  • Scaling Considerations: Adding a language requires embedding its development set, computing cluster weights, sampling auxiliary data, and training one adapter.For 3,000 development examples, the first three steps require 496 seconds: 248 seconds, 174 seconds, and 74 seconds, respectively.
  • Statistical Evaluation: Permutation tests shuffle method labels across languages 10,000 times to estimate whether observed mean differences reflect systematic method effects.
  • Statistical Evaluation: Cohen’s d quantifies practical improvement magnitude using pooled cross-language standard deviation, with small, medium, and large thresholds near 0.2, 0.5, and 0.8.
Loading 2604.20720v1…