Source-linked AI summary
Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressors
Mantas Lukauskas
TL;DR
The paper asks whether extractive prompt compression closes or widens the token-cost and quality gap faced by non-English languages. It conducts a controlled audit using parallel multilingual data, tokenizer-matched budgets, multiple compressors, and eleven target models. The transfer gap is real for English-supervised learned compressors but absent for XProvence v1, while translated-data retraining can over-prune Chinese and safe budgets are smaller outside English.
Problem
Non-English passages cost 1.3–1.8× more tokens than English, but evidence is limited on whether English-supervised prompt compressors transfer across languages.
Method
The study compares four learned compressors and four deterministic baselines on fully parallel data in ten languages, with target-tokenizer budget matching across eleven models.
Results
The transfer gap replicates across English-supervised compressors and target models, while multilingually trained XProvence v1 shows no gap; its translated-data v2 empties 92% of Chinese contexts at an aggressive threshold.
Takeaways & Limitations
Safe compression budgets are much smaller outside English, whereas multilingual supervision can close the gap but does not guarantee stable behavior across releases.
Takeaways & Limitations
The audit covers ten languages and five scripts, while auxiliary long-context, MultiEURLEX, and arbitrage analyses cover only six core European languages; translationese risks remain.
Abstract
from arXiv · showhide
Extractive prompt compression promises to cut LLM inference costs by removing low-information tokens, and learned compressors such as LLMLingua-2 report strong results on English benchmarks. Most other languages already pay a token premium: the same content costs 1.3-1.8x more tokens than in English. We ask whether compression closes or widens this gap. Using fully parallel data in ten languages spanning five scripts, with controls budget-matched in the target model's tokenizer, we audit four learned compressors against four deterministic baselines, on eleven target models from ten vendors (over 250,000 evaluation calls). Three of the compressors are trained with English supervision (LLMLingua-2 XLM-R/mBERT; Kompress-v2 from the production Headroom stack); the fourth, XProvence, is trained multilingually. First, the transfer gap is real, replicates across target models and compressor backbones, and is strongly rate-dependent: at a 0.33 keep-rate English retains 57-62% of normalized context utilization while Lithuanian retains 10-24% and Chinese essentially none, despite Chinese having the smallest token premium. Second, the gap tracks compression supervision data, not architecture. All three English-trained compressors show it, deterministic methods show no comparable gap, and the multilingually trained XProvence v1 shows none. Its v2 release, retrained on translated data, empties 92% of Chinese contexts at its aggressive threshold without any warning. Third, in a harder long-context setting, aggressive learned compression drives compressed contexts to or below no-context utility in three of five non-English languages. A translate-then-compress pipeline matches or beats native compression at roughly half the token cost in three of five tested languages. We release all code, compressions, and model outputs. Safe compression budgets are much smaller outside English.
1 Introduction
This paper audits whether extractive prompt compression alleviates or worsens non-English token costs and quality disparities. It uses controlled parallel evaluations to test transfer across methods, rates, languages, and target models.
- Motivation: 1.3–1.8× more o200k tokens are required for the same passages in nine non-English languages than in English.The premium reaches 4.5× under Qwen2.5’s tokenizer for Hindi.
- Research question: English-supervised compression could penalize non-English users twice through higher token costs and larger quality losses.The paper frames this as the central risk motivating the audit.
- Audit design: The audit fixes semantic content and target-tokenizer budgets while comparing four learned compressors with deterministic baselines across ten languages and eleven target models.The evaluation uses fully parallel items spanning five scripts and controls matched to achieved budgets.
- Main findings: At a 0.33 keep-rate, the transfer gap is significant in 8 of 9 non-English languages and replicates across target models and English-supervised compressor backbones.Deterministic methods show no comparable gap, and neither does multilingually trained XProvence v1.
- Main findings: Learned compression can reduce non-English contexts to no-context performance in long-context stress tests.The paper separately tests long-context behavior and task dependence.
- Main findings: Translate-then-compress can dominate native compression in the translation-arbitrage analysis, with translationese as a caveat.The comparison concerns compression performed after translation versus compression in the original language.
2 Related Work
Prior extractive compressors and tokenizer studies establish an English-centered field with higher non-English token prices. The paper positions its audit as testing whether multilingual backbones and multilingual supervision actually close that transfer gap.
- Prompt compression: Extractive prompt compression includes perplexity-based pruning, distilled token classification, query-aware reordering, self-information filtering, and production systems.The audited family retains plain-text prompts compatible with target APIs.
- Tokenizer inequality: Subword tokenizers price non-English text higher, and this study treats that premium as measured context rather than a contribution.The related work establishes tokenizer inequality as the economic background for the audit.
- Multilingual compression: Almost all extractive compressors use English supervision, while XProvence v1 is trained multilingually and is the only audited compressor that closes the transfer gap.The paper also audits XProvence v2, retrained on translated MS MARCO, because the two checkpoints behave differently on Chinese.
- Multilingual evaluation: LLMLingua-2 has a multilingual XLM-R backbone but English-only evaluation and English GPT-4 distillation supervision.The audit therefore tests whether multilingual pretraining alone transfers the compression skill.
3 Audit Protocol
The audit uses parallel multilingual data, tokenizer-matched compression conditions, normalized utility metrics, and a broad target-model panel. Its design separates compressor behavior from language, tokenization, task, and model effects.
- Languages and data: Ten fully parallel languages span four families, seven branches, five scripts, and o200k token premiums from 1.28 for Chinese to 1.83 for Latvian.The primary task is Belebele reading comprehension with 300 parallel items per language.
- Conditions: Each item is evaluated full, no-context, and under learned or deterministic compression conditions at requested keep-rates ρ∈{0.75, 0.5, 0.33}.The conditions include LLMLingua-2, TF-IDF sentence extraction, deterministic lemmatization plus stopword removal, and XProvence.
- Conditions: XProvence uses a multilingual BGE-M3 reranker trained on 16 languages with silver labels from a multilingual LLM, enabling a multilingual-training comparison.Both v1 and v2 checkpoints are audited because they behave differently on Chinese.
- Metric: Normalized context utility measures usable value retained relative to full and no-context anchors, with transfer gap defined as τℓ,c=UEN,c−Uℓ,c.The denominator is comparable across languages, and uncertainty uses 2,000-resample paired bootstraps with per-comparison 95% CIs.
- Uncertainty: The study emphasizes patterns replicated across models, methods, and rates rather than isolated significance intervals.The analysis reports many comparisons without multiplicity correction.
- Target models: Primary targets are gpt-5.4-mini and claude-haiku-4-5, with replication across nine further models from proprietary, open-weight, and Chinese-vendor families.The roster is designed to test whether target-model family or Chinese pretraining protects against the observed Chinese failure.
4 Results: Transfer Gap by Rate, Method, and Target Model
Transfer gaps emerge under aggressive learned compression, replicate across methods and target models, and are tied more closely to supervision language than compressor architecture. The resulting failures are especially severe for Chinese and other non-English languages in long-context settings.
- Transfer gap by rate: At ρ=0.33, transfer gaps are significant in 8/9 non-English languages, while Lithuanian and Ukrainian retain 10% of normalized context utilization and Chinese falls below no-context utility.Chinese has the smallest token premium at 1.28× under o200k, yet suffers the largest compression penalty; safe compression is about 2× for English versus approximately 1.3× or less elsewhere.
- Transfer gap by method: At the same achieved budgets, deterministic methods show no comparable cross-lingual gap on GPT, unlike LLMLingua-2.On GPT-5.4-mini, deterministic means range from −0.20 to −0.09 at ρ=0.5, versus +0.27 for LLMLingua-2.
- Transfer gap by method: The three English-supervised learned compressors show large gaps across backbones, whereas multilingual XProvence v1 shows none.Kompress-v2 has significant gaps in 8/8 auditable non-English languages, while XProvence v1 has no significant gap at either tested threshold on GPT.
- Transfer gap by method: The XProvence v2 release reintroduces a gap at its aggressive threshold, deleting 92% of Chinese contexts because its Chinese relevance scores are degenerate.At threshold 0.5, v2 achieves a Chinese keep-rate of 0.08, compared with 0.21–0.24 in other languages; its GPT gap is +0.16 and Claude gap +0.29.
- Target-model replication: Across eleven target models, English is consistently safest and Chinese hardest, with mean LLMLingua-2 gaps at ρ=0.5 ranging from +0.26 to +0.40.Chinese-vendor models still show the largest Chinese gaps, indicating that stronger target-model language pretraining does not recover information deleted by the compressor.
- Long-context stress test: In the long-context stress test, LLMLingua-2 at ρ=0.33 keeps English useful but drives non-English contexts to or below no-context utility, while deterministic compression preserves 44–66% everywhere.The long-context setting places the target passage among seven distractors and uses question-agnostic compression.
5 Long-Context Stress Test
In the long-context stress test, aggressive learned compression can reduce non-English contexts to no better than providing no context, while task structure determines whether compression is costly.
- Long-context stress test: Compressed Lithuanian, Latvian, and Polish contexts perform at or below the no-context anchor with GPT.At U≤0, compressed context provides no benefit over sending no context while still incurring token cost.
- Long-context stress test: Claude preserves the same language ordering, with English at 0.56 versus 0.14–0.33 elsewhere.
- Task-dependence: On MultiEURLEX classification, compression is nearly free across languages down to 20×, and a 5% title-only heuristic matches or beats full documents.Micro-F1 is 0.47–0.51 for the title-only heuristic versus 0.45–0.48 for the full document.
- Task-dependence: The cross-lingual penalty appears task-dependent: distributed-content answers suffer, whereas surface-signal classification favors positional heuristics at lower cost.
6 Translation Arbitrage
The paper tests translating non-English contexts to English before compression as a token-saving alternative. This pipeline matches or beats native-language compression in three of five languages, but its evidence is directional and its latency and translation-source caveats matter.
- Translation arbitrage: At approximately 0.18× native token cost, translate-then-compress matches or beats native compression at 0.33× cost in three of five languages.The gains are LT +10 pp, FI +6.6, and ET +5.
- Translation arbitrage: Translation alone saves 35–44% of tokens at a 5–8 pp quality cost.
- Caveats: The translation-arbitrage results are directional rather than definitive because each language uses n=60 items and the source passages derive from translated FLORES data.A native-source replication remains future work.
- Caveats: Translation adds approximately 1–2 seconds per passage, compared with 0.1–0.3 seconds for LLMLingua-2.The latency may be negligible for batch or cache-warm workloads but matter for interactive first-token latency.
7 Discussion and Limitations
The audit links learned-compression failures to disrupted grammatical cohesion and compression supervision language, while recommending task- and language-specific deployment safeguards. Its scope remains limited by language coverage, translationese, task differences, and clustered evaluation items.
- Discussion: Case endings and agreement morphology carry thematic roles in Baltic, Slavic, and Finnic languages, so dropping function morphology can disrupt grammatical structure.Chinese requires a different account because character-level deletion can produce different words or non-words; its diagnostic remains future work.
- Discussion: At a U≥0.8 fidelity bar, no English-supervised learned compressor qualifies below ρ= 0.75 in any non-English language, while Chinese has no qualifying tested rate.For deep budgets outside English, the audit recommends deterministic methods such as lemmatization+stopword removal where morphological resources exist.
- Discussion: Query-aware multilingual pruning is the only tested method reported safe in all ten languages, but XProvence v2 returns empty contexts for 92% of Chinese inputs at its aggressive threshold.This recommendation is subject to a non-commercial license and per-language calibration check.
- Limitations: The study covers ten languages and five scripts, with six Indo-European languages and no non-Uralic agglutinative language such as Turkish.Auxiliary arms cover only six core European languages, limiting generalization across the world’s languages.
- Limitations: Non-English passages are translations from English, so translationese may make them syntactically closer to English and make the measured transfer gap a plausible lower bound for native text.The authors mitigate but do not eliminate contamination and translationese risks using no-context anchoring and normalized scoring.
8 Conclusion
The conclusion finds that English-centric evaluation overstates multilingual usefulness: English-supervised compressors require much milder budgets outside English, whereas multilingual supervision avoids the gap but is not consistently stable. The paper releases its code, compressed contexts, and model outputs.
- 8 Conclusion: Safe compression budgets outside English are roughly half those in English for every audited English-supervised learned compressor.The comparison spans three compressor backbones and a production system.
- 8 Conclusion: Deterministic baselines transfer with little or no gap and match or beat learned methods at deep budgets outside English.The conclusion recommends budget-matched deterministic methods, multilingual supervision, or translate-then-compress pipelines for non-English contexts.
- 8 Conclusion: The released package includes code, 25,000+ cached compressed contexts with achieved budgets, and 178,000+ raw model-output records from the main arms.Pipeline details are provided in Appendix E.
- 8 Conclusion: The XLM-R compressor-internal tokenizer prices all ten languages nearly equally, ruling out input length as the driver of the transfer gap.Its measured relative pricing range is 0.97–1.25.
B Full Results Tables
The full-results materials provide raw condition-level accuracies, transfer-gap estimates with confidence intervals, and cross-model significance summaries, with table notes defining budgets, scoring, and replication limits.
- Full Results Tables: Tables 3 and 4 report raw accuracies for every condition, requested rate, and language on GPT-5.4-mini and Claude Haiku 4.5.Table 3 uses 300 paired items and matched achieved o200k budgets; Table 4 records refusals as incorrect.
- Full Results Tables: Table 3 marks lemma+stopword as undefined for ZH/AR/HI because simplemma lacks support for those languages.Truncation and random pruning are matched per item to LLMLingua-2’s achieved o200k budget.
- Full Results Tables: Table 5 reports LLMLingua-2 transfer gaps τvs. EN with 95% item-level paired bootstrap confidence intervals from 2,000 resamples.Bold entries indicate confidence intervals excluding zero.
- Full Results Tables: Table 6 summarizes mean transfer gaps across nine non-English languages, Chinese gaps, and significant-language counts for each target model.Replication models use 150 items on a reduced grid, making deep-budget significance counts noisier; Chinese is worst-affected for every model.
C MultiEURLEX Task-Dependence Results
The MultiEURLEX classification arm shows that compression is nearly free across its six core languages, contrasting with harder answer-bearing probes where aggressive LLMLingua-2 compression harms Lithuanian more than English.
- C MultiEURLEX Task-Dependence Results: Compression is nearly free at every rate in all languages on level-1 EUROVOC classification, and a 5% title-only heuristic matches or beats the full document.The task therefore does not probe the cross-lingual transfer gap.
- C MultiEURLEX Task-Dependence Results: The Lithuanian probe embeds an OVS minimal pair among filler sentences and asks who performed the action, testing whether case endings preserve thematic roles.It uses six nouns, five transitive verbs, gender-agreeing options, and native-speaker review.
- C MultiEURLEX Task-Dependence Results: At ρ= 0.33, English accuracy remains 98.7% while Lithuanian falls to 82.7% on the 75-item case-marking probe.Both languages score 100% with the full passage; no response selected the patient, but the question format disfavors that option.
E Reproducibility Details
The reproducibility details specify multilingual token accounting, script-aware TF-IDF extraction, and implementation choices for sensitivity analysis, random deletion, and Kompress-v2.
- o200k token budgets are computed with tiktoken, while achieved compression rates are recorded per item.
- TF-IDF extraction is sentence-level, order-preserving, and adapted to Han, Arabic, and Devanagari punctuation, with character-level term units for Chinese.IDF is estimated per language on the full aligned data.
- The audited LLMLingua-2 checkpoints use an MPS backend and force newline and question-mark tokens.
- Random deletion is budget-matched but script-dependent because Chinese uses multi-word chunks rather than whitespace-delimited words.
- Passage-cluster bootstrap sensitivity uses multinomial passage weights with the same seed and resample count.The analysis is implemented in analyze cluster sensitivity.py.