Source-linked AI summary
Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy
Ayoub Kirouane, Georgios Giaples, Christos Petrocheilos
TL;DR
The paper asks whether a robot policy can acquire Greek without Greek demonstrations or architecture changes, and how such transfer should be measured. It machine-rephrases instructions and evaluates language dependence with guaranteed-null controls and replicated seeds. Bilingual training yields consistent Greek transfer, whereas Greek-only training remains near its control, and the authors emphasize null calibration and replication as practical requirements.
Problem
Robot foundation models and their demonstration corpora are predominantly English, leaving most languages without robot-policy evidence or datasets.
Method
The study machine-rephrases an existing corpus into Greek without changing the architecture, then compares bilingual, Greek-only, and control policies using guaranteed-null and multi-seed evaluations.
Results
On ninety tasks, bilingual training produced a 6.7–7.1-point margin over its wrong-instruction control on every seed, while Greek-only training reached at most 2.7 points.
Takeaways & Limitations
Reliable low-resource robot-policy localization requires a guaranteed null before trusting metrics and replication across seeds.
Takeaways & Limitations
The evaluation covers trained goals rather than generalization to novel tasks, and three seeds per ninety-task arm leave the exact permutation test at p = 0.100.
Abstract
from arXiv · showhide
Robot foundation models are trained and evaluated predominantly in English, and robot demonstration corpora do not exist for most languages. We study the addition of Greek to an open vision-language-action stack using only machine-rephrased instructions and no architecture changes. The main challenge is measurement rather than translation. Several plausible instruments produce false conclusions: a color-histogram metric rewards noise, a single-goal benchmark scores 84.6% under correct Greek and 82.6% under deliberately wrong instructions, training loss fails to predict Greek success, and single-run comparisons are dominated by seed variation. On a discriminative ninety-task suite with three seeds per arm, a multilingual text tower without Greek demonstrations remains at its wrong-instruction floor, while Greek-only training exceeds its control by at most 2.7 points. Bilingual training yields a consistent 6.7-7.1 point margin over its control and reaches about two fifths of English performance. The policy also overfits the translator's phrasing; training on seven phrasings per task approximately halves this penalty. Warm-starting from a language-adapted world model and unfreezing the text tower both degrade performance. The results support two practical requirements for low-resource robot-policy localization: build a guaranteed null before trusting a metric, and replicate low-resource-language results across seeds.
1. Introduction
This study evaluates whether machine-rephrased Greek instructions can localize an English-centered robot policy without architectural changes, finding that measurement and controls are central to interpreting transfer. A multilingual tower and bilingual demonstrations enable partial Greek behavior, while several intuitive interventions and weak benchmarks fail.
- Machine translation took hours, but measuring whether a policy follows Greek became the study’s main challenge.The authors frame the work as a study of controls rather than a system recipe.
- A multilingual tower is required for coherent Greek-conditioned generation; an English-centric tower remains noisy despite supervised fine-tuning.The multilingual result used a subsequent 6,836-clip run built from Greek captions and LIBERO renders.
- 64% of Greek prompts produced coherent localized-world-model scenes, but 0% matched specific prompt content and 30% partially matched.Greek conditioning transmitted domain information more reliably than task-specific details.
- Warm-starting from a Greek-adapted world model degraded both languages, with English falling from 96.4% to 79.2% and Greek from 48.6% to 13.6%.The result suggests video-generation fine-tuning can erode action-relevant representations.
- 84.6% Greek success versus 82.6% under wrong instructions shows that a single-goal benchmark can obscure language dependence.The authors therefore use a language the policy is independently unable to follow as a guaranteed null.
2. Related Work
Prior work shows that multimodal policies can exploit non-language cues, making standard instruction-following benchmarks unreliable. This paper extends that methodological concern to Greek localization with a guaranteed language-null control, while noting machine-generated target-language data as a live confound.
- Scene identity can predict demonstrated actions better than instruction text, producing causal confusion in imitation policies.
- LIBERO and related benchmarks vary goals or chain instructions to reduce affordance-based shortcuts that bypass language.
- A guaranteed language-null control uses an identically trained policy that is independently unable to follow the probe language, complementing perturbation-based controls.
- The paper studies machine-rephrased Greek instructions without changing the architecture, positioning localization as an embodied analogue of multilingual transfer.
- Machine-generated target-language instructions remain a live confound, and the paper reports no Greek robot demonstration corpus authored by humans.
- Unfreezing pretrained representations can degrade out-of-distribution performance, while this paper reports a corresponding cross-lingual tower-unfreezing result.
- Warm-starting from a video-generation checkpoint degraded both languages here, despite differing prior setups that reported benefits from videogenerative pretraining.
3. Setup
The studies use Cosmos3’s shared language pathway to test Greek localization in generated video and closed-loop manipulation, using machine-rephrased data and fixed architecture. Evaluation emphasizes correct, wrong, and cross-seed comparisons, while acknowledging tokenizer, bootstrap, calibration, multiplicity, and seed limitations.
- Model and evaluation setup: Cosmos3 combines a video world model with an action policy whose instruction pathway uses the same text tower and whose actions are evaluated in closed-loop LIBERO.
- Model and evaluation setup: Figure 1 traces pipe-separated English and Greek instructions through tokenization and a shared text tower before video or action evaluation.
- Language data and tokenization: The target-language corpus is machine-rephrased Greek, covering structured scene captions and 53,207 unique task instructions from two suites.
- Language data and tokenization: Greek instructions produce 274 tokens versus 72 English tokens across ten evaluations, an inflation of 3.81× with near character-level segmentation.
- Evaluation and uncertainty: Policies are evaluated on correct English, correct Greek, and deliberately wrong Greek instructions using the same tasks, seeds, and initial states.
- Evaluation and uncertainty: The ten-goal percentile bootstrap is mildly anti-conservative, with nominal 95% intervals covering roughly 89–91% under the authors’ rate profiles.
- Evaluation and uncertainty: 31.6 points separate the three Greek success rates of a reseeded reference policy, while English varies by only 1.0 point.
- Evaluation and uncertainty: The authors report no multiplicity correction and use a p = 0.100 ordering rule for cross-policy claims rather than treating it as significance.
4. Study 1: Localizing the World Model
Study 1 tests whether Greek can be localized in Cosmos3’s video world model and separates tower comprehension from generation-pathway adaptation. English-centric towers remain unable to learn Greek through SFT, whereas a multilingual tower plus generation training produces coherent but only partially grounded Greek scenes.
- Tower localization: Caption SFT cannot induce Greek in an English-centric text tower: Greek-conditioned generation remains structureless noise across the dose ladder.
- Tower localization: A multilingual tower alone is insufficient because its generation pathway has never seen Greek-derived conditioning.
- Generation-pathway adaptation: A stock generation-pathway fine-tuning recipe followed by a 6,836-clip mixed run produces coherent Greek-conditioned robot scenes without architecture changes.
- Grounding transfer: English generation was fully coherent with a good content match on 90% of held-out clips.
- Grounding transfer: 64% of Greek-conditioned clips were judged coherent, but none achieved a good content match and 30% achieved a partial match.
- Grounding transfer: Greek conditioning transferred scene-level grounding more reliably than prompt-specific content grounding.
5. Study 2: Does the Policy Read?
The study shows that apparent Greek success can arise from benchmarks, translator phrasing, or motion priors rather than reliable language reading. Discriminative controls and replicated comparisons reveal partial, non-compositional grounding with substantial phrasing sensitivity.
- Benchmark validity: 84.6% Greek success versus 82.6% under deliberately wrong instructions shows that single-goal suites cannot attribute language.The matched suite hides whether the policy follows the instruction because each scene admits one trained goal.
- Benchmark validity: 48.6% Greek success on ten goals is misleading because per-goal behavior is bimodal and some Greek results match or fall below the guaranteed null.The bilingual policy reaches 0.72–0.98 on five goals and 0.00–0.24 on five, while its largest genuine gains are +0.84 on two goals.
- What the policy reads: Greek-only training remains weak and inconsistent: its Greek and English success correlate at 0.95 because neither instruction changes behavior.The wrong-instruction residue is command-dependent, but the study could not identify the feature governing which wrong command elicits which behavior.
- Translator dependence: The apparent translator penalty is not monotonic with visual or structural distance: one replicated comparison reverses by 36 points.Training on multiple phrasings improves performance broadly, but the amount lost under rewording is not predicted by how distant the phrasing looks.
- Translator dependence: Changing phrasing costs Greek accuracy far more than English accuracy: Greek drops average 23.3 points versus 3.9 points for English.The matched English perturbation still costs 4.7 points, showing that perturbation size contributes, but Greek remains substantially more sensitive.
- Interpretation: A large fraction of the headline Greek result is translator-specific, so the study does not treat it as general Greek competence.The conclusion is partial, non-compositional referent-level grounding rather than a uniform language capability.
6. The Real-Robot Corpus We Cannot Score
The large real-robot corpus could be translated cheaply but could not be evaluated with the paper’s closed-loop success protocol. A proxy metric detects a small English instruction signal while Greek remains indistinguishable from its null at the tested training stage.
- Corpus scope: 53,096 translated DROID instructions cover 57,639 real-robot success episodes, but the corpus cannot be scored with closed-loop success without a physical Franka arm.Every reported number instead comes from the much smaller LIBERO suite.
- Measurement: The available proxy is action-prediction error on held-out episodes under correct, unrelated, and Greek instructions, gated by English separability.If the proxy cannot distinguish correct English from an unrelated instruction, its Greek values are not interpreted.
- Measurement: English instructions improve action prediction by 2.28% (MSE 0.290 versus 0.297), while Greek is indistinguishable from the null.The English result is treated as directional evidence that the metric responds to language, not as a measured effect size.
- Scope: At 500 iterations of a recipe intended for 10,000, English conditioning is detectable but Greek conditioning is not.Whether Greek emerges later remains untested because the additional compute was judged disproportionate to the proxy’s limitations.
- Implication: The study’s practical lesson is that proxy metrics need a null before their results can be trusted.Translation was produced in hours, whereas evaluating the real-robot corpus was the bottleneck.
7. Lessons for Evaluation
The study finds that several common evaluation instruments can report apparent language understanding where none exists. Reliable evaluation requires guaranteed null controls, multi-goal suites, and replicated low-resource runs.
- A color-histogram metric reported improvement while both generations remained structureless noise whose palette drifted toward the reference.
- 84.6% Greek success versus 82.6% under deliberately wrong instructions shows that single-goal benchmarks cannot falsify language-following claims.
- Greek success varied from 48.6% to 80.2% across seeds while English moved by one point, making single-run low-resource comparisons unreliable.
- Across six policies, loss varied by 1.4% while Greek success varied by a factor of 7.2, so training loss did not rank instruction following.
- On ninety tasks, Greek-only training exceeded its wrong-instruction floor by at most 2.7 points, overturning the stronger ten-goal interpretation.
- The five retractions show that positive results need controls or replication because cheaper measurements repeatedly reported success where none existed.
8. Limitations
The evidence is limited to machine-generated Greek, one model family, simulation, and studied goals, with several unresolved measurement and statistical boundaries. The authors also leave key mechanisms and broader generalization unanswered.
- The study uses one language, one model family, simulation only, and machine-generated Greek rather than human-authored Greek.
- The independent evaluation set uses a different language model rather than native speakers, so the idiolect result is bounded to machine-rendering differences.
- World-model binding relies on one calibrated VLM judge, with calibration validating coherence but not the reported three-way content-match judgment.
- Every evaluated goal was seen in training, so the study measures no generalization to novel tasks; unseen-phrasing performance also remains seed-dependent.
- The Greek–English gap remains mechanistically unresolved because representational depth and tokenizer fragmentation, measured at 3.81×, are entangled.
- Three seeds per arm separate recipes on ninety tasks but cannot push the exact permutation test below its p = 0.100 floor.
- The phrasing-diversity benefit was tested only on ten goals, leaving its extension to ninety-task training sets untested.
9. Conclusion
The study concludes that Greek localization is feasible only when the underlying language tower already supports it and when target-language demonstrations are added. Its broader contribution is an evaluation discipline built around guaranteed nulls and replication.
- Machine-produced Greek plus bilingual demonstrations reached roughly half of English performance on ten goals and two fifths on ninety tasks, while target-only training barely exceeded its control.
- Greek success was bimodal across goals, with five goals at 0.72–0.98 and five at 0.00–0.24 at the profiled seed.
- The practical localization order is to check the tower, add target-language and high-resource demonstrations, leave the pretrained representation alone, and use a guaranteed null.
- The authors expect early work in this area to calibrate instruments because prior negative results are scarce.
- Five preliminary conclusions were overturned by controls or replication, supporting the transferability of the evaluation controls over the reported numerical results.
10. Availability
The paper provides a bilingual Greek policy checkpoint for direct use in the Cosmos3 policy server.
- The released model is a Hugging Face safetensors export that loads directly into the Cosmos3 policy server.
A. Every inferential comparison
The paper’s recipe pairs prerequisite checks and null-controlled evaluation with replicated comparisons. Its results show that multilingual support alone is insufficient, Greek demonstrations have limited isolated benefit, and bilingual training performs better while several seemingly helpful interventions hurt.
- Inferential comparisons: At α = 0.05 over twenty comparisons, Bonferroni leaves only within-policy task-level contrasts significant; three runs per arm cannot support corrected cross-policy claims.Within-policy contrasts use tasks as the sampling unit, while cross-policy contrasts use training runs and have a smallest attainable two-sided p of 0.100.
- Measurement and recipe: Tokenizer fertility is only a cheap first look; the text tower must already handle the target language before training can proceed.The English-centric tower produces noise on Greek, whereas the multilingual tower yields coherent Greek scenes on 64% of prompts.
- Measurement and recipe: Warm start 79.2 / 13.6% (EN / EL); from base 96.4 / 48.6%.The result supports training from the base checkpoint rather than a language-adapted world model.
- Replicated comparisons: Greek margin over the wrong-instruction floor 6.7–7.1 points on every seed; Greek demonstrations alone: at most 2.7.The replicated margins distinguish bilingual training from the limited benefit of Greek-only training.
- Measurement and recipe: Unfrozen: 82.4 / 18.0%, worse in both languages, replicated.Leaving the text tower frozen is the better-performing intervention in this comparison.
- Measurement and recipe: Single-goal suite: 84.6% Greek against 82.6% wrong, showing why evaluation needs multiple goals and a wrong-instruction control.The control costs one evaluation run and exposes the weak discrimination of the single-goal result.
B. Per-goal profiles and the Greek instructions
Greek performance is bimodal and seed-sensitive across goals, so one run cannot define the recipe’s profile. Persistent failures are not explained by translation adequacy alone, because a clear translation can fail while ambiguous translations can succeed.
- Per-goal profiles: The number of goals above 50% is five, nine, and six across seeds, with only goal 0 below 50% in every seed.Thus, “five succeed and five fail” describes one run rather than the bilingual recipe.
- Greek instructions: Goal 0 persistently fails despite a literal, unambiguous translation, and an independent translator produces the same sentence.This weakens the explanation that its repeated failure is caused by incorrect translation.
- Greek instructions: Ambiguous stove translations yield 24/60/58 for goal 1 and 72/96/96 for goal 5, so translation adequacy does not predict either per-goal profile.The two ambiguity cases produce sharply different outcomes across the reported runs.