Source-linked AI summary
StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translation
Rohit Dhaipule, Sukhdeep Singh Kharbanda, Prasanth Bathala, Pradyumna Lanka, Anubhav Shrimal
TL;DR
When stronger machine-translation models replace earlier ones, available human post-edits may be stale, while collecting fresh edits for every version is prohibitively expensive. StalePO combines downward likelihood movement, base-response anchoring, and token-level constraints; it improves all-pass rates on two language pairs, with Hindi gains also confirmed by human evaluation.
Problem
Method
StalePO combines downward likelihood movement on both responses, anchoring to the model’s base response, and per-token KL decomposition.
Source: §1 · p. 2Results
Takeaways & Limitations
StalePO learns to avoid failure modes in stale preferences rather than imitating them, with gains concentrated in style adherence and fluency in the reported evaluations.
Source: §7 · p. 8The primary evaluation relies on LLM-as-Judge MQM; human evaluation covers an English→Hindi subset, and English→Turkish results lack human evaluation.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
Machine translation systems are periodically upgraded to stronger models, but the available preference signal is human post-edits of an older system's outputs, which the newer model may already surpass. Moreover, collecting fresh post-edits for every new model is prohibitively expensive. We call this the Stale Preference problem. Standard DPO can fail in this setting: it may increase the likelihood of inferior post-edits, erode the model's existing quality, and fail to provide the per-token control needed to correct localized errors. We introduce StalePO, an objective derived from three requirements this regime imposes. Likelihood movement must be downward on both responses, the policy must be anchored to its own base response, and the KL constraint must apply at the token level. These requirements are jointly necessary. In ablations, each mechanism in isolation leaves the model's performance indistinguishable from the base model, and only their combination converts stale feedback into gains. On English-to-Hindi and English-to-Turkish localization data, StalePO improves the fraction of segments passing all LLM-as-judge MQM quality checks by 14.9 and 4.6 percentage points, respectively, with gains concentrated on style and fluency. A human evaluation under the same framework confirms these gains on English-to-Hindi, raising the fraction of segments passing all seven human checks by 13.8 percentage points.
1 Introduction
The stale preference problem arises when post-edits from weaker, earlier systems remain available because collecting fresh feedback for each model is costly. StalePO combines downward likelihood movement, base-response anchoring, and token-level constraints to learn from that feedback.
1 Introduction
- Older-system post-edits can be inferior overall to the current model’s outputs while still correcting genuine errors on dimensions such as target-style adherence.Fresh post-edits for every model version are prohibitively expensive in high-volume pipelines.Source: §1 · p. 1
- DPO constrains relative likelihood changes but not their absolute direction, its pair-only KL penalty does not preserve the model’s own generations, and its sequence-level signal lacks per-token control.These limitations matter when post-edits are inferior overall but contain localized corrections.Source: §1 · p. 2
- The paper formalizes stale preferences as inferior model-relative outputs that nevertheless carry useful signal on dimensions where the model underperforms.StalePO is introduced as an objective addressing this setting.Source: §1 · p. 2
- Stale preference optimization requires combining downward movement on both responses, base-response anchoring, and token-level KL constraints; each mechanism alone leaves performance indistinguishable from the base model.The downward movement is a tendency, not a guaranteed update direction.Source: Figure 1 · p. 1, §1 · p. 2
- StalePO improves all-pass rate by 14.9 pp on English→Hindi and 4.6 pp on English→Turkish, with professional-translator evaluation confirming a 13.8 pp gain.The results are reported for localization data.Source: §1 · p. 2
2 The Stale Preference Data Problem
Stale post-edits arise because fresh human editing is costly and remain tied to earlier systems’ wording. They can lose to the current model on most quality dimensions while winning on a small subset.
2 The Stale Preference Data Problem
- Stale post-edits typically lose to the current model on most quality dimensions but retain an advantage on a small subset S.The formalization requires a non-negligible fraction of examples with this pattern.Source: §2 · p. 2
- The stale-data setting reflects the economics of localization pipelines, where millions of monthly segments make fresh post-editing after each model replacement unaffordable.Because editors revise rather than retranslate, their edits remain tied to the earlier model’s lexical and syntactic patterns.Source: §2 · p. 2
- Learning from stale preferences requires using the preference ordering without imitating either response.The paper identifies this as a requirement existing methods do not address.Source: §2 · p. 2
3 Background and Related Work
DPO optimizes preference pairs through a contrastive objective, but its relative-likelihood constraint does not control absolute update direction. Related approaches address parts of this problem while leaving other limitations or the stale-preference setting unresolved.
3 Background and Related Work
- DPO optimizes a contrastive objective over preferred and dispreferred outputs using their implicit rewards relative to a frozen reference policy.The objective increases relative preference but does not constrain the absolute direction of either update.Source: §3 · p. 3
- APO-Down discourages likelihood increases for both responses while preserving preference separation, but lacks a base-response anchor and operates at sequence level.Repeated application can erode baseline generation quality, and sequence-level optimization cannot selectively target localized token errors.Source: §3 · p. 3
- Prior preference losses still treat the preferred response as a target, while online methods refresh data for a different form of staleness and token-level methods govern update locations rather than target imitation.The related-work discussion also distinguishes MT methods designed for high-quality chosen translations from this stale-preference setting.Source: §3 · p. 3
4 Stale Preference Optimization
StalePO combines downward likelihood movement, anchoring to the model’s own base response, and token-level regularization in one objective. Each component addresses a distinct limitation of stale-preference optimization.
- StalePO combines directional bias, base-response anchoring, and token-level regularization to address the limitations of stale-preference optimization.Source: §4 · p. 3
- The preference term discourages increasing the preferred response’s likelihood while assigning a larger penalty to the dispreferred response, preserving their relative ordering.Source: §4 · p. 4
- The one-sided anchor activates only when the policy assigns its base response lower likelihood than the reference, pushing likelihood back toward the reference.Source: §4 · p. 4
- Sequence-level log-ratios do not control where displaced probability mass goes, allowing updates to concentrate next-token distributions on dominant tokens and erode base-model diversity.Source: §4 · p. 4
- Sequential forward KL applies per-token constraints so local deviations cannot cancel across positions, while its mass-covering effect discourages mode collapse and preserves diversity.Source: §4 · p. 4
5 Experiments
The experiments evaluate translation quality with seven-category MQM judgments and test sets for English→Hindi and English→Turkish. The setup also describes filtered and unfiltered post-edit data, training procedures, and baseline component configurations.
5.1 Evaluation
- The primary evaluation uses seven binary LAJ-MQM judgments per segment and defines all-pass as passing all seven; Table 1 covers two LAJ-MQM test sets and a human-judged English→Hindi subset.Source: §5.1 · p. 4, Table 1 · p. 5
- Professional translators evaluate a 2,000-segment English→Hindi subset under the same seven-category taxonomy, with pass-rate changes agreeing with LAJ-MQM in direction and concentration.Source: §5.1 · p. 4
5.2 Setup
- English→Hindi post-edits are filtered for stylistic modernization and punctuation normalization, concentrating supervision on style and fluency; English→Turkish post-edits are retained unfiltered.Source: §5.2 · pp. 4–5
- Training examples pair a source prompt with a human post-edit, raw NMT output, and sampled GPT-OSS 120B base response; evaluation uses temporally disjoint, stratified test sets for both language pairs.Source: §5.2 · p. 5
- The baselines vary the downward, anchor, token-KL, and post-edit-supervision components; BAPO-Down combines downward control with the anchor but omits token-level KL.Source: §5.2 · p. 5, Table 2 · p. 5
- Figure 2 plots pass-rate change versus the base model across token-level KL coefficient α and anchor strength λ sweeps, with the other coefficient held fixed in each panel.Source: Figure 2 · p. 6
6 Results
StalePO improves translation quality across English→Hindi and English→Turkish, with gains concentrated in style and fluency. Ablations and training dynamics show that performance depends on combining anchoring, directional guidance, and token-level constraints.
6 Results
- English→Hindi all-pass rate improves by 14.9 pp under LAJ-MQM and by 13.8 pp under human MQM; English→Turkish performance improves by 4.6 pp.Hindi gains are concentrated in Style (+20.9 pp) and Fluency (+16.5 pp), while LAJ-MQM Accuracy decreases by 5.4 pp; Turkish gains are concentrated in Style (+2.3 pp) and Fluency (+1.3 pp).Source: §6 · p. 6
6.1 Baseline Ablation
- SFT and single-component methods remain near the base model, while StalePO combines anchoring, directional guidance, and token-level constraints to improve quality.Tuned DPO shifts all-pass by −0.4 pp, TDPO by 0.5 pp, BAPO by −0.2 pp, and APO-Down by 0.6 pp; SFT raises all-pass by only 1.8 pp.Source: §6.1 · pp. 6–7
- StalePO improves all-pass rate by 14.9 pp, compared with 7.4 pp for BAPO-Down; its per-position divergence constraint partially recovers Accuracy.BAPO-Down gains 15.4 pp in Style and 16.9 pp in Fluency, with Accuracy down 6.4 pp; StalePO's Accuracy decrease is 5.4 pp.Source: §6.1 · p. 7
6.2 Hyperparameter Ablation
- The all-pass rate peaks at α = 0.25 and remains substantially above the base model at higher α values; adding an anchor improves metrics, with effects plateauing at higher λ.Without an anchor, metrics remain near the base model; even λ = 0.5 produces substantial gains.Source: §6.2 · p. 7
6.3 Training Dynamics
- Anchoring stabilizes directional variants: StalePO and BAPO-Down meet the anchor constraint within 70 steps and end two orders of magnitude below unanchored runs on LAnchor.StalePO also retains a slightly larger late-training KL margin and early learning signal than BAPO-Down, consistent with token-level KL sustaining asymmetric adaptation.Source: Figure 3 · p. 7, §6.3 · p. 8
7 Conclusion
The conclusion formalizes stale preferences in machine translation and proposes StalePO. It reports gains on two language pairs and a human evaluation, while identifying broader application and richer preference data as future directions.
7 Conclusion
- StalePO improves the proportion of error-free segments by 14.9 pp on English→Hindi and 4.6 pp on English→Turkish.The largest gains are in style adherence and fluency, and human MQM evaluation corroborates the English→Hindi result at 13.8 pp.Source: §7 · p. 8
- The conclusion frames stale preferences as arising whenever model updates outpace fresh preference-data collection, beyond machine translation alone.It presents learning to avoid stale-preference failure modes, rather than imitating them, as StalePO's mitigation approach.Source: §7 · p. 8
- Extending StalePO to other domains and richer, multi-error preference data is identified as a promising future direction.Source: §7 · p. 8
Limitations
The evaluation relies primarily on LLM-as-Judge MQM, with human validation on a subset; StalePO also has specific boundaries on when stale preference data provides a learning signal.
Limitations
- Budget constraints prevent professional human MQM annotation at test-set scale, so the primary evaluation uses LLM-as-Judge MQM validated against human annotations.The human validation is limited to an English→Hindi subset.Source: Limitations · p. 8
- StalePO cannot improve outputs when the updated model is already better than the post-edits on every dimension, because the advantage subset is empty.In that case, the preference pair carries no signal to learn from, so the objective can only preserve existing behavior.Source: Limitations · p. 8
- StalePO is not designed for cases where post-edits are uniformly better than the model; methods that imitate the preferred output are more appropriate there.Source: Limitations · p. 8
A Lexical Bias of Post-Edits Toward the Legacy NMT System
On English→Hindi training data, post-edits are closer to legacy NMT outputs than to the base model’s outputs, while reference-based metrics can penalize systems with higher all-pass gains.
A Lexical Bias of Post-Edits Toward the Legacy NMT System
- Across 24,220 English→Hindi training segments, the comparison measured TER from post-edits to legacy NMT outputs and GPT-OSS 120B base outputs.Source: A Lexical Bias of Post-Edits Toward… · p. 10
- 2.3× more segments were closer to legacy NMT than to GPT-OSS base (55.3% vs. 23.7%); mean TER was 0.426 vs. 0.517.The remaining 21.0% of segments were equidistant.Source: A Lexical Bias of Post-Edits Toward… · p. 10, Table 3 · p. 10
- SFT leads post-edit-reference metrics with the lowest TER and highest chrF and BLEU, including an 8.4-point BLEU gain over base, but its all-pass gain is only 1.8 pp.Table 4 reports TER, COMET, chrF, and BLEU on the 5,000-segment test set, averaged over five runs.Source: A Lexical Bias of Post-Edits Toward… · p. 10, Table 4 · p. 11
- StalePO and BAPO-Down have the largest all-pass gains (+14.9 pp and +7.4 pp) yet the worst TER; StalePO is also worst on COMET, chrF, and BLEU.The paper attributes this inversion to higher-quality translations diverging from NMT-anchored post-edits, including style-guide-mandated conversational transliterations.Source: A Lexical Bias of Post-Edits Toward… · p. 11