Retrieve What Helps, Only When It Helps: Agentic Reinforcement Learning for Selective Retrieval-Augmented Machine Translation

1 Introduction

• Translation is a partial-evidence regime: parametric knowledge suffices for some inputs, while retrieval can help others or introduce noisy, misleading context.Similarity-based retrieval may miss actionable evidence even when it finds semantically close documents.Source: §1 · pp. 1–2
• SRAT treats reasoning, retrieval, and final translation as actions in a unified policy optimized using complete trajectories and final translation quality.GRPO lets trajectories with different retrieval decisions and queries compete for the same input.Source: §1 · p. 2
• Across idiom, knowledge-intensive, and document-level translation, SRAT consistently outperforms never-retrieve and always-retrieve baselines.The settings include offline retrieval for the first two and offline or online retrieval for document-level translation.Source: §1 · p. 2
• Quality-aligned retriever training provides further gains when similarity-based retrieval is insufficient.The retrieval policy can also be optimized with a frontier black-box translator, outperforming its direct translation and supporting modular optimization for external models.Source: §1 · pp. 2–3
• Joint Plackett–Luce optimization and translation-quality contrastive training align retrieval with downstream translation quality rather than semantic similarity alone.The former propagates trajectory-level reward to retrieval; the latter derives direct supervision from controlled evidence interventions.Source: §1 · p. 2

2 Related Work

• Translation-specific adaptation improves translators, while inference-time information acquisition often remains outside the modeling problem.Deliberation approaches add analyses or refinement, but their form and amount are typically fixed in advance.Source: §2 · p. 3
• Retrieval-augmented MT uses examples, idiom knowledge, documents, or knowledge graphs, but systems often retrieve by default or route with predefined rules.Retrieval is also often optimized separately from the final translation objective.Source: §2 · p. 3
• Unlike answer-oriented retrieval, translation can produce plausible outputs without retrieval, and useful evidence may guide wording without containing the target translation.This motivates selective retrieval aligned with translation quality.Source: §2 · p. 3

3 Method

• SRAT frames selective retrieval as part of the translation policy and aligns document selection with downstream translation quality.The method addresses both when to retrieve and what evidence to return.Source: §3 · p. 3, §3.3 · p. 4

3.1 Selective Retrieval as a Translation Policy

• The policy can reason, query a retriever when evidence is insufficient, or terminate with a translation, including directly from the initial state.Retrieved documents enter the interaction state, where the model can reason further, query again, or translate.Source: §3.1 · pp. 3–4

3.2 Learning When to Retrieve

• Relative trajectory rewards train retrieval decisions without labels specifying which inputs require external knowledge.The reward combines translation quality with penalties for malformed outputs and excessive redundancy; the curriculum initializes retrieval behavior before optimizing the final objective.Source: §3.2 · p. 4

3.3 Learning What to Retrieve

• The retriever is aligned to translation quality through joint policy-gradient training and a separate contrastive approach.The contrastive approach evaluates candidates by their effect on translation, then uses the best-performing candidate as direct supervision while the translator is fixed.Source: §3.3 · p. 4, §3.3.2 · p. 5

3.3.1 Joint Policy-Gradient Alignment

• Plackett–Luce sampling exposes retriever selection probabilities so trajectory advantage can update both translator and retriever.The stochastic ranking distribution supports gradient-based updates despite discrete document selection.Source: §3.3.1 · pp. 4–5, Figure 2 · p. 5

3.3.2 Separate Contrastive Alignment

• Contrastive alignment selects documents by measured translation effect, giving the retriever direct, lower-variance supervision.Unlike joint optimization, it requires additional candidate evaluations while keeping the translator fixed.Source: §3.3.2 · p. 5, Figure 3 · p. 5

3.4 Inference

• At inference, the learned policy decides whether retrieval is needed, queries when needed, and translates without a reference or quality evaluator.A maximum retrieval-call limit and output-token budget bound worst-case latency.Source: §3.4 · p. 5

4.1 Experimental Setup

• The experiments cover idiom translation with short explanations, knowledge-intensive translation with multilingual documents, and document-level En→Ru subtitle translation with online search.The first two settings use offline retrieval corpora; Document-MT-LLM has no predefined collection and requires policy-generated queries to an online search engine.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 RQ1: Does Selective Retrieval Improve Translation?

• SRAT achieves the strongest COMET, task-specific LLM-judge, and MQM results across settings, improving over the strongest SFT baselines on IdiomKB, RAGtrans, and Document-MT-LLM.Reported gains include I5 by 0.096 and MQM by 1.27 points on IdiomKB Zh→En; GRF by 0.019 and MQM by 0.82 points on RAGtrans En→Zh; and GRF by 0.020 and MQM by 0.42 points on Document-MT-LLM.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 7
• Against matched never- and always-retrieve policies, SRAT performs better across all three benchmarks while learning retrieval rates that vary by setting.Relative to SRAT-A, MQM improves by 20.6% and 17.4% on the two IdiomKB directions, by 8.0–12.8% on RAGtrans, and by 5.4% on Document-MT-LLM.Source: §4.1 · p. 6, §4.2 · p. 7

4.3 RQ2: When Does Utility-Aligned Retriever Training Help?

• On RAGtrans, contrastive retriever training raises retrieval accuracy from 56.3% to 70.7% on En→Zh and from 72.6% to 81.4% on En→De, with MQM gains of 6.8% and 7.4%.On IdiomKB, where the default retriever is already about 95% accurate, alignment improves accuracy only to 97–98%, with limited downstream gains.Source: §4.3 · p. 7, Table 3 · p. 8
• On RAGtrans, contrastive training improves MQM over joint training from −4.41 to −4.27 on En→Zh and from −5.89 to −5.54 on En→De.The paper attributes the difference to contrastive training’s direct document-level supervision from controlled candidate interventions, rather than a shared trajectory reward across many decisions.Source: §4.3 · pp. 7–8

4.4 RQ3: Can Selective Retrieval Improve Strong Black-Box Translators?

• A separate Qwen-based SRAT policy learns retrieval decisions using a frozen Claude translator and Claude’s final translation quality as reward, without modifying Claude’s parameters.The paper reports the same overall trend with frozen GLM 5: prompted retrieval degrades translation, while a trained SRAT policy improves over direct translation.Source: §4.4 · p. 8
• Prompted retrieval decisions from Claude are not consistently beneficial, whereas SRAT-Claude provides more reliable gains over direct Claude translation.SRAT-Claude improves most metrics on IdiomKB with 1.9–10.2% retrieval; on RAGtrans, MQM improves by 12.4% on En→Zh and 16.8% on En→De.Source: §4.4 · p. 8

• SRAT learns when and what to retrieve for LLM-based translation.Source: §5 · p. 8
• Across offline and online retrieval settings, SRAT consistently outperforms fixed never- and always-retrieve strategies.Source: §5 · p. 8
• Utility-aligned retriever training improves difficult retrieval settings, and adaptive retrieval enhances a frozen black-box translator beyond direct translation.Source: §5 · p. 8

A.1 Results on Document-MT-LLM

• SRAT is best among non-oracle systems on COMET, GRF, GRB, and MQM, improving over SRAT-A by 1.6% COMET, 2.7% GRF, and 5.4% MQM.The policy retrieves on 88.6% of examples, yet its advantage over always retrieving indicates external evidence is not uniformly beneficial in this online, document-level setting.Source: §A.1 · p. 11, Table 5 · p. 11

A.2 Technical Details

• Training uses GRPO with a Flat E5 retriever, up to five interaction turns, and three-seed mean reporting with paired t-tests for significance.The reported training runs use Qwen2.5-7B-Instruct by default and take 47–382 GPU hours across the listed dataset directions.Source: §A.2 · p. 11

A.3 Comparison with Oracle Retrieval Policy

• The Oracle approaches SRAT on some settings but shows the largest headroom on RAGtrans En→De, where it reaches 0.869 COMET and −5.25 MQM versus SRAT’s 0.857 and −5.98.On IdiomKB Zh→En, SRAT reaches 0.822 COMET and −2.74 MQM compared with the Oracle’s 0.831 and −2.67.Source: §A.3 · p. 11, Table 7 · p. 12
• The Oracle retrieves less often than SRAT on RAGtrans—37.7% vs. 52.2% for En→Zh and 30.4% vs. 43.0% for En→De—so remaining gains may depend on better instance selection, not more retrieval.Because it selects using the blended training score, the Oracle is not a strict upper bound on every evaluation metric.Source: §A.3 · p. 12

A.4 Learned Policy Behaviors

• Multi-retrieval is rare on IdiomKB (0.6–1.4%), rises on RAGtrans (1.9–2.1%), and reaches 3.1% on Document-MT-LLM.Explicit reasoning is nearly absent on IdiomKB (0–0.1%), present on 89–92% of RAGtrans examples, and used on 66.3% of Document-MT-LLM examples.Source: §A.4 · p. 12, Table 8 · p. 12

A.5 Generalization to Another Black-Box Translator

• GLM-Ret worsens MQM from −1.666 to −2.076 (En→Zh) and −1.668 to −1.956 (En→De), while SRAT-GLM retrieves on 32.3%/24.0% and slightly beats direct GLM.GLM-Ret retrieves on 68.1% and 79.7% of inputs, respectively.Source: §A.5 · p. 13

B.1 Contribution of SFT and RL

• On IdiomKB Zh→En, RL raises I5 from 0.640 to 0.731 and improves MQM from −3.95 to −2.74, after an SFT warm start that performs similarly to CSC.BLEU decreases slightly after RL, consistent with reward optimization for COMET and LLM-judged quality rather than reference n-gram overlap.Source: §B.1 · p. 13

B.2 Single-Round vs. Multi-Round Optimization

• On En→De, round-one GRF/GRB gains are 0.175/0.177 versus 0.000/0.002 in round two; En→Zh GRF gains are 0.073 then 0.001.The reported gains diminish substantially after the first round.Source: §B.2 · p. 13
• Most first-round improvement comes from translator RL, and the negligible gains from another costly full cycle motivate using one round by default.The additional round requires both a full RL stage and retriever training.Source: §B.2 · p. 14

B.3 Robustness to LLM Evaluators

• Across both datasets and both alternative judges, SRAT achieves the best score on every reported metric, with system ordering highly consistent with the Sonnet-based evaluation.The reported consistency indicates the gains are not artifacts of optimizing toward one particular LLM judge.Source: §B.3 · p. 14

B.4 Robustness to a Stronger Backbone

• On RAGtrans, MQM improves by 7.6% over CSC and 4.7% over SRAT-A; on IdiomKB, MQM rises from −1.87 for CSC to −1.33 with Qwen3-8B.SRAT achieves the best COMET, task-specific judge, and MQM scores in both settings while retrieving selectively.Source: §B.4 · p. 14

B.5 Effect of the Number of Retrieved Documents

• On IdiomKB, k = 1 gives the strongest COMET, I5, I3, and MQM, while k = 5 degrades all metrics; on RAGtrans, k = 3 trades slight BLEU and COMET gains for lower GRF, GRB, and MQM than k = 1.Increasing k further to 5 also decreases RAGtrans performance; retrieval rates decrease as more documents are returned per action.Source: §B.5 · p. 14

B.6 Robustness to Online Retrieval Backends

• Replacing DuckDuckGo with Google Search improves every translation metric on Document-MT-LLM while keeping the learned policy unchanged.The result suggests SRAT can exploit higher-quality retrieved evidence rather than depending on a particular search backend.Source: §B.6 · p. 15

B.7 Human Evaluation

• The blinded evaluation covers 32 En→Zh examples from IdiomKB and RAGtrans, assessed by 20 annotators across accuracy, fluency, style, terminology, and non-translation.Annotators were native Chinese speakers with advanced English proficiency; scores are negative error penalties, with values closer to zero indicating better quality.Source: §B.7 · p. 15
• Human MQM ranks SRAT (−1.94) above SRAT-A (−2.77), CSC (−3.56), and Direct (−6.32); SRAT also reduces accuracy and terminology penalties versus Direct.Relative to SRAT-A, SRAT scores better across all four reported dimensions, with the largest absolute improvement in accuracy, from −1.15 to −0.81.Source: §B.7 · p. 15

C Limitations

• The evaluation spans several settings and language directions, but broader language and specialized-domain coverage is needed to establish applicability.Human evaluation is relatively limited in scale, and retriever alignment is learned for a particular translator; transfer across models and retrieval from more sources remain open directions.Source: C Limitations · pp. 15–16

D Ethical Considerations

• Retrieved sources may be inaccurate or biased, online queries may expose source text, and policy training and external evaluations consume computational resources.The authors identify careful source selection, human review, privacy-aware deployment for confidential material, and more economical adaptation as relevant considerations.Source: D Ethical Considerations · p. 16

E.1 When Retrieval Helps

• When parametric knowledge is insufficient, retrieved evidence can resolve idioms, names, or technical concepts and improve translation quality.Examples include evidence clarifying an idiom’s intended sense, supplying a canonical name and rank, or correcting a technical terminology error.Source: §E.1 · pp. 16–17

E.2 When Retrieval Hurts

• Retrieval can worsen a translation when it overrides a better direct rendering, matches a query to unrelated content, or introduces factual distortion.The examples include loss of figurative register, character-overlap mismatch for a technical term, and a retrieved description inducing an erroneous claim.Source: §E.2 · p. 17
• Even correct evidence can encourage literal transfer of a culture-specific image when the direct translation already gives a natural English equivalent.In the cited idiom case, the retrieved explanation was correct, but its use produced a less suitable literal rendering.Source: §E.2 · p. 17

E.3 Semantic Similarity Does Not Imply Translation Utility

• A lexically stronger match can be less useful for translation than a slightly lower-ranked document that covers the source’s broader concept or needed details.The equation-of-state example contrasts a higher-similarity document about gas equations with a lower-similarity document addressing the broader thermodynamic concept; a North Korea example likewise contrasts crisis history with relevant agricultural geography.Source: §E.3 · p. 18

E.4 When BLEU Disagrees with Translation Quality

• High BLEU can favor surface overlap despite incorrect terminology, while semantic and error-based metrics can favor a paraphrase with the correct term.In the Go example, CSC receives higher BLEU despite using an incorrect term; SRAT uses the correct term and ranks better under GRF and MQM.Source: §E.4 · p. 19

E.5 How Black-Box Translators Use Retrieved Evidence

• With the same retrieved context, Claude incorporates useful information more selectively than GLM, which can copy regional style or let off-topic evidence overwrite terminology.The examples show both translators extracting a useful founding name, but GLM also copying the document’s style; in another case GLM changes a correct term while Claude preserves it.Source: §E.5 · p. 19

F LLM-as-Judge Evaluation Templates

• The reported evaluation templates define I3, I5, MQM, GRF, and GRB scoring procedures, with GRF and GRB raw scores divided by 100 for reporting.I5 uses a five-level scale, and MQM follows the ThinMQM template.Source: F LLM-as-Judge Evaluation Templates · p. 19

MQM (ThinMQM)

• The template lists translation error categories, classifies errors as critical, major, or minor, and deducts −25, −5, or −1 points per error span, respectively; no-error receives 0.Source: MQM (ThinMQM) · p. 20