PaperlayerView exact revision on arXiv

Source-linked AI summary

Length-Adaptive Decoding for Masked Diffusion Machine Translation

Yan Zhan, Mengkai Hou, Wanting Zhang, Zhijun Gao

arXiv:2608.22274v1cs.CLcs.AIcs.LG

TL;DR

Fixed-canvas masked-diffusion translation must supply target length before denoising, but this decision has received less attention than token reveal order. The paper introduces Entropy-Valley, which probes candidate canvases using all-mask predictive entropy and selects the lowest-entropy option. EV recovers substantial COMET-22 gains over corpus-ratio lengths across three directions, while its effectiveness varies by language pair and backbone.

  • Problem

    Fixed-canvas masked-diffusion MT must choose target length before denoising, while prior decoding work has focused mainly on token reveal order.

  • Method

    Entropy-Valley uses all-mask forward passes and mean predictive entropy to select a denoising-friendly canvas without additional training, reference lengths, or a length predictor.

  • Results

    64.9%, 65.3%, and 33.0% of the COMET-22 improvement from reference target lengths is recovered over corpus-ratio lengths on En→Zh, Zh→En, and En→De, respectively.

  • Takeaways & Limitations

    Target-canvas choice is a structural bottleneck: EV improves fixed-ratio decoding, preserves source content, and need not match reference lengths exactly.

  • Takeaways & Limitations

    Performance varies with backbone, tokenizer, language pair, and candidate range; broader generalization requires more model families and language pairs.

Abstract

from arXiv · show

Machine translation tests masked diffusion language models (dLLMs) because every source token must be rendered faithfully, while fixed canvas decoding must choose target length before denoising. Existing masked diffusion decoding work mainly studies token unmasking order, leaving this length decision under-explored despite its direct effect on coverage and redundancy. We introduce Entropy-Valley (EV), a training-free length selector that scores candidate target canvases by mean predictive entropy from all-mask forward passes and selects the canvas the backbone is most prepared to fill. Relative to a baseline using training corpus length statistics, EV recovers 64.9%, 65.3%, and 33.0% of the COMET-22 gain from reference target lengths on En$\to$Zh, Zh$\to$En, and En$\to$De. Our diagnostics show that denoising-friendly lengths need not match reference lengths. Evaluation by three translation experts supports the En$\leftrightarrow$Zh adequacy gains, with stronger evidence on Zh$\to$En. Compared with a LLaMA-3-8B autoregressive (AR) model trained on the same fine-tuning data, the EV system ties on En$\to$Zh and leads on Zh$\to$En; an oracle-length diagnostic further shows that, in this masked diffusion MT setting, deciding which tokens to reveal first matters less than how the target length is supplied.

1 Introduction

Fixed-canvas masked-diffusion MT must choose target length before denoising, making length selection a distinct bottleneck from reveal order. Entropy-Valley uses model uncertainty to select a denoising-friendly canvas and improves content retention and COMET-22 performance over corpus-ratio lengths.

  • Motivation: Machine translation requires a canvas long enough to preserve source content but not so long that it creates repetition or unsupported slots.The length decision is made before denoising, unlike autoregressive decoding, which generates until EOS.
  • Results: 89.1% placeholder retention and 81.3% number retention are achieved by EV on WMT22 En→Zh, versus 66.9% and 77.6% for the ratio baseline.The gains show that sentence-level length choice can preserve literal source-side content.
  • Research gap: Target length selection is a measurable bottleneck in fixed-canvas masked diffusion MT, alongside decoding-order choices.The paper compares these decisions under matched budgets.
  • Approach: EV addresses this bottleneck without additional training by using all-mask entropy to choose a canvas the backbone can denoise well.It probes candidate lengths before decoding and then decodes only the selected canvas.
  • Results: 64.9%, 65.3%, and 33.0% of the COMET-22 improvement from reference target lengths is recovered by EV on En→Zh, Zh→En, and En→De, respectively.EV does not simply match reference lengths.
  • Evaluation: EV is evaluated with statistical tests, expert judgments, coverage and error analyses, a matched-data LLaMA-3-8B AR baseline, reveal-order controls, and cross-backbone checks.These experiments test both the length-selection finding and its relation to alternative decoding choices.

2 Related Work

Related work addresses masked diffusion generation, MT length modeling, variable-length diffusion, autoregressive MT, and token reveal order. EV differs by selecting a fixed canvas at test time from the frozen backbone’s all-mask entropy without adding a length predictor or changing the decoder architecture.

  • Masked diffusion decoding: Masked diffusion language models use continuous-embedding, absorbing-state, or masked objectives, while fixed-canvas MT supplies target length before denoising.Recent LLaDA and Dream models report strong open-ended generation and reasoning results.
  • Length selection in machine translation: Non-autoregressive MT methods predict lengths, search length beams, alter training objectives, or use insertion and deletion operations.These approaches are designed around architectures or objectives that explicitly model length.
  • Variable-length diffusion language models: Variable-length diffusion methods adjust, expand, contract, calibrate, correct, or estimate output length during or before denoising.The cited methods include DAEDAL, ρ-EOS, CAL, LR-DLLM, and SmartCrop.
  • Autoregressive LLMs for MT: A matched-data LLaMA-3-8B LoRA-SFT baseline positions EV against autoregressive MT while targeting the separate bottleneck of fixed-canvas selection.The comparison calibrates the masked-diffusion setting rather than replacing EV’s objective.
  • Reveal-order methods: Reveal-order methods decide which masked positions to expose first, whereas EV decides which canvas to expose them on.The paper uses matched-budget controls to compare canvas-length sources with tested unmasking orders.

3 Method: Entropy-Valley

Entropy-Valley selects a target canvas before masked diffusion decoding by probing a compact set of candidate lengths with all-mask forward passes. It chooses the minimum-entropy candidate and applies the baseline MED schedule to that canvas without reference lengths or a trained length predictor.

  • Candidate canvases: EV constructs candidate canvas lengths from a fixed source-to-target ratio set and removes duplicate integer lengths.The candidate set is sentence-specific because lengths depend on the source tokenizer length.
  • Decoding: After selection, EV decodes the chosen canvas with the standard MED schedule for the same number of steps as the baseline.No additional decoding architecture is introduced.
  • Problem formulation: The selector operates on a frozen LoRA-SFT masked-diffusion MT system and returns a positive length from a finite candidate set.The final +1 reserves the designated EOS slot.
  • Cost and constraints: At most five distinct candidate lengths are probed before decoding, adding at most five probe passes to the baseline’s 32 denoising passes at T=32.Probes may be evaluated sequentially without extra peak memory or batched when memory permits.
  • Selection target: EV selects the length the backbone appears most prepared to denoise rather than recovering the reference length L∗.The selected canvas therefore need not match the reference exactly.
  • Limitations: The fixed candidate grid limits EV when the needed compression falls outside its range, although the compact grid avoids extra training.The candidate range is therefore an explicit inference-time boundary.
  • Entropy scoring: For each candidate L, EV runs one all-mask forward pass, computes mean predictive entropy over the first L−1 slots, and selects L⋆ = arg minL∈C(x) H̄(L).The EOS-designated slot is excluded so its near-zero entropy does not dominate comparisons across lengths.

4 Experiments

Experiments show that Entropy-Valley improves fixed-backbone masked-diffusion translation across directions and backbones, with strongest evidence on En↔Zh. Diagnostics attribute the gains to per-sentence canvas selection rather than reference-length matching, extra compute, or reveal-order choices alone.

  • Main results: 64.9% and 65.3% of the COMET-22 oracle gaps are closed by EV on En→Zh and Zh→En, compared with 33.0% on En→De.EV also gains +0.0172, +0.0165, and +0.0070 COMET-22 on the three directions, respectively.
  • Controls for the EV gain: Extra forward passes recover at most 0.001 COMET for Ratio, far below the En↔Zh EV gains, so additional probing does not explain the main result.A completed multi-candidate decoding alternative is also lower-scoring in the tested setting despite higher cost.
  • Scope across backbones: EV remains above Ratio across tested Dream-Base, DiffuLLaMA, and LLaDA systems, although oracle-gap closure varies by backbone.These cross-backbone comparisons support the criterion beyond LLaDA but remain scoped to the tested systems.
  • Controls for the EV gain: With reference lengths fixed, reveal-order schedules span less COMET variation than the tested length choices, though poor reveal orders can still hurt.The controlled En→Zh diagnostic therefore assigns the larger variation to target-canvas choice among the evaluated decisions.
  • Controls for the EV gain: EV closes more of the length-oracle COMET gap despite remaining far from the oracle in target-length error, showing that it selects denoising-friendly rather than reference-matching canvases.A fixed ratio cannot reproduce the gain because EV changes the canvas per source sentence.
  • Human evaluation and limitations: Human evaluation supports adequacy gains, especially on Zh→En, while En→De remains a boundary case and EV is scoped to fixed-backbone canvas selection.EV raises adequacy by +0.50 on Zh→En and +0.18 on En→Zh; fluency changes are small.

5 Conclusion

The paper identifies target-canvas selection as a structural bottleneck in fixed-canvas masked-diffusion MT and proposes Entropy-Valley as a training-free, model-internal test-time length decision. Across tested directions, EV improves over a fixed corpus ratio, aligns with En↔Zh adequacy judgments, and shows that canvas choice can matter more than reveal order.

  • Target-canvas selection is a structural bottleneck because short canvases can drop source content while overlong canvases create unsupported slots.
  • Entropy-Valley probes candidate all-mask canvases and selects the one the backbone appears most prepared to denoise, without adding a length head or using reference lengths.
  • EV improves over a fixed corpus ratio across the tested directions and preserves literal source content that short canvases can lose.
  • In the controlled En→Zh comparison, target canvas choice produces larger variation than the evaluated decoding-order choices.

Limitations

The method's scope is bounded by its candidate canvas set, backbone and tokenizer, language-pair behavior, and evaluation coverage. En→De is the clearest current boundary, while human evaluation is limited to En↔Zh.

  • Absolute scores and oracle-gap closures vary with the backbone and tokenizer, so broader generalization requires more model families and language pairs.
  • En→De is the clearest boundary for the current LLaDA system, with weaker sentence-level evidence and Ratio slightly exceeding EV at T≥64.
  • Human evaluation covers only En↔Zh, leaving En→De and De→Fr conclusions dependent on automatic metrics.
  • EV cannot select a needed canvas when the required compression falls outside its fixed candidate set.With R={0.70, . . . , 0.90} on En→Zh, EV cannot choose a canvas below 0.70|x|.

Ethical Considerations

The study uses public benchmark data and professional bilingual translators, with privacy protections and no private user-data collection. It also notes that public news text may contain names or sensitive events.

  • The paper uses public WMT news benchmarks under their public research terms or licenses.
  • The authors do not collect private user data or attempt to identify individuals.
  • Three professional Chinese-English bilingual translators evaluated anonymised En→Zh and Zh→En outputs with system identities hidden.
  • Evaluators consented to research use of ratings without personal identifiers and were compensated at local professional translator hourly rates.

A Reproducibility and Configuration

The experiments use public WMT22 test data, LoRA-adapted LLaDA and matched AR configurations, fixed decoding protocols, and three-run evaluations. EV adds a bounded inference-time probing cost while holding the reveal-order protocol fixed for length comparisons.

  • LLaDA-8B-Base is adapted with LoRA targeting seven Transformer module families, using bf16 training and inference.The adaptation trains approximately 157M parameters, or 1.95% of the backbone.
  • Training uses AdamW, cosine learning-rate decay with warm-up, three epochs per direction, and masked-diffusion cross-entropy.
  • The matched LLaMA-3-8B AR baseline uses the same SFT pairs, prompt family, LoRA scale, and evaluation data, with beam size 4 and native EOS termination.
  • Three independent training runs per direction yield nine checkpoints.
  • Default decoding uses MED at T=32, with EOS unconstrained and output truncated at the first EOS.
  • The main experiments compare Length Oracle, Ratio, and Entropy-Valley on WMT22 News Translation test sets with N=2037 per direction.Fixed ratios are 0.8 for En→Zh, 1.2 for Zh→En, and 1.8 for En→De.
  • EV's probe cap adds at most 15.6% to the forward pass count at T=32, while En→Zh decoding takes roughly 1.7 seconds per sentence versus 1.5 seconds for Ratio.
  • Length comparisons hold reveal order fixed, and the En→Zh EV–Ratio difference is 2.1× the tested order span.The reference-length-inclusive span is 3.3× the order span.

C.6 Paired significance and compute budget

Paired tests support EV’s gains most consistently for Zh→En, while En→De shows weaker sentence-level evidence. Budget controls indicate that the improvement is attributable to length selection rather than extra forward passes.

  • Zh→En is reliable across all three runs, whereas En→Zh is reliable in run A and En→De passes Wilcoxon but not bootstrap.This matches the weaker sentence-level evidence reported for En→De.
  • At most 0.001 COMET is recovered by increasing Ratio from EV’s maximum budget T=37 to T=40.The comparison uses Ratio at T=32, T=37, and T=40.
  • The negligible budget effect indicates that EV’s En↔Zh gains come from its selected length rather than additional computation.

D Coverage losses from short canvases

EV improves source-content retention over Ratio, especially for En→Zh placeholders and numbers, but its benefits vary with compression and the fixed candidate grid can miss needed shorter canvases.

  • Literal retention: EV raises En→Zh placeholder retention from 66.9% to 89.1% and number retention from 77.6% to 81.3% over Ratio.Both retention gains have bootstrap confidence intervals strictly above zero.
  • Fixed-grid boundary: Individual high-compression sentences can fail when the required canvas lies below the fixed candidate grid.The “Please give me a moment” case uses an EV L=7 canvas although the needed canvas lies below [0.70, 0.90].
  • Aggregate losses: EV scores below Ratio on 596 En→Zh sentences, with most losses caused by within-grid selection errors rather than grid-boundary failures.

F Reveal order controls and negative diagnostics

Reveal-order variants do not close the length-controlled gap in the tested settings, while cross-backbone and Dream-Instruct diagnostics delimit where the conclusions apply.

  • Reveal-order controls: With reference length fixed, six source-guided variants span only 0.0081 COMET, and none reliably improves over MED.Strict left-to-right and random order also score below MED.
  • Negative diagnostics: On a 400-sentence challenge subset, SIG-first averages −0.0004 COMET relative to MED, with no useful SIG-Concentration subset identified.The correlation is ρ = −0.0675 with p = 0.13.
  • Backbone scope: Dream-Instruct does not produce usable MT outputs under the LLaDA-compatible protocol, so the cross-backbone check uses Dream-Base instead.Plain-prompt LoRA-SFT fails to override its chat-aligned predictive distribution, while chat-template training yields mostly empty outputs.
  • Backbone scope: Zero-shot Dream-Instruct decoding produces empty outputs at rates of 97.4% for EV, 96.6% for Ratio, and 94.8% for Oracle.Non-empty outputs are short repetition patterns, indicating that length choice cannot repair the shared EOS failure.
  • Cross-backbone check: EV exceeds Ratio in every Dream-Base and DiffuLLaMA aggregate COMET cell, with paired tests significant across all 18 comparisons.Gap closure varies by backbone and direction, ranging from 54% to 69% for Dream-Base.

I Bilingual human evaluation protocol

The human evaluation compares anonymized EV and Ratio outputs with independent three-expert ratings on adequacy, fluency, and preference across En↔Zh samples.

  • Study design: Three bilingual professional translators independently rated two direction-specific packs of 100 WMT22 sentences.Each annotator saw the same sentences without cross-annotator discussion during rating.
  • Study design: Each direction contains 25 EV-best cases, 25 Ratio-best cases, 25 near ties, and 25 random cases.The cases are selected using COMET differences and random sampling.
  • Related evaluation: EV remains above Ratio in all six Dream-Base and DiffuLLaMA aggregate COMET cells, providing a separate cross-backbone comparison from the human study.
  • Rating criteria: Adequacy rates how completely and accurately the translation conveys source meaning on a 1–5 scale, while fluency rates naturalness and grammaticality on a 1–5 scale.
  • Aggregation: The aggregate preference uses per-sentence majority vote, and aggregate Zh→En adequacy uses a paired Wilcoxon test on the three-expert mean.Figure 7 additionally reports En→Zh per-annotator adequacy boxplots and sentence-level COMET–adequacy correlation.
  • Rating criteria: Preference is a single A/B/Tie judgment combining adequacy and fluency, with ties used when the difference is tiny.
Loading 2608.22274v1…