EDWARD-NMT: Elastic Dimensions With Adaptive Routing and Dual-consistency for Neural Machine Translation

1 Introduction

• The motivating heterogeneity includes mixed lexical and stylistic distributions across training corpora, while MoE routes tokens to experts specializing in distinct patterns.MoE is described as scaling capacity while keeping per-token computation tractable.Source: §1 · p. 1
• Rigid top-k routing fixes per-token compute, while keeping the full expert pool in memory limits MoE deployment on resource-constrained devices.The paper also notes that existing approaches do not provide a unified way to adjust memory and compute footprints on the fly.Source: §1 · p. 1
• EDWARD-NMT uses Matryoshka-style expert dimensions and an adaptive sigmoid router with a learned, task-conditioned threshold to expose sub-networks and vary active experts per token.Its nested sub-networks share weights and are extracted at inference without post-hoc fine-tuning.Source: §1 · pp. 1–2
• A dual-consistency objective aligns predictive distributions across nested structural scales to stabilize aggressively truncated sub-networks without post-hoc adaptation.The paper presents this as an in-place regularization mechanism.Source: §1 · pp. 1–2
• Across the evaluated benchmarks, EDWARD-NMT matches recent lightweight MoEs, improves on Transformer Big, and its smallest variant outperforms Transformer Base.The experiments cover PhoMT, WMT-14, multilingual OPUS-16, and a multi-domain De↔En benchmark; variants come from a single checkpoint.Source: §1 · pp. 1–2

2 Related Work

• Prior MoEs have fixed memory footprints; post-hoc compression yields hardware-specific models, while multi-sub-network designs require fine-tuning [Shazeer et al., 2017; Fedus et al., 2022; Dai et al., 2022; Jiang et al., 2024; Liu et al., 2024; Lagunas et al., 2021; Li et al., 2025; Cai et al., 2020; Yu et al., 2019; Hou et al., 2020].Existing elastic designs are described as limited to dense networks or token-level granularity [Cappellazzo et al., 2026].Source: §2 · p. 2
• EDWARD-NMT embeds Matryoshka-style elasticity in sparse experts and uses variance-preservation scaling to stabilize activations across nested widths without post-hoc adaptation.The resulting sub-networks share a single trained MoE.Source: §2 · p. 2
• Other dynamic-routing approaches adapt token compute through null experts, auto-tuned activations, or differentiable ReLU routing, whereas EDWARD-NMT couples adaptive sigmoid routing with elastic experts to adjust compute and memory [Zeng et al., 2024; Guo et al., 2025; Wang et al., 2025].Concurrent MoSE uses slimmable experts but retains rigid top-k routing and test-time training for width adaptation [Tastan et al., 2026].Source: §2 · p. 2
• Unlike consistency methods aligning perturbed views or dense elastic sub-networks aligned to a full-capacity teacher, EDWARD-NMT uses symmetric KL divergence to transfer knowledge bidirectionally between nested widths [Xie et al., 2020; Liang et al., 2021; Yao et al., 2025; Yu and Huang, 2019; Cai et al., 2020].The paper describes this as anchoring the full-capacity network while preserving aggressively truncated variants.Source: §2 · p. 2
• EDWARD-NMT conditions its shared expert on translation direction through FiLM modulation, extending shared-expert designs that isolate common knowledge [Yang et al., 2025; Perez et al., 2018].Figure 1 shows the direction-aware shared expert alongside the router and nested Matryoshka expert slices.Source: §2 · pp. 2–3, Figure 1 · p. 3

3 EDWARD-NMT

• EDWARD-NMT replaces Transformer feed-forward sublayers with MoE blocks combining elastic experts, adaptive routing, dual-consistency, and a parallel direction-aware shared expert.The encoder-decoder architecture is designed to expose sub-networks at inference.Source: §3 · p. 3

3.1 Preliminaries: Standard Mixture-of-Experts

• Standard MoEs fix each token’s compute budget with top-k routing, while memory footprint scales with the full expert pool even when fewer experts are evaluated.Source: §3.1 · p. 3

3.2 Elastic Experts via Matryoshka Dimensions

• Nested expert widths share one set of parameters; dynamically truncating intermediate activations produces selectable sub-networks, with width-dependent rescaling preserving expected squared activation norm.The rescaling supports stable joint training across nested capacities.Source: §3.2 · pp. 3–4

3.3 Adaptive Sigmoid Routing

• A token-adaptive sigmoid gate uses a dynamic threshold to activate a variable number of experts, while an auxiliary sparsity loss targets a specified active-expert budget.A straight-through estimator supports backpropagation through the discrete activation decision.Source: §3.3 · p. 4

3.4 Dual-Consistency Objective

• Dual-consistency samples two widths, compares their predictive distributions with symmetrized KL divergence, and supports both cross-capacity self-distillation and same-width consistency.The larger view guides the smaller one, while the smaller view structurally regularizes the larger; equal widths yield consistency regularization across stochastic passes.Source: §3.4 · pp. 4–5

3.5 Direction-Aware Shared Expert

• A direction-conditioned shared expert runs alongside sparse routing; its damped contribution provides a stable grammatical backbone while routed experts can specialize across domains.Source: §3.5 · p. 5

3.6 Training Objective and Inference

• Training combines label-smoothed cross-entropy, routing constraints, and dual-consistency; at inference, deterministic matrix slicing sets memory capacity while adaptive routing sets per-token compute.Cross-entropy and routing losses are computed independently for the sampled widths and averaged.Source: §3.6 · p. 5

4.1 Datasets

• The multi-domain De↔En benchmark spans five diverse domains and uses macro-averaging; OPUS-16 has 16 languages grouped as high (> 0.9M pairs), low (< 0.1M pairs), and medium resource.Source: §4.1 · p. 5

4.2 Architecture and Sub-network Family

• The encoder-decoder uses four routed experts and one direction-aware shared expert per MoE layer, with nested dimensions D = {2048, 1024, 512, 256}.The four variants share trained parameters and differ by their inference-time dimension slice.Source: §4.2 · p. 5, Table 1 · p. 5
• Dual-consistency samples two dimensions uniformly with replacement, while routing uses target sparsity ρtarget = 1.25 and damping α = 0.1.The direction embedding is learned per translation direction; the router thresholds span 0.05 to 0.60.Source: §4.2 · p. 6

4.3 Baselines

• Baselines include dense Transformer BASE/BIG, mBART, Switch Transformer top-1 MoE, and GShard top-2 MoE.Models are jointly trained on translation directions; OPUS-16 baseline results are reported from Zhao et al. (2024).Source: §4.3 · p. 6

4.4 Implementation Details

• Training uses AdamW on a TPU v5-64 pod, while evaluation uses beam search and dataset-specific BLEU metrics, with COMET also reported for WMT-14 and multi-domain.All four EDWARD-NMT variants are extracted and evaluated from the same checkpoint.Source: §4.4 · p. 6

5.1 Main Translation Quality

• Across PhoMT and WMT-14, XL beats Transformer Big in all directions; it gains +0.62 BLEU on multi-domain, while OPUS-16 M beats ST-MOE and GS-MOE in both directions.On multi-domain, XL uses 161M active parameters versus 210M for Transformer Big.Source: §5.1 · p. 6, Table 2 · p. 7, Table 3 · p. 7
• On WMT-14, EDWARD-NMT-S (59M active parameters) beats Transformer Base (61M active) by +2.70 COMET; on multi-domain, S scores 80.96 COMET versus GS-MOE’s 78.20.At full capacity, XL exceeds Transformer Big by +0.61 COMET on average.Source: Table 4 · p. 7, §5.1 · p. 7
• EDWARD-NMT-L surpasses Transformer Big with 102M versus 210M active parameters and 61.5 versus 48.0 TPS; S uses 4× fewer total parameters and is 2.2× faster than XL.Across benchmarks, adaptive routing activates 1.34 to 1.70 experts per token, with active expert count controlled by target density ρ.Source: §5.1 · pp. 7–8, Table 5 · p. 8

5.2 Ablation Studies

• Adding Matryoshka Experts raises XL to 40.96 BLEU, while the full system reduces routing from Top-2 to approximately 1.38 experts and improves L, M, and S quality.On this benchmark, integrating sigmoid routing and dual-consistency slightly lowers XL quality; dual-consistency prevents smaller sub-networks from collapsing.Source: Table 6 · p. 8, §5.2 · p. 8
• On multi-domain, Common Shared Expert adds +0.2 to +0.3 BLEU over No SE, while Direction-Aware Shared Expert adds approximately +1.0 BLEU on XL and +0.6 on S.Source: Table 7 · p. 8, §5.2 · p. 8
• Dual-Consistency balances cross-width quality: R-Drop lowers L by −0.6 BLEU, while unidirectional distillation lacks mutual regularization and anchoring only to the maximum width is sub-optimal.The proposed loss couples capacity views, using larger variants as distillation targets and smaller cores as regularizers.Source: Table 8 · p. 8, §5.2 · pp. 8–9
• Widths interpolated between the four trained dimensions, including 1536 and 768, retain robust quality without fine-tuning, but D = 128 triggers severe performance breakdown.The reported elasticity is bounded by the trained envelope.Source: §5.2 · p. 9
• Removing variance-preservation scaling degrades every extracted sub-network, with a −0.37 BLEU drop at D = 256.The drop is more pronounced at smaller capacities.Source: §5.2 · p. 9

6 Conclusion

• The architecture combines Matryoshka-style elastic experts, adaptive sigmoid routing, and a direction-aware shared expert.A symmetric dual-consistency objective aligns predictive distributions across nested widths.Source: §6 · p. 9
• The full-capacity model matches or exceeds strong MoE baselines, and smaller extracted sub-networks outperform comparably sized dense Transformers.Evaluations cover PhoMT, WMT-14, multilingual OPUS-16, and a multi-domain benchmark.Source: §6 · p. 9
• The framework is presented as a practical, scalable path to resource-aware sequence modeling and adaptable on-device deployment.Source: §6 · p. 9

• EDWARD-NMT is evaluated only on encoder-decoder machine translation; massively multilingual settings, decoder-only models, and broader sequence-generation tasks remain future work.Source: Limitations · p. 9
• Elasticity currently targets feed-forward widths, leaving dynamic depth and variable hidden-state dimensions unexplored.Source: Limitations · p. 9
• Performance degrades below the minimum trained capacity, so the reported interpolation applies within the trained envelope.Source: Limitations · p. 9

Ethical Considerations

• EDWARD-NMT uses public NMT benchmarks and baselines reproduced under their licenses, but may propagate corpus biases or inaccuracies and should not inform high-stakes decisions without human review.Source: Ethical Considerations · pp. 9–10

A Experimental Details

• The appendix outlines datasets, training configurations, and hyperparameters for reproducing Section 5 results with publicly available data.Source: A Experimental Details · p. 13

A.1 Datasets and Preprocessing

• The evaluation uses PhoMT (about 3M pairs), filtered WMT-14 (about 3.9M pairs), multi-domain De↔En (about 1.3M pairs across five domains), and 16-language OPUS-16.Source: §A.1 · pp. 13–14

A.2 Architecture

• The encoder-decoder has eight layers per side, d_model=512, eight heads, and four routed Matryoshka experts plus a direction-aware shared expert in each MoE block.Source: §A.2 · p. 14

A.4 Model Selection and Evaluation

• Checkpoints are selected by validation BLEU averaged across directions, and paired significance tests report p < 0.001 for sub-network gains over baselines.Source: §A.4 · p. 14

A.5 Baselines

• Comparisons include dense Transformers, mBART, Switch Transformer, and GShard; PhoMT mBART results come from Doan et al. (2021).Source: §A.5 · pp. 14–15

A.3 Hyperparameters

• Router and elasticity settings are shared across benchmarks, while hyperparameter values and search spaces are reported in Table 10.Source: Table 10 · p. 15

A.6 Software and Hardware Settings

• Dual-consistency raises training to about 1160 ms per step versus 523 ms for standard MoEs, but one run yields a family at 2.01× Transformer Base’s per-model cost.Source: §A.6 · p. 15

A.7 Step-by-Step Reproduction Guide

• Reproduction samples two widths for shared-parameter forward passes, then extracts S/M/L/XL from one checkpoint without fine-tuning, distillation, or post-hoc adaptation.Source: §A.7 · p. 16

B.1 Computational Results

• EDWARD-NMT-S achieves 104.9 TPS, slightly below TRANSFORMER BASE at 117.7 TPS, but outperforms TRANSFORMER BIG at 48.0 TPS; at B=128, it delivers a 3.2× throughput speedup over BASE.Source: §B.1 · p. 16

B.2 Multilingual Results

• At L=6, XL reaches 34.28 BLEU versus THOR-MOE’s 33.31 and LINGUAL-SMOE’s 32.71; at L=8 it peaks at 35.29 Avg, while S scores 32.87 Avg.At L=6, EDWARD-NMT uses 60 total experts versus 192 in the baselines.Source: Table 12 · p. 17, §B.2 · pp. 17–18

B.3 Architecture Elasticity

• Unseen widths such as D=1536 and 768 interpolate robustly within the trained envelope, while extrapolating below it to D=128 breaks down.Source: Table 14 · p. 18, §B.3 · p. 18

B.4 Ablation on Variance Preservation

• Removing deterministic rescaling ψ(Dk) degrades translation quality across nested capacities, with a wider gap at smaller widths.Source: Table 15 · p. 18, §B.4 · p. 18

B.5 Detailed Multi-domain Results

• Extracted L and XL variants achieve state-of-the-art results on almost all individual multi-domain test domains while using fewer active parameters.Source: §B.5 · p. 18, Table 16 · p. 19

B.6 Additional WMT-14 Results

• On WMT-14 chrF2, the 59M-active S variant exceeds 61M-active Transformer Base by +1.70 En→De and +2.20 De→En points; XL also surpasses BIG and mBART.Source: §B.6 · pp. 18–19, Table 17 · p. 19

B.7 Additional PhoMT Results

• 42.13 En→Vi and 42.42 Vi→En TER: EDWARD-NMT-XL achieves the lowest overall post-editing effort, outperforming 448M-parameter mBART with nearly 3× fewer active parameters.EDWARD-NMT-S (51M active) also achieves lower TER than TRANSFORMER BIG (213M active) in both directions, despite using 4× fewer active parameters.Source: §B.7 · p. 19
• Dual-Consistency achieves the highest scores across XL, L, M, and S, outperforming the “No loss” baseline; asymmetric KL distillation mitigates symmetric R-Drop’s degradation but remains below baseline.Symmetric R-Drop degrades performance across all scales.Source: §B.7 · pp. 19–20, Table 19 · p. 19
• +0.59 BLEU on XL and +0.18 on S: Matryoshka Experts improve over Top-2 MoE, while Dual-Consistency adds +0.38 BLEU on S over Matryoshka Experts alone.Adaptive Sigmoid Routing yields the best overall performance for XL, L, and M; all settings use DASE.Source: Table 20 · p. 19, §B.7 · p. 20

B.8 Additional Hyperparameter Ablations

• The XL variant is evaluated without Dual-Consistency for ablations of target density, minimum routing threshold, and router jitter noise.Tables 21–23 cover these settings on the multi-domain benchmark.Source: §B.8 · p. 20, Table 21 · p. 20, Table 22 · p. 20
• Figures report average active experts per token and learned threshold over training, plus active-expert percentages by layer across elastic dimensions.The layer-wise distributions are for the multi-domain benchmark.Source: §B.8 · p. 20, Figure 2 · p. 20, Figure 3 · p. 20
• λdc = 1.0 provides the best trade-off, maximizing L, M, and S sub-network quality; λdc = 0.1 favors XL but under-transfers to smaller variants, while λdc = 5.0 degrades all scales.The ablation is on the multi-domain benchmark, averaged over both translation directions.Source: Table 24 · p. 20, §B.8 · p. 21
• ∼1.15 experts at ρ = 1.0 and ∼1.69 at ρ = 1.5: empirical active experts scale predictably with target density across capacities.The auxiliary loss provides fine-grained control of inference compute budget without architectural modifications.Source: Table 25 · p. 20, §B.8 · p. 21