ESTS at WMT26: Routing-Informed Expert Pruning for Model Compression

1 Introduction

• Routing mass ranks experts within layers, while cross-lingual routing divergence allocates retained capacity across layers before low-importance experts are physically removed.The resulting specialists are recovery-tuned on synthetic translation data.Source: §1 · p. 1
• Calibration and recovery data cover news, social, and speech content to better match the varied task distribution.The framework is applied to informal social-media exchanges and colloquial dialogue alongside instruction-following and format-preservation requirements.Source: §1 · p. 1
• The retained expert projection weights receive additional MXFP4 quantization.Source: §1 · p. 1
• The six systems span 4.186B–7.770B parameters and 4.55–6.33 GiB packed artifact sizes, with internal xCOMET-XL comparisons using GPT-5.1 pseudo-references.There are three compression operating points each for English–Simplified Chinese and English–Egyptian Arabic.Source: §1 · p. 1

2.1 Task Categories

• WMT26 inputs are grouped as news, social, speech, and JSON/software, with distinct translation, style, and structure requirements.Recovery data covers news, social, and speech; JSON handling is implemented at inference rather than included in recovery SFT.Source: §2.1 · pp. 1–2

2.2 Source Corpus Construction

• The English source corpus contains 44,000 examples from four datasets selected to approximate the three recovery categories.News Crawl supplies news; Reddit post bodies and grouped Bluesky posts supply social examples; grouped podcast turns supply speech examples, with filtering and deduplication across sources.Source: §2.2 · p. 2

2.3 Synthetic Recovery Data

• GPT-5.1 generates translation targets using each example’s category-specific WMT26 instruction, with incomplete or failed generations excluded.Source: §2.3 · p. 2
• ZHO recovery data retains 43,997 examples from the full 44,000-example corpus, while ARZ retains 9,999 from a 10,000-example subset.The ARZ subset preserves category and source-family composition while approximately matching the full corpus’s share of source-token mass.Source: §2.3 · p. 2

3.1 Routing-Based Calibration

• Task-specific routing mass ranks experts within each layer, and cross-lingual routing divergence determines each layer’s retained capacity.The calibration uses 660 examples per translation direction for expert rankings and paired English–target sequences for divergence estimates.Source: §3.1 · pp. 2–3
• The dynamic allocation gives more retained capacity to layers with greater language-specific routing divergence while preserving the total capacity budget.The method initializes each layer at the minimum valid capacity, then distributes the remaining budget proportional to divergence.Source: §3.1 · p. 3

3.2 Expert Removal and Submitted Models

• After allocating capacities, the method removes each layer’s lowest-routing-mass experts and physically slices expert and router parameters to create the submitted models.Per-layer expert counts are stored in the model configuration so the sliced weights can load through the modified model implementation.Source: §3.1 · p. 3, §3.2 · p. 3

3.3 Recovery Fine-Tuning

• Recovery uses full-parameter SFT on 43,997 ZHO pairs for one epoch and 9,999 ARZ pairs for three epochs.Training uses BF16 and a maximum sequence length of 4096.Source: §3.3 · p. 3

3.4 MXFP4 Quantization

• MXFP4 quantization packs retained gate_up_proj and down_proj weights in 32-value blocks, while other parameters remain BF16.The resulting checkpoints use mixed-precision storage.Source: §3.4 · p. 4

4.1 Category Inference and Prompting

• Deterministic source heuristics assign inputs to news, social, speech, or JSON, then apply category-specific instructions and preprocessing.JSON string values are translated using the news instruction while keys, structure, and non-string values are preserved.Source: §4.1 · p. 4

4.2 Generation and Output Validation

• The generator makes up to ten whole-unit attempts, rejecting outputs that fail structural validation or contain other specified errors.For news and social, validation checks preservation of source paragraph and line-break structure.Source: §4.2 · p. 4

4.3 Segmented Fallback

• After ten failed whole-unit attempts, non-JSON inputs are segmented progressively, translated in parts, and reassembled; unresolved cases fall back to the best available completion or source text.HTML is split at paragraph and, if needed, line-break boundaries; long units may be divided into sentence or complete-word spans.Source: §4.3 · pp. 4–5

4.4 JSON Projection and Reconstruction

• For JSON, the system translates extracted string values and maps them back into a copy of the source structure rather than generating a JSON object directly.Validation preserves keys and their order, array structure, non-string values, and placeholders; output is serialized from the source-owned reconstruction.Source: §4.4 · p. 5

• The internal xCOMET-XL evaluation scored 198 released blindset documents as source–hypothesis–reference units, using GPT-5.1 pseudo-references and blindset instructions.Source: §5 · p. 5
• xCOMET-XL scores were 0.4130, 0.4094, and 0.3840 for ZHO k26, k27, and k28, and 0.2705, 0.2647, and 0.2380 for ARZ k22, k24, and k26.Source: §5 · p. 5
• GPT-OSS-20B scores are reference points, not controlled estimates of pruning effects, and shared GPT-5.1 generation may bias scores toward its outputs.Source: §5 · p. 5
• Robust inference produced a final output for every evaluated input with no hard errors; all ARZ inputs used whole-unit generation, while some ZHO inputs needed fallback processing.One k27 input used raw-completion fallback, and one k28 JSON output retained source text for an unresolved span.Source: §5 · p. 5
• Pruning reduces the stored expert pool but leaves active-expert MLP computation unchanged; retries and fallback add work, and packed size is not peak inference VRAM.Source: §5 · pp. 5–6

• Six WMT26 systems use routing-informed expert removal from GPT-OSS-20B, followed by recovery fine-tuning and MXFP4 quantization.Source: §6 · p. 6
• The specialists range from 4.186B to 7.770B parameters and from 4.55 to 6.33 GiB in packed size.Source: §6 · p. 6
• The submitted inference system handles instruction and format requirements through category inference, robust generation, segmented fallback, and deterministic JSON reconstruction.Source: §6 · p. 6
• Internal GPT-5.1-pseudo-reference evaluation compares submitted compression operating points; organizer-measured translation quality, decoding speed, and memory are deferred to shared-task findings.Source: §6 · p. 6

• Expert importance weights are averaged over tokens in a sequence to produce a sum-1 expert distribution for the layer.Source: A Routing Divergence Metric · p. 8
• For each calibration sequence, the metric compares target-language and English routing distributions using Jensen–Shannon divergence, without entropy normalization as in Bandarkar et al. (2026).Source: A Routing Divergence Metric · p. 8
• Mean aggregation across paired calibration sequences gives a 0-to-1 measure of how language-specialized routing is in that layer.Source: A Routing Divergence Metric · p. 8

• Each layer starts at minimum capacity K; remaining capacity is allocated among non-full layers, with E as the upper bound.The total retained capacity budget is C = L(E − k).Source: B Dynamic Capacity Allocation Details · p. 8
• When provisional capacity exceeds E, the layer is capped and its overflow returned to the remaining budget for reallocation.Source: B Dynamic Capacity Allocation Details · p. 8
• Hamilton’s method rounds real-valued capacities to integers while preserving the total budget and bounds; each layer then drops its lowest-scoring experts.Source: B Dynamic Capacity Allocation Details · p. 8