Source-linked AI summary
Neural Machine Translation Decoding with Terminology Constraints
Eva Hasler, Adrià De Gispert, Gonzalo Iglesias, Bill Byrne
TL;DR
NMT systems still face the open problem of adhering to user-provided terminology constraints. This paper uses constrained beam search with source-aligned constraints and attention to guide placement and reduce duplication. Across four language pairs, the approach respects customer-provided terminology while maintaining overall translation quality, and its computational-complexity improvements translate into faster decoding.
Problem
NMT systems have improved translation quality, but controlling their output to adhere to user-provided terminology remains an open problem.
Source: §1 · p. 1Method
The decoder combines finite-state constrained beam search with source-aligned target constraints, using attention to place constraints and mask covered spans.
Source: §1 · p. 1Results
Across four language pairs, customer-provided terminology is respected while overall translation quality is maintained, and computational-complexity improvements translate into faster decoding.
Source: §5 · p. 5Takeaways & Limitations
Compared with existing methods, attention-based decoding reduces output duplication and improves constraint placement.
Source: §5 · p. 5Aligned-span decoding depends on source-target pair quality; incomplete source or target phrases in terminology can cause over- or under-translation.
Source: §2.3 · p. 3
Abstract
from arXiv · showhide
Despite the impressive quality improvements yielded by neural machine translation (NMT) systems, controlling their translation output to adhere to user-provided terminology constraints remains an open problem. We describe our approach to constrained neural decoding based on finite-state machines and multi-stack decoding which supports target-side constraints as well as constraints with corresponding aligned input text spans. We demonstrate the performance of our framework on multiple translation tasks and motivate the need for constrained decoding with attentions as a means of reducing misplacement and duplication when translating user constraints.
1 Introduction
Domain adaptation and guided decoding can encourage terminology use but do not reliably enforce it, although strict enforcement matters for consistent brand information. The paper aligns target constraints with source spans to improve placement and reduce duplication.
1 Introduction
- Existing domain adaptation and guided decoding do not reliably enforce terminology, despite companies needing brand information translated correctly and consistently.The passage says this requirement can be more important than translation quality alone.Source: §1 · p. 1
- Earlier constrained methods enforce target words, but grid beam search lacks source-word correspondence and can misplace constraints or translate the corresponding source words more than once.Entity tagging can translate entities correctly given good alignments, but requires entity-tag training data and excludes entities from model scoring.Source: §1 · p. 1
- The proposed decoder aligns target-side constraints with source words and uses NMT attention to place them and mask covered constraints to reduce duplication.The approach adapts constrained beam search to machine translation while seeking to maintain unconstrained translation quality.Source: §1 · p. 1
2 Constrained Beam Search
The decoder uses finite-state acceptors and multiple stacks to force constraint satisfaction during beam search. With source-aligned constraints, attention guides placement, reduces search complexity, and masks already covered spans.
2.2 Multi-Stack Decoding
- Multiple stacks group hypotheses by satisfied constraints, allowing low-scoring constraint extensions to continue instead of being pruned from the ordinary beam.Each stack maps to an acceptor state; stacks are pruned to beam size k, so actual beam size depends on the number of acceptor states.Source: §2 · p. 2, §2.2 · p. 2
2.1 Finite-state Acceptors for Constraints
- The target-language acceptor encodes all constraint permutations interleaved with vocabulary loops; phrase constraints use one state per token, with intermediate states restricted to the next phrase token.Alternatives are represented by different arcs connecting the same states, and multi-token alternatives have intermediate states.Source: §2.1 · p. 2
- Figure 1 illustrates a phrase constraint C1 and two single-token alternatives C2; either constraint order reaches the final state with both satisfied.The figure depicts the FSA for C1 = ab and C2 = {x, y}.Source: §2.1 · p. 2, Figure 1 · p. 2
2.3 Decoding with Attentions
- Attention selects when to insert a constraint from its aligned source span, reducing complexity from O(tk2^c) to O(tkc) by ignoring other constraint permutations.The maximum-attention source position must fall within the constraint span and match an outgoing arc; disabling vocabulary loops saves additional time.Source: §2.3 · p. 2
- After a constraint is completed, the decoder masks its source span in future expansions by zeroing attention weights on covered positions.This excludes covered positions from the attention context vector.Source: §2.3 · pp. 2–3
- Aligned-span decoding depends on source-target pair quality, and incomplete source or target phrases in terminology can cause over- or under-translation.Source: §2.3 · p. 3
- The maximum-attention placement rule is strict; neighboring source-token translations may require allowing extra tokens, while secondary-attention thresholding can help when the maximum never falls on a constraint span.Without relaxation in the latter case, the decoder may produce empty translations.Source: §2.3 · p. 3
3 Experimental Setup
Experiments use English–German and English–Chinese translation in both directions, with ensembles and LNMT systems. They evaluate reference-derived gold constraints and automatically extracted dictionary constraints matched to the reference.
3 Experimental Setup
- Experiments use attention-based English–German and English–Chinese models in both directions, ensemble three models per language pair, and combine them with SMT-lattice n-grams as LNMT.Decoding uses beam size 12 and length normalization, backs off to decoding without attention when attention-based decoding fails, and reports lowercase BLEU.Source: §3 · p. 3
3.1 Data
- Training data follows WMT 2017; English–German is tokenized and truecased, German source compounds are split, and NMT training is augmented with backtranslation.English–Chinese is tokenized and lowercased, and byte-pair encoding is applied to all data.Source: §3.1 · p. 3
3.2 Terminology Constraints
- Gold constraints are up to two reference tokens missing from the baseline, or phrases of up to five subwords around a missing token; aligned source words are unavailable.The setup tests incorporating user corrections to baseline output.Source: §3.2 · p. 3
- Dictionary entries are automatically extracted from reference terms and phrases, filtered for English stop words and verbs, and checked against a pruned phrase table for likely pairs.For evaluation, entries are ensured to match the reference when applied to an input sentence.Source: §3.2 · p. 3