Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling

1 Introduction

• Soft attention weights all tokens but can dilute attention to significant elements, while hard attention selects a subset but is difficult and inefficient to train.Hard attention is non-differentiable and often relies on policy gradients; sequential sampling also makes it time-inefficient.Source: §1 · p. 1
• ReSA integrates hard and soft attention: hard attention trims sequences for soft attention, whose reward signals help train the hard component.The paper presents this combination as addressing the limitations of both mechanisms.Source: §1 · pp. 1–2
• RSS selects tokens in parallel, and ReSA uses two RSS modules to model sparse dependencies between selected head and dependent tokens.ReSAN builds on ReSA as a sentence encoder without CNN or RNN structure.Source: §1 · p. 2
• ReSAN achieves the best test accuracy among sentence-encoding models on the official SNLI leaderboard and state-of-the-art performance on SICK.The authors also report better efficiency and prediction quality than recurrent/convolutional networks, self-attention networks, and well-designed models.Source: §1 · p. 2

2.1 Attention

• Vanilla attention scores each token against a query, then softmaxes the scores into a categorical distribution whose expected token representation is the output.Multi-dimensional attention extends token-level scoring to a separate alignment score for each feature.Source: §2.1 · p. 2

2.2 Self-Attention

• Self-attention derives its query from the input sequence, allowing it to model dependencies among tokens in that sequence.Source: §2.2 · p. 2
• Token-to-token self-attention builds a context-aware representation for each token; the paper uses masked self-attention as its soft module.The mask can block token-to-token attention or encode directionality through asymmetric entries.Source: §2.2 · p. 2
• Source-to-token self-attention removes the query from the compatibility function and compresses the sequence into a vector representation.Source: §2.2 · p. 3

3.1 Reinforced Sequence Sampling (RSS)

• RSS samples token-selection variables in parallel from probabilities computed by a learned attention mechanism over context-aware features.Its context-fusion layer already incorporates sequential information, so each token’s selection probability does not depend on the previous selection; RSS is trained with policy gradients because its discrete choices are non-differentiable.Source: §3.1 · p. 3

3.2 Reinforced Self-Attention (ReSA)

• ReSA uses two RSS modules to select head and dependent tokens, then applies soft self-attention to model dependencies among selected token pairs.The resulting sparse mask reduces self-attention computation, and the soft module’s prediction correctness supplies the reward signal used to train RSS.Source: §3.2 · pp. 3–4

3.3 Applications of the Proposed Models

• ReSAN combines ReSA with source2token self-attention to produce a sentence representation, using neither recurrent nor convolutional networks.Two simplified variants discard unselected-token information for greater time efficiency, but cannot perform context fusion because their input and output sequences differ in length.Source: §3.3 · p. 4

• The remaining ReSAN parameters are learned end-to-end by back-propagation using cross-entropy loss plus L2 regularization.These parameters include word embeddings, soft self-attention, and classification or regression layers.Source: §4 · p. 4
• RSS parameters are optimized with REINFORCE, using the classification loss as negative reward and penalizing selection of too many tokens.The penalty weight λ is fine-tuned from {0.005, 0.01, 0.02}.Source: §4 · p. 4
• Training first fixes RSS to select every token while optimizing the remaining parameters, then jointly updates both once development loss stops decreasing significantly.The staged start addresses inaccurate reward feedback early in training.Source: §4 · p. 5
• Experiments use TensorFlow on a GTX 1080Ti, Adadelta, and 300D GloVe 6B embeddings, with the best result selected from tested dropout keep probabilities.The tested keep probabilities are {0.65, 0.70, 0.75, 0.8}.Source: §4 · p. 5

5.1 Natural Language Inference; 5.2 Semantic Relatedness

• ReSAN achieves the best test accuracy among SNLI sentence encoders and state-of-the-art or competitive performance across all three SICK metrics.On SNLI, it also outperforms the listed recurrent, convolutional, and attention-based models; the SICK results are reported in Table 3.Source: §5.1 · pp. 5–6, Table 3 · p. 6, §5.2 · p. 6

5.1 Natural Language Inference

• Ablations attribute 0.2% to unselected head tokens, 0.5% to separate RSS selection, 0.3% to hard attention, and 2.9% to soft self-attention.The two variants are more time-efficient but perform worse; applying RSS to self-attention or attention improves both performance and time efficiency.Source: Table 2 · p. 6, §5.1 · p. 6
• ReSAN exceeds 600D Bi-LSTM + intra-attention by 3.0%, 300D NSE encoders by 1.7%, 600D Deep Gated Attn by 0.8%, and 300D SPINN-PI by 3.1%.SPINN-PI is described as a recursive model using an external semantic parsing tree.Source: §5.1 · p. 6

5.2 Semantic Relatedness

• On SICK, ReSAN also achieves the best results among the listed recurrent, convolutional, and self-attention baselines, and outperforms recursive models and feature-engineering methods.The table reports accuracies averaged over five runs, with standard deviations in parentheses.Source: Table 3 · p. 6, §5.2 · p. 6

5.3 Case Study

• In the case study, RSS selects more dependents than heads; selected dependents include non-trivial words, while most stop words are selected as neither.The authors describe dependents as modifying corresponding heads and key heads as composing the sentence trunk.Source: §5.3 · pp. 6–7
• The soft self-attention visualization gives greater attention to semantically important words and to token pairs that form sense-groups.The examples include nouns and verbs such as “sit” and “talk,” and the pairs “adults, waiting” and “waiting, event.”Source: Figure 2 · p. 7, §5.3 · p. 7

6 Related Work

• Prior NLP applications of reinforcement learning target either non-differentiable objectives or faster model execution.Examples include optimizing machine-comprehension F1 and simplifying sentences for simplicity, relevance, and fluency [Hu et al., 2017; Zhang and Lapata, 2017].Source: §6 · p. 7
• Selection and skimming methods use reinforcement learning to choose relevant text or skip less important input in sentiment analysis and reading tasks [Lei et al., 2016; Yu et al., 2016].Source: §6 · p. 7
• Other examples apply reinforcement learning to constituency parsing, machine translation, question-answering summaries, memory access, and information gathering in dialogue [Yogatama et al., 2016; He et al., 2016a; Choi et al., 2017a; Shen et al., 2017; Li et al., 2017a].Source: §6 · p. 7

7 Conclusions

• ReSA combines RSS-based, highly parallelizable hard attention with soft self-attention to address the intrinsic weaknesses of both mechanisms.Hard attention trims long sequences and encodes dependency information for soft attention; soft attention supplies reward signals that improve hard-attention training feasibility.Source: §7 · p. 7
• ReSAN is an RNN/CNN-free sentence-encoding model based on ReSA and source2token self-attention.Source: §7 · p. 7
• Across natural language inference and semantic relatedness, ReSAN achieves a new best SNLI test accuracy among sentence-encoding models and state-of-the-art SICK performance with equal or fewer parameters and less time.Source: §7 · p. 7

A Comparison to a Iterative Sampling

• Iterative sampling parameterizes conditional selection with a basic RNN whose latent state encodes context and selection history, producing discrete token selections across the sequence.The comparison tests whether parallel discrete RSS can trim long sentences and model dependencies.Source: A Comparison to a Iterative Sampling · p. 9
• Two iterative-sampling modules select dependent and head tokens, then use their outputs as a mask on the soft self-attention compatibility function.The comparison integrates iterative sampling with soft self-attention in the same way as ReSAN.Source: A Comparison to a Iterative Sampling · p. 9
• The SNLI comparison evaluates parameter count, training and inference time, and training and test accuracy for ReSAN with RSS versus iterative sampling.The models are compared on SNLI, described as a large dataset designed to test sentence-encoding models.Source: A Comparison to a Iterative Sampling · p. 9, Table 4 · p. 9
• 5× less training time and 3× less inference time let RSS-based ReSAN achieve competitive test accuracy versus iterative-sampling ReSAN on SNLI.The table caption notes that accuracy differences are expected to be experimental noise from neural-network randomness.Source: Table 4 · p. 9, A Comparison to a Iterative Sampling · p. 9