Source-linked AI summary
Simple and Effective Text Matching with Richer Alignment Features
Runqi Yang, Jianhai Zhang, Xing Gao, Feng Ji, Haiqing Chen
TL;DR
Text matching models must balance alignment quality with efficiency across several tasks. RE2 preserves original, previous aligned, and contextual features while simplifying other components, and it reaches state-of-the-art-level performance across four datasets with fewer parameters and faster inference. The paper concludes that this combination is suitable for broad text-matching applications, while its implementation assumes variance-preserving initialization for deeper residual connections.
Problem
The paper asks what components are sufficient for fast, effective general-purpose text matching across multiple tasks.
Method
RE2 uses repeated simple alignment blocks that retain original point-wise, previous aligned, and contextual features while simplifying other components.
Results
RE2 performs on par with state-of-the-art models across four datasets and three tasks, with fewer parameters and the fastest inference among similarly performing models.
Takeaways & Limitations
RE2 is suitable for a wide range of text-matching applications because it combines fast speed with strong performance.
Takeaways & Limitations
The implementation uses a variance-preserving initialization assumption when connecting residual outputs for n ≥3.
Abstract
from arXiv · showhide
In this paper, we present a fast and strong neural approach for general purpose text matching applications. We explore what is sufficient to build a fast and well-performed text matching model and propose to keep three key features available for inter-sequence alignment: original point-wise features, previous aligned features, and contextual features while simplifying all the remaining components. We conduct experiments on four well-studied benchmark datasets across tasks of natural language inference, paraphrase identification and answer selection. The performance of our model is on par with the state-of-the-art on all datasets with much fewer parameters and the inference speed is at least 6 times faster compared with similarly performed ones.
1 Introduction
RE2 argues that effective text matching can retain only essential alignment features while simplifying other components, yielding a fast architecture with strong benchmark performance.
- Text matching covers natural language inference, paraphrase identification, answer selection, and related tasks that predict relationships between two text sequences.
- Prior models compensate for a single alignment process with external syntactic features, complicated alignment mechanisms, or extensive post-processing layers.
- Multiple alignment layers can refine predictions, but deeper architectures are harder to train because lower-level features propagate inefficiently and gradients vanish.
- RE2 retains previous aligned, original point-wise, and contextual features for alignment while questioning the need for slower components used in earlier approaches.
- The architecture keeps individual layers simple and combines them into a lightweight model that remains powerful.
- Across four datasets and three tasks, RE2 performs on par with state-of-the-art models while using the fewest parameters and achieving the fastest inference among similarly performing models.
2 Our Approach
RE2 processes two sequences through repeated simple blocks that encode, align, and fuse representations. Augmented residual connections preserve original, previously aligned, and contextual features for subsequent alignment and prediction.
- RE2 embeds tokens, processes both sequences through consecutive same-structured blocks, pools the final representations, and predicts the target.
- The alignment and fusion layers receive original point-wise, previous aligned, and contextual features as complementary inputs.
- Each block contains encoding, alignment, and fusion layers, while augmented residual connections connect the blocks.
- RE2 uses simple word embeddings, vanilla multi-layer convolutional encoders, and max-over-time pooling without character embeddings or syntactic features.
- Attention-based alignment computes similarity between positions in the two sequences and produces representations by weighted summation of the other sequence.
- The fusion layer compares local and aligned representations using subtraction and element-wise multiplication to highlight differences and similarities.
- The prediction layer combines pooled vectors, their difference, and their element-wise product before a feed-forward network produces the output.
3 Experiments
Experiments show that RE2 matches state-of-the-art performance across four datasets and three tasks while using few parameters and fast inference. Ablations and occlusion analyses identify the three alignment features as complementary, with stacked blocks and a necessary fusion design supporting performance.
- Inference efficiency: RE2 is 6 times faster than CSRAN and at least 17 times faster than DIIN in CPU inference among similarly performed methods.Measurements use batches of eight sentence pairs on Intel Core i7 CPUs.
- Benchmark results: RE2 achieves performance on par with the state-of-the-art on SNLI, SciTail, Quora Question Pairs, and WikiQA.These datasets cover natural language inference, paraphrase identification, and answer selection.
- Ablation analysis: Removing richer alignment inputs significantly degrades performance across all datasets, establishing them as a key model component.The alignment inputs retain original point-wise, previous aligned, and contextual features.
- Ablation analysis: A simpler fusion layer performs worse, and parallel blocks underperform stacked blocks, whereas alignment and prediction layers can be simplified on some datasets.
- Occlusion analysis: Contextual features support entailment, while previous aligned features and original point-wise features support neutral and contradiction decisions.The analyses associate these features with phrase-level encoding, focusing on aligned pairs, and comparing semantic differences, respectively.
4 Related Work
Prior text-matching models improve alignment through richer features, more complex matching operations, post-processing, or repeated alignment, but deeper architectures introduce training difficulties.
- Earlier models encode each sequence independently before classifying the resulting vectors.
- Matching-aggregation models align sequences at lower levels and aggregate aligned representations with attention mechanisms.
- Three improvement strategies add syntactic or hand-designed features, increase alignment complexity, or add post-processing layers.
- Multiple alignment layers refine predictions through intermediate states but suffer from inefficient feature propagation and vanishing gradients.
5 Conclusion
RE2 is a highly efficient general-purpose text-matching model that retains three alignment features while simplifying most other components. It matches state-of-the-art performance across four datasets and three tasks with few parameters and fast inference.
- RE2 retains previous aligned, original point-wise, and contextual features for inter-sequence alignment.
- The model simplifies most other components while targeting high efficiency for general-purpose text matching.
- RE2 achieves state-of-the-art-level performance on four datasets spanning three text-matching tasks with few parameters and very high inference speed.