Source-linked AI summary
Self-Adaptive Hierarchical Sentence Model
Han Zhao, Zhengdong Lu, Pascal Poupart
TL;DR
AdaSent addresses the challenge of representing sentences across word, phrase, and sentence levels without a fixed structure. It builds multi-scale representations through gated local composition and adaptively selects task-suitable representations, achieving effective and robust short-sequence modeling with superior classification performance across five benchmark datasets.
Problem
Modeling sentences at multiple scales is difficult, while recurrent and recursive networks face gradient vanishing and fixed-structure limitations.
Method
AdaSent recursively composes adjacent segments into a multi-scale hierarchy and uses gating networks to adaptively combine and select representations.
Results
AdaSent consistently outperforms competitor models across five benchmark datasets while using fewer parameters than CNN.
Takeaways & Limitations
AdaSent can learn task-dependent representations and effectively model short sequences through variable-length multi-scale representations.
Abstract
from arXiv · showhide
The ability to accurately model a sentence at varying stages (e.g., word-phrase-sentence) plays a central role in natural language processing. As an effort towards this goal we propose a self-adaptive hierarchical sentence model (AdaSent). AdaSent effectively forms a hierarchy of representations from words to phrases and then to sentences through recursive gated local composition of adjacent segments. We design a competitive mechanism (through gating networks) to allow the representations of the same sentence to be engaged in a particular learning task (e.g., classification), therefore effectively mitigating the gradient vanishing problem persistent in other recursive models. Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets.
1 Introduction
Sentence modeling seeks vector representations that support downstream language tasks, but fixed-length models may discard useful intermediate structure. AdaSent addresses this with a task-adaptive multiscale hierarchy and reports strong benchmark performance.
- Sentence representations are used as inputs for classification, machine translation, and semantic matching.
- cBoW pools word vectors globally, capturing sentence topics but not the sequential structure of words.
- Order-aware neural models incorporate word interactions through recursive, recurrent, convolutional, or gated architectures.
- AdaSent forms a task-adaptive multiscale hierarchy, weighting phrase and sentence representations through gating instead of relying on a predefined parse tree.
- The study reports qualitative task-specific representation learning and quantitative superiority over previous approaches on 5 benchmark data sets.
2 Background
Background models sentence meaning by transforming word inputs through pooling or sequential composition. Their structures range from order-insensitive global pooling to recurrent chains, recursive trees, and gated pyramids.
- A sequence is encoded as one-hot token vectors and mapped to distributed word vectors using an embedding matrix U.
- cBoW represents a sentence by average- or max-pooling its word vectors across the sequence.
- cBoW ignores word order and sentence length, allowing semantically different sentences to share a vector representation.
- An RNN computes each hidden state from the current word vector and previous hidden state, producing a chain-based sentence representation at h_T.
- Recursive neural networks compose representations according to a predefined binary parsing tree.
- Gated recursive convolutional networks learn a pyramid-shaped composition structure on the fly between recurrent chains and recursive trees.
3 Self-Adaptive Hierarchical Sentence Model
AdaSent builds a multiscale hierarchy from words to phrases and sentences, then uses local and global gating to select task-suitable representations. Its recursive compositions also implicitly average alternative substructures, while identity connections support gradient flow.
- 3.1 Structure: AdaSent forms a directed-acyclic pyramid whose units summarize consecutive phrases, from word vectors at level 1 to the whole sentence at the top.Each level contains representations of phrases with a corresponding length, and level pooling summarizes all such phrases.
- 3.1 Structure: Word vectors are linearly mapped from R^d to R^D before entering the pyramid, allowing phrase and sentence representations to occupy a higher-dimensional space.The transformation uses U′ and the pretrained embedding matrix U; their factorization can reduce effective parameters when d ≪ D.
- 3.2 Local Composition and Level Pooling: At each local composition, gating weights choose among transforming the children or forwarding their representations, with recursive composition continuing to the pyramid’s top.The three nonnegative gate coefficients sum to 1, and the gating mechanism can use softmax or another system producing a three-category multinomial distribution.
- 3.2 Local Composition and Level Pooling: AdaSent’s recursive local composition represents each unit as a convex combination of possible substructures, making the model more robust to local noise than rigid recursive structures.The paper presents this weighted model averaging as an implicit property of the recursive composition process.
- 3.3 Gating Network: Each level is pooled into a multiscale summary, and a gating network weights level-specific classifier decisions to form the final consensus.The level summaries can use average or max pooling, while the belief scores indicate which level is suitable for the current instance and task.
- 3.4 Back Propagation through Structure: Identity terms in the backpropagation equations provide constant-error paths, while local composition weights act like forgetting gates to mitigate gradient vanishing.The paper explicitly compares these effects with mechanisms in LSTM memory blocks.
4 Experiments
AdaSent is evaluated on five benchmark datasets for sentence and short-phrase classification against diverse competing models. The experiments examine training stabilization, classification accuracy, variance, and whether the model adaptively selects useful representations.
- Experimental Setting: Experiments compare AdaSent with probabilistic, recursive, convolutional, paragraph-vector, and recurrent sentence models on five benchmark datasets.The datasets include MR, CR, SUBJ, MPQA, and TREC, covering sentiment, subjectivity, and question classification tasks.
- Training: AdaSent uses recurrent-matrix regularization, norm clipping, and minibatch AdaGrad to mitigate gradient-exploding training behavior.The regularizer penalizes the Frobenius norm as a surrogate upper bound for the recurrent matrix’s spectral norm; λ typically ranges from 0.01 to 5 × 10^-5.
- Experiment Results: AdaSent consistently outperforms P.V., cBoW, RNN, BRNN, and GrConv while using 10K–100K parameters versus about 400K for CNN.It outperforms all other models on MPQA, where instances average three words, which the authors attribute to short-phrase modeling.
- Experiment Results: AdaSent outperforms BRNN across all datasets because level-wise global pooling explicitly models phrases of different lengths.The authors also report that variable-length multiscale representation is key to AdaSent’s success, whereas GrConv does not retain intermediate representations.
- Experiment Results: Across ten runs with varied hyperparameters and random initializations, AdaSent consistently outperforms every competing model on all datasets.The evaluation reports mean classification accuracy and standard deviation for each model and dataset.
- Qualitative Analysis: Belief-score distributions vary by input sentence and dataset, while the gating network selects the hierarchy level it considers most appropriate.In one MR example, the gating network assigns the largest belief score to the first level, producing a correct final consensus despite incorrect higher-level predictions.
- Qualitative Analysis: PCA visualizations show that AdaSent learns representations that support better prediction without an explicit separation constraint.The visualization compares AdaSent and cBoW on SUBJ, MPQA, and TREC using the highest-weight hierarchical representation for each sentence.
5 Conclusion
AdaSent represents sequences with a multiscale hierarchy rather than a flat fixed-length vector. The reported analyses and experiments support its effectiveness and robustness for short-sequence modeling and its ability to adapt representations to the task.
- 5 Conclusion: AdaSent uses a multiscale hierarchy instead of a flat, fixed-length continuous representation for sequence modeling.The model forms hierarchical abstractions and feeds them as a multiscale summary to the classifier.
- 5 Conclusion: Qualitative results show that AdaSent learns input-sequence representations according to the task at hand.The conclusion characterizes this task-dependent representation learning as a central qualitative finding.
- 5 Conclusion: Empirical results demonstrate AdaSent’s effectiveness and robustness in short-sequence modeling.The conclusion summarizes the paper’s experimental evidence without narrowing it to a single dataset or metric.