Source-linked AI summary
Learning Structured Text Representations
Yang Liu, Mirella Lapata
TL;DR
The paper addresses the difficulty of learning structure-aware document representations without annotated discourse data or external parsers. It introduces differentiable non-projective structured attention for hierarchical document encoding and reports competitive performance with meaningful induced structures, while noting substantial computational costs for an alternative structured-attention method.
Problem
Annotated discourse data and external parsers make structure-aware document modeling costly and can introduce pipeline error propagation.
Method
The model uses differentiable non-projective dependency-tree marginals as attention weights at sentence and document levels, enabling end-to-end hierarchical representation learning.
Results
Experiments show competitive performance against strong comparison systems, while induced trees are meaningful though different from linguistically motivated structures.
Takeaways & Limitations
The approach learns structural information without an external parser and produces interpretable intermediate document and sentence structures.
Takeaways & Limitations
The inside-outside comparison is computationally prohibitive for document classification, requiring 5 hours of training for one epoch.
Abstract
from arXiv · showhide
In this paper, we focus on learning structure-aware document representations from data without recourse to a discourse parser or additional annotations. Drawing inspiration from recent efforts to empower neural networks with a structural bias, we propose a model that can encode a document while automatically inducing rich structural dependencies. Specifically, we embed a differentiable non-projective parsing algorithm into a neural model and use attention mechanisms to incorporate the structural biases. Experimental evaluation across different tasks and datasets shows that the proposed model achieves state-of-the-art results on document modeling tasks while inducing intermediate structures which are both interpretable and meaningful.
1 Introduction
Document modeling supports many downstream NLP applications, but discourse-aware representations often require costly annotations and external parsers. The paper proposes learning structural document representations directly from data with differentiable dependency-tree attention.
- Document modeling supports topic labeling, summarization, sentiment analysis, question answering, and machine translation.
- Discourse-structure representations include trees, graphs, entity transitions, and combinations of these forms.
- Annotated corpora and external NLP tools make linguistically motivated document representations difficult and expensive to produce.
- External discourse parsers can create pipeline architectures in which errors propagate to later processing stages.
- The proposed model learns document representations with structural information, trains efficiently on documents of arbitrary length, and induces interpretable intermediate structures.
2 Background
Basic intra-sentence attention models word-to-word relationships but does not impose richer structures such as trees. Structured attention addresses this limitation while retaining differentiable end-to-end training, though the inside-outside alternative has high computational cost.
- Intra-sentence attention estimates normalized scores between word pairs and uses them to collect contextual information for each word.
- Basic attention captures shallow word-word dependencies and cannot represent more elaborate structures such as trees or graphs.
- Structured attention imposes dependency-tree constraints on attention distributions while remaining differentiable and parser-free.
- The inside-outside algorithm has O(n3) time complexity, is difficult to parallelize, and can make structured attention impractical for real-world applications.
3 Encoding Text Representations
The model builds hierarchical sentence and document representations while using structured attention to induce non-projective dependency trees over words and sentences. Differentiable operations allow end-to-end training, with structured attention computed efficiently on GPUs.
- Sentence Model: The sentence model separates each bidirectional LSTM output into semantic and structure vectors for task-specific encoding and structured attention.The semantic vector encodes task-relevant information, while the structure vector is used to calculate structured attention.
- Structured Attention Mechanism: Structured attention constrains pairwise attention weights to posterior marginals of a non-projective dependency tree.The model uses Matrix-Tree Theorem operations to calculate edge marginals and uses them as attention weights.
- Structured Attention Mechanism: Structured attention updates each word’s semantic vector with context gathered from possible parent and child nodes, including a special root embedding.The parent and child context vectors are concatenated with the semantic vector and transformed into a new representation.
- Document Model: The model composes documents hierarchically by representing words, pooling them into sentence vectors, and then composing sentence vectors into a document representation.A sentence-level bi-LSTM and structured attention produce updated word representations before pooling; a document-level bi-LSTM and structured attention then produce the final representation y.
4 Experiments
The experiments evaluate structured attention for sentence-level inference, document classification, and induced dependency structures. Results show performance gains, faster inference than inside-outside attention, and interpretable structures that are not trained from annotated trees.
- Natural Language Inference: Structured attention further improves natural language inference, and the matrix-inversion model slightly outperforms Kim et al.’s inside-outside model.The model achieves results in the same ballpark as related LSTM-based systems.
- Natural Language Inference: 10–20 times slower than our model on the same platform, inside-outside attention is considerably less efficient than matrix inversion.Matrix inversion does not reduce running speed relative to simpler attention.
- Document Classification: The combined sentence- and document-level structured-attention model achieves the best results on three of four document-classification datasets.The comparison covers word-based, hierarchical, and parser-based representations.
- Induced Structures: On SNLI, the model induces simpler dependency trees than Stanford’s parser, with average height 5.78 versus 8.99 and most trees at depth 2–4.Almost half of the induced trees are projective despite no enforced projectivity constraint.
- Induced Structures: Although trained without annotated trees, the model produces plausible dependency structures whose edges often agree with Stanford’s parser and are generally shorter and more direct.The induced structures use direct links to collect information from other words.
- Induced Structures: Document-level trees are usually shallow, with over 70% projective trees on Yelp, IMDB, and CZ Movies, but they cannot be directly compared with discourse-parser output.The model builds trees over entire sentences and currently lacks sentence segmentation.
5 Conclusions
The proposed model learns document representations with rich structural dependencies through differentiable non-projective parsing and hierarchical sentence- and document-level structure. Its induced trees are meaningful but differ from linguistic structures, while future work targets linguistically meaningful trees and summarization applications.
- The model automatically learns rich structural dependencies while representing documents hierarchically at both sentence and document levels.Its non-projective dependency-tree marginals are differentiable, enabling efficient end-to-end training.
- Experiments show competitive performance against strong comparison systems on sentence and document modeling tasks.
- The induced tree structures are meaningful despite differing from linguistic ones and requiring neither linguistic annotations nor an external parser.
- Future work will investigate whether appropriate training objectives can induce linguistically meaningful dependency trees.
- The authors also plan to use document-level trees to represent or extract important content for summarization.