PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Language Modeling by Jointly Learning Syntax and Lexicon

Yikang Shen, Zhouhan Lin, Chin-Wei Huang, Aaron Courville

arXiv:1711.02013v2cs.CLcs.AI

TL;DR

Existing language models have limited access to syntactic structure, while supervised parsers and tree-structured models require structural annotation. PRPN jointly induces syntax from unannotated sentences and uses it for language modeling, achieving or approaching state-of-the-art language-modeling performance and outperforming some unsupervised parsing baselines.

  • Problem

    Existing recurrent models do not efficiently use syntactic information, while tree-structured approaches commonly require costly expert structural annotation.

  • Method

    PRPN uses a differentiable Parsing Network to infer syntactic structure and Reading and Predict Networks to use syntactically related memories for language modeling.

  • Results

    PRPN achieves or approaches state-of-the-art performance on word-level and character-level language modeling and outperforms some strong unsupervised parsing baselines.

  • Takeaways & Limitations

    The experiments show that PRPN can discover syntactic structure from unannotated sentences while leveraging that structure for language modeling.

  • Takeaways & Limitations

    The model assumes that each node depends only on its parent and its left siblings.

Abstract

from arXiv · show

We propose a neural language model capable of unsupervised syntactic structure induction. The model leverages the structure information to form better semantic representations and better language modeling. Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information. On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation. In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network. Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks.

1 INTRODUCTION

Neural language modeling benefits from syntactic structure, but supervised parsers require costly annotation and generalize poorly beyond treebanked, well-formed language. PRPN addresses this by inducing structure from unannotated sentences while using it to improve language modeling.

  • Syntactic knowledge supports hierarchical representations, long-term dependency modeling, compositional effects, and gradient back-propagation.
  • Supervised parsers provide structure information but require expensive expert annotation and have limited generalization to languages without treebanks or informal sentences.
  • Unsupervised syntactic structure induction is studied to parse languages without annotated treebanks, tailor dependency structures, investigate language learning, and examine the poverty of the stimulus.
  • PRPN simultaneously induces syntactic structure from unannotated sentences and uses the inferred structure to form a better language model.
  • Its Parsing Network estimates syntactic distances and soft constituent decisions, while Reading and Predict Networks use syntactically related memories for representation and next-token prediction.
  • The model reaches or approaches state-of-the-art performance on word-level and character-level language modeling and outperforms some strong unsupervised parsing baselines.

2 RELATED WORK

Prior work incorporates tree structure through externally supplied parses, supervised structural signals, recursive models, multiscale recurrent features, and syntactic language models. PRPN differs by inferring tree structure without supervision through recurrent attention.

  • Tree-based language understanding has used bottom-up encoders with external parse trees or models requiring supervised structural signals during training.
  • Recursive models have also been analyzed for learning tree structure without exposure to grammar trees.
  • PRPN infers tree structure unsupervisedly as a recurrent network that implicitly models structure through attention.
  • Earlier and recent recurrent approaches model long-term dependencies through predefined delays, multiple updating frequencies, or recurrent layers at different scales.
  • Syntactic language models include top-down parsing and dependency-based approaches, with some neural models trained using parsed-sentence supervision.

3 MOTIVATION

Direct tree modeling can require supervision and may be brittle on ungrammatical input, while ordinary recurrence overlooks natural-language dependencies. PRPN combines recurrence with syntactically guided skip-connections.

  • In Figure 1, hard arrows denote parent-to-child tree dependencies, while dashed arrows denote sibling dependencies.
  • Directly modeling tree structure is challenging and usually requires supervision, whereas tree reliance can reduce robustness to ungrammatical sentences.
  • Standard recurrent models are more robust to nonconforming sequential data but neglect dependency relations that dominate natural-language sentence structure.
  • PRPN integrates structured dependency relations with recurrent networks through skip-connections to previous syntactically related hidden states.
  • The model represents non-leaf constituents with spans of hidden states and controls their skip-connections using gates derived from latent structure.

i. In order to define gt

The latent structural context determines which earlier states can connect to the current token. These gates support skip-connections, recurrent updates, next-token prediction, and structured attention.

  • The latent variable l_t represents the local structural context of token x_t.
  • For a non-leftmost subtree child, l_t is the position of x_t’s leftmost sibling.
  • For a subtree’s leftmost child, l_t is the leftmost child belonging to that subtree’s leftmost sibling.
  • The gate definition allows selected earlier positions to form valid skip-connections while excluding others.
  • Sibling dependencies receive at least one skip-connection, which forwards information and propagates gradients backward; parent-to-child relations are modeled implicitly.
  • The Reading Network recurrently updates hidden states and the model approximates the next-word distribution from those structured representations.
  • Structured attention in the Reading Network captures dependency relations by summarizing previous recurrent states.

4 MODELING SYNTACTIC STRUCTURE

The model represents local syntactic structure with a probability distribution over possible structures and a differentiable syntactic-distance signal. These quantities support soft gating during learning and allow a tree structure to be inferred from learned distances.

  • At time step t, p(l_t|x_0, ..., x_t) represents the probability of choosing one of t possible local structures.
  • The model uses a Stick-Breaking Process to parameterize the distribution over local structures.The process assigns probability sequentially to candidate positions, with α variables controlling the allocation.
  • The expected gate value is the cumulative distribution function of the local-structure distribution, enabling replacement of discrete gates with expectations.
  • Soft gating vectors approximate the hidden-state update and next-token prediction while preserving differentiability.
  • Syntactic Distance assigns scalar values to adjacent-token pairs, with larger distances identifying constituent boundaries and supporting tree inference.The model compares each current distance with previous distances to locate nearby constituent beginnings.
  • A convolutional kernel computes each syntactic distance from consecutive previous token representations, with window size L defining the look-back range.The distance is produced from convolutional features followed by a ReLU transformation; sequence-wide computation uses front padding.

5 MODELING LANGUAGE

The Reading Network combines recurrent memory with structured attention over previous states, while the Predict Network uses these representations and gates to model the next word. Structured attention links the current token to syntactically related memories and produces adaptive summaries for recurrent updating.

  • The Reading Network generates new states from the current input, previous memory states, and gates.
  • The Reading Network maintains hidden and memory tapes and uses structured attention to capture dependency relations among previous states.Each hidden state is represented by hidden and memory vectors, and the tapes retain bounded spans of prior states.
  • Structured attention produces a probability distribution over previous hidden states and adaptive summaries of the hidden and memory tapes.
  • The recurrent update computes new memory and hidden states from the current input and the adaptive summaries, then appends them to the tapes.
  • The Predict Network models the next-word distribution from hidden states m_0, ..., m_t and gates g_{t+1}.
  • Because x_{t+1} is unavailable at time t, the model temporarily estimates d_{t+1} from x_{t-L}, ..., x_t.

6 EXPERIMENTS

The experiments evaluate PRPN on character-level and word-level language modeling, unsupervised constituency parsing, and ablations. Results indicate strong language-modeling performance, benefit from structured attention, and syntactic structures consistent with human annotation.

  • Experimental setup: PRPN is evaluated on character-level language modeling, word-level language modeling, and unsupervised constituency parsing.The character-level evaluation uses Penn Treebank, while word-level experiments use Penn Treebank and Text8; parsing uses WSJ10.
  • Character-level language modeling: Character-level visualizations show higher syntactic distance at word-final characters followed by spaces, separating words and restricting letter-level attention within words.The model uses separator hidden states to summarize preceding information.
  • Character-level language modeling: PRPN achieves state-of-the-art character-level performance and significantly outperforms baseline models.The supplied results passage reports the comparison without providing the corresponding numerical values.
  • Word-level language modeling: PRPN produces results comparable to state-of-the-art methods on word-level Penn Treebank modeling and outperforms baseline methods on Text8.The authors note that broader hyperparameter tuning might further improve performance.
  • Ablation analysis: Removing the Parsing Network causes a significant performance drop, providing empirical evidence that structure information benefits attention control.The ablation compares variants that remove parsing or structured attention components.
  • Unsupervised constituency parsing: On WSJ10, PRPN significantly outperforms the RANDOM baseline and has comparable performance to CCM, while DMV+CCM and UML-DOP perform better.The latter methods use additional dependency information or model contiguous and non-contiguous lexical dependencies.

7 CONCLUSION

The paper presents PRPN as a neural language model that jointly induces syntax from unannotated sentences and uses it to improve language modeling. It reports near-state-of-the-art language-modeling performance and inferred structures highly correlated with human annotation.

  • Conclusion: PRPN simultaneously induces syntactic structure from unannotated sentences and uses the inferred structure to learn a better language model.Its differentiable parsing decisions allow language-model loss to back-propagate into the neural Parsing Network.
  • Conclusion: PRPN achieves or approaches state-of-the-art performance on word-level and character-level language modeling tasks.The conclusion also reports that inferred syntactic structure is highly correlated with human expert annotation.

A INFERRED TREE STRUCTURE

PRPN infers tree structure from syntactic distances and models local dependencies with probabilistic, softly gated connectivity. The resulting mechanism lets the model retain structure-sensitive attention while remaining differentiable.

  • Tree induction: The Parsing Network computes syntactic distances between successive words, which determine soft constituent decisions and inferred tree structure.The tree is built recursively by sorting distances and splitting constituents around the largest distances.
  • Local structure: At time t, a local-structure variable selects a dependency range over previous hidden states, determining which earlier information the current token can use.The selected structure defines dependencies from m_t′ through m_t.
  • Probabilistic model: A Dirichlet Process provides a nonparametric prior whose number of possible structures grows with sentence length.This allows attention to as many words as occur in the sentence.
  • Differentiable relaxation: The model replaces discrete graph decisions with soft attention and multiplicative gating, so the expected operation can update hidden and predictive functions differentiably.The soft gate approximates the expected value of the discrete masking operation.

C NO PARTIAL OVERLAPPING IN DEPENDENCY RANGES

PRPN’s dependency ranges are designed to avoid partial overlap, a property required for valid tree structure. The gating temperature controls the hierarchy, while syntactic distances distinguish sibling tokens from tokens in different subtrees.

  • No partial overlap: PRPN’s dependency ranges cannot partially overlap under the proposed parameterization, which is essential for recovering a valid tree.The proof derives contradictory distance inequalities from the assumption of partial overlap.
  • Dependency interpretation: A dependency range for x_t contains tokens treated as siblings or as offspring of a sibling, while disjoint or nested ranges correspond to different tree relations.Partially overlapping ranges cannot form a valid tree.
  • Binary recovery: Taking the temperature parameter τ toward infinity converts the soft gates into a binary version based on comparisons of syntactic distances.The resulting gate preserves the dependency-range structure while removing soft overlap.
  • Gating properties: The gate remains monotonic and provides soft truncation of each token’s dependency range.This monotonicity supports the ordering needed for tree-structured dependencies.
  • Temperature effect: When τ is zero, the model yields a flat tree; increasing τ increases hierarchical levels, while intermediate values may allow overlapping truncation ranges.The hard-tree limit is reached as τ approaches positive infinity.
  • Syntactic distance: Syntactic distance is near zero for adjacent sibling words and larger for words in different subtrees, approaching one when they share no subtree.These distances determine the tree shown in the paper’s figure.

E BASELINE METHODS FOR UNSUPERVISED CONSTITUENCY PARSING

The paper evaluates PRPN against standard unsupervised constituency-parsing baselines and other unsupervised parsing models. These comparisons include simple branching strategies, a random binary-tree baseline, an upper bound, and models spanning constituency and dependency parsing.

  • RANDOM selects a binary tree uniformly at random and serves as the unsupervised baseline.
  • LBRANCH and RBRANCH choose completely left-branching and right-branching structures, respectively.
  • The UPPER BOUND measures how well a binary system can perform against Treebank sentences.Because Treebank sentences are generally flatter than binary trees, added brackets count as incorrect and limit attainable precision.
  • PRPN is also compared with DEP-PCFG, CCM, DMV, and DMV+CCM, covering dependency-structured, constituent-context, dependency, and combined constituency-dependency models.
Loading 1711.02013v2…