PaperlayerView exact revision on arXiv

Source-linked AI summary

Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks

R. Thomas McCoy, Robert Frank, Tal Linzen

arXiv:2001.03632v1cs.CL

TL;DR

Finite training data can support multiple generalizations, so architectural inductive biases may shape how sequence-to-sequence models handle syntax. The paper tests architectural factors on English question formation and tense reinflection, finding that only tree-structured models showed a consistent hierarchical bias across tasks.

  • Problem

    The paper asks which architectural properties shape generalization when the same training data is consistent with both hierarchical and linear rules.

    Source: §1 · p. 1

  • Method

    The authors compare neural sequence-to-sequence architectures on English question formation and tense reinflection, with training sets compatible with hierarchical and linear generalizations.

    Source: §1 · p. 2

  • Results

    Only models built around the correct tree structure showed a robust hierarchical bias across tasks; other tested factors also qualitatively affected generalization.

    Source: §1 · p. 2, §9 · p. 10

  • Takeaways & Limitations

    The findings suggest that human-like syntactic generalization may require specific high-level structure, at least when learning from text alone.

    Source: §1 · p. 2

    Extending the conclusions to humans is limited because humans may share processing across phenomena more strongly than neural networks do.

    Source: §9.2 · p. 11

Abstract

from arXiv · show

Learners that are exposed to the same training data might generalize differently due to differing inductive biases. In neural network models, inductive biases could in theory arise from any aspect of the model architecture. We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection. For both tasks, the training set is consistent with a generalization based on hierarchical structure and a generalization based on linear order. All architectural factors that we investigated qualitatively affected how models generalized, including factors with no clear connection to hierarchical structure. For example, LSTMs and GRUs displayed qualitatively different inductive biases. However, the only factor that consistently contributed a hierarchical bias across tasks was the use of a tree-structured model rather than a model with sequential recurrence, suggesting that human-like syntactic generalization requires architectural syntactic structure.

1 Introduction

Because finite training data can support multiple generalizations, a learner’s inductive biases help determine how it handles unseen examples. The paper tests architectural sources of hierarchical bias and finds that only tree-based models generalized hierarchically across both tasks.

1 Introduction

  • Because any finite training set can support multiple generalizations, the learner’s properties as well as its data matter to generalization.The paper uses “inductive biases” for factors that guide a learner toward one generalization rather than another.Source: §1 · p. 1
  • The question-formation task tests hierarchical versus linear generalization on examples where MOVE-MAIN and MOVE-FIRST make different predictions.The training data is consistent with both rules, while examples containing a relative clause on the subject distinguish them.Source: §1 · p. 1
  • Recurrent-unit type, attention type, and model structure all qualitatively affected question-formation generalization, but only tree-based models generalized hierarchically across both tasks.The models’ choices between MOVE-MAIN and MOVE-FIRST varied with these architectural factors.Source: §1 · p. 2
  • LSTMs and GRUs displayed qualitatively different inductive biases, a difference that may partly reflect GRU hidden-state values being bounded.The authors connect this possibility to the effect of squashing on counting generalization described by Weiss et al. (2018).Source: §1 · p. 2
  • Only models using the correct tree structure showed a robust hierarchical bias across tasks; sequential models did not, even with explicit hierarchical input marking.The authors conclude that human-like syntactic generalization may require specific high-level structure, at least when learning from text alone.Source: §1 · p. 2

2 The question formation task

The paper shifts attention from the content of a bias favoring MOVE-MAIN to the learner properties that might produce it. It frames question formation as a comparison between hierarchical and linear rules, then describes the data, evaluation, and architectures used to test that comparison.

2.1 Background

  • Earlier accounts focus on what favors MOVE-MAIN, whereas this study asks which learner factors give rise to that bias, including architectural constraints.The background frames this question against proposals involving substitutability, simplicity, statistical patterns, and semantic knowledge.Source: §2.1 · p. 2

2.2 Framing of the task

  • The task trains models to transform or preserve sentences generated from a context-free grammar, with the final input token indicating the requested operation.Question-formation training examples are consistent with both MOVE-FIRST and MOVE-MAIN; the generalization set includes relative-clause cases where the rules diverge.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3 Evaluation metrics

  • First-word accuracy on the generalization set distinguishes the rules because the moved auxiliary is the output’s first word, while test-set full-sentence accuracy checks exact outputs.The authors use first-word accuracy because models often make other errors on generalization examples; low accuracy indicates consistency with MOVE-FIRST when the output begins with one of the two relevant auxiliaries.Source: §2.3 · p. 3

2.4 Architecture

  • In the sequential sequence-to-sequence model, an encoder reads the input word by word and passes its final hidden state to a decoder that generates the output word by word.The encoder and decoder learn their weights via gradient descent.Source: §2.4 · p. 3, Figure 3 · p. 4

2.5 Overview of experiments

  • With the task held constant, experiments varied recurrent unit and attention type—two aspects with no clear connection to question formation—and tested tree-based rather than sequential models as a more task-relevant modification.Figure 4 reports results for each combination of recurrent unit and attention type; all numbers are medians over 100 initializations.Source: §2.5 · pp. 3–4, Figure 4 · p. 4
  • For the generalization set, cases with identical auxiliaries or a singular–plural mismatch are excluded to prevent uninformative or number-based heuristics.Figure 4 grays out architectures scoring below 50% on the test set.Source: §2.5 · pp. 3–4
  • For most models, the proportion of outputs beginning with the first auxiliary is almost exactly one minus first-word accuracy.The caption notes an exception for the SRN in one attention condition.Source: §2.5 · pp. 3–4

3 Recurrent unit and attention

Recurrent-unit and attention choices shape generalization, and their effects interact. Squashing shifts MOVE-MAIN preference, while outcomes also vary across random seeds and hyperparameters.

3.1 Recurrent unit

  • The recurrent unit updates the encoder and decoder hidden state after each word; SRNs, GRUs, and LSTMs differ in state representation and gating.Source: §3.1 · p. 4

3.2 Attention

  • Attention lets the decoder use a weighted sum of encoder states, with location-based and content-based variants differing in how their weights are derived.Source: §3.2 · p. 4

3.3 Results

  • With SRN content-based attention, models preferred MOVE-MAIN; GRU location-based attention also preferred MOVE-MAIN, while the remaining architectures preferred MOVE-FIRST.The SRN without attention failed on the test set; the other architectures exceeded 50% full-sentence accuracy.Source: §3.3 · p. 5
  • Attention had opposite effects across recurrent units: content-based attention aligned SRNs with MOVE-MAIN and location-based attention with MOVE-FIRST, with the reverse pattern for GRUs.Source: §3.3 · p. 5

3.4 Differences between LSTMs and GRUs

  • LSTMs and GRUs have qualitatively different inductive biases, and adding squashing made each unit type choose MOVE-MAIN more often than its unsquashed counterpart.The authors suggest bounded GRU hidden-state values may partly explain the difference.Source: §3.4 · p. 5

3.5 Hyperparameters and random seed

  • Generalization varied across random seeds, and learning rate and hidden size also appeared to affect it qualitatively, but their effects were difficult to characterize systematically.For SRNs with content-based attention, generalization-set first-word accuracy ranged from 0.17 to 0.90 across seeds.Source: §3.5 · p. 5

4 Tree models

The section tests whether models designed to encode hierarchy actually favor hierarchical generalization. The implicit ON-LSTM did not, while the fully tree-based model was the only tested model that succeeded on the test set and favored MOVE-MAIN.

4.1 Tree model that learns implicit structure

  • The ON-LSTM imposes stacklike constraints on LSTM cell-state updates, creating an implicit soft tree rather than receiving sentence trees as input.Each soft tree level corresponds to a depth in the cell state.Source: §4.1 · pp. 5–6
  • ON-LSTM achieved 0.93 test-set full-sentence accuracy but only 0.05 generalization-set first-word accuracy, favoring MOVE-FIRST rather than MOVE-MAIN.Its behavior shows that structured processing alone did not produce the expected hierarchical generalization.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Tree models given explicit structure

  • Tree-RNNs use supplied parse trees: the Tree-GRU encoder composes sister nodes bottom-up, while the tree decoder expands a parent into left and right children.The experiments compare tree encoders, tree decoders, and models using both.Source: §4.2 · p. 6
  • Only Tree/Tree both succeeded on the test set and favored MOVE-MAIN; models with tree decoders favored MOVE-MAIN, but Sequential/Tree failed on the test set.Tree/Tree reached 69% generalization-set full-sentence accuracy, while all other models had close to 0%.Source: §4.2 · pp. 6–7, Figure 7 · p. 7

5 Tense reinflection

The tense-reinflection task tests whether models generalize by hierarchical subject agreement or by the most recent noun. Sequential models and the ON-LSTM favored recent-noun agreement, whereas Tree/Tree favored subject agreement, making it the only tested model consistent with a hierarchical bias across both tasks.

5 Tense reinflection

  • The authors use a second task because a MOVE-MAIN preference on question formation alone might reflect task-specific factors rather than a general hierarchical bias.They treat consistent hierarchical generalization across tasks as evidence of such a bias.Source: §5 · p. 7

5.1 Reinflection task

  • The task transforms past-tense sentences into present tense or leaves them unchanged, testing agreement with either the hierarchical subject or the linearly most recent noun.The models were trained where AGREE-SUBJECT and AGREE-RECENT agreed, then evaluated on examples where the rules differed.Source: §5.1 · p. 7
  • All sequential models and the ON-LSTM favored AGREE-RECENT, while Tree/Tree favored AGREE-SUBJECT; only Tree/Tree behaved consistently with a hierarchical bias across both tasks.The relevant bias of the sequential models remains unclear; the authors claim only that it is not hierarchical.Source: §5.1 · p. 8

6 Are tree models constrained to generalize hierarchically?

The authors test whether tree models are structurally forced to generalize hierarchically by training sequential and Tree/Tree models on datasets that unambiguously support linear or hierarchical transformations.

6 Are tree models constrained to generalize hierarchically?

  • The four unambiguous training sets support either MOVE-MAIN or MOVE-FIRST for question formation, and either AGREE-SUBJECT or AGREE-RECENT for tense reinflection.The authors trained 100 sequential GRUs and 100 Tree/Tree GRUs, all without attention, on each dataset.Source: §6 · p. 8
  • Both sequential and tree-based models achieved 100% first-word accuracy on MOVE-MAIN and MOVE-FIRST datasets, while sequential and tree-based models achieved 91% and 99% main-verb accuracy, respectively, on both agreement datasets.These results show that both architectures learned linear and hierarchical transformations with similar accuracy.Source: §6 · p. 8
  • Tree/Tree’s hierarchical preference under ambiguous training did not arise from a constraint imposed by tree structure; appropriate training data could override its inductive bias.The models learned linear and hierarchical transformations with similar accuracy.Source: §6 · p. 8

7 Tree structure vs. tree information

The section separates the effects of tree-based computation from the effects of providing parse information. The results indicate that both the model structure and the correctness of the tree structure matter for hierarchical generalization.

7 Tree structure vs. tree information

  • To separate parse information from computation structure, the authors tested Tree/Tree with right-branching trees and sequential GRUs given bracketed parse information in their inputs and outputs.Right-branching trees supplied structure without linguistically informative parses; brackets supplied parse information without guiding the sequential network’s computation.Source: §7 · pp. 8–9
  • Neither a bracket-informed sequential GRU nor Tree/Tree using right-branching trees displayed a hierarchical bias on either task.The bracketed-input experiment used patience 6 because the added brackets lengthened sentences; other hyperparameters remained as described in Appendix A.Source: §7 · p. 9
  • The results indicate that input/output parse information alone was insufficient, and tree-based computation alone was also insufficient without the correct parse structure.The authors conclude that parse information must be integrated into model structure and that the tree structure must be correct.Source: §7 · p. 9

8 Will models generalize across transformations?

The authors test whether an unambiguous task transfers hierarchical generalization to an ambiguous task. Multi-task learning alone had little effect; adding overt auxiliaries produced an asymmetric result, which the authors attribute tentatively to agreement cues rather than transfer of the transformation itself.

8 Will models generalize across transformations?

  • The multi-task test addresses the possibility that learners use hierarchical evidence from other linguistic phenomena when a transformation itself is ambiguous.The motivation is that human learners encounter a language with many phenomena, not an isolated transformation.Source: §8 · pp. 9–10
  • With overt auxiliaries added, ambiguous tense reinflection generalized hierarchically, while ambiguous question formation still generalized linearly.Figure 10 reports generalization-set performance on the ambiguous task.Source: Figure 10 · p. 10, §8 · p. 10
  • Without added auxiliaries, multi-task behavior differed only minimally from single-task behavior in both settings.The two settings made either question formation or tense reinflection the ambiguous task.Source: §8 · p. 10
  • The authors hypothesize that the asymmetric transfer comes from unambiguous long-distance subject–verb agreement in question formation, not the question-formation transformation itself.They call the apparent tense-reinflection improvement likely spurious because its training set lacks the relevant withheld output types.Source: §8 · p. 10

9 Discussion

The discussion argues that model generalization reflects interacting architectural factors, but robust hierarchical generalization appeared only with explicit syntactic structure. The authors qualify implications for human acquisition and identify tree-based models with correct parses as the most viable tested approach.

9 Discussion

  • All tested factors qualitatively affected inductive biases, but hierarchical bias arose only when inputs and computations were governed by syntactic structure.The authors connect this result to the claim that hierarchical bias has been argued to underlie children’s syntax acquisition.Source: §9 · p. 10

9.1 Relation to Rethinking Innateness

  • Low-level architectural changes affected generalization, yet the robust hierarchical preference came from explicit linguistic structure rather than low-level properties.Attention type had different effects in SRNs and GRUs; the authors therefore treat support for fine-grained architectural constraints as partial.Source: §9.1 · p. 11

9.2 Relation to human language acquisition

  • Only tree-based models with correct parses showed hierarchical bias; extending this result to humans is limited by possible differences in input and cross-phenomenon sharing.The authors note that children may hear unambiguous hierarchical examples and receive cues such as prosody and real-world grounding, which their training data omit; the restricted setup may even make learning easier than richer input.Source: §9.2 · p. 11

9.3 Practical takeaways

  • Tree-based architectures with correct parses were the most viable tested route; no explored hyperparameter setting yielded cross-task hierarchical generalization in non-tree models.Multi-task learning mostly brought modest improvements, though further secondary tasks might help; the authors leave tree-based learning without correct parses for future work.Source: §9.3 · pp. 11–12

A Architecture and training details

The appendix specifies shared model and optimization settings, along with dataset sizes and the training procedure.

A Architecture and training details

  • Models used 256-dimensional word embeddings and hidden states, a learning rate of 0.001, and batch size 5.Word embeddings were learned from scratch.Source: A Architecture and training details · p. 12
  • Training used 100,000 examples, with 10,000 examples each for validation, test, and generalization; teacher forcing was used on 50% of examples.Evaluation occurred every 1,000 batches, and training stopped after at least 30,000 batches if validation failed to improve across three evaluations.Source: A Architecture and training details · p. 12

B Equations for squashing experiments

The appendix describes equations for experiments that modify LSTM and GRU gating to alter whether their states are squashed.

B Equations for squashing experiments

  • The LSTM squashing experiment modifies the standard cell-state update so its new coefficients add to 1.The change was intended to make the LSTM cell state exhibit squashing.Source: B Equations for squashing experiments · p. 12
  • The non-squashed GRU modification adds an input gate and changes the hidden-state update so z acts solely as a forget gate.The authors changed gate structure rather than adding a squashing nonlinearity to avoid vanishing gradients.Source: B Equations for squashing experiments · p. 12

Related papers

Loading 2001.03632v1…