A Scaled-Up Empirical Study of Syntactic Language Models

1 Introduction

• SLMs jointly model surface sentences and linearized parse trees, adding explicit hierarchical constituency structure to otherwise flat token sequences.Standard causal models can learn distributional proxies for syntax, but leave hierarchical structure implicit.Source: §1 · p. 1
• Earlier SLM studies mostly evaluated models up to 300M parameters, trained on up to 9B tokens, and tested on only two or three tasks.Prior work also left the complete design space of tree linearizations and attention masks underexplored.Source: §1 · p. 1
• The study evaluates selected models up to 1B parameters and approximately 100B terminal training tokens across 11 downstream tasks and two syntactic benchmarks.It covers 12 configurations: one terminal baseline and 11 SLM variants with different tree linearizations and attention masks.Source: §1 · p. 1
• Standard-causal Tree and TGTree outperform structural-mask variants on XSum in both BBC settings; TGTree also achieves the highest BLiMP score at 500M, while hybrid attention improves document perplexity.At 1B on FineWeb-Edu, TGTree and Pause-2 have similar average downstream accuracy, while tree models perform better on syntactic evaluation.Source: §1 · p. 1
• The study systematically examines tree linearization and attention design on BBC News at 100M and deduplicated BBC News at 500M.The full configuration family is evaluated at 100M, with eight configurations at 500M.Source: §1 · pp. 1–2
• Pause-token baselines are included to assess explicit structure alongside the additional token positions it introduces.Source: §1 · p. 2

2 Related Work

• Transformer Grammars encode constituency trees as action sequences with hierarchical attention, and a later study extends the approach to dependency trees.These approaches are part of a line of work showing syntactic inductive bias can improve language modeling and syntactic evaluation.Source: §2 · p. 2
• Zhao et al. (2025) provide the most systematic comparison of SLM design choices, including tree linearization and composition mechanisms.Prior studies typically couple linearized-tree modeling with specialized attention mechanisms.Source: §2 · p. 2
• Other approaches incorporate syntactic bias through latent structure induction, stack-based recursive state tracking, or dependency-graph-informed attention.Source: §2 · p. 2

3.1 Syntactic Language Models

• A syntactic language model predicts a sentence and its constituency structure through a linearized token sequence, whose probability is factorized autoregressively.When linearization is one-to-one, the sequence probability is the joint probability of the sentence and tree; lossy linearizations can map multiple trees to one sequence.Source: §3.1 · p. 2

3.2 Tree Linearization

• LIN1, LIN2, and LIN3 vary in the structural tokens emitted, while LIN1-shuf and Pause-1/2 provide non-structural controls.LIN1 resembles bracket notation with labeled closing brackets; LIN2 defines the STACK/COMPOSE mask, and LIN3 tests whether additional closure tokens help. LIN1-shuf preserves terminal order; Pause controls insert learned PAUSE tokens after terminals.Source: §3.2 · pp. 2–3

3.3 Structural Attention

• Attention choices include standard causal attention, structural masks, and hybrid attention that assigns different mask types to different heads.TGNomask retains COMPOSE attention at CNT1 but removes the post-composition mask; its default hides CNT2 from later positions, while TGNomask-Aug exposes CNT2. Hybrid attention uses half the heads for each of two mask types.Source: §3.3 · pp. 3–4, Table 3 · p. 4

• Multiple-choice tasks use the OLMES completion/cloze formulation with multi-shot demonstrations, parsing both context and continuation with 1-best constituency parses.Answers are treated as candidate completions rather than labels selected from a displayed option list.Source: §4 · p. 4
• Full scoring includes terminal and non-terminal continuation tokens, whereas terminal scoring includes only answer-content tokens.The two score equations are given in the passage; terminal scoring focuses the comparison on answer content while retaining the parse as context.Source: §4 · p. 4
• Prediction applies task-specific normalization to the terminal score, then selects the candidate with the highest resulting score.Tasks using length normalization divide by the terminal continuation's character count; other tasks use the raw sum.Source: §4 · p. 4

5.1 Setup

• Training uses a shared OLMo-based decoder-only Transformer family; structural and hybrid variants use specialized attention kernels, while Pause-K and Pushdown provide comparison models.BBC settings differ in article versions, cumulative training exposure, and optimization, so cross-setting changes cannot be attributed to scale alone.Source: §5.1 · pp. 4–5, §5.3 · p. 5

5.2 Evaluation Methods

• Evaluation combines five-seed BBC XSum and BoolQ fine-tuning with FineWeb-Edu’s 11-task OLMES suite and SG/BLiMP; document perplexity marginalizes over up to 300 parsed candidates.For document perplexity, only the current sentence is marginalized for a fixed history selected from the highest-probability sequence per completed sentence; truncation gives a lower bound on that sentence probability.Source: §5.2 · p. 5

5.3 Experiment 1: Disentangling Tree Linearization and Attention

• Across BBC settings, Tree and TGTree have the highest XSum means; at dedup 500M they exceed TGNomask by more than 1.4 points and TG by more than 2.8, while TGNomask-Aug leads BLiMP at 100M.TGTree also leads BLiMP at dedup 500M.Source: §5.3 · p. 6, Table 4 · p. 6
• Hybrid attention lowers TGTree Doc-PPL from 14.709 to 14.482 at 100M and from 13.947 to 13.785 at dedup 500M, but lowers XSum, SG, and BLiMP.At 500M it also improves BoolQ by 0.98 points; hybrid attention has the lowest perplexity but does not improve summarization over causal Tree models.Source: §5.3 · p. 6
• Tree and TGTree exceed their pause-token controls on XSum, SG, BLiMP, and document perplexity, with both tree models gaining more than 11 BLiMP points at 500M.Pause tokens still improve XSum over Terminal in both settings, and BoolQ favors pause controls at 500M by 0.21 and 0.43 points.Source: §5.3 · p. 6
• Tree and TGTree improve document perplexity over Terminal in both BBC settings, while the full STACK/COMPOSE mask performs worse and TGNomask recovers much of the loss.NoONT and Compress have higher perplexity than Tree in their respective settings; hybrid attention has the lowest perplexity.Source: §5.3 · pp. 6–7
• Linearization choices trade off tasks: doubling closing tokens gives the highest XSum mean in both experiments, while removing opening non-terminals improves SG but lowers XSum.At 100M, tripling closing tokens further improves SG and BLiMP over TGTree but lowers XSum; shuffling structural tokens substantially lowers BLiMP.Source: §5.3 · p. 7

5.4 Experiment 2: Scaling to 1B on FineWeb-Edu

• At 1B, TGTree averages 55.88, 2.68 points above Terminal, but nearly matches Pause-2 (55.88 vs. 55.80); Tree leads both syntactic benchmarks.Tree exceeds Terminal by 2.14 points on BLiMP and 1.94 points on SG.Source: §5.4 · p. 7

6.1 Why Does Tree Linearization Help Without Structural Attention?

• The implicit multi-task hypothesis is that joint prediction makes constituency boundaries explicit; Sartran et al. (2022) attribute joint-modeling perplexity gains to additional syntactic supervision.The proposed compositional-bottleneck hypothesis says causal attention can learn what to route through constituent positions while retaining direct access to earlier words, but the mechanism is not directly tested.Source: §6.1 · p. 7
• The adaptive-computation hypothesis contrasts a fixed number of learned pause tokens after each terminal with parsedependent tree tokens placed at phrase boundaries.The authors suggest this placement may allocate additional computation near constituent composition.Source: §6.1 · p. 8

6.2 Syntactic Generalization and Downstream Performance

• The best linearization differs by evaluation: Tree-NoONT has the highest SG score in both BBC settings, while TGTree leads XSum and has the higher downstream average on FineWeb-Edu.Tree leads both syntactic benchmarks on FineWeb-Edu, so predicting more structural tokens does not improve every form of syntactic generalization equally.Source: §6.2 · p. 8
• At 1B, Pause-2 approaches TGTree’s downstream average but remains below it on SG and BLiMP, motivating evaluation on both targeted syntax tests and broad downstream tasks.Additional token positions can support downstream performance without producing the same syntactic behavior as explicit tree tokens.Source: §6.2 · p. 8

7 Conclusion

• The study examines tree linearization and attention design at scales up to 1B parameters, supporting tree linearization as a practical way to introduce syntactic bias.The conclusion emphasizes comparison with additional non-structural computation when assessing its benefits.Source: §7 · p. 8
• At 1B, TGTree and Pause-2 have similar aggregate downstream performance, while tree models retain an advantage on syntactic evaluation.The conclusion frames this as a need to compare tree-linearization benefits with those of additional non-structural computation.Source: §7 · p. 8
• Across BBC 100M and dedup 500M, causal tree models achieve strong generation and syntactic performance while remaining compatible with FlashAttention.Hybrid attention provides the best document perplexity.Source: §7 · p. 8

Limitations

• Parse quality is unquantified, the study covers only English, and parsing errors may propagate into models.The proposed compositional-bottleneck and adaptive-computation explanations also remain hypotheses requiring interventions on representations or attention patterns.Source: Limitations · p. 8
• Each pretraining configuration has one run, so fine-tuning seeds and evaluation bootstraps do not measure variation across pretraining runs.BBC rankings are descriptive; FineWeb-Edu per-task tests are exploratory and lack multiple-comparison correction.Source: Limitations · pp. 8–9
• BBC and FineWeb-Edu experiments differ in data, token budgets, and optimization, so they cannot isolate deduplication or scale effects.Pause-token controls approximately match sequence lengths rather than exact FLOPs, and structural-mask models are not evaluated at 1B on FineWeb-Edu.Source: Limitations · p. 9
• The historical BBC perplexity test substantially overlaps training data, while checks for normalized-text, near-duplicate, and local-fragment overlap remain incomplete.A token-level audit found no exact document matches against the deduplicated training arrays.Source: Limitations · p. 9
• Document PPL approximates the parse marginal with a finite candidate set and one selected history; multiple-choice evaluation conditions on silver parses and omits structural-token likelihoods.Full data and checkpoint downloads remain pending.Source: Limitations · p. 9

A.1 Implementation and Hyperparameters

• The OLMo-based decoder-only Transformer uses SwiGLU, RoPE, RMSNorm, and tied input/output embeddings; the 500M models have 508M non-embedding and 579M total parameters.Table 7 lists three scale configurations.Source: §A.1 · p. 11, Table 7 · p. 11
• Models train from scratch for one epoch with AdamW, cosine learning-rate decay, 2,000 warmup steps, and a 2,048-token maximum sequence length, except dedup Pause-2.Pause-2 uses 683 terminal positions expanded to 2,049 positions; the Step Law sets learning rate and batch size.Source: §A.1 · p. 11
• For BBC 500M, Tree uses 1.226 times as many input positions as Pause-1, while TGTree uses 1.059 times as many as Pause-2.Their learning rates and batch sizes also differ under the Step Law; Table 8 reports batch, length, and final-checkpoint steps.Source: §A.1 · p. 11, Table 8 · p. 12
• Deduplicated BBC data contains 1,466,132 articles and 0.801B terminal tokens; training loss includes structural or pause targets, while document PPL is normalized by terminal and EOS tokens.The final checkpoint from each run is evaluated.Source: §A.1 · p. 11

A.2 Computational Costs

• Historical H800 throughput used FlashAttention-2 for causal models and FlexAttention for structural masks, but does not characterize the new RTX 5090 typed-kernel runs.The historical 100M mixing-head measurement was 3,375 token/s, approximately 18× slower than causal.Source: §A.2 · p. 12, Table 9 · p. 12

A.3 Tokenizer and Structural Tokens

• Tree-linearization tokenizers reserve opening and closing non-terminal tokens so constituency structure is preserved during pretraining and evaluation.The tokenizer is based on GPT-2 or Qwen3 tokenization.Source: §A.3 · p. 12

B.1 Downstream Fine-Tuning on BBC News: XSum and BoolQ

• BBC XSum uses three fine-tuning epochs and beam search, while BoolQ uses five epochs and validation accuracy; both report means and sample standard deviations over five seeds.XSum reports ROUGE-1/2/L and their arithmetic mean; BoolQ uses a global batch size of 40.Source: §B.1 · p. 12

B.2 Syntactic Generalization (SG)

• Syntactic Generalization scores use target-region surprisal, with tree models approximating prefix probabilities through word-synchronous beam search.At each terminal step, models emit the observed terminal and may expand non-terminal hypotheses; the beam size is 300 and decoding length is capped at max(6L, 10).Source: §B.2 · p. 13

B.3 BLiMP

• BLiMP scores whether a model assigns higher likelihood to grammatical sentences in minimal pairs across 12 phenomena, using 300 parse proposals for tree-linearization models.Terminal and pause-token models use K = 1; some Tree-NoONT and Tree-Compress scores were not reevaluated with sequence deduplication.Source: §B.3 · p. 13

B.4 Model-Specific Document-PPL Protocols

• Document-PPL scoring deduplicates projected Tree-NoONT and Tree-Compress sequences, but their BLiMP candidate checks remain incomplete and those scores are excluded from marginalized-likelihood claims.NoONT has 34,053,187 distinct candidates after deduplication; Compress has 37,177,859.Source: §B.4 · pp. 13–14
• Tree-Shuffle and Pushdown use distinct probability and candidate spaces, so they are contextual comparisons rather than entries in the controlled Tree/TG PPL ranking.Neither model is included in the new ten-model experiment.Source: §B.4 · p. 14

B.5 OLMES Multiple-Choice Evaluation on FineWeb-Edu

• FineWeb-Edu multiple-choice evaluation treats each answer as a completion and scores candidate-specific suffixes using one silver parse per textual component, without tree marginalization.The primary score is Sterm, with task-specific option-score normalization.Source: §B.5 · p. 14

C Bootstrap Confidence Intervals and Paired Tests

• FineWeb-Edu uncertainty uses 10,000 paired bootstrap resamples; TGTree ranks first in 52.18% of macro-average resamples, versus 42.27% for Pause-2 and 5.55% for Pause-1.Paired contrasts use uncorrected two-sided bootstrap p-values, with p < .05 treated only as a descriptive threshold.Source: C Bootstrap Confidence Intervals and… · p. 14

D Historical BBC Experiments and Clean Reevaluation

• An exact token comparison found training copies of 4,752 of 4,966 original PPL test documents (95.69%), prompting replacement with reserved-clean-v1 perplexities.Scans of deduplicated training found no complete token-identical matches to the replacement development or test documents, though other overlap checks remain incomplete.Source: D Historical BBC Experiments and Clean… · p. 15
• Historical Tree worsens from 100M to 500M on SG (63.13 vs. 81.40) and clean PPL (16.001 vs. 14.736), while deduplicated 500M Tree reaches 83.43 SG and 13.864 PPL.Because data, budgets, and optimization settings changed together, this comparison does not identify the cause.Source: D Historical BBC Experiments and Clean… · p. 15

E SG Subtask Breakdown

• Tree and TGTree each improve Agreement by 10.53 points over Terminal, but differ across other SG subtasks; Tree-NoONT leads Garden Path Effects and Licensing and has the highest overall SG score.TGTree does better on Garden Path Effects than Tree, while Tree does better on Center Embedding and Long-Distance Dependencies.Source: E SG Subtask Breakdown · p. 16

F Full-versus-Terminal Score Sensitivity

• Omitting structural-token scores raises raw-score macro averages by 2.16 points for Tree and 0.62 for TGTree, whose interval includes zero.The analysis reuses evaluation examples and uses different normalization, so it does not independently validate the main scoring rule.Source: F Full-versus-Terminal Score Sensitivity · p. 16
• Sfull scores the candidate with one parser-selected silver tree, while Sterm scores terminal factors only but retains that tree as context; neither is the exact marginal sentence likelihood.Source: F Full-versus-Terminal Score Sensitivity · p. 16

F.1 Alternative Ten-Task Comparison

• Validation-10 scores 22,207 examples; eight tasks overlap with the main evaluation (20,461 examples, 92.1%), so it is a sensitivity comparison, not an independent evaluation set.OpenBookQA and MMLU use different splits.Source: §F.1 · p. 16
• The supplied SG subtask breakdowns cover 500M models on deduplicated BBC News and 1B models on FineWeb-Edu.Their captions identify scope but do not provide subtask results.Source: Table 13 · p. 17, Table 14 · p. 17
• Tree’s macro ∆ interval was [1.35, 2.99], while TGTree’s was [−0.09, 1.35] on the eight-task sensitivity suite.∆=Term−Full; intervals are paired 95% bootstrap intervals.Source: Table 15 · p. 17
• TGTree’s terminal-score accuracy exceeded Pause-2’s by 0.26 points on Validation-10, with a 95% CI of [−0.68, 1.20].The interval includes zero.Source: Table 16 · p. 17