PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

Haris Riaz, Hyungji Kim, Mihai Surdeanu

arXiv:2608.06111v1cs.CLcs.AI

TL;DR

Transformers’ positional embeddings encode token order but largely miss syntactic structure. SiPE learns a lightweight dependency-based prior and improves syntactic generalization, perplexity, and language understanding across encoder and decoder models.

  • Problem

    Transformers’ positional embeddings represent token order but lack the syntactic relations needed for richer compositional representations.

  • Method

    SiPE learns a lightweight prior from linearized dependency trees and injects it through the positional pathway across absolute, relative, and rotary embeddings.

  • Results

    SiPE improves syntactic generalization and language understanding while reducing perplexity; decoder SyntaxGym reaches 80.60, a +10.3% relative improvement over the no-syntax baseline.

  • Takeaways & Limitations

    SiPE establishes a more favorable tradeoff between syntactic supervision and inference cost by conditioning on a single parse without changing downstream model usage.

  • Takeaways & Limitations

    SiPE requires dependency-parser tags at inference and inherits parser errors, which may reduce benefits in domains where parsing is less accurate.

Abstract

from arXiv · show

Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}. We introduce \textbf{S}yntax-\textbf{i}nformed \textbf{P}ositional \textbf{E}mbeddings (\textbf{SiPE}), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate \emph{where} and \emph{how} the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly---while for encoders it is best added directly to the input embeddings, composing with each encoder's native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to $10.3\%$ while simultaneously reducing perplexity by $9.0\%$ over a base model with no syntactic supervision---a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to $8.2\%$ over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost.

1 Introduction

SiPE adds a lightweight dependency-tree prior to positional embeddings, addressing Transformers’ limited representation of syntactic relations without changing self-attention or other components. Across encoders and decoders, it improves syntactic generalization, perplexity, and language understanding while conditioning on a single parse at inference.

  • Motivation: Positional embeddings encode token order but fail to capture syntactic relations underlying semantic composition, contributing to weak compositional generalization.Prior approaches often impose syntactic structure through quadratic token-token interactions and hard-coded attention masks, increasing architectural complexity and computational overhead.
  • Method: SiPE augments the positional pathway with coarse directional indicators from linearized dependency trees, learned through an auxiliary indicator-prediction objective during pre-training.The method composes with absolute, relative, and rotary positional encoding families without modifying self-attention or other Transformer components.
  • Decoder results: +10.3 relative improvement yields the best SyntaxGym score, 80.60, while perplexity decreases by 9.0% relative for Transformer-XL decoders.For autoregressive decoders, positional-pathway injection alone is the strongest configuration.
  • Downstream results: +8.2% relative improvement on GLUE is achieved by the same Transformer-XL model, while encoders improve BLiMP by up to +2.3% relative for ModernBERT.Continued pre-training on BLLIP-LG adds +3% for DeBERTav3 on out-of-domain evaluation, and the same encoder models improve GLUE at small and large scale.
  • Inference efficiency: SiPE conditions on a single parse at inference, moving the Pareto frontier between syntactic supervision and inference cost without adding asymptotic cost over self-attention.Unlike prior approaches that parse at inference or discard syntax at runtime, SiPE retains syntactic information during inference.

2 Related Work, Background, and Notations

Prior syntax-aware Transformers often constrain self-attention with pairwise interactions or hard-coded masks, increasing parameters and training complexity. This work instead represents dependency structure with lightweight Hexatagging-derived priors applied through the positional-embedding pathway.

  • Related work: Prior work finds that formal-language pretraining and tree-based signals can improve syntactic generalization and data efficiency.
  • Related work: Most existing methods inject syntax by constraining self-attention with pairwise token interactions and hard-coded attention masks, adding parameters and training complexity.
  • Hexatagging: Hexatagger assigns each token a governor-position label and a dependency-relation label, enabling dependency structure to be represented through embedding lookups without modifying self-attention.
  • Hexatagging: The implementation uses terminal/non-terminal hexatags alone, omits dependency-relation labels, and adds one dedicated EOS non-terminal label per token.

3 Approach

SiPE represents dependency syntax as hexatags attached to each word and injects the resulting learned prior at different locations in Transformer positional and attention pathways. For Transformer-XL decoders, multiplicative coupling with the relative-position term is strongest, while ungated or redundant alternatives perform worse.

  • Hexatag representation: SiPE parses each input into a hexatagged sequence, trains encoders with masked language modeling and decoders with next-token prediction, and preserves tags under prefix truncation.Each token carries its tags, while continuation subwords contribute no tag term.
  • Injection sites: The learned syntactic prior can be entangled with token embeddings, composed with native positional encoding, or disentangled as an additive attention-score term.Two learned embedding tables encode terminal and non-terminal tags, with first-subword masking injecting each word’s hexatag once.
  • Relative-position decoder: Transformer-XL supplies the decoder testbed, using sinusoidal relative-position encoding directly in attention scores with cross-segment caching disabled.This makes relative positions apply only within the current context window.
  • Positional-pathway injection: Multiplicative coupling of the syntactic correction with the relative-position term outperforms all other tested injection approaches on syntactic evaluation.The correction is gated by query–offset alignment: strong alignment produces a larger syntactic adjustment, while near-zero alignment suppresses it.
  • Ablations: Ungated additive attention bias is weaker than multiplicative coupling, per-layer attention projections outperform shared projections, and combining positional and attention-side injections hurts.The joint variant is redundant rather than complementary for Transformer-XL.

4 Experimental Results

SiPE’s best injection route depends on architecture: positional-pathway injection is strongest for the relative-PE Transformer-XL decoder, whereas input-pathway injection is most reliable for encoders. Across intrinsic and downstream evaluations, SiPE improves syntactic performance, perplexity, and GLUE results while using a single parse at inference.

  • Inference cost: SiPE uses a single parse tree at inference instead of expensive marginalization.This is an explicit design trade-off in the evaluation setup.
  • Intrinsic evaluation: 10.29-point SyntaxGym improvement accompanies a perplexity reduction from 18.63 to 16.95 for positional-pathway SiPE on Transformer-XL.The decoder reaches SyntaxGym 80.60 versus the vanilla token baseline, while perplexity improves on BLLIP-LG.
  • Extrinsic evaluation: Input-pathway SiPE consistently outperforms encoder baselines on GLUE, while positional-pathway injection also helps relative and rotary encoders with smaller gains.The three encoder families are RoBERTa with absolute PE, DeBERTa-v3 with relative PE, and ModernBERT with RoPE.
  • Extrinsic evaluation: +8.2% relative GLUE macro-average improvement raises Transformer-XL from 68.17 to 73.78 with positional-pathway SiPE.SiPE improves every Transformer-XL GLUE task and shows the positional pathway best augments its relative positional encoding.
  • Intrinsic evaluation: +1.41% (RoBERTa), +2.27% (DeBERTa-v3), and +2.32% (ModernBERT) are the encoder BLiMP gains from SiPE.After continued pretraining on BLLIP-LG, these rise to +1.87%, +3.06%, and +4.21%, respectively, indicating out-of-distribution generalization.

5 Where Should Syntax Enter? A Layerwise Injection Study

The layerwise study finds that SiPE syntax should enter a decoder at the earliest layers, with full injection from the first layer producing the strongest syntactic generalization. Attention analysis further shows that encoders visibly redistribute syntactic attention, whereas Transformer-XL converts the prior mainly into downstream semantic gains.

  • Layerwise Injection: 80.6 versus 73.5: skipping Transformer-XL’s first layer sharply reduces SyntaxGym performance compared with full SiPE injection.The study injects SiPE from layer k onward and compares k ∈ {2, 4, 6, 8, 10, 12, 14} with full injection from k=1.
  • Layerwise Injection: Syntactic information matters most in Transformer-XL’s lower layers, while injecting it only in later layers consistently weakens syntactic generalization.Full injection beginning at the first layer is clearly best in the sweep.
  • Layerwise Injection: BLiMP drops are expected because its full-sentence scoring is dominated by language-modeling ability rather than the ungrammatical span alone.Most baselines in Table 2 likewise fall below the vanilla token baseline on BLiMP.
  • Attention Analysis: Encoders redistribute verb→object attention toward syntactic adjacency, whereas Transformer-XL shows the prior weakly in attention and yields the largest downstream GLUE gains.This comparison comes from BLiMP object–verb agreement attention analysis.

6 Conclusion

The paper proposes a lightweight strategy for infusing syntactic information into any Transformer architecture. Its additive prior embedding adds only O(1,000) parameters per model while yielding reliable gains across intrinsic and extrinsic evaluations and multiple encoder and decoder variants.

  • The proposed strategy infuses syntactic information into any Transformer architecture.
  • O(1,000) parameters per model are added through an additive prior embedding.
  • Reliable gains appear across intrinsic and extrinsic evaluations and multiple encoder and decoder Transformer variants.

Limitations · A Preliminaries · A.1 Subword-Level Tag Injection

The paper’s limitations concern inference-time parsing, restricted syntactic supervision, incomplete exploration of injection designs, decoder-generation costs, and limited scale and linguistic coverage. Its subword implementation attaches each word’s hexatag only to the first subword, using the same mask to control prior injection and auxiliary supervision.

  • Limitations: At inference, every input must be dependency-parsed into hexatags, so the method inherits tagger errors and adds preprocessing overhead despite using a lightweight 184M-parameter DeBERTa-v3-base tagger.The tagger runs efficiently relative to encoder pre-training and downstream fine-tuning.
  • Limitations: The study uses only coarse directional terminal and non-terminal priors; preliminary DEPREL supervision produced no meaningful GLUE gains and was not investigated further.This leaves the value of richer dependency-relation labels unresolved.
  • Limitations: The injection search is broad but incomplete, so reported configurations are the strongest found under a fixed compute budget rather than guaranteed global optima.The unexplored space is especially large for relative and rotary attention-side variants.
  • Limitations: Autoregressive generation is hindered because each new token requires re-tagging, while parser revisions to earlier tags can invalidate injected representations and standard KV-caching.The authors identify fast decoder generation as their biggest limitation and a target for future work.
  • Limitations: Experiments cover only small models, English, and the studied pre-training budgets, leaving scaling to larger models, other languages, and longer training unresolved.The relative benefit of an explicit syntactic prior at billion-parameter scale remains unclear.
  • A.1 Subword-Level Tag Injection: Because hexatags are word-level but modeling is subword-level, the implementation attaches each word’s tag pair only to its first subword and leaves continuation subwords untagged.Continuation subwords receive syntactic information indirectly through self-attention.
  • A.1 Subword-Level Tag Injection: The first-subword mask controls both prior injection and auxiliary supervision, giving each word one input tag signal and one prediction while avoiding length-dependent over-weighting.The auxiliary head predicts next tagged positions in autoregressive models and tags at masked positions in masked language models.
  • A Preliminaries: The paper compares input, positional, and attention-side syntactic injections across RoBERTa, DeBERTa-v3, ModernBERT, and Transformer-XL.These architectures span absolute, relative, and rotary positional-encoding schemes across encoders and an autoregressive decoder.

A.2 SiPE Position-Pathway Design Choices for Transformer-XL

For Transformer-XL, SiPE’s positional-pathway design depends on how syntax is coupled to position and on pairing layer-specific projections with the projection target. The adopted configuration uses per-layer projections into the full model dimension, while multiplicative coupling is strongest overall.

  • Projection and layer-sharing choices: 74.01 BLiMP and 80.60 SG identify per-layer projection into the full model dimension as Transformer-XL’s best configuration.The gain comes from pairing the two design choices rather than from either axis alone.
  • Projection and layer-sharing choices: Per-layer projections let each layer specialize syntax’s modulation of positional preference, while the full model dimension provides a richer projection target than a single head subspace.The per-layer correction is scaled by each layer’s own positional term, and the full-dimensional output is split across heads.
  • Projection and layer-sharing choices: The best projection target depends on coupling: multiplicative injection works better with dmodel, whereas fully disentangled coupling works better with the dhead head subspace.Thus, the projection target should not be selected independently of how the syntactic prior enters the position pathway.
  • Coupling form: Multiplicative coupling outperforms both input-side and disentangled injection, while disentangled injection already outperforms input-side injection.Multiplicative coupling scales the position term by tag alignment; disentangled coupling adds alignment as a standalone logit term.

A.3 Fully-Disentangled Syntax Injection Design Choices for Transformer-XL … B Time Complexity

The paper compares syntax-injection design choices across Transformer-XL, DeBERTa-v3, and ModernBERT, identifying stronger configurations while preserving compatibility with native positional mechanisms. It also shows that SiPE adds few parameters and can retain lower-order time complexity for input-side injection.

  • A.3 Fully-Disentangled Syntax Injection Design Choices for Transformer-XL: Per-layer projection into each attention head’s subspace is the best fully-disentangled configuration, averaging strongest across BLiMP and SyntaxGym.The four combinations vary layer sharing and projection target; this configuration is used in the main results.
  • A.3 Fully-Disentangled Syntax Injection Design Choices for Transformer-XL: 74.72 BLiMP and 78.72 SG are achieved by the best disentangled configuration, exceeding input-side injection at 73.82 BLiMP and 76.97 SG but trailing multiplicative coupling.The performance ordering is input-side < fully disentangled < multiplicative.
  • A.4 Attention-Side and Combined Injection Design Choices for Transformer-XL: Per-layer attention-side projections outperform shared projections on syntactic evaluations, both alone and combined with input-side injection.Each layer can specialize its tag-to-attention routing, so the per-layer form is used wherever attention-side biases appear.
  • A.5 DeBERTa-v3: Disentangled Relative Positional Embeddings and Tag Injection: In DeBERTa-v3, input-side tag priors enter through the content stream, while the relative-position table and its projections remain unmodified.The two streams meet only when their score terms are summed into the attention logit, allowing portability across positional schemes.
  • A.5 DeBERTa-v3: Disentangled Relative Positional Embeddings and Tag Injection: DeBERTa’s position-pathway variant multiplies both relative-position terms, c2p and p2c, by tag–query alignment while leaving content-to-content untouched.This is the direct analogue of multiplicative Transformer-XL injection, adapted to DeBERTa’s two-term position pathway.
  • A.6 ModernBERT: Rotary Injection Details: Input-level injection commutes with the native positional mechanism across RoBERTa’s absolute, DeBERTa’s relative, and ModernBERT’s rotary encodings.This establishes portability of the input-side design across the three encoder architectures.
  • B Time Complexity: 7d parameters per pathway, approximately 5K at d = 768 and 7K at d = 1024, keep SiPE under 0.01% of total language-model parameters.SiPE uses separate terminal and non-terminal tag tables, totaling 7d parameters per pathway.
  • B Time Complexity: O(Ld) lookup and addition cost for input-side injection is lower-order than self-attention’s O(L^2d), whereas position-pathway interactions cost O(L^2d).Position-pathway variants therefore match self-attention asymptotically, while input-side injection preserves lower-order overhead.

C Analyzing Attention Patterns on Object–Verb Agreement Tasks

SiPE increases BLiMP accuracy across absolute, rotary, and relative positional-encoding models, with encoders strongly redirecting verb→object attention toward syntactically adjacent objects. Transformer-XL shows a weaker attention-map effect but the largest downstream GLUE gain, suggesting it routes more syntactic information beyond attention.

  • BLiMP attention analysis: Across all three positional-encoding families, SiPE achieves higher overall BLiMP accuracy than the corresponding base model.The comparison includes RoBERTa-base, ModernBERT-base, and Transformer-XL.
  • Decoder attention patterns: 55% of Transformer-XL cases shift verb→object attention toward the object, a much weaker effect than in the encoders.The Transformer-XL result is based on 11/20 cases and uses positional-pathway injection averaged over all 16 layers.
  • Downstream implications: +8.2% relative on GLUE is Transformer-XL’s largest downstream gain despite its weak attention-map shift.This suggests the decoder routes more enhanced syntactic knowledge beyond attention toward semantic processing.

D Hyperparameters

Fine-tuning uses standardized AdamW settings across backbones, with task-specific learning-rate and weight-decay variation limited to ModernBERT.

  • D Hyperparameters: AdamW uses (β1, β2) = (0.9, 0.999) and ϵ = 10−6 unless otherwise noted.These optimizer settings apply across the task-specific fine-tuning configurations.
  • D Hyperparameters: 20 epochs are used for all GLUE fine-tuning runs.The runs also use AdamW with ϵ = 10−6.
  • D Hyperparameters: RoBERTa and DeBERTa-v3 use LR 1e−5 and WD 0.1 throughout, while ModernBERT varies these by task.Only ModernBERT’s learning rate and weight decay vary across GLUE tasks.

E Alternative Embedding Combination Strategies

The experiments find that directly adding terminal and non-terminal syntactic priors at the input embedding is the strongest strategy for absolute positional embeddings in RoBERTa-base. Adding full dependency-relation labels or extra fusion capacity does not improve downstream performance, while the finding is limited to this encoder and positional scheme.

  • Adding dependency relations: Richer dependency-relation labels do not translate into stronger downstream performance in the tested fusion forms.The study uses a substantially larger label inventory than the terminal and non-terminal priors, but its variants remain no better than ADD_T_NT.
  • Fusion mechanism: Directly adding terminal and non-terminal priors to the input embedding achieves the best macro-average among fusion strategies.This ADD_T_NT strategy uses no extra parameters and outperforms concatenation and residual injection.
  • Fusion mechanism: Adding fusion capacity through concatenation, residual injection, or learned interpolation does not surpass direct addition.The learned interpolation variant never improves over ADD_T_NT at any initialization, and residual injection yields only modest gains.
  • Adding dependency relations: Every dependency-relation variant improves over the no-prior baseline on macro-average, but ADD_T_NT still attains the highest mean.Layering DEPREL priors onto terminal and non-terminal priors does not improve results, whether added directly, concatenated, or interpolated.
  • Summary and scope: The conclusion is limited to absolute positional embeddings with the RoBERTa encoder, while optimal strategies for other positional schemes and larger budgets remain open.Per-dataset rankings vary, so no single variant is optimal on every task; ADD_T_NT is adopted as a simple, robust default.
Loading 2608.06111v1…