PaperlayerView exact revision on arXiv

Source-linked AI summary

Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails

Sai Krishna Arthanari, JaeHyeong Chang, Chengzhe Sun, Siwei Lyu

arXiv:2608.20647v1cs.CL

TL;DR

The paper asks whether directional structure in contextual representations helps dependency relation classification, especially when comparing opposite-direction states. Using a frozen-trunk diagnostic methodology alongside controlled representation comparisons, it finds that unfused directional splitting helps while cross-direction pairing fails increasingly with distance, with the mechanism only partially explained. Its claims replicate across genre, but the core ablation remains limited to English and a single-layer BiLSTM substrate.

  • Problem

    The paper investigates whether directional structure carries recoverable relational information and whether cross-direction comparisons are informative in dependency relation classification.

  • Method

    The study uses a single-layer, architecturally unmixed BiLSTM, controlled pairwise classifier heads, frozen-trunk probes, backbone comparison, and replication on a second English treebank.

  • Results

    Unfused F+B beats either direction alone and fused self-attention, while Fi-vs-Bj is weakest and its distance penalty grows; 93% of the gap survives trunk freezing.

  • Takeaways & Limitations

    Directional splitting is useful and generalizes across genre, whereas cross-direction pairing is robustly harmful with distance and remains only partially mechanistically understood.

  • Takeaways & Limitations

    The core splitting ablation uses one single-layer BiLSTM architecture and two English treebanks; directional splitting on Transformer-derived representations or another language remains untested.

Abstract

from arXiv · show

Splitting a bidirectional LSTM's contextual representation into a forward-only $F_i$ (strictly a function of tokens $1..i$) and a backward-only $B_i$ (strictly a function of tokens $i..n$) beats either alone and beats a fused self-attention representation for dependency relation-type classification. But a specific, natural extension of this idea -- pairing a token's forward state against a \emph{candidate}'s backward state (``cross-direction'' pairing, $F_i$ vs.\ $B_j$) -- consistently \emph{underperforms} same-direction pairing, and the penalty \emph{grows}, not shrinks, with token distance, both paired-bootstrap significant. We diagnose why using a frozen-trunk methodology: architectural information leakage between directions is impossible by construction (a single-layer BiLSTM, verified by code inspection); 93\% of the same-vs-cross gap survives freezing the trunk and training only fresh heads, ruling out training-co-adaptation as the primary cause; linear regression shows partial representational redundancy between $F_i$ and $B_i$ ($R^2{=}0.324$ vs.\ $0.028$ for a shuffled control) and a linear probe shows partial anticipatory encoding of upcoming tokens in $F_i$ (36.5\% vs.\ 17.2\% majority baseline) -- real effects, but neither alone, nor combined, cleanly explains the full gap. Extended frozen-trunk diagnostics (a positional probe and a distance-decay probe) show directional information is genuinely stored but not exactly positioned, and propagates only a few tokens before decaying to baseline -- consistent with, and mechanistically underneath, the distance-growth finding.

I. INTRODUCTION

The paper isolates directional structure in contextual representations and tests whether keeping directions separate improves dependency relation classification. Unfused forward-plus-backward representations help, but cross-direction pairing is consistently weaker and increasingly penalized by distance, with only a partial mechanistic diagnosis.

  • Directional structure is rarely isolated before relational comparison because most contextual architectures fuse directional or all-to-all information into one representation.
  • Unfused F+B beats either direction alone and a fused self-attention representation for relation-type classification, despite a small standardized effect.
  • Cross-direction Fi-vs-Bj pairing is the weakest construction, and its penalty relative to same-direction pairing grows monotonically with token distance.
  • 93% of the same-vs-cross gap survives freezing the trunk, while architectural leakage is ruled out by the single-layer, unmixed BiLSTM design.
  • Representational redundancy and anticipatory encoding are real partial effects, but positional and distance probes indicate that stored directional information is imprecisely positioned and short-range.
  • The findings replicate on GUM, and a parameter-matched Transformer narrows but does not close the fresh BiLSTM’s advantage.

II. RELATED WORK

The paper situates its directional-comparison ablation against self-attention, bidirectional representations, and biaffine parsing, while using a single-layer BiLSTM because it makes directional separation exact and inspectable.

  • Self-attention provides fused all-to-all contextualization, whereas this paper keeps forward and backward states separate through pairwise comparison.
  • Prior bidirectional representations combine forward and backward directions for transfer, but the paper specifically tests cross-direction pairwise comparison.
  • The backbone-comparison arc scorer follows biaffine parsing’s structured-prediction recipe while using an explicit hand-specified pairwise feature.
  • The literature search found no directly relevant prior work on comparing one token’s forward state with another token’s backward state, though the search was not exhaustive.
  • A single-layer BiLSTM is chosen because Fi and Bi are architecturally disjoint, unlike Transformer states mixed by self-attention from layer one.
  • Fi is computed from tokens 1..i and Bi from tokens i..n, with both states sliced directly from the unmixed BiLSTM output.

B. Pairwise representations tested

The study compares pairwise feature constructions using identically shaped classifier heads so that performance differences primarily reflect the representations being compared.

  • Each candidate pair uses an identically shaped single-hidden-layer classifier head, isolating representation differences from classifier capacity.
  • The strongest f+b+p variant is reused unchanged for full-pairwise arc scoring in the backbone comparison.
  • The fused self-attention baseline forms pairwise features from each token’s fused Hi representation, elementwise product, and absolute difference.

C. Per-token diagnostic representation

The per-token diagnostic representation Hi is used to probe what directional states encode about absolute position and neighboring tokens independently of a specific comparison pair.

  • Hi supports a positional probe asking whether directional subparts encode absolute position and how precisely they do so.
  • Hi also supports a distance-decay probe measuring how far directional information about specific neighboring tokens propagates.

D. Backbone comparison: BiLSTM vs. Transformer, parameter-matched

The backbone comparison trains full-pairwise dependency arc scorers under an identical structured-prediction recipe, contrasting BiLSTM and Transformer representations while controlling a parameter-count confound.

  • Method: Full-pairwise arc scorers predict unlabeled dependency heads with an MLP over pair representations and sentence-level softmax cross-entropy.The trunk and scorer are optimized jointly.
  • Backbones: Three backbones are compared: the BiLSTM, the standard fused-attention Transformer, and a parameter-matched Transformer.The matched Transformer is reduced until its total parameter count is within 3% of the BiLSTM’s.

IV. EXPERIMENTAL SETUP

The study uses controlled dependency experiments on English EWT, with GUM for genre-diverse validation, fixed optimization settings, and targeted replication of its main claims.

  • Datasets: Experiments use Universal Dependencies English EWT as the primary testbed and English GUM for cross-dataset validation across substantially different genres.EWT contains informal web, blog, email, and review text, while GUM includes academic, fiction, how-to, news, interview, and travel-guide text.
  • Protocol: Training is held constant with AdamW, learning rate 10^-3, weight decay 10^-5, batches of 32 sentences, and 8 epochs.The default uses seeds 42–44, with seeds 42–46 for the relation-classification ablation and backbone comparison.
  • Research questions: The experiments trace whether F/B splitting beats fused representations, whether cross-direction pairing adds signal, why it fails, and whether the findings withstand scrutiny.The questions are organized as A through E across the paper.
  • Evaluations: Relation-type classification evaluates 16-way gold-arc predictions using test accuracy and macro-F1, with single-seed primary runs and five-seed confirmation.The classification table reports a five-seed mean±standard deviation for confirmation.
  • Headline result: Unfused F+B produces the largest jump over either direction alone and beats fused self-attention by ∼0.7 points with ∼35% fewer parameters.Paired bootstrap comparisons against f_only, b_only, and standard_attention are all significant but have small standardized effects.
  • Evaluations: Binary edge-existence detection uses balanced gold-arc and random-pair examples, while distance analysis reports cross_fb − same_f accuracy/F1 gaps by token-distance bucket.The binary results average over three seeds, as does the distance analysis.

B. Does cross-direction pairing add signal?

Cross-direction pairing does not add signal: it underperforms same-direction pairing, and its penalty increases with token distance, a trend supported by paired bootstrap testing.

  • Overall comparison: Cross_fb trails same_f by −0.0308 accuracy points, with a 95% CI of (−0.0336, −0.0279) and Cohen’s d=−0.100.The result is statistically significant and has a small standardized effect.
  • Overall comparison: Cross_fb trails same_f and same_b by ∼3 points, while cross_bf trails them by ∼7–8 points.These gaps are 30–80× larger than seed-to-seed noise, whose standard deviation is ≤0.0025 throughout.
  • Distance analysis: The cross-direction penalty grows monotonically with distance and is nearly 7× larger in F1 at 21+ than at 1–2 tokens.This contradicts the account that short-range dominance masks a long-range cross-direction advantage.
  • Diagnostic framing: The frozen-trunk comparison evaluates whether the same-direction advantage persists without further trunk training, alongside architectural and representational diagnostics.The supplied table passage identifies the comparison as frozen-trunk versus end-to-end same_f−cross_fb accuracy gap.
  • Distance analysis: The distance-growth trend is statistically robust: the 21+ minus 1–2 gap difference is −0.0371, with a 95% CI of (−0.0516, −0.0227).The test uses stratified within-bucket sentence-independent resampling over 5000 bootstrap iterations.

C. Why does cross-direction pairing fail?

Frozen-trunk probes rule out architectural leakage and largely rule out training co-adaptation, while identifying real but incomplete redundancy, anticipatory, positional, and distance-decay effects.

  • Candidate explanations: R^2=0.324 versus R^2=0.028 for a shuffled-partner control shows partial representational redundancy, but not the full collapse required to explain the gap.The regression is 11.5× higher than the shuffled control, yet far below R^2→1.
  • Candidate explanations: 93% of the end-to-end same-versus-cross gap survives trunk freezing, largely ruling out training co-adaptation as the primary cause.Fresh heads reproduce 2.96 of the original 3.18-point gap.
  • Candidate explanations: 36.5% next-token UPOS accuracy versus a 17.2% majority baseline confirms partial anticipatory encoding in F_i, but not why cross-direction pairing is actively worse.The same-token ceiling is 98.1%, so the anticipatory signal is real but incomplete.
  • Extended diagnostics: Position is encoded without explicit P_i, yet adding P_i improves F+B by approximately 19 points, showing stored position is useful but not redundant.The MLP probe reaches approximately three times the majority baseline without P_i; the linear head trails the MLP, especially for H_i.
  • Extended diagnostics: Directional information reaches the majority baseline by Δ=5 forward and Δ=3 backward, indicating propagation over only a few tokens before decaying.This provides a concrete mechanism underneath the task-level distance-growth finding: cross-direction pairing has progressively less directional signal at larger distances.
  • Backbone validation: Parameter matching narrows but does not close the BiLSTM advantage, while paired bootstrap confidence intervals exclude zero and the standardized effect becomes small.The supplied diagnostics also include full-pairwise arc-scoring and comparison tables, but their captions alone do not provide additional quantitative results.

D. Does this hold under a fresh, parameter-matched backbone?

A parameter-matched Transformer still underperforms the BiLSTM on real dependency head-finding, extending the directional-representation result beyond classification.

  • Backbone comparison: +3.04 points unmatched versus +2.60 points matched shows parameter matching narrows the test UAS gap without eliminating it.The paired bootstrap uses sentence-level resampling with 5000 iterations.
  • Backbone comparison: A parameter-matched Transformer remains significantly below the BiLSTM on real dependency head-finding, despite nearly identical parameter counts.This extends the classification-only comparison to a structured-prediction task under a fair parameter budget.
  • Backbone comparison: The standardized effect shrinks from d=0.216 to d=0.085 after matching, changing from moderate to small while remaining statistically significant.Both confidence intervals exclude zero.

E. Does this hold on a different genre?

The two core directional findings replicate on GUM, a substantially different-genre English treebank: unfused splitting remains strongest and cross-direction pairing remains weakest.

  • Replication: On GUM, the full five-variant ordering matches EWT: f_plus_b_plus_p>f_plus_b>f_only>standard_attention>b_only.GUM includes academic, fiction, how-to, news, interview, and travel-guide text, unlike EWT’s informal web, blog, email, and review text.
  • Replication: Cross-direction pairing again loses by a comparable, slightly larger margin on GUM.The replication uses the identical protocol and seeds.

VI. DISCUSSION

The discussion treats directional splitting and cross-direction failure as robust empirical findings, while keeping the mechanism and generality claims explicitly limited.

  • What replicated: Directional splitting generalizes across genre, and cross-direction pairing remains the weakest construction with a distance-growing penalty.Both findings are bootstrap-confirmed and replicate on GUM.
  • What replicated: The BiLSTM still outperforms a fresh, parameter-matched Transformer on structured prediction, so the result is not tied entirely to one backbone.The comparison is an external validation point rather than a Transformer-based directional-splitting ablation.
  • What remains only partially explained: Architectural leakage is ruled out and 93% of the gap survives trunk freezing, but redundancy and anticipatory encoding do not fully explain the failure.Positional and distance-decay probes sharpen the mechanism without closing the explanatory gap.
  • Novelty and framing: The authors report no prior work on this specific cross-direction ablation, while qualifying the novelty claim because the search was targeted rather than exhaustive.This is presented as a focused empirical result, not a general theory of directional representations.
  • Scope and limitations: The study covers two English datasets and tests directional splitting primarily with a single-layer BiLSTM, leaving other languages and Transformer-derived splits open.The targeted literature search for this ablation was also not exhaustive.
Loading 2608.20647v1…