Syntax vs. Semantics: How Transformers Learn Deep Dependencies

Jiangrui Zhao; Xiaoting Du

Source-linked AI summary

Syntax vs. Semantics: How Transformers Learn Deep Dependencies

Jiangrui Zhao, Xiaoting Du

arXiv:2608.26139v1Jun 26, 2026cs.CLcs.AI

TL;DR

The paper asks why Transformers learn local syntax before deep semantic dependencies and why semantic reasoning can emerge abruptly. It proposes a mechanistic framework based on gradient competition, validates it across model scales, and introduces topology-aligned interventions. The framework accounts for CoT’s benefit and reports that the proposed objective outperforms standard fine-tuning by more than 2x on variable binding tasks.

Problem
The optimization dynamics governing Transformers’ acquisition of deep semantic dependencies remain poorly understood, despite their importance for systematic generalization and evidence of delayed, abrupt emergence.
Method
The paper models learning as competition between Surface Statistics and Deep Semantics, analyzes gradient starvation and alignment dynamics, and evaluates CoT and topology-aligned interventions across scales.
Results
The experiments support Gradient Starvation, Softmax Saturation, and phase-transition emergence governed by the semantic alignment ratio ρ(t).
Takeaways & Limitations
CoT supervision and the topology-aligned contrastive objective alleviate the syntactic barrier by rectifying gradient geometry.
Takeaways & Limitations
LLM experiments are concentrated in the code domain because code provides explicit AST-based dependency ground truth, and the objective increases training memory consumption.

Abstract

from arXiv · show

Large Language Models demonstrate remarkable syntactic fluency, yet the optimization dynamics governing their acquisition of deep semantic dependencies remain poorly understood. We propose a mechanistic framework that models this learning process as a competition between Surface Statistics and Deep Semantics. Our theoretical analysis identifies a ``Gradient Starvation" phenomenon where the error signals for sparse semantic dependencies are actively suppressed during early optimization. This suppression impedes the learning of structural reasoning and causes its emergence to manifest as a sudden phase transition. Furthermore, this framework offers a mechanistic basis for the effectiveness of Chain-of-Thought (CoT) strategies. By externalizing intermediate reasoning steps into concrete tokens, CoT effectively bypasses the suppression regime inherent to implicit reasoning. We validate these findings across scales ranging from toy transformers to production models (Llama-3.1-8B, Qwen2.5-Coder-7B). Finally, guided by this theory, we propose a topology-aligned contrastive objective that explicitly rectifies the gradient geometry. Experiments on variable binding tasks demonstrate that our method achieves an improvement that is over 2x larger than that obtained via standard cross-entropy fine-tuning.

1. Introduction

The paper frames deep-dependency learning as a dynamical competition in which frequent surface patterns initially suppress semantic signals, producing delayed and abrupt reasoning emergence. It develops predictions about staged dependency learning and explains how CoT supplies additional gradient pathways, validated across controlled and production-scale models.

Deep dependencies require recovering latent relational structure, but their optimization dynamics remain poorly understood despite evidence that models learn them only after mastering local structure.
Gradient Starvation describes how high-curvature gradients from frequent local patterns suppress weaker long-range semantic signals until a critical phase transition.
The framework models syntax and semantics as competitors in the gradient landscape rather than as static representational or architectural properties.
CoT externalizes intermediate states, injecting additional gradient pathways that partially bypass starvation constraining implicit reasoning.
The experiments trace dependency circuits from toy models through Pythia checkpoints to Qwen2.5-Coder-7B and Llama-3.1-8B using code and ASTs as controlled semantic-topology proxies.

2. Mechanistic Characterization: The Dynamics of Semantic Emergence

The mechanistic account attributes semantic emergence to curvature-driven gradient competition, softmax-mediated suppression, and alignment with a low-rank dependency operator. This alignment develops through unstable crossover into systematic reasoning, while CoT adds a parallel learning pathway.

2. Mechanistic Characterization: Surface Statistics and Deep Semantics compete through an alignment-ratio dynamic, with high-curvature syntax initially suppressing updates along semantic directions.
2.2. The Geometry of Competition: Gradient Starvation: Assumption 2.1 posits λsyn ≫ λsem, so syntactic curvature dominates gradient magnitude when residual projections are comparable.
2.3. Suppression Mechanism: Softmax Saturation: Softmax saturation creates an exponential gradient barrier: when Asyn approaches 1, Asem approaches 0 and semantic learning stalls until syntactic confidence decreases.
2.4. Emergence via Subspace Alignment: Expected attention updates align WQK with the rank-1 outer product µuseµTsrc, representing a topological edge in the computation graph.
2.5. Phase Transitions and the Critical Threshold: The transition proceeds from syntactic dominance through a volatile crossover to systematic emergence once ρ(t) exceeds τstable and WQK locks onto the topological direction.
2.6. Accelerating Semantic Emergence via CoT: CoT injects a local gradient independent of downstream correctness, short-circuiting long credit assignment and producing large relative gains when implicit signals are starved.

3. Controlled Experiments: Verifying the Signal-to-Noise Dynamics

Controlled synthetic-code experiments test whether syntax–semantics competition produces gradient starvation, staged learning, and alignment-ratio-driven transitions. The results support suppressed semantic updates early in training, rapid improvement after attention shifts toward operands, and acceleration under syntax suppression.

Experimental Setup: Synthetic code separates syntax tokens from variable dependencies, enabling controlled tests of gradient starvation, Softmax Saturation, and alignment-ratio effects.The training data covers variable assignment and arithmetic operations.
3.1. Visualizing Gradient Starvation: Early training suppresses variable-token gradients despite high semantic error, while later syntactic-gradient decay triggers a sharp surge in variable gradients.This two-stage pattern is consistent with active semantic inhibition rather than an absent semantic circuit.
3.2. Escaping Softmax Saturation: During curriculum training, attention initially favors syntactic anchors, then operand attention crosses syntactic attention as accuracy rises rapidly.The curriculum switches from variable assignment to addition at step 1500.
3.2. Escaping Softmax Saturation: The transition appears in both variable assignment and addition when the alignment ratio ρ(t) crosses the critical threshold τstable ∈(0, 1).This links semantic alignment and task improvement to a thresholded attention dynamic.
3.3. Causal Validation of alignment ratio Thresholds: Suppressing syntactic gradients accelerates semantic alignment, while variable-gradient noise disrupts alignment; removing syntactic markers also shortens the early plateau.These interventions support a competitive relationship between syntactic and semantic update directions.
3.4. Summary of Findings: The same transition occurs in a natural-language entity-binding task, and the experiments identify Gradient Starvation, Softmax Saturation, and alignment ratio ρ(t) as framework pillars.The authors further connect syntax suppression with accelerating reasoning emergence and recommend prioritizing logic density over syntactic perfection.

4. Mechanistic Experiments: The Dynamics of Emergence and Composition

Experiments on Pythia models trace semantic dependency learning from gradient starvation through unstable competition to systematic emergence. They also show that pointer mechanisms form earlier in natural language than code, while CoT restores gradient selectivity across long dependency chains.

Alignment-ratio dynamics: Pythia-160M experiments reconstruct the effective attention-gradient geometry using the product rule to measure alignment ratio ρ(t).The metric respects the model’s factored query-key parameterization while remaining mathematically equivalent to the theoretical definition.
Alignment-ratio dynamics: Initially ρ(t) ≈0, a transient spike appears at 5k steps, and a monotonic rise after 20k marks systematic emergence.The transient spike reflects unstable heuristic alignment before consolidation onto the semantic operator.
Pointer formation: Pythia-1.4B attention heads show stable diagonal predecessor pointing in natural-language identity chains, with amplified first-column attention at Step 143k.The late first-column concentration indicates multi-hop aggregation through iterative propagation.
Pointer formation: Code exhibits delayed and shallower reference-mechanism formation than natural language.The authors attribute this gap to stronger syntactic curvature and reduced code exposure during pre-training.
CoT signal restoration: Implicit reasoning loses gradient signal rapidly with chain depth, whereas CoT maintains high gradient selectivity regardless of total depth.Intermediate supervision restores local learning signals and alleviates the long-range bottleneck.

5. Generalization to LLMs

The paper evaluates topology-aligned semantic binding interventions in code and natural language using production-scale models and structurally targeted datasets. Binding is concentrated in specialized heads, whose targeted intervention improves reference resolution and transfers across programming languages.

Experimental setup: Variable Cloze Completion masks AST-targeted variable usages in Python150k and JavaScript150k to test semantic dependency resolution beyond local n-grams.The evaluation uses native ASTs to surgically target usage nodes rather than randomly dropping tokens.
Topology-aligned objective: The proposed objective adds a topological auxiliary loss that explicitly optimizes alignment ratio ρ(t) against syntactic noise.Its contrastive formulation separates the true source score from average distractor scores.
Topology-aligned objective: The steering update aligns with the semantic edge and orthogonalizes against distractor directions, accelerating movement through the instability region.The active margin-violation term adds a rank-1 component to the interaction-matrix update.
Universality and structural sparsity: Binding capabilities concentrate in sparse specialized heads and remain invariant across Llama-3.1-8B, Qwen2.5-Coder-7B, JavaScript, and Python.Figure 8 visualizes the corresponding attention heatmaps for JavaScript and Python.
Causal verification: Ablating the Top-8 binding heads causes a substantially larger performance drop than random-8 masking, supporting their functional role in reference resolution.The comparison is reported for Variable Cloze Completion.
Generalization: Topology-aligned tuning outperforms standard supervised fine-tuning on WinoGrande across Qwen2.5-7B and Llama3-8B, and transfers across Python and JavaScript without additional fine-tuning.The cross-language result supports language-agnostic binding circuits rather than syntax-specific behavior.

6. Related Works

Prior work has studied syntax–semantics tension through architectural separation, structural priors, and probing, while attention research spans interpretability and training-dynamics analyses.

Syntax and semantics: Existing approaches address syntax and semantics through architectural separation, dependency-tree priors, or probes of whether syntactic competence entails semantic generalization.These lines of work largely treat syntax and semantics as static representations or architectural choices.

7. Conclusion

The paper frames semantic emergence as spectral competition in which syntax initially suppresses deep dependency learning, then reports evidence for a sharp phase transition and targeted interventions that alleviate the barrier.

Conclusion: The proposed framework characterizes deep dependency learning as spectral competition, with high-curvature syntax masking semantic directions through gradient starvation.The experiments span toy models, intermediate Pythia checkpoints, and production-scale language models.
Conclusion: CoT supervision and topology-aligned contrastive training rectify gradient geometry and help alleviate the syntactic barrier.The conclusion motivates objectives and data strategies that encourage relational structure beyond surface statistics.

Impact Statement

The paper aims to connect gradient dynamics with reasoning-circuit emergence, supporting more transparent and controllable pre-training while identifying practical trade-offs.

The framework links gradient dynamics to reasoning-circuit emergence to support more transparent and controllable pre-training.
Circuit polysemanticity makes aggressive fine-tuning of binding heads risk interference with other latent capabilities.
The topology-aligned objective increases training memory consumption compared with standard objectives.
LLM experiments are concentrated in the code domain because code supplies explicit AST-based ground truth for semantic dependencies.

A.1. Experimental Verification of Spectral Disparity

The experiments test whether syntactic directions have substantially higher curvature than semantic directions, using normalized perturbations and loss-landscape measurements.

The hypothesis assigns large Hessian eigenvalues to syntactic directions and small eigenvalues to semantic directions.The analysis uses iso-energy perturbations to probe anisotropic local geometry.
Normalized perturbations convert loss changes into direct proxies for directional curvature through the Rayleigh quotient.Steeper parabolic growth indicates larger Hessian eigenvalues.
At |α| = 0.2, syntactic perturbations increase loss by ∆L ≈1.4, indicating high curvature.
Within the same perturbation range, semantic loss increases remain ∆L < 0.2, indicating near-zero semantic curvature.
The resulting curvature ratio λsyn/λsem ≫1 supports the spectral-disparity postulate.Figure 9 depicts the syntactic direction as a steeper valley than the semantic direction.

A.2. Validation of Low-Rank Topological Operators

The analyses support a sparse, low-rank semantic binding mechanism alongside high-rank syntactic processing, with gradients and representations becoming organized around semantic directions.

The semantic head has effective rank reff ≈1.19, consistent with a rank-1 topological operator.
The syntactic head has effective rank reff ≈62.18, indicating high-dimensional, diffuse information processing.
Low-rank binding operators are rare and localized to a specialized subset of attention heads.
Across depth, gradients transition from noisy and context-dependent at L < 15 to a single dominant direction at L > 20.The concentrated update accumulates constructively along the topological edge.
At Layer 20, variable references form distinct clusters for each of 26 variables despite randomized surrounding contexts.The visualization uses hidden-state projections after filtering and manifold reduction.
The semantic subspace provides a context-independent variable code that supports pointer-passing mechanisms.
Attention heatmaps evaluate semantic binding across JavaScript and Python for Qwen2.5-Coder-7B.

B.1. Experiments in Sec. 3

The experiments combine controlled Transformer studies, checkpoint analyses, gradient probes, and topology-aligned objectives to investigate semantic learning dynamics and cross-language generalization.

The controlled experiments use a causal NanoTransformer with fixed vocabulary partitioning, AdamW optimization, and seed 42.
Cross-language evaluation reports that the proposed method consistently outperforms Base and SFT across models and directions.
Experiment 1 monitors embedding-layer gradient norms separately for syntax tokens and variable identifiers during pointer-binding learning.
Experiment 2 switches from variable copying to arithmetic addition at step t = 1500 to visualize release from syntactic attention concentration.
Experiment 3 masks syntactic-token gradients or applies competing interventions to test the causal role of the alignment ratio ρ(t).
Pythia analyses use dense checkpoint sampling for alignment dynamics and checkpoints t ∈{1k, 20k, 143k} for structural and CoT analyses.
The alignment ratio ρ(t) measures projection of query-key interactions onto a low-rank semantic operator and tracks transition from starvation to semantic locking.
Gradient saliency compares causal-source and distractor gradients across checkpoints and random graph instances.

C.6. Accelerating Semantic Emergence via CoT

The framework explains CoT as making intermediate reasoning states explicit, improving semantic retrieval at inference and adding a parallel gradient pathway during training. These mechanisms are predicted to benefit deep compositional tasks more than shallow pattern matching.

Inference-time CoT: CoT prompting externalizes the intermediate state z, replacing P(y | x) with P(z | x)P(y | x, z).The explicit state stabilizes semantic retrieval under softmax saturation.
Training-time CoT: Training-time CoT supervision adds a local loss term Lz for predicting z alongside Ly|z.This directly supervises the intermediate state and creates a more local signal for binding-related parameters.
Gradient bottlenecks: Implicit reasoning can suffer downstream insensitivity and upstream gating, causing semantic gradients to vanish or be suppressed by softmax saturation.These failure modes weaken credit assignment from the final output to binding-circuit parameters.
Gradient bottlenecks: CoT supervision adds an injected gradient that bypasses downstream sensitivity and remains active when the implicit pathway is weak or gated.The total update therefore contains a parallel route for learning semantic dependencies.
Semantic emergence: CoT increases the semantic projection of updates, tends to raise the alignment ratio ρ(t), and accelerates crossing the stability threshold τstable.This connects the injected gradient directly to earlier semantic emergence.
Task dependence: Deep compositional tasks can achieve Gain ≫1 because injected gradients dominate weak implicit semantic signals, whereas shallow pattern matching yields Gain ≈1.The predicted contrast reflects each task’s reliance on the semantic subspace.