PaperlayerView exact revision on arXiv

Source-linked AI summary

Three-Phase Transformer

Mohammad R. Abu Ayyash

arXiv:2604.14430v1cs.CLcs.AIcs.LG

TL;DR

Three-Phase Transformer asks whether a structurally partitioned residual stream can improve decoder-only Transformers while composing with RoPE. It introduces phase-aware operations and a DC-position side-channel, achieving lower perplexity than a matched RoPE-Only baseline while showing that N is not a unique optimum.

  • Problem

    Three-Phase Transformer addresses whether a three-phase structural prior can organize the residual stream and compose with RoPE rather than compete with it.

  • Method

    The method partitions the hidden vector into cyclic channels with phase-aware normalization, inter-layer rotations, phase-aligned GQA, and a Gabriel's horn profile injected into an orthogonal DC subspace.

  • Results

    -7.20% perplexity over a matched RoPE-Only baseline at 123M parameters, with 1.93x step-count convergence speedup and 1.64x wall-clock speedup.

  • Takeaways & Limitations

    The load-bearing design is the partitioned residual stream with per-block rotation, per-phase normalization, and horn DC injection, while N functions as a parameter-sharing knob rather than a unique optimum.

  • Takeaways & Limitations

    Short-horizon ablations can reverse the long-horizon assessment of PhaseAwareRMSNorm, and the 17% wall-clock overhead makes step-matched and compute-matched baselines distinct.

Abstract

from arXiv · show

We present Three-Phase Transformer (3PT), a residual-stream structural prior for decoder-only Transformers on a standard SwiGLU + RMSNorm + RoPE + GQA backbone. The hidden vector is partitioned into N equally-sized cyclic channels, each maintained by phase-respecting ops: a per-channel RMSNorm, a 2D Givens rotation between attention and FFN that rotates each channel by theta + i*(2*pi/N), and a head-count constraint aligning GQA heads with the partition. The architecture is a self-stabilizing equilibrium between scrambling and re-imposition, not a bolted-on module. The partition carves out a one-dimensional DC subspace orthogonal to the channels, into which we inject a fixed Gabriel's horn profile r(p) = 1/(p+1) as an absolute-position side-channel composing orthogonally with RoPE's relative-position rotation. The canonical N=3 borrows its metaphor from balanced three-phase AC, where three sinusoids 120 degrees apart sum to zero with no anti-correlated pair. At 123M parameters on WikiText-103, 3PT achieves -7.20% perplexity (-2.62% bits-per-byte) over a matched RoPE-Only baseline at +1,536 parameters (0.00124% of total), with 1.93x step-count convergence speedup (1.64x wall-clock). N behaves as a parameter-sharing knob rather than a unique optimum: at 5.5M an N-sweep over {1,2,3,4,6,8,12} is near-monotone with N=1 winning; at 123M a three-seed sweep finds N=3 and N=1 statistically indistinguishable. The load-bearing mechanism is the channel-partitioned residual stream, per-block rotation, per-phase normalization, and horn DC injection. We characterize (a) self-stabilization of the geometry without explicit enforcement, a novel instance of the conservation-law framework for neural networks; (b) a U-shaped depth profile of rotation-angle drift at 12 layers; (c) orthogonal composition with RoPE, attention, and FFN.

1 Introduction

3PT imposes a three-phase structural geometry on the residual stream while preserving vanilla Transformer components, using orthogonal rotation, phase-aware normalization, aligned GQA, and a fixed absolute-position side-channel. The design composes these operations as an equilibrium between residual-stream scrambling and repeated phase re-imposition.

  • 1 Introduction: 3PT partitions the hidden residual stream into three equal-width phases inspired by balanced 120° three-phase signals.The partition is structural rather than learned or sinusoidal.
  • 1 Introduction: The four phase-respecting operations are DC-replacing phase splitting, PhaseAwareRMSNorm, per-block PhaseRotationLayer, and phase-aligned GQA.Embedding, SwiGLU, attention, RoPE, residual connections, and cross-entropy otherwise remain vanilla.
  • 1 Introduction: Gabriel's horn injects r(p) = 1/(p+1) into the partition's one-dimensional DC subspace as an absolute-position side-channel.The profile is fixed, registered as a buffer, and adds zero trainable weights.
  • 1 Introduction: The PhaseRotationLayer applies independent 2D orthogonal Givens rotations between attention and FFN, using learnable theta plus fixed 120° phase offsets.Its orthogonal Jacobian preserves gradient singular values despite non-residual placement.
  • 1 Introduction: The complete combination composes a structural channel partition, phase-aligned GQA, per-phase normalization, per-layer rotation, and horn injection on top of RoPE.The authors report that this exact combination had not been published previously.

2 Related Work

Related work situates 3PT among structural channel decompositions, orthogonal rotations, partitioned normalization, positional side-channels, conservation laws, depth profiles, and lightweight priors. The cited precedents overlap with individual components or broader principles, but the supplied passages distinguish 3PT's combined geometry and mechanisms from them.

  • 2.1 Three-phase 120° channel partition: The Dual-Stream Transformer is the closest architectural analog, but uses two functional streams without 120° offsets or rotation layers.Neural collapse and ETF-Transformer support 120° simplex geometry in different settings rather than decoder-only residual partitioning.
  • 2.2 PhaseRotationLayer between attention and FFN: SpinQuant and QuaRot use rotations between Transformer components, while 3PT uses per-channel rotations to structure representations rather than optimize quantization or suppress outliers.LieRE applies related SO(n) machinery to Q/K positional encoding inside attention.
  • 2.3 Phase-aligned grouped-query attention: Differential Transformer provides a per-head normalization precedent, but 3PT applies independent normalization to fixed residual-stream phases and aligns GQA boundaries with them.The alignment is described as a novel configuration constraint.
  • 2.5 Gabriel's horn DC injection: The fixed analytic horn occupies a rank-one residual subspace as an absolute-position side-channel, unlike conventional additive encodings, emergent position recovery, learned registers, or content-dependent position gates.The supplied related work identifies this component as lacking a clear prior-art precedent.
  • 2.7 Self-stabilization and conservation laws: Prior work establishes general symmetry-based conservation laws and other depth-dependent or lightweight structural priors, while the paper positions 3PT's self-stabilization and U-shaped rotation drift as distinct findings.The cited literature includes uniformizing, prescribed, bell-shaped, and cross-scale architectural or geometric effects.

3 Method

3PT partitions the residual stream into cyclic phase channels and maintains them with phase-aware normalization, rotations, and GQA alignment. A fixed DC-subspace horn signal supplies absolute position while RoPE retains its standard relative-position role.

  • Cyclic phase geometry: The hidden dimension is partitioned into N equal phase blocks with offsets φ_i = 2πi/N; the canonical configuration uses N = 3 and 120° spacing.The shared common-mode subspace is reserved for Gabriel's horn injection.
  • Phase rotation: PhaseRotationLayer applies shared per-pair learnable Givens angles plus fixed 2πi/N offsets independently within every phase.The rotations are inserted between attention and FFN and are norm-preserving and invertible.
  • DC side-channel: The horn replaces the cross-phase DC component with the fixed profile r(t) = 1/(t+1), shifting only the common-mode subspace.The final configuration uses horn injection without auxiliary zero-sum loss.
  • Phase normalization: PhaseAwareRMSNorm normalizes each phase block with its own mean-square statistic while retaining the same total parameter count as standard RMSNorm.Independent scale vectors are concatenated after per-phase normalization.
  • Attention alignment: Phase-aligned GQA requires N to divide both query and key-value head counts, so heads partition evenly into phase groups while attention still mixes across heads.The 123M configuration uses N = 3, n_q = 12, and n_kv = 3.
  • Backbone composition: RoPE remains unchanged in attention and supplies relative-position encoding, complementing the horn's absolute-position signal in a disjoint subspace.SwiGLU, residual connections, and cross-entropy remain standard components of the backbone.

4 Experiments

Experiments locate the main gain in the embedding-side three-phase structure and show that it becomes competitive only when stacked with RoPE. The strongest reported combination improves substantially over RoPE alone and accelerates convergence.

  • Experiment 1: 11.23% lower PPL: 3Phase-Embed beats the vanilla baseline at step 2,000 and stays ahead from step 200 onward.It also reaches PPL 72.89 at step 1,000 versus the baseline's step-2,000 final PPL of 73.17.
  • Experiment 1: The embedding-side machinery carries almost all the gain; attention-only three-phase is essentially noise, and adding it to the embedding does not improve the result.The experiment therefore retires the attention-side mechanism.
  • Stage 1: 15.7% worse PPL: three-phase without RoPE loses to RoPE alone, showing that the embedding structure is not a substitute for RoPE.The three-phase variant nevertheless beats plain additive sinusoidal encoding.
  • Stage 2: 15.36% lower PPL: G, combining learnable-frequency three-phase embedding with RoPE, improves from 53.85 to 45.58 versus RoPE alone.Fixed-frequency stacking also works, with F reaching PPL 46.20.
  • Stage 2: Roughly 2× convergence: G reaches PPL 53.44 at step 1,000, already surpassing RoPE-alone's step-2,000 final PPL of 53.85.Learnable frequencies improve over fixed frequencies by only 0.62 PPL in the reported comparison.
  • Interpretation: The learnable-frequency advantage was small at every checkpoint, foreshadowing the later finding that the embedding's sinusoidal-PE apparatus is vestigial.This conclusion concerns the subsequent experiments rather than the initial Stage 2 comparison.

4.3 Experiment 3 - Cumulative stress test

The cumulative stress test finds that only phase-aligned heads consistently help, while several added constraints hurt. Because cumulative variants confound effects, an orthogonal grid supplies cleaner marginal-effect estimates.

  • Cumulative stress test: Only Align is a clean winner in the six-refinement comparison; PhRMS gives small context-dependent gains, while HardZS, PhDrop, Shrink, and LrnOff hurt or add noise.The six flags are Align, PhRMS, HardZS, PhDrop, Shrink, and LrnOff.
  • Cumulative stress test: HardZS cuts effective embedding capacity by one third because phase C becomes −(A+B), leaving the embedding to learn only 128 dimensions.Later cumulative variants inherit this reduced-capacity representation.
  • Cumulative stress test: PhDrop, Shrink, and LrnOff were evaluated on top of the already capacity-reduced model, so their isolated effects were not fairly identified by the cumulative chain.This motivates the orthogonal 2^6 grid.
  • Orthogonal grid: −0.74 PPL: Align has the only clean marginal win, whereas Shrink (+9.06), PhDrop (+4.12), PhRMS (+1.91), and HardZS (+1.22) hurt at 2,000 steps.LrnOff changes PPL by only −0.005 and is treated as pure noise.
  • Orthogonal grid: At 20,000 steps, PhRMS reverses from a +1.91 PPL short-horizon penalty to a consistent 0.02–0.05 PPL lead from step 8,000 onward.This demonstrates that the 2,000-step verdict for PhRMS is horizon-dependent.

4.5 Experiment 5 - Long-horizon 20k-step runs

Long-horizon training strengthened the three-phase advantage over RoPE-Only, while removing embedding-side positional and scaling scaffolding further improved the simpler architecture.

  • Long-horizon comparison: 13.30% lower final PPL was achieved by the three-phase winner than the RoPE-Only baseline at 20,000 steps.The gap expanded relative to 2,000-step experiments, and the winning variant reached PPL 16.98 by step 8,000 versus the baseline's converged 17.06.
  • Long-horizon comparison: 2.5× convergence speedup let the winning three-phase variant reach PPL 16.98 at step 8,000, below RoPE-Only's step-20,000 final PPL of 17.06.The comparison uses the same 20,000-step training horizon.
  • Embedding simplification: Removing sinusoidal positional encoding improved PPL by 0.39 (2.61%) over the previous winner and produced a 15.57% improvement over RoPE-Only.The simpler ChannelStructure trailed only at step 1,000, then led for 18 consecutive checkpoints.
  • Embedding simplification: Phase_scale and the × √d_model multiplier were effectively inverse operations: phase_scale fell from 0.5774 to approximately 0.07 during training.The learned scale approached 1/√192, cancelling the embedding multiplier and matching vanilla embedding magnitude.
  • Embedding simplification: LearnableOffsets contributed only a 0.0022 PPL difference, while removing sinusoidal PE also eliminated its associated learnable frequencies and phase offsets.The evidence supports treating these embedding-side parameters as unnecessary in the canonical model.

4.7 Experiment 7 - Removing phase_scale and the √d_model multiplier

Removing phase_scale and the embedding multiplier simplified the model while improving quality and convergence, with only the rotation parameters adding trainable overhead.

  • Quality and efficiency: 13.9712 PPL beat the previous best by 0.43 PPL (2.99% relative improvement) and RoPE-Only by 18.09%.The resulting three-phase architecture added 128 parameters, or 0.0023%, over RoPE-Only.
  • Parameter cost: The entire three-phase architecture added 128 parameters because PhaseAwareRMSNorm, zero-sum subtraction, phase splitting, and phase-aligned heads added no extra parameter count.The 128 parameters were exactly 32 theta values across four PhaseRotationLayers.
  • Quality and efficiency: 3.33× convergence speedup let the new model match RoPE-Only's step-20,000 PPL 17.06 at step 6,000.The model reached PPL 17.04 within 30% of the training budget.
  • Mechanistic effect: 30–40% lower theta drift showed that PhaseRotationLayer worked less after phase_scale and × √d_model were removed.The prior configuration spent optimization steps learning an identity-like rescaling.

4.8 Experiment 8 - Is zero-sum enforcement actually needed?

The three-phase geometry largely self-stabilizes without explicit zero-sum enforcement, freeing its DC subspace for a fixed positional signal while preserving phase balance.

  • Self-stabilization: 0.03 PPL was the cost of removing zero-sum enforcement, indicating that the phases self-organized close to balance.The no-zero-sum variant scored 14.0015 PPL versus 13.9712 for the enforced baseline.
  • Self-stabilization: Residualizing PhaseRotationLayer worsened PPL by approximately 0.06, whereas the non-residual overwrite preserved the rotation's intended work.The residual variant scored 14.0602 versus 14.0015 without zero-sum enforcement.
  • DC tunnel: 0.07 PPL improvement came from Gabriel's horn injection over plain mean subtraction, with zero additional trainable parameters.The horn variant scored 13.9015 versus 13.9712 for hard mean subtraction.
  • DC tunnel: The freed one-dimensional DC subspace can carry a structured scalar-per-position signal without disturbing the self-stabilized phase geometry.The paper presents the DC tunnel as a general slot for profiles such as ramps, exponentials, or learned scalars.
  • DC tunnel: The horn's zero-sum residual remained at 3 × H_128 / 128 ≈ 0.1273 across evaluations, matching its analytic per-phase contribution.Subtracting that contribution recovered the same intrinsic phase asymmetry as the no-horn variant.

4.10 Experiment 10 - Learnable horn versus fixed horn

The learnable horn softens its head while preserving the analytic tail, but its small perplexity gain is within seed noise, supporting the fixed profile. At 123M, the canonical architecture improves matched RoPE-Only performance while exhibiting stable phase geometry and depth-dependent rotation drift.

  • Learnable versus fixed horn: 0.02 PPL separates the learnable horn from the fixed horn, within the 0.046 PPL five-seed noise floor.The learnable variant uses 129 additional parameters, so the fixed horn was retained.
  • Learnable versus fixed horn: horn[0] converges from 1.0 to 0.532 while the tail remains near 1/(p+1), indicating a softer head and preserved tail.Past position ~40, the learned profile becomes non-monotonic and first crosses zero at position 51.
  • Rotation drift: At 12 layers, rotation drift is U-shaped: block 2 has minimum drift 0.069, while block 11 reaches 1.833.Blocks 1–6 remain near initialization, whereas the deepest block applies an approximately 9.3° mean shrinkage.
  • 123M scale-up: -7.20% PPL and -2.62% BPB distinguish ThreePhase 123M from the matched RoPE-Only baseline at +1,536 parameters.ThreePhase reaches matched quality with 1.93× fewer steps and 1.64× lower wall-clock time, despite 17.2% per-step overhead.
  • 123M scale-up: The zero-sum residual remains fixed at 3 × H_1024 / 1024 ≈ 0.0220, including without hard mean enforcement.At final evaluation, phase means are [0.0086, 0.0088, 0.0046].

4.12 Experiment 12 - 5.5M seed sweep (noise floor)

A five-seed 5.5M sweep estimates the single-seed noise floor and shows that the canonical reported seed is conservative rather than unusually favorable.

  • Experimental design: The five runs use identical architecture and 5,463,872 parameters while varying only model-init seed.The data seed remains fixed at 42.
  • Noise floor: 13.8500 PPL ± 0.046 is the five-seed mean, with a range of 13.7940–13.9015.The spread is 0.1075 PPL, establishing 0.046 PPL as the approximate single-seed standard deviation.
  • Noise floor: Seed 42 is the worst of five seeds at 13.9015 PPL, making the canonical headline conservative.Seed 1 is best at 13.7940 PPL.

N Thetas/blk Final PPL Val Loss Time

The number of phase channels behaves primarily as a parameter-sharing choice rather than a uniquely optimal geometric setting. Results vary with scale and seeds, weakening strong claims that N=3 is universally superior.

  • 5.5M N sweep: N=1 wins at 5.5M with 13.6268 PPL versus 13.9015 for N=3, while the sweep is otherwise near-monotone.The N=4 versus N=3 inversion is only 0.034 PPL, below the 0.046 PPL single-seed standard deviation.
  • N sweep interpretation: The N–PPL curve is roughly log-linear in rotation-theta count, with N=12 additionally penalized by narrow d_head=8 attention heads.This pattern is interpreted as parameter-sharing regularization with diminishing returns.
  • 123M comparison: At 123M, N=3 beats N=1 by 0.10 PPL in the single-seed comparison, but the advantage does not persist across three seeds.N=1 wins two of three seeds, and the paired mean difference is +0.0778 PPL in N=1's favor with 0.0929 PPL standard error.
  • 123M comparison: The 123M three-seed sweep reproduces the U-shaped depth profile and shows N=3 and N=1 are statistically indistinguishable at this scale.The joint runs support the channel-partitioned residual, per-block rotation, phase-aware RMSNorm, and horn injection as the load-bearing mechanism.

5 Key findings

3PT combines a structurally partitioned residual stream with phase-aware operations and an orthogonal DC side-channel. The geometry self-stabilizes, composes with RoPE and standard Transformer components, and produces depth- and direction-specific learned behavior.

  • Architecture: The residual stream remains vanilla in dimension, while four operations make channel stripes meaningfully distinct: horn replacement, phase-aware normalization, phase rotation, and head divisibility.Attention and SwiGLU still mix across stripe boundaries, so the architecture alternates scrambling with re-imposition.
  • Self-stabilization: The three phases self-balance without hard mean subtraction or an auxiliary zero-sum loss, with residuals stabilizing near 7 × 10^-3 at 5.5M.At 123M, the residual remains pinned at the analytic horn value even when explicit enforcement is disabled.
  • Horn orthogonality: The horn residual equals exactly NUM_PHASES × mean(horn), matching N × H_1024 / 1024 across six independent 123M runs.This supports the horn’s one-dimensional subspace being orthogonal to the N-phase decomposition.
  • Phase geometry: All three phases have equal activation radius, so their specialization is angular rather than magnitude-based.The phase-2-versus-phase-0 difference is therefore a rotation-direction difference.
  • Depth profile: At 12 layers, rotation drift becomes U-shaped and reverses direction at block 2, with early blocks expanding and later blocks shrinking the initialized angles.The deepest block’s 1.833 L2 drift corresponds to an approximately uniform 9.3° shrinkage.
  • DC side-channel: The fixed horn preserves a closed-form positional side-channel, while a learnable horn softens the head but yields no measurable PPL improvement.The learnable profile becomes non-monotonic after about position 40 and first crosses zero at position 51.

6 Discussion

3PT is framed as a lightweight residual-stream structural prior whose phase-respecting operations leave core Transformer components intact. Its empirical advantages are accompanied by compute, baseline-comparison, and phase-selection boundaries.

  • Discussion: 3PT extends lightweight structural priors with a geometric residual-stream intervention while leaving attention, FFN, and RoPE unchanged.The contribution is positioned outside conventional positional encoding, embedding, attention, normalization, optimizer, tokenizer, and data-mixture categories.
  • Discussion: 1.93× fewer optimizer steps translate to 1.64× faster wall-clock convergence because rotation adds ~17% per-step compute overhead.Step-matched and compute-matched comparisons are therefore distinct.
  • Discussion: External GPT-2 comparisons are not clean architecture A/B tests because distilgpt2 uses WebText pretraining and differs in depth.3PT beats zero-shot GPT-2 small by 0.136 BPB but trails distilgpt2-finetuned by 0.086 BPB.
  • Discussion: At 123M, N=1 and N=3 are statistically indistinguishable across three seeds, limiting directional claims about the best phase count.At 5.5M, N=1 clearly wins by 6× the noise floor, but the 123M result remains noise-limited.

7 Future work

Future work targets the learned depth schedule, broader DC-side-channel designs, larger N sweeps, and scaling beyond the current 123M WikiText-103 setting. These directions address observed structural patterns and unresolved scale dependence.

  • 7. Future work: The U-shaped theta-drift profile suggests replacing the linear initialization with a non-linear schedule whose minimum occurs early and whose tail decays.The minimum is near block 2, while later blocks shrink their initial angles.
  • 7. Future work: The minimum-block location has across-seed standard deviation below 0.016 for both N=3 and N=1, supporting a robust architectural pattern.This stability motivates schedule redesign rather than treating the profile as seed noise.
  • 7. Future work: The DC subspace can host scalar-per-position functions beyond Gabriel's horn, with candidate shapes including ramps, exponentials, log-spaced codes, and learned scalars.The horn is favored for absolute-position language modeling, while other tasks may require different shapes.
  • 7. Future work: A 123M sweep with at least five seeds per N is needed to resolve whether the phase-count optimum changes with scale or remains in the noise.The proposed larger sweep includes N ∈ {1, 2, 3, 4}.
  • 7. Future work: Scaling to 350M–1B parameters on a larger corpus with matched compute is required to test whether the observed gap shrinks, holds, or grows.The current paper cannot answer this above 1B parameters.

8 Conclusion

3PT introduces a low-overhead geometric organization of the residual stream that composes with standard positional encoding and Transformer components. Its reported gains coexist with self-stabilizing geometry, a U-shaped rotation-drift profile, and phase-count results that depend on scale.

  • 8. Conclusion: 3PT adds a residual-stream structural prior through phase-respecting operations, costing +1,536 trainable parameters (0.00124%) at 123M scale.The intervention includes PhaseAwareRMSNorm, PhaseRotationLayer, phase-aligned GQA, and Gabriel's horn DC injection.
  • 8. Conclusion: The architecture structures token representations geometrically before and during training while allowing standard RoPE to remain on top.The three-phase structure is not itself a positional encoding or embedding.
Loading 2604.14430v1…