Parcae: Scaling Laws For Stable Looped Language Models

Motivation

• Looped architectures increase FLOPs by repeatedly sending activations through a layer block, potentially matching larger fixed-depth models without increasing parameters.They also offer potential for latent reasoning and per-token adaptive compute.Source: §1 · p. 1
• Prior looped models suffer residual-state explosion, loss spikes, and sensitivity to hyperparameter selection and residual normalization.Variable-depth training can produce additional late-training loss spikes.Source: §1 · pp. 1–2

Approach

• Parcae recasts looping as a nonlinear time-variant dynamical system and uses a linear approximation to connect instability with injection-parameter spectral norms.The framework identifies unstable parameterizations of A as inducing residual-stream explosion.Source: §1 · p. 2
• Parcae constrains A through a discretized negative diagonal parameterization, normalizes input injection, and uses per-sequence depth sampling to reduce training instability.These changes target residual explosion and loss spikes while preserving looped computation.Source: §1 · p. 2

Results

• 6.3% lower validation perplexity is achieved against parameter- and data-matched RDMs, while 1.3B-parameter Parcae improves Core and Core-Extended by 2.99 and 1.18 points versus matched Transformers.The latter result matches Transformers up to twice the size.Source: §1 · p. 2
• FLOP-optimal training increases looping and data together through empirical power laws, while test-time looping follows predictable exponential decay with an irreducible loss.The study evaluates end-to-end quality, training FLOP scaling, and test-time scaling.Source: §1 · pp. 2–3

Looped Architectures

• Prior looped architectures vary in loop placement, halting mechanism, topology, and differentiation; this work focuses on implicit-halting middle-looped models with explicit differentiation.The review distinguishes pre-, mid-, and post-looping, explicit routers and stochastic depth, and single-block and hierarchical topologies.Source: §2 · p. 3
• A middle-looped architecture consists of a prelude P, recurrent unit R, and coda C, with the recurrent unit repeatedly transforming the embedded input.The formulation maps input s to e = P(s), iterates h, and produces output p = C(h_T).Source: §2.1 · p. 3
• Training samples recurrence depth T from a distribution Λ, while truncated backpropagation limits the backward pass to a constant depth.Variable-depth training is intended to support stronger test-time scaling.Source: §2.1 · p. 3

Dynamical Systems

• The paper uses an LTI dynamical system as a tractable linear surrogate for analyzing complex nonlinear looped models.The surrogate describes hidden-state evolution under input-driven dynamics.Source: §2.2 · pp. 3–4
• Discrete LTI stability depends on the spectral radius of A: systems are stable when ρ(A) < 1 and unstable when ρ(A) > 1.The reviewed regimes are stable, marginally stable, and unstable.Source: §2.2 · p. 4

Scaling Laws

• Scaling-law analysis fits loss as a function of model size and training tokens under fixed FLOP budgets using parabolic and parametric methods.The parametric form includes irreducible loss and power-law terms for parameters and data.Source: §2.3 · p. 4

Training Instability

• Residual normalization can mitigate state explosion but requires sensitive hyperparameter tuning, and large-scale runs can still exhibit late-training loss spikes.The paper reports such spikes after 170k steps and addresses them by normalizing the input to B.Source: §3 · pp. 4–5

State Explosion from Unconstrained A and B

• Table 1 compares prior loop update rules by their LTI stability properties, emphasizing whether their A parameterizations satisfy the spectral-radius condition.The comparison covers addition and concatenation injection within the same framework.Source: Table 1 · p. 5, §3 · p. 5

Dynamical System over Residual Stream

• An exact nonlinear residual-stream update is linearized into a discrete LTI system, making stability conditions analytically tractable.The full recurrent update includes transformer operations through R, while the linearized form drops R.Source: §3 · p. 5

State Explosion from Unconstrained A and B

• Prior looped architectures use A parameterizations with ρ(A) = 1 or unconstrained ρ(A), corresponding to marginally stable or unstable systems.These choices do not enforce the conditions required for bounded recurrent dynamics.Source: §3 · p. 5
• Divergent runs learn ρ(A) ≥ 1, whereas convergent runs maintain ρ(A) < 1, empirically linking spectral-radius constraints to residual stability.The analysis identifies ρ(A) as the critical factor governing instability.Source: §3 · p. 5

Architecture and Training

• Parcae explicitly satisfies the dynamical-system stability constraints and adds per-sequence depth sampling to stabilize variable-depth training.The method targets both recurrent instability and variance introduced by sampled depth.Source: §4 · p. 6

Stable Recurrent Parameterization

• Continuous A and B are discretized with a learned step size using ZOH or Euler schemes, yielding discrete recurrent parameters.The resulting update uses A and B in the loop recurrence.Source: §4.1 · p. 6
• A is parameterized as a negative diagonal matrix before discretization, ensuring negative eigenvalues while keeping the stability mechanism simple and inexpensive.The learnable vector is transformed through Diag(−exp(·)); B remains unconstrained and the input e is normalized.Source: §4.1 · p. 6

Architecture and Training

• Parcae normalizes the embedded input e before recurrence and computes outputs by applying the coda projection after T loops.Its recurrent update is ht+1 = Aht + Be + R(ht, e).Source: §4.1 · p. 6
• Parcae matches prior RDM and Transformer setups while differing through residual normalization choices and dynamical-system parameters such as A, B, C, and ∆.Against Transformers, the middle third of layers is looped and a prelude norm is included.Source: §4.1 · p. 6

Stable Training Algorithm

• Per-sequence depth sampling provides more depth samples within a micro-batch, and the training design is intended to reduce loss spikes and improve recurrence extrapolation.The method parameterizes Λ using µ_rec and sets µ_bwd = ⌈µ_rec^2⌉.Source: §4.2 · p. 6

• Parcae is compared with parameter- and data-matched RDMs and Transformers using held-out perplexity and downstream benchmarks.Source: §5.1 · p. 7, Table 5 · p. 8

Comparison against RDMs

• Parcae reduces perplexity by up to 6.2% and 9.1% on validation and WikiText, respectively, versus prior RDMs.Source: §5.1 · p. 7

Ablations

• Each Parcae modification contributes to stability or quality: constrained A enables convergence at high T, sampling stabilizes lower depths, and prelude normalization improves quality.Source: §5.1 · p. 7

Comparison Against Transformers

• Parcae reduces validation perplexity by 4.3–9.2% and improves Core and Core-Extended scores by up to 2.99 and 1.18 points versus Transformers.Source: §5.1 · p. 7
• A 770M Parcae model achieves Core quality comparable to a 1.3B Transformer with roughly half the parameters.Source: §5.1 · p. 7

Training FLOP Scaling

• Compute-optimal training increases mean recurrence µrec and data in tandem, establishing looping as an orthogonal scaling axis.Source: §5.2 · p. 8, Figure 4 · p. 8
• At 140M and 370M scales, increasing µrec while proportionally reducing tokens lowers validation loss under fixed FLOPs.Source: §5.2 · p. 8
• Optimal recurrence and token budgets follow predictable power laws with exponents γµ ≈0.40 and γD ≈0.78.Source: §5.2 · p. 9, Figure 5 · p. 9
• The scaling-law fit predicts held-out validation loss with 1.3% error at 140M and 0.8% error at 370M.Source: §5.2 · p. 9

IsoFLOP Comparison

• The optimal looped frontier achieves strictly lower loss than fixed-depth models and yields 1.2–2.0 points higher Core scores.Source: §5.2 · p. 9, Figure 6 · p. 9, Table 6 · p. 9

Test-Time Scaling

• Test-time looping follows a saturating exponential decay, with gains plateauing near the training recurrence µrec.Source: §5.3 · p. 10

Unified Scaling Law

• A unified fit predicts held-out test-time loss within 0.85–1.31% average error, or 0.1–0.17% when using empirical loss at T = µrec.Source: §5.3 · p. 11

Looped Architectures

• Larger-scale investigation is needed for loop-unit placement, recurrent-unit composition, and extreme looping at deeper recurrence depths.Source: §6 · p. 11
• Different discretizations, full-rank parameterizations, and recurrent update rules may enable recurrence at larger depths.Source: §6 · p. 11

Scaling

• The study’s scaling observations are limited to small architectures, so performance at large FLOP budgets and parameterizations remains unresolved.Source: §6 · p. 11
• As µrec increases, more test-time steps are required to achieve equivalent quality, motivating methods that reduce inference steps.Source: §6 · p. 11

Conclusion

• Parcae prevents residual explosion by constraining the spectral norm of looped-model injection parameters.The paper presents this as the central stability mechanism for its looped architecture.Source: §7 · p. 11

Dynamical-systems formulation

• Looped-model recurrence can be represented as a nonlinear time-variant dynamical system over the residual stream.The framework identifies A, B, and C as dynamical-system parameters and R as the nonlinear transformer update.Source: C Derivation of Instability Conditions… · p. 22
• Linearizing the recurrence connects stability to the spectral properties of the residual-state transition matrix.For Pre-Norm additive injection, the linearized transition has A = I, making the system marginally stable.Source: C Derivation of Instability Conditions… · p. 22

FLOP estimation

• Parcae’s effective training FLOPs account separately for parameters that are and are not backpropagated through recurrent depth.The formulation is C = (2 N̂1 + 6 N̂2)D, with embedding parameters excluded from N̂ and attention FLOPs additionally estimated.Source: D FLOP Estimate of Parcae · p. 22

Implementation

• The implementation provides forward-pass and per-sequence stochastic-depth procedures for efficiently training variable recurrent depths.Batched sequences use the maximum sampled depth while avoiding initial state updates to preserve efficient gradient flow.Source: E Parcae Forward Pass and Training… · p. 23

Stability across learning rates

• Parcae remains stable across the tested learning-rate sweep for baseline and residual-normalized RDM comparisons.The sweep covers learning rates from 2e−4 through 1e−3.Source: F Additional Stability Ablations · p. 23, Figure 9 · p. 24

Per-sequence sampling

• Per-sequence sampling eliminates loss spikes that occur with per-micro-batch sampling in 350M-parameter Parcae training.The spikes correspond to overly large recurrent residual jumps at the final recurrence.Source: G Per-sequence Sampling Reduces Loss… · p. 25, Figure 10 · p. 25
• Per-sequence depth sampling supports stronger fixed-point behavior by reducing recurrent residual jumps and improving the training objective’s estimate.Recurrent residual and state-norm metrics are used to assess this behavior.Source: G Per-sequence Sampling Reduces Loss… · p. 25, Figure 11 · p. 25

Truncated recurrence sampling

• The proposed recurrence-sampling method decouples the sampled recurrence distribution from the backpropagation-depth parameter.Prior sampling truncated recurrence based on µbwd, producing a shifted or compressed distribution relative to the desired pretraining distribution.Source: H Sampling of Truncated Recurrence · pp. 27–28, Figure 13 · p. 27, Figure 12 · p. 27
• At T = µrec = 8, the ablated sampling methods have nearly identical training and validation trajectories, but the proposed method performs better at T ∈ [4, 16, 64].The reported difference is attributed to the prior method’s constricted effective recurrence range during pretraining.Source: H Sampling of Truncated Recurrence · p. 28

Varying µrec

• µrec = 8 performs best when µbwd = 4, including at higher test-time recurrences.Models with larger µrec performed worse at both lower and higher test-time recurrence values in the reported experiment.Source: I Selecting µrec and µbwd · p. 29, Figure 15 · p. 29, Table 9 · p. 29

Varying µbwd

• Increasing µbwd improves validation performance at T = 16 and T = 64 when µrec = 20.The improvement is observed across the tested lower and higher recurrence settings, although lower µbwd can perform better at lower validation recurrences.Source: I Selecting µrec and µbwd · p. 30, Figure 16 · p. 30

Open question

• The authors leave FLOP-optimal choices of µrec and µbwd for future work.The reported experiments establish coupling between the two quantities but do not determine their jointly optimal settings.Source: I Selecting µrec and µbwd · p. 30

• The 1.3B run was stable for 150k optimizer steps before exhibiting state explosion and loss spikes.Source: J Ablation of Prelude Normalization · p. 31
• Spectral norms of A and B remained low, while C’s norm grew but produced a relatively small, slowly decreasing empirical expansion ratio.Source: J Ablation of Prelude Normalization · p. 31, Figure 18 · p. 31, Figure 19 · p. 32
• The recurrent state norm jumped on the first recurrence and then remained relatively stable, indicating that the main explosion occurred before recurrent iteration.Source: J Ablation of Prelude Normalization · p. 31, Figure 20 · p. 32
• Prelude-output normalization stabilized recurrent norms and enabled the stable 1.3B training run.Source: J Ablation of Prelude Normalization · p. 32
• A single layer in the prelude created a residual-norm explosion, while the nonlinear recurrent blocks had little effect on the explosion.Source: Figure 22 · p. 33, Figure 21 · p. 33
• Prelude normalization improved convergence for 140M and 370M models, but yielded only negligible improvement for 770M and 1.3B models.Source: J Ablation of Prelude Normalization · p. 33, Figure 24 · p. 34

• The authors compare four functional forms for the saturating relationship between test-time recurrence T and validation loss, fitting each curve independently.Source: L Fitting Parametric Functions to… · p. 36
• The exponential decay form achieves the lowest Huber loss, performing 2.3× better in-distribution and 7.1× better under extrapolation than the shifted power law.Source: L Fitting Parametric Functions to… · p. 36
• Removing the irreducible loss floor increases error by over 40×, showing that test-time scaling saturates to a finite loss determined by training.Source: L Fitting Parametric Functions to… · p. 36
• The fitted irreducible loss closely matches empirical loss at T = µrec, motivating the training scaling law as the unified law’s floor.Source: L Fitting Parametric Functions to… · pp. 36–37
• Conditioning the decay rate on training recurrence is necessary: removing µrec conditioning increases training error by 3.5× and held-out error by approximately 33%.Source: L Fitting Parametric Functions to… · p. 37
• The unified fit predicts held-out validation loss within 0.85–1.31% average error, while replacing its floor with an oracle reduces error to 0.10–0.17%.Source: L Fitting Parametric Functions to… · p. 37

• The isoFLOP setup fixes training FLOPs and varies µrec while adjusting token budgets, using 36 models at 140M and 18 at 370M.Source: O Expanded Setup For Training and… · p. 38
• The evaluation includes Core and Core-Extended downstream tasks, with three seeds used for baseline-transformer comparisons.Source: Table 15 · p. 39
• Fixed-depth Parcae models with µrec = 1 provide baselines for comparing looping frontiers against pure data scaling.Source: O Expanded Setup For Training and… · pp. 39–40
• The test-time setup trains 140M and 370M models on 11.2B tokens with µrec from 2 to 12, then evaluates recurrences T from 1 through 24.Source: O Expanded Setup For Training and… · p. 40
• The unified-law setup reuses isoFLOP models, evaluates ten test-time recurrence values, and fits all eight parameters jointly using Huber loss and L-BFGS.Source: O Expanded Setup For Training and… · p. 40
• Parcae and baseline Transformers use separate architectural setups, while the comparison reports parameter definitions and parameter counts explicitly.Source: P Model Definitions · p. 40, §P.2 · p. 41, Table 17 · p. 41
• Parcae training uses specified warmup, cooldown, optimizer, learning-rate, clipping, precision, and FineWeb-edu data configurations.Source: §Q.1 · pp. 42–43, §Q.2 · pp. 43–44
• Baseline Transformers receive learning-rate sweeps, whereas Parcae does not; the authors expect a more optimal Parcae learning rate could further improve performance.Source: §Q.2 · p. 43, §Q.1 · p. 43