PaperlayerView exact revision on arXiv

Source-linked AI summary

p-Spin Glass Network Efficient Single-Batch Continual Learning

Vladimer Khasia

arXiv:2608.14774v1cs.LG

TL;DR

Minimal-batch sequence learning remains limited by memory demands and optimization instability. The p-Spin Glass Network combines implicit equilibrium dynamics with ternary internal quantization, matching Transformer validation loss at 12.5% of the training sequences while supporting stable single-batch learning.

  • Problem

    Existing sequence models remain limited by large memory footprints and severe gradient variance and optimization instability at minimal batch sizes.

  • Method

    The p-Spin Glass Network combines continuous-time state-space integration with implicit spin-glass equilibrium attractors and ternary internal projection matrices.

  • Results

    Validation loss was 4.8004 versus 4.8025 for optimized Transformers while using 12.5% of the training sequences, alongside smooth monotonic convergence at B = 1 across subword and raw-byte streams.

  • Takeaways & Limitations

    The architecture establishes a foundation for continuous learning and memory-constrained applications through low-bit compression, sample efficiency, and single-batch optimization stability.

Abstract

from arXiv · show

Modern sequence models heavily rely on massive memory footprints and large-batch stochastic optimization, barriers that restrict sample efficiency and continual learning. We introduce the $p$-Spin Glass Network, a novel architecture that overcomes these limitations, structurally manages optimization variance and yields four noticeable capabilities: 1. It enforces memory efficiency: native ternary quantization compresses internal parameters by $8\times$, while exact implicit gradients strictly bound activation memory to $\mathcal{O}(B \cdot T \cdot D)$. 2. it demonstrates sample efficiency, matching the asymptotic performance of a Transformer baseline while utilizing $8\times$ fewer training sequences. 3. Method enables single-batch stability and smooth, monotonic convergence at a stochastic micro-batch size of $1$. 4. Finally, this stability proves modality-agnostic, maintaining robust temporal credit assignment across both discrete subword and long horizon uncompressed raw byte streams. Ultimately, this work removes large batch requirement for stable deep learning, establishing a foundation for continuous learning and edge AI.

1. Introduction

Transformers’ memory and batch-size demands obstruct low-resource and continual learning, while existing sequence-memory and quantization advances do not resolve minimal-batch gradient variance and instability (Vaswani et al., 2017; Radford et al., 2019; Parisi et al., 2019). The p-Spin Glass Network addresses these barriers through ternary-quantized implicit thermodynamic projections, enabling memory efficiency, sample-efficient optimization, single-batch stability, and modality-agnostic sequence modeling.

  • Introduction: The p-Spin Glass Network targets minimal-batch optimization instability by combining implicit fixed-point learning with ternary-quantized thermodynamic projection matrices.The quantization bounds the equilibrium layer’s Lipschitz constant and structurally dampens gradient variance, building on implicit deep learning and spin-glass principles (Bai et al., 2019; El Ghaoui et al., 2021; M´ezard et al., 1987; Parisi, 1980; Crisanti & Sommers, 1992; Gross & M´ezard, 1984).
  • Introduction: 8× parameter compression and O(B · T · D) activation memory establish the network’s architectural memory efficiency.Ternary quantization applies to the implicit attractor’s projection matrices, while SRAM-fused state-space chunking and Implicit Function Theorem gradients make activation memory independent of fixed-point iteration count K.
  • Introduction: 12.5% of the required training sequences achieves asymptotic performance parity with optimized standard Transformers.This establishes sample-efficient optimization relative to the Transformer baseline.
  • Introduction: B = 1 yields smooth, monotonic convergence, eliminating the standard requirement for massive batch accumulation.The bounded fixed-point solver provides structural gradient-variance damping for pure stochastic micro-batch training.
  • Introduction: Temporal credit assignment remains robust from discrete subword spaces with |V | = 49152 to continuous raw byte streams with |V | = 256.The architecture is presented as a generalized, modality-agnostic sequence modeling framework spanning fundamentally different representation scales.

2. Methodology · 2.1. Mathematical Formulation and Derivation

The p-Spin Glass Network is formulated as a sequence-modeling framework whose forward and backward dynamics couple continuous-time state-space integration with p-spin glass equilibrium attractors. Its derivation uses chunked causal processing, ternary-quantized equilibrium projections, and implicit fixed-point gradients evaluated without iterative graph unrolling.

  • 2.1. Mathematical Formulation and Derivation: The architecture couples continuous-time state-space integrations with p-spin glass equilibrium attractors for sequence modeling on discrete language modalities.The formulation defines B, T, and D as batch size, sequence length, and hidden feature dimension, with T = N · C for chunk size C and N chunks.
  • 2.1.1. CONTINUOUS ODE INTEGRATION VIA GATED STATE-SPACES: The gated state-space module bounds continuous-time temporal growth to negative domains through a data-dependent ODE decay gate.Inputs are RMS-normalized with ϵ = 10^-6, and state-space projections Q, K, and V are computed across batch, sequence, head, and head-dimension axes.
  • 2.1.1. CONTINUOUS ODE INTEGRATION VIA GATED STATE-SPACES: Chunking avoids O(T^2) memory materialization by constructing exact intra-chunk causal masks in SRAM and combining local attention with cross-chunk global memory.The sequence is partitioned into N chunks of length C, while the residual update forms Xmid by adding the contextualized state-space output to the layer input.
  • 2.1.2. HIGHER-ORDER INTERACTIONS (p-SPIN GLASS GATED ATTRACTORS): The p-spin glass MLP uses ternary projection weights in {−1, 0, 1}, scaled by an absolute mean and trained with a Straight-Through Estimator.The residual tensor is normalized before the implicit equilibrium computation, and dense external and internal projection matrices are quantized for parameter efficiency.
  • 2.1.2. HIGHER-ORDER INTERACTIONS (p-SPIN GLASS GATED ATTRACTORS): The higher-order attractor projects representations into a 2Dint space, separates value bounds from spin gates, and bounds frustration with γ = 0.5 tanh(γparam).Across K equilibrium micro-steps, the network seeks the energy-minimizing fixed point Y* = YK, which is then down-projected and added residually to produce the layer output.
  • 2.1.3. EXACT IMPLICIT GRADIENTS: Backpropagation through the iterative attractor uses the Implicit Function Theorem and a Neumann-series approximation to reconstruct gradients without O(K) graph unrolling.The equilibrium derivatives ϕu and ϕg define the iterative gradient Vk from the incoming gradient at the fixed point.
  • 2.1.3. EXACT IMPLICIT GRADIENTS: The implicit gradients are strictly evaluated on the fixed-point manifold, completing the attractor’s exact backward computation.This fixed-point evaluation accompanies the Neumann-series gradient reconstruction rather than differentiating through every equilibrium iteration.

2.2. Algorithmic Specification

The p-Spin Glass Network is specified in two modality-agnostic formulations that share identical internal dynamics while differing in input encoding, embeddings, and evaluation metrics.

  • 2.2. Algorithmic Specification: The complete specification provides distinct subword-tokenized and continuous raw-byte formulations corresponding directly to the code implementations.This establishes the network as a general sequence-modeling framework across modalities.
  • 2.2. Algorithmic Specification: Both formulations use the same p-spin attractor and hardware-fused state-space dynamics, with differences limited to encoding spaces, embedding matrices, and final evaluation metrics.The subword formulation uses |V| = 49152, while the raw-byte formulation uses |V| = 256.

2.3. Complete Complexity Analysis

The complexity analysis separates subword and byte modalities, showing that internal computation and memory remain controlled while vocabulary size determines projection and logit costs. Quantized p-spin interactions, SRAM chunking, and implicit gradients provide addition-based execution, linear activation memory, and approximately 8× parameter compression.

  • Time Complexity: Internal p-spin interactions use ternary weights, replacing floating-point multiplications with hardware additions and subtractions while requiring O(B · T · Dint · (D + K · Dint)) time per layer.The quantized matrices WQint are restricted to {−1, 0, 1}.
  • Time Complexity: Vocabulary projection costs O(B · T · D · |V|), with subwords requiring O(B · T · D · 49152) and bytes requiring O(B · T · D · 256).The subword projection dominates computation, whereas byte-level execution is primarily bounded by internal hidden dimensions.
  • Space Complexity: Activation memory remains O(B · T · D), avoiding standard global attention’s O(B · T^2 · H) scaling through SRAM-resident causal chunking.The chunk decay mask is materialized in GPU SRAM and immediately reduced.
  • Space Complexity: IFT-based backward computation makes p-spin activation memory invariant to micro-step count K, replacing O(K · B · T · Dint) graph storage with dependence on the converged state and intermediate bounds.The backward pass uses a Neumann-series approximation of the inverse Jacobian.
  • Space Complexity: Theoretical parameter compression shrinks the internal MLP footprint by approximately 8× versus equivalent 16-bit architectures using 1.58-bit packed p-spin interactions.State-space matrices remain 16-bit, while Wext and Wint use packed 2-bit blocks.
  • Space Complexity: Peak logit memory similarly scales as O(B · T · 49152) for subwords versus O(B · T · 256) for bytes, enabling longer byte-level contexts under equal HBM.The categorical distribution vector is the bottleneck during cross-entropy gradient formation.

3. Experiments

Across subword and byte-level settings, p-Spin Glass achieves strong sample efficiency and stable optimization at micro-batch size B = 1. The experiments also show modality-agnostic behavior and substantially lower step latency than the Transformer baseline.

  • Sample efficiency: 4.8004 validation cross-entropy loss (Perplexity 121.56) marginally outperforms the Transformer’s 4.8025 (Perplexity 121.81) despite using 8× fewer sequences.The Subword p-Spin model reaches the baseline’s asymptotic performance with 12.5% of the training data.
  • Execution efficiency: The Subword p-Spin model executes at 0.44 seconds per step versus 18.9 seconds for the Transformer baseline.The comparison uses B = 1 for p-Spin and B = 64 for the baseline.
  • Modality agnosticism: At B = 1, the Byte-level p-Spin model reaches 2.12 Bits-Per-Byte and an equivalent subword loss of 5.5 after 8,000 sequences.Byte-level processing reduces the vocabulary from 49,152 to 256 and collapses the VRAM projection footprint by approximately 192×.
  • Gradient stabilization: At B = 1, p-Spin Glass exhibits a perfectly smooth, monotonic convergence curve without the stochastic jumps typical of single-sample training.The fixed-point solver is described as a variance damper because implicit gradients follow a stable energy-minimizing manifold.
  • Gradient stabilization: Ternary internal projection matrices bound the equilibrium layer’s Lipschitz constant, preventing catastrophic gradient scaling and reducing the need for massive gradient accumulation.The matrices are quantized to {−1, 0, 1}.

4. Conclusion

The p-Spin Glass Network bridges continuous-time state-space integration with implicit physical attractors, combining memory efficiency, sample efficiency, single-batch stability, and modality-agnostic temporal credit assignment. These properties establish a foundation for continuous learning and memory-constrained applications.

  • Conclusion: Together, native low-bit compression, high sample efficiency, and single-batch optimization stability establish a foundation for continuous learning and memory-constrained applications.This conclusion synthesizes the architecture’s principal advantages over standard baselines.
  • Sample Efficiency: 4.8004 vs. 4.8025 validation loss is achieved while processing exactly 12.5% of training sequences, or 8,000 vs. 64,000, matching optimized Transformers asymptotically.Continuous routing of the implicit equilibrium enables this sample efficiency.
  • Knowledge Acquisition and Continual Learning: At stochastic micro-batch size B = 1, fixed-point solving and ternary-bounded Lipschitz constants produce smooth, monotonic convergence without gradient accumulation.The mechanism is intended to support stable continuous knowledge retention.
  • Modality Agnosticism: Robust temporal credit assignment persists across compressed subword inputs with |V| = 49152 and extreme-horizon uncompressed raw byte streams with |V| = 256.The architecture therefore operates across varying input topologies as a general sequence-modeling framework.

A. Extended Experimental Details and Parameter Analysis

This appendix details the architectures, optimization hyperparameters, and memory footprints of the baseline and proposed p-Spin Glass Sequence models.

  • A. Extended Experimental Details and Parameter Analysis: The appendix provides complete architectural specifications for both the baseline and proposed p-Spin Glass Sequence architectures.
  • A. Extended Experimental Details and Parameter Analysis: It reports the optimization hyperparameters used for the baseline and proposed architectures.
  • A. Extended Experimental Details and Parameter Analysis: It breaks down the memory footprint of both model architectures.

A.1. Dataset and Preprocessing

The models were trained sequentially on the sample-10BT split of HuggingFaceFW/fineweb-edu using a fixed shuffle buffer and seed. Preprocessing covered both tokenized subword sequences and raw UTF-8 byte sequences with modality-specific vocabularies and context lengths.

  • A.1. Dataset and Preprocessing: All models used sequential streaming from the sample-10BT split of HuggingFaceFW/fineweb-edu, with shuffle buffer size 500 and random seed 1337.The training pipeline iteratively streamed the data.
  • A.1. Dataset and Preprocessing: Subword preprocessing used the SmolLM2-135M tokenizer with vocabulary size |V| = 49152 and maximum context length T = 1024.Sequences were packed to the specified context limit.
  • A.1. Dataset and Preprocessing: Byte preprocessing encoded raw UTF-8 bytes with vocabulary size |V| = 256, reserving byte 255 for EOS, and maximum context length T = 2048.Sequences were packed to the specified context limit.

A.2. Architectural Configurations and Optimization

The evaluation used approximately 60M logical parameters across models, with fixed p-Spin Glass and State-Space configurations. Subword and Byte models were dimensioned to match total parameters, and all methods followed specified AdamW optimization settings and learning-rate scheduling.

  • Model configuration: All models were scaled to approximately ∼60M logical parameters for fair evaluation, while p-Spin Glass used L = 4 layers, H = 8 attention heads, and K = 5 equilibrium micro steps.The State-Space continuous integration used a fixed chunk size of C = 128.
  • Model configuration: The Byte model expanded its dimensions to D = 704 and Dint = 2048, compared with D = 544 and Dint = 1408 for the Subword model, to match total parameter number.
  • Optimization: Optimization used AdamW with weight decay 0.05, global gradient clipping at 1.0, and a learning-rate schedule combining 500-step linear warmup, an 8×10−4 peak, a plateau, and final cosine decay to a 0.1 minimum ratio.

A.3. Parameter Footprint and Memory Analysis

The p-Spin Glass models reduce parameter and logical memory footprints through native low-bit quantization and redistribution of parameters into continuous logic matrices. The Byte-level model reaches 41.37 MB versus 116.62 MB for standard Transformer or other natively uncompressed models.

  • Parameter Footprint: 8.0× theoretical MLP compression is achieved by quantizing internal thermodynamic interactions to 1.58-bit representations, logically packed as 2-bit hardware structures.The quantized interactions are Wext and Wint.
  • Memory Analysis: The Byte-level p-Spin model reallocates parameters from vocabulary embedding projection into core continuous logic matrices, including the Ternary MLP and State-Space Attention.This redistribution reduces the model’s global memory footprint.
  • Memory Analysis: 41.37 MB is the Byte-level p-Spin model’s global memory footprint, compared with 116.62 MB for a standard Transformer or other natively uncompressed model.Tables 2 and 3 report the models’ parameter distributions and logical memory footprints.
Loading 2608.14774v1…