PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARCL: Spectral Partitioned Analytic Continual Learning

James Hartley, Zeropy Surio, Daniel Whitmore, Hannah Clarke, Thomas Reed

arXiv:2608.21307v1cs.LG

TL;DR

Analytic continual learners can forget old classes because incoming features reshape a shared inverse autocorrelation operator, causing spectral interference and old-logit drift. SPARCL partitions features into a frozen high-energy core and an adaptable residual block, and experiments show it closes most of the classical-analytic gap to strong representation matchers across four datasets.

  • Problem

    Analytic continual learning still exhibits old-class degradation because new-task samples reshape the shared inverse autocorrelation operator, diluting old discriminative directions even without gradient-based overwriting.

  • Method

    SPARCL partitions the feature space into a high-energy core and residual complement, freezes old-class core components, and restricts subsequent updates to the residual block with optional random-projection expansion.

  • Results

    SPARCL closes most of the classical-analytic gap to strong representation matchers under a frozen-ViT protocol across CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A.

  • Takeaways & Limitations

    Spectral partition preserves old-class core logits exactly while retaining residual plasticity, and remains complementary to sparse feature-decorrelation methods such as Fly-CL.

  • Takeaways & Limitations

    The evaluation assumes exemplar-free class-incremental learning with a frozen pre-trained visual encoder, updating only the linear classifier across tasks.

Abstract

from arXiv · show

Analytic continual learning has emerged as a strong exemplar-free alternative to gradient-based class-incremental learning because it replaces iterative optimization with closed-form ridge updates. Yet the usual forgetting narrative, centered on stochastic gradient overwriting, does not explain why analytic methods still drift on old classes despite exact recursive solvers. We identify the culprit as spectral interference: the joint ridge classifier for all tasks shares the inverse autocorrelation operator $(R+λI)^{-1}$, so incoming task samples that load onto old dominant eigendirections dilute the spectrum and perturb old-class logits even when old labels are never revisited. Based on this view, we propose SPARCL, a spectral partitioned analytic continual learner that decomposes the running autocorrelation into a high-energy core and a residual complement, freezes old-class classifier components in the core subspace, and updates only the residual block through recursive least squares with an optional residual random-projection expansion. This yields a simple closed-form update with a provable invariance guarantee for the core contribution of old logits. Across CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A under a frozen ViT-B/16 protocol, SPARCL closes most of the gap from classical analytic learners to strong representation matchers, while remaining complementary to sparse feature-decorrelation approaches such as Fly-CL.

1. Introduction

Analytic continual learners avoid iterative optimization but still forget because new-task data reshapes a shared inverse autocorrelation operator. SPARCL protects high-energy directions while confining adaptation to residual features.

  • Motivation: Analytic continual learning uses recursive ridge updates with frozen backbones, yet old-class degradation remains unexplained by gradient overwriting.The central question is what changes when no gradients rewrite classifier parameters.
  • Motivation: New-task samples aligned with old principal directions reshape the shared inverse autocorrelation operator, diluting old modes and shifting old-class logits.This occurs even when old cross-correlation statistics remain unchanged.
  • Method: SPARCL decomposes the feature space into a stable high-energy core and a residual complement, freezing old-class core components while updating the residual block.The residual block can optionally be expanded with low-cost random projections.
  • Contributions: SPARCL provides residual-only updates that preserve old-class core logits exactly and bound total logit drift by incoming residual energy.These are stated as the paper’s core theoretical guarantees.
  • Contributions: SPARCL is evaluated under a frozen-ViT protocol across four datasets and is reported to close most of the gap between classical analytic learners and representation matchers.The introduction identifies this as an extensive experimental contribution.

2. Related Work

Prior continual-learning methods use replay, regularization, architecture expansion, prompting, or analytic classifier fitting. SPARCL instead targets interference inside the analytic classifier update while complementing strong frozen-feature pipelines.

  • Continual learning beyond analytics: Replay-, regularization-, and architecture-based continual learners are effective but commonly require rehearsal buffers, task-specific tuning, or iterative optimization.Examples include exemplar replay, distillation, parameter-importance regularization, and classifier calibration.
  • Prompting and pre-trained learners: Prompt-based and subspace-expansion methods isolate plasticity in prompts, tokens, or expanded representations while retaining frozen backbones.The cited approaches include L2P, DualPrompt, CODA-Prompt, and related parameter-efficient adapters.
  • Prompting and pre-trained learners: Strong frozen features combined with careful classifier fitting recover substantial CIL performance in pipelines such as SLCA, SLCA++, and RanPAC.SPARCL is positioned as complementary because it modifies the analytic update rule rather than the backbone adaptation recipe.
  • Analytic continual learning: Analytic continual learning builds on ACIL’s recursive ridge regression and extends it to kernels, online streams, long-tailed regimes, large models, speech, and other settings.The related-work discussion presents analytic learning as exemplar-free and computationally efficient.

3. Preliminaries

The preliminaries define exemplar-free class-incremental learning with a frozen visual encoder and a recursively updated linear ridge classifier. All classes share the inverse autocorrelation geometry, creating coupling across tasks.

  • Class-incremental setting: Each task introduces disjoint classes, each sample is observed once, and only the linear classifier is updated across tasks.The visual encoder remains frozen throughout the class-incremental stream.
  • Class-incremental setting: The task feature matrix Φt and one-hot label matrix Yt represent features and active classes at step t.Φt has nt rows and d feature columns, while Yt has nt rows and ct active-class columns.
  • Analytic ridge formulation: ACIL-style methods maintain autocorrelation and cross-correlation statistics incrementally for recursive ridge fitting.These sufficient statistics support closed-form updates without revisiting earlier samples.
  • Analytic ridge formulation: Woodbury identities update (Rt + λI)^-1 without repeated d × d inversions, yielding efficient deterministic classifier updates.This efficiency motivates analytic CIL in privacy-sensitive or compute-limited settings.
  • Analytic ridge formulation: Because every class shares one inverse autocorrelation operator, old classifiers can change when Rt’s geometry changes even if old label statistics stay fixed.This shared inverse creates the coupling that motivates spectral partitioning.

4. SPARCL

SPARCL partitions accumulated feature energy into core and residual eigenspaces, freezes old-class core coefficients, and updates only residual statistics analytically. Optional residual projections add capacity without altering the protected core.

  • Spectral partition: SPARCL eigendecomposes accumulated autocorrelation and selects the smallest core width k meeting an energy threshold τ.The leading eigenvectors define the core projector P∥, and the orthogonal complement defines the residual projector P⊥.
  • Spectral partition: The core stores dominant accumulated representation directions where inverse-gain changes most strongly affect old logits, while the residual captures lower-energy or newer directions for plasticity.This assigns stability to the core and adaptation to the complement.
  • Core freeze and residual-only update: SPARCL freezes old-class core coefficients and updates only the residual block, while new classes can use the full residual pathway.New classes do not alter old-class core coefficients.
  • Residual update: Residual features and sufficient statistics support an analytic update whose final classifier combines frozen old core terms with updated residual and new-class terms.The supplied method passages identify the residual block as the updated component.
  • Core freeze and residual-only update: Equation (11) enforces stability only on spectrally dominant components while leaving analytic adaptation unconstrained in the complementary block.The design targets geometric interference rather than freezing the entire classifier.
  • Residual expansion: Residual capacity can be expanded with m random orthonormal directions constrained to the residual space, increasing flexibility without disturbing the core.The expansion is applied only to the plastic residual component, unlike full-feature random projection pipelines.
  • Positioning: SPARCL retains running sufficient statistics and a frozen encoder like prior analytic learners but separates memory-bearing directions from adaptation directions.Its distinction is a core/residual partition rather than a homogeneous classifier refit.
  • Positioning: The residual inverse update reduces cubic cost from the full feature dimension to the residual width, with rank-nt Woodbury corrections inside that block.The spectral partition is recomputed periodically or when residual conditioning crosses a threshold.

5. Theory

The theory attributes analytic continual-learning forgetting to spectral interference in the shared ridge inverse and proposes partitioning stable core directions from a plastic residual block.

  • SPARCL update: SPARCL builds a spectral partition from accumulated autocorrelation energy, freezing old-class core components while restricting later updates to the residual subspace.Optional residual random projections restore capacity as task shift accumulates.
  • Interference mechanism: Spectral interference arises when new-task covariance loads onto old eigendirections, shrinking inverse gain and displacing old logits without gradient overwriting.The mechanism applies even when old-class cross-correlation and labels remain unchanged.
  • Guarantee: Residual-only updates preserve the core contribution to old-class logits exactly, while remaining drift is confined to residual feature energy.Because the core captures most accumulated variance, the resulting bound is typically smaller than unrestricted drift.
  • Guarantee: If an input has no residual component, its old-class logits remain exactly unchanged under SPARCL updates.This is the direct special case of the core-logit invariance guarantee.

6. Experiments

Under a frozen ViT-B/16, exemplar-free protocol across four class-incremental benchmarks, SPARCL closes most of the gap between classical analytic learners and strong representation matchers while trading more compute for spectral invariance.

  • Main results: SPARCL closes most of the gap from ACIL–REAL to RanPAC and Fly-CL, and outperforms the strongest matchers on all four datasets.The largest relative lift over REAL occurs on ImageNet-R/A, where spectral interference is expected to be strongest.
  • Domain shift: On ImageNet-R/A, spectral partition recovers several points lost by unrestricted ACIL-style inversion without requiring Fly-CL’s 104-dim KC expansion.These datasets involve faster covariance-geometry changes across sessions.
  • Efficiency: SPARCL lies between F-OAL and Fly-CL in wall-clock training time per task.Partial eigendecompositions and a residual Woodbury solve cost more than Fly-CL’s sparse streaming path but less than RanPAC’s dense RP + CV ridge.
  • Ablations: Removing spectral partition collapses performance to the ACIL-scale regime, while disabling core freeze or residual RP causes smaller but clear losses.The energy threshold peaks near 0.95 because weaker cores leak interference and stronger cores reduce residual plasticity.
  • Mechanism analysis: SPARCL freezes dominant eigenspace directions inside the ridge solve, whereas Fly-CL reduces prototype correlation through sparse expansion before matching.Early sessions benefit most from core freeze, while residual random projections help later sessions needing capacity.
  • Temporal performance: After several CIFAR-100 sessions, ACIL-style curves separate from the compact performance of RanPAC, Fly-CL, and SPARCL.The separation appears after sufficient off-task mass enters the shared inverse.
  • Complementarity: SPARCL and Fly-CL are complementary because SPARCL targets the shared ridge inverse while Fly-CL targets multicollinearity in an expanded feature map.The paper frames these as orthogonal controls on where interference is broken.

7. Conclusion

The paper concludes that analytic continual-learning forgetting is spectral rather than optimization-based and addresses it by partitioning classifier updates into stable core and plastic residual components.

  • Conclusion: SPARCL preserves old-class coefficients in the high-energy core, updates only the residual block, and can expand residual capacity.This directly targets eigenvalue dilution in the shared inverse autocorrelation operator.

A.1. Proof of Proposition 1 (Spectral Interference)

The proof attributes analytic continual-learning drift to changes in shared inverse gains along old-class spectral directions and formalizes residual-only updates that preserve core logits. It also identifies computational and practical boundaries for spectral partitioning.

  • Spectral interference: New energy δi on an eigenmode changes its ridge inverse gain by −δi/((λi+λ)(λi+δi+λ)), producing spectral interference.The resulting perturbation is strongest on modes with large prior eigenvalues.
  • Spectral interference: If δi=0 on every mode carrying old-class energy, the old-logit drift vanishes.This is the proof's zero-interference condition for unaffected old-class directions.
  • Core invariance: Residual-only updates leave old-class logits exactly unchanged whenever the input has no residual component, P⊥ϕ(x)=0.The guarantee follows directly from the residual bound.
  • Core invariance: SPARCL freezes old-class core coefficients while solving projected residual normal equations, including residual–core coupling without altering the frozen block.The constrained formulation uses projected residual operators and a parameterization that separates core and residual coordinates.
  • Complexity and scope: SPARCL costs O(d^2k) for partial eigendecomposition and O(r^2nt+r^3) for residual updates, versus ACIL's O(d^2nt+d^3), when r≪d.Residual random-projection expansion adds O((r+m)^2nt), and infrequent refreshes preserve the computational advantage.
  • Complexity and scope: If the base session is small or atypical, early core directions may be suboptimal, so aggressive freezing can lock in a poor basis.The paper suggests warm-up, trust-region refreshes, or multiple cores, while leaving adaptive repartitioning to future work.

C. Detailed Experimental Settings

The experiments use frozen ViT-B/16 features across class-incremental benchmarks and compare SPARCL with analytic, rehearsal, prompt, and representation-based baselines. Settings also examine longer streams, residual expansion, refresh frequency, spectral diagnostics, efficiency, and failure cases.

  • Protocol: ViT-B/16 features are ImageNet-21K-pretrained and ℓ2-normalized, with a linear or residual-expanded classifier.The classifier protocol keeps the backbone frozen.
  • Configuration: The default configuration uses ridge λ=1.0, energy threshold τ=0.95, residual RP width m=128, two-session partition refreshes, feature dimension d=768, and three seeds.λ is selected from {0.1,1,10}, while τ is selected from {0.85,0.90,0.95,0.98} on a held-out base split.
  • Evaluation: Evaluation reports Avg Acc, final average accuracy, average forgetting, and mean absolute old-logit drift on a fixed old-class probe set.These metrics jointly measure class-incremental performance and the proposed interference mechanism.
  • Results: Longer streams amplify spectral accumulation, and Table 4 reports a larger relative gain for SPARCL consistent with the interference analysis.Table 4 evaluates CIFAR-100 with T=20 and five classes per session.
  • Results: Beyond m=128, returns diminish while memory grows linearly in m.The residual expansion therefore has a capacity–memory trade-off.
  • Results: Refreshing every session improves accuracy by +0.2 but increases wall-clock cost, whereas refreshing only after the base session underperforms under substantial spectrum rotation.Refreshing every two sessions is reported as the practical compromise.
  • Results: SPARCL reaches 94.60 overall Acc on CIFAR-100 under Fly-CL's longer-split protocol, versus 92.80 for REAL and 90.55 for ACIL.Published Fly-CL reaches 94.22 overall Acc under the same protocol.

J. Implementation Notes and Pseudocode Details

SPARCL implements analytic continual learning by partitioning features into frozen core and plastic residual subspaces, with residual-only updates and optional random-projection expansion. The design preserves old-class core logits while retaining efficient recursive ridge computation and controlled capacity.

  • Numerical implementation: Residual Gram factors of size r × r, or (r + m) × (r + m) with random projections, avoid dense d × d inverses.For d = 768 and τ = 0.95, r is typically 40–120 depending on the stream.
  • Residual expansion: Residual random-projection directions are projected onto the complement of the core and QR-orthonormalized, guaranteeing orthogonality to frozen directions.The construction samples G, applies P⊥, and then performs QR orthonormalization.
  • Class bookkeeping: Old-class columns retain frozen core coefficients, while new-class columns start at zero and are fitted in residual and optional random-projection coordinates.Only the columns corresponding to newly active classes expand as the class count grows.
  • Design alternatives: Full feature orthogonalization removes useful shared structure, while SPARCL’s soft residual updates with a large but incomplete core worked better in practice.Maintaining private per-class covariances also sacrifices pooled analytic efficiency and scales poorly with class count.
  • Update rule: SPARCL freezes the core for old classes and restricts subsequent updates to the residual block, with optional residual expansion restoring capacity under task shift.The method preserves plasticity in the residual while treating the core as analytic memory for old classes.
  • Worked example: A 44% relative change can affect an old classifier coordinate without old labels, whereas SPARCL freezes the selected core coordinate and absorbs new mass residually.In the toy example, the core coordinate remains 0.4 and the core logit contribution stays invariant for features aligned with e1.

Q. Dataset and Split Cards

The evaluation spans class-incremental image benchmarks with different visual and covariance challenges under a frozen-feature protocol, while acknowledging deployment and dataset-use boundaries. The dataset cards cover CIFAR-100, ImageNet-100, ImageNet-R, and CUB-200-2011.

  • CIFAR-100: CIFAR-100 contains 100 natural-image classes and uses B0-10 or B0-20 sessions, with average accuracy measured over all seen classes after each session.B0-10 uses 10 classes per session, while B0-20 uses 5.
  • ImageNet-100: ImageNet-100 is a 100-class subset evaluated with the same session protocol as CIFAR-100 and 224 × 224 crops for ViT features.
  • ImageNet-R: ImageNet-R is a rendition and domain-shift benchmark where covariance geometry changes more than class semantics alone would suggest.
  • CUB-200-2011: CUB-200-2011 is a fine-grained bird benchmark testing whether residual capacity suffices when classes are visually close and share many principal directions.
  • Use and ethics: The study frames SPARCL as an algorithmic contribution to exemplar-free learning and notes that privacy-preserving statistics are a side benefit relative to rehearsal.The paper does not endorse surveillance uses and asks that dataset licenses and intended use accompany released code.
  • Positioning: SPARCL is positioned within analytic continual learning while borrowing geometric vocabulary from subspace isolation and remaining complementary to prompt-based and expansion methods.
Loading 2608.21307v1…