Source-linked AI summary
Score-Preserving Continual Lasso under Task Shift
Jingyi Guo, Runze Li, Lingzhou Xue
TL;DR
In high-dimensional sparse regression, continual learning must use sequential tasks without access to earlier raw data, but recursively reused Lasso estimates can carry forward shrinkage. SP-CL preserves the historical squared-loss score, recovering pooled-Lasso estimation and supporting current-task correction under related-task conditions; its evaluation decompositions separate information loss, task conflict, and specialization.
Problem
In high-dimensional sparse regression, recursively reusing an ℓ1-penalized estimator can propagate earlier shrinkage across tasks when historical data are inaccessible.
Source: §1 · pp. 1–2Method
SP-CL separates sparse estimation from state transmission by preserving the historical squared-loss score and applying a current-task correction without changing the transmitted state.
Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 5Results
SP-CL's cumulative estimator exactly matches offline pooled Lasso, while under an oracle related-task condition its specialist attains the offline source-assisted rate.
Source: §1 · p. 2, §4.1 · p. 7, §7 · p. 10Takeaways & Limitations
Overall, BWT, and FWT decompositions distinguish historical information loss, task conflict, and current-task specialization.
Source: §1 · p. 3, §7 · p. 10The source-assisted rate guarantee assumes an oracle related-task regime in which all historical tasks inform the current target.
Source: §4.1 · p. 7
Abstract
from arXiv · showhide
Continual learning (CL) aims to learn from sequential tasks while mitigating catastrophic forgetting when historical data are no longer accessible. We study continual learning for high-dimensional sparse linear regression, where $\ell_1$ regularization creates a difficulty absent from continual ridge regression: that recursively reusing an $\ell_1$-penalized estimator can propagate earlier shrinkage and accumulate regularization bias across tasks. We propose score-preserving continual Lasso (SP-CL), which separates sparse estimation from state transmission by preserving the historical squared-loss score. Under squared loss, its cumulative estimator exactly recovers a full-history Lasso minimizer. When task coefficients coincide, it inherits the full-history sparse rate. For related task-specific coefficients, SP-CL uses a regularized current-task correction that leaves the transmitted state unchanged and attains the offline source-assisted rate under an oracle related-task condition. We also derive Overall/BWT/FWT decompositions that distinguish historical information loss, task conflict, and current-task specialization, and validate the resulting regimes in simulation.
1 INTRODUCTION
Continual learning must balance performance on observed tasks, retention of earlier knowledge, and adaptation to new tasks. This paper addresses how sparse regression can preserve historical information under task shift without transmitting accumulated Lasso shrinkage.
1 INTRODUCTION
- Continual learning seeks strong performance on observed tasks while preserving earlier knowledge and exploiting it on new tasks.These aims are reflected by Overall, BWT, and FWT, respectively.Source: §1 · p. 1
- Prior continual-regression theory covers high-dimensional and heterogeneous tasks, but its regularization mechanism remains quadratic rather than sparse.Related work also treats sequential ℓ1 estimation without task shift and sparse transfer with source samples still accessible.Source: §1 · pp. 1–2
- Recursive transmission of an ℓ1-penalized estimate can propagate penalty subgradients that do not belong to the historical data objective.The paper identifies this as a sparse-regression state-transmission problem, unlike quadratic regularization methods.Source: §1 · p. 2
- Overall, BWT, and FWT must be interpreted jointly because old-task degradation can reflect information loss or movement toward a different current-task optimum.The paper's decompositions distinguish historical information loss, task conflict, and current-task specialization.Source: §1 · pp. 2–3
- SP-CL separates sparse estimation from state transmission, preventing earlier Lasso shrinkage from contaminating the historical state.Its cumulative estimator matches offline pooled Lasso, and a current-task correction adapts without changing the transmitted state.Source: §1 · p. 2
- Under related-task conditions, a sparse current-batch correction attains the offline two-stage rate using the continual state and current batch.The correction adapts the cumulative estimate toward the current coefficient without modifying the transmitted state.Source: §1 · p. 2
2 PROBLEM SETUP AND EVALUATION
The setup is sequential high-dimensional sparse regression: each arriving task supplies a batch, while earlier raw observations cannot be revisited. Evaluation uses prediction error across observed tasks and standard transfer comparisons.
2 PROBLEM SETUP AND EVALUATION
- At each sequential task, the learner observes a batch and may use it with the transmitted state, but cannot revisit earlier raw observations.The regime allows dimension p to exceed per-task sample size, with sparse task-relevant coefficients.Source: §2 · p. 3
- Task performance is measured by population mean-squared prediction error, with excess prediction error differing from MSE by the same-task irreducible baseline.For tasks observed through time t, the best single population coefficient minimizes their weighted MSE.Source: §2 · p. 3
- Overall performance averages task MSE across tasks observed so far, weighted by each task's share of the cumulative sample size.With equal batch sizes, this is the usual average task MSE.Source: §2 · p. 3
- Overall evaluates the current predictor on observed tasks, while BWT compares old-task performance with its task-time reference and FWT compares against current-only fitting.Positive BWT or FWT indicates improvement relative to its reference; the metrics need not rank predictors identically when task parameters differ.Source: §2 · p. 3
3 SCORE-PRESERVING CONTINUAL LASSO
SP-CL separates sparse estimation from state transmission, preserving historical data scores without carrying forward taskwise Lasso shrinkage. Its cumulative estimator matches an offline pooled-Lasso minimizer, while a current-task correction supports adaptation under task shift.
3 SCORE-PRESERVING CONTINUAL LASSO
- Direct-center transmission mixes the full-history data score with accumulated taskwise L1-penalty subgradients, which need not grow monotonically.A score carrier represents the historical squared-loss objective while excluding regularization contributions.Source: §3 · p. 4
- With a fixed nonzero sparse signal, aligned inherited shrinkage can eventually zero every active coordinate despite more tasks; offline pooled Lasso's threshold falls with cumulative sample size.The construction assumes noiseless tasks with equal batch sizes and a design orthogonal on active coordinates.Source: §3 · p. 4
- Figure 1 shows substantial attenuation of true active coefficients under direct-center transmission, while SP-CL remains aligned with offline pooled Lasso.The noisy, correlated-design experiment uses p = 200, s = 10, n_t = 40, and T = 30.Source: §3 · p. 5, Figure 1 · p. 5
3.2 SCORE-PRESERVING CONTINUAL LASSO
- SP-CL retains L1 regularization for sparse fitting but transmits historical curvature and data score without accumulated penalty contributions, making each cumulative fit an offline pooled-Lasso minimizer.The pooled-Lasso benchmark requires historical raw batches, whereas SP-CL reconstructs its objective from the transmitted state.Source: §3.2 · pp. 5–6
3.3 CONSEQUENCE UNDER A SHARED COEFFICIENT
- With a shared task coefficient, task mismatch disappears; the section compares sparse estimation in this setting under pooled-design, noise, and sample-size conditions.The stated conditions include sparsity and a well-conditioned pooled design and noise.Source: §3.3 · p. 6, Table 2 · p. 7
4 CURRENT-TASK TRANSFER UNDER RELATED TASK SHIFT
The transfer analysis considers task-specific coefficients under a shared-covariance design and an oracle regime where historical tasks are informative for the current target. SP-CL corrects its cumulative predictor using the current batch, with estimation scales reflecting both accumulated information and task discrepancy.
4.1 DESIGN CONDITION AND THE RELATED-TASK CLASS
- The related-task regime assumes independent sub-Gaussian design rows with common well-conditioned covariance and errors; h measures the largest L1 discrepancy from the current task.Smaller h means more transferable history, and the oracle regime treats all historical tasks as informative for the current target.Source: §4.1 · p. 7
4.2 CURRENT-TASK TRANSFER
- The adapted predictor adds a sparse correction estimated from the current batch to the cumulative predictor, while the current-only Lasso provides a comparison.The cumulative predictor uses historical information through the score-preserving state.Source: §4.2 · p. 7
- The specialist's estimation scale is sL_t,δ/N_t + (h q_t ∧ h^2); more history reduces its first term, while larger task discrepancy increases its second.Under polynomial growth of task count and inverse confidence, L_t,δ is of order log p; the correction inherits the offline two-stage rate without revisiting history.Source: §4.2 · p. 8
5 CONTINUAL EVALUATION UNDER TASK HETEROGENEITY
The section relates generalist and specialist performance across Overall, FWT, and BWT, while separating unavoidable task conflict from avoidable deviation from the best cumulative single head.
Task conflict
- Task conflict TCt is the unavoidable loss from representing heterogeneous tasks with one coefficient; the remaining deviation measures avoidable departure from the best cumulative single head.The analogous historical decomposition centers conflict at w⋆_{t−1}.Source: §5 · p. 8
Generalist–specialist metric relations
- The specialist’s FWT gain equals its current-task gain, while Overall performance expresses a trade-off between current gain and historical cost.Under equal batch sizes, BWT also isolates task conflict; even with perfect historical information, BWT can equal −TC_{t−1}.Source: §5 · p. 8
Task conflict
- Old-task degradation need not indicate lost transmitted information, because task conflict alone can produce negative BWT.A budgeted interpolation between the generalist and specialist outputs is given in Appendix F.Source: §5 · p. 8
6 EXPERIMENTS
The experiments evaluate sparse estimation, transfer under task shift, and continual performance using analytic population risks over 50 streams. SP-CL is compared with pooled Ridge and current-task fitting, while generalist and specialist outcomes are assessed across task heterogeneity.
Setup
- The experiments use Gaussian AR(1) covariates, sparse coefficients, and analytic population risks over 50 streams; tuning, solver, and sensitivity details are in Appendix B.The sparse-estimation setting uses a shared s=10 coefficient with p=1000, nt=40, T=20, and NT=800<p.Source: §6 · p. 8
Sparse high-dimensional estimation
- At the final task, SP-CL excess risk is 0.115 versus 2.92 for pilot-tuned pooled Ridge, with about 20 effective coordinates versus roughly 950–990.The experiment has p=1000 and NT=800<p; pooled Lasso matches SP-CL.Source: Figure 2 · p. 9, §6 · p. 9
Transfer under task shift
- Pooling benefits small task mismatch; at intermediate mismatch, current-task correction can rescue negative transfer, but severe shift with little history can leave the corrected estimator worse than current-only fitting.The transition reflects a reduction in estimation error from more history and increased correction cost from larger task mismatch.Source: Figure 3 · p. 9, §6 · p. 9
Continual-learning performance
- At h=12, specialist FWT_S=8.23 vs. FWT_G=6.45 and BWT_S=−3.23 vs. BWT_G=+0.98; Overall favors the generalist throughout.The comparison covers final-task Overall, FWT, and BWT as h varies.Source: §6 · p. 9, Figure 4 · p. 10
7 CONCLUSION
The conclusion presents SP-CL as separating sparse estimation from state transmission, matching offline pooled Lasso cumulatively while supporting current-task adaptation and decomposing continual-learning metrics.
Conclusion
- SP-CL separates sparse estimation from state transmission to prevent recursively transmitted ℓ1-penalized estimates from carrying past shrinkage.The conclusion treats regularization for estimation and information retained for future tasks as distinct objects.Source: §7 · p. 10
- The cumulative SP-CL estimator exactly matches offline pooled Lasso, and a current-task correction provides statistically efficient adaptation under task shift.Source: §7 · p. 10
- Overall, FWT, and BWT decompositions distinguish information retention, task conflict, and specialization.Source: §7 · p. 10
AI USE STATEMENT
The appendix documents the authors’ use of generative AI and develops results on score transmission, pooled-Lasso equivalence, and its statistical consequences. It also reports noisy-design diagnostics and sensitivity to the penalty constant.
AI USE STATEMENT
- The authors used generative AI to help develop the framework, organize claims, design the initial protocol, and draft and edit the manuscript.They report manually reviewing AI-assisted content and taking responsibility for the final text, mathematical claims, and empirical artifacts.Source: AI USE STATEMENT · p. 10
A.3 EXACT POOLED-LASSO EQUIVALENCE
- SP-CL’s cumulative estimator is exactly a full-history pooled-Lasso minimizer, supporting the standard pooled-Lasso sparse rate under the stated conditions.The proof identifies the smooth part of the update with the full-history squared loss before applying the Lasso penalty.Source: §A.3 · p. 12, §A.5 · p. 12
A.4 EXACT COMPRESSED STATE UNDER COMMUTING EMPIRICAL TASK HESSIANS
- Under commuting empirical task Hessians with a fixed known common eigenbasis, the offline pooled-Lasso minimizer can be represented by a 2p + 1-dimensional transmitted state.If the dense basis must also be stored, total persistent storage remains O(p^2).Source: §A.4 · p. 12
B.1 PERSISTENT-SIGN LOWER BOUND
- Persistent signed-support recovery and ℓ2 consistency are incompatible whenever st log p/n0 does not tend to zero on the usual score-concentration event.The obstruction concerns the active block even when the full cumulative Gram matrix remains singular.Source: §B.1 · p. 13
B.2 SUPPORT EXTINCTION
- In the underdetermined construction, direct-center transmission eventually drives every active coordinate to zero and keeps it there.The construction remains batchwise underdetermined for arbitrarily large p because only the active block must be orthogonal.Source: §B.2 · p. 13
B.3 ADDITIONAL STATE-TRANSMISSION DIAGNOSTICS
- At the final task, direct-center’s mean population MSE is 10.99 and its true-positive rate is 0.046, versus 1.06 and 1.000 for both SP-CL and offline pooled Lasso.The SP-CL/offline pooled-Lasso objective gap remains at floating-point precision in the noisy, correlated experiment.Source: §B.3 · p. 14
- Across tested penalty constants, SP-CL’s score discrepancy remains at floating-point precision while direct-center’s remains positive; support collapse and near-null MSE occur at cλ ≳ 0.75 in this design.At cλ = 0.25, direct-center retains 66% of true coefficient magnitude on average and has MSE 2.2.Source: §B.3 · p. 14
B.4 ADDITIONAL SPARSE-EFFICIENCY RESULTS
The appendix adds sparse-efficiency results, specifies the task-shift simulation, and establishes a conflict bound for a single coefficient across tasks.
B.4 ADDITIONAL SPARSE-EFFICIENCY RESULTS
- SP-CL’s squared estimation-error ordering agrees with the risk comparison in Figure 2; its final-task false discovery rate is approximately 0.47 in this experiment.The experiment quantifies finite-sample prediction benefit when p > NT, rather than exact variable selection.Source: §B.4 · pp. 14–15, Figure 7 · p. 15
B.5 TASK-SHIFT SIMULATION CONSTRUCTION
- The task-shift experiment varies mismatch h and historical-information ratio m across fixed-support coefficient paths, averaging 50 common-random-number replicates per cell.The current-only fit is shared across m values for each fixed (h, seed).Source: §B.5 · p. 15
B.6 UNAVOIDABLE SINGLE-HEAD CONFLICT
- For two tasks with distinct population optima, any single coefficient incurs nonzero loss on at least one task.This is the stated unavoidable single-head conflict.Source: §B.6 · p. 15
C PROOF OF THEOREM 2
The proof reduces SP-CL’s first stage exactly to offline pooled Lasso and transfers the related-task argument to the continual setting. The appendix also states conditions for signed-support recovery under a shared coefficient.
C PROOF OF THEOREM 2
- By Theorem 1, the continual pair equals the offline two-stage source-assisted procedure on the same data, making this exact reduction the streaming-specific step in the proof.The remaining argument follows the offline proof after making its confidence level explicit.Source: C PROOF OF THEOREM 2 · p. 16
- The related-task bound is established on a joint high-probability event under the stated sample-size and mismatch conditions, without treating the pooled estimator as independent of the current design.The pooled estimator uses the current batch through its objective.Source: C PROOF OF THEOREM 2 · p. 16
D.1 CONDITIONS AND PROOF FOR SHARED-COEFFICIENT SIGNED-SUPPORT RECOVERY
- The shared-coefficient signed-support claim uses additional irrepresentability, beta-min, and sample-size conditions, after which the usual primal–dual-witness argument applies.The proof relies on the exact pooled-Lasso reduction.Source: §D.1 · p. 17
D.2 RELATED-TASK SELECTION
The appendix gives related-task selection conditions, explains metric-decomposition proofs and partial correction, and reports a climate-stream application. In that application, SP-CL matches offline pooled Lasso numerically, while the short future-block comparisons are descriptive.
D.2 RELATED-TASK SELECTION
- Under the stated related-task selection conditions, the appendix reports a support-recovery result with probability 1 − o(1).The conditions include a sparse anchor and support compatibility between the anchor and contrast.Source: §D.2 · p. 18
E PROOF OF THE METRIC DECOMPOSITION
- The metric-decomposition proof derives the identities by subtracting the FWT definitions and reweighting historical and current MSE terms; for BWT, irreducible terms cancel under equal batch weights.Source: E PROOF OF THE METRIC DECOMPOSITION · p. 18
F OPTIONAL PARTIAL CORRECTION UNDER A HISTORICAL BUDGET
- A partial correction interpolates between the generalist at η = 0 and the specialist at η = 1 without changing the transmitted state.Its population trade-off separates unavoidable single-head conflict from the choice of how far to adapt toward the new task.Source: F OPTIONAL PARTIAL CORRECTION UNDER A… · p. 19
G.1 DATA AND STREAM CONSTRUCTION
- The climate application predicts one-week-ahead Los Angeles temperature anomalies from North Pacific/North America sea-level pressure, using eight chronological tasks and a separate 2025–2026 future block.The future block contains 41 weekly pairs from two winters.Source: §G.1 · p. 19
G.2 EVALUATION AND TUNING √
- The climate evaluation compares four Lasso estimators and two baselines, tunes penalties on training blocks only, and uses paired or moving-block bootstrap uncertainty estimates.Source: §G.2 · pp. 19–20
G.3 PREDICTION AND HISTORICAL RISK
- On the future block, SP-CL coincides with offline pooled Lasso to numerical tolerance, with score-transmission gap 3.6 × 10^-11 versus 0.391 for direct-center; future comparisons are descriptive given the short block.On historical test blocks, direct-center’s average excess MSFE relative to offline pooled Lasso is 0.430, with 95% interval [0.059, 0.861].Source: §G.3 · p. 20
G.4 SPATIAL DIAGNOSTICS AND LIMITATIONS
Climate-stream diagnostics offer exploratory views of future-block performance, historical forgetting and regret, covariance dissimilarity, and spatial support and sign patterns. The application uses correlated pressure-grid proxies from one chronological stream and does not establish causal attribution or confirm common covariance.
G.4 SPATIAL DIAGNOSTICS AND LIMITATIONS
- The covariance-distance means (0.268 between tasks; 0.295 within-task split halves) and calibrated values (0.306 versus 0.313) are close, making the shift diagnostic descriptive rather than confirmatory.The application illustrates score transmission on one real chronological stream; it neither verifies the common-covariance approximation nor rules out broader climate-regime drift.Source: §G.4 · p. 20
- Figure 8 reports future-block MSFE against zero-anomaly and persistence baselines, historical forgetting and regret, and pairwise covariance dissimilarities across chronological tasks.The common penalty is λ_t = 0.20/.Source: Figure 8 · p. 21
- Figure 9 compares final supports spatially, marking predictors selected only by Offline pooled Lasso/SP-CL versus Direct center and locating the SLP window edge and Los Angeles.Its heatmaps show [A_t,j − A_j,j]+ for SP-CL and Direct center on a common color scale.Source: Figure 9 · p. 22
- Figure 10 maps positive and negative final coefficients in red and blue, with point sizes on a common absolute-magnitude scale.The sign maps separate coefficient sign from the method-overlap comparison in Figure 9.Source: Figure 10 · p. 22