PaperlayerView exact revision on arXiv

Source-linked AI summary

MePo++: Unifying Representation Refinement and Reconciliation for General Continual Learning

Guanglong Sun, Kanglei Zhou, Liyuan Wang, Qi Cheng, Hongwei Yan, Shuang Cui, Hang Su, Jun Zhu, Yi Zhong

arXiv:2609.05075v1cs.AI

TL;DR

PTM-based GCL must learn from evolving, blurry streams despite gaps between static pretraining and continual adaptation, and unreliable output alignment. MePo++ addresses these gaps with MetaPrep for unsupervised pre-deployment refinement and StreamAlign for online reconciliation to a stable geometry. It consistently outperforms strong baselines across diverse PTMs, datasets, learners, and continual-learning settings, while requiring additional preparation cost and unlabeled upstream data and relying on a fixed geometry prior.

  • Problem

    PTM-based GCL must handle upstream-downstream misalignment and unreliable alignment under blurry streams, where task identities, boundaries, and repeated data access are unavailable.

  • Method

    MePo++ combines MetaPrep, which uses pseudo-sequence construction and bi-level meta-refinement, with StreamAlign, which reconciles online features against a stable geometry prior.

  • Results

    MePo++ surpasses the strongest baselines in all 18 combinations of three PTMs, three datasets, and two continual-learning metrics, averaging 7.93 percentage points of improvement.

  • Takeaways & Limitations

    MePo++ provides a general post-training framework for PTM-based GCL across standard, few-shot, and CLIP-based continual-learning settings.

  • Takeaways & Limitations

    MetaPrep adds pre-deployment cost and requires unlabeled upstream data, while StreamAlign's fixed geometry prior may become less representative under severe or long-term distribution shifts.

Abstract

from arXiv · show

General continual learning (GCL) aims to learn from evolving data streams without task identities, explicit boundaries, or repeated access to previous data, making it a realistic yet challenging setting for continual intelligence. Although pretrained models (PTMs) provide rich prior knowledge for addressing the limited supervision and non-stationary nature of GCL, existing PTM-based methods often directly adapt pretrained representations and overlook two critical gaps: the misalignment between upstream pretraining and downstream continual adaptation, and the unreliability of conventional output alignment under blurry streams. Here we propose MePo++, a unified post-training framework that bridges pretrained knowledge and downstream GCL through representation refinement and reconciliation. MePo++ introduces two complementary components: MetaPrep, which improves representation plasticity for continual adaptation through unsupervised meta-refinement over pseudo continual sequences; and StreamAlign, which reinforces representation stability by reconciling evolving online features with a stable pretrained geometry. By improving representation learnability before adaptation and preserving alignment during continual learning, MePo++ enables PTMs to remain both plastic for new concepts and stable over evolving streams. Experiments across diverse PTMs, datasets, and continual learning baselines demonstrate the consistent effectiveness and generality of MePo++ for PTM-based GCL. Our code is available at https://github.com/SunGL001/MePo_Plus.

I. INTRODUCTION

General continual learning targets evolving, blurry streams without task identities or repeated data access, exposing limits in existing continual-learning assumptions. MePo++ addresses PTM-based GCL through pre-deployment representation refinement and online representation reconciliation, with consistent gains across evaluated settings.

  • Motivation: GCL removes explicit task identities and boundaries, requiring adaptation and retention as concepts and distributions evolve in single-pass streams.These conditions make GCL more realistic but harder than conventional task-incremental continual learning.
  • Motivation: Existing methods often rely on task-wise data or train representations from scratch, reducing transferability, sample efficiency, or storage and privacy efficiency.PTMs offer transferable prior knowledge, motivating the question of how pretrained knowledge can support realistic GCL.
  • Challenges: PTM-based GCL faces upstream-downstream misalignment because static pretrained representations may lack continual adaptability, while output alignment is unreliable under blurry streams.The two gaps motivate balancing plasticity for new concepts with stability during evolving streams.
  • MePo++: MePo++ prepares representations before deployment with MetaPrep and aligns them during online learning with StreamAlign, without task identities or task boundaries.MetaPrep uses unlabeled-feature clustering and bi-level meta-refinement; StreamAlign aligns online features to a stable geometry while preserving semantic consistency.
  • Evaluation: MePo++ consistently improves representative GCL baselines across three PTMs, three datasets, and two metrics, including standard, few-shot, and CLIP-based settings.The reported evaluation includes comparisons, ablations, sensitivity tests, and representation analyses across diverse continual-learning conditions.

B. Continual Learnability of Pretrained Representations

PTM-based GCL requires representations that are prepared for continual adaptation and remain aligned during blurry, evolving streams. MePo++ addresses these needs through MetaPrep before deployment and StreamAlign during online learning.

  • Continual Learnability: Existing post-training strategies provide limited or inconsistent continual-learning gains across CIFAR-100, ImageNet-R, and CUB-200.Joint Tune, Seq. Tune, and OML are compared against the original DualPrompt baseline using the same Sup-21K backbone.
  • Continual Learnability: MetaPrep consistently improves both AAUC and ALast, indicating that pretrained representations benefit from dedicated continual-adaptation optimization.The empirical analysis reports performance changes relative to the original DualPrompt baseline.
  • Representation Alignment: Conventional output alignment is unreliable under blurry streams because class-wise statistics accumulated during adaptation can be incomplete and unstable.SLCA-Out consistently degrades performance, while MVP-Out and MISA-Out show dataset-dependent behavior.
  • Representation Alignment: StreamAlign consistently achieves larger AAUC and ALast gains by maintaining stable representation alignment during GCL.Across CIFAR-100, ImageNet-R, and CUB-200, it improves AAUC by 2.74%, 3.84%, and 1.78%, and ALast by 7.67%, 6.21%, and 5.80%, respectively.
  • MePo++ Design: MePo++ uses MetaPrep to prepare representations before deployment and StreamAlign to preserve stable geometry during online adaptation.MetaPrep produces a GCL-oriented initialization and stable geometry prior; StreamAlign reconciles transient online features with that prior.

B. MetaPrep: Continual-Oriented Representation Refinement

MetaPrep converts unlabeled upstream data into evolving pseudo-continual experiences and refines the pretrained encoder for post-adaptation performance. It also produces a stable geometry prior for downstream reconciliation.

  • Unsupervised Continual Sequence Construction: MetaPrep clusters current pretrained features into pseudo-classes and organizes them into sequential pseudo tasks with a held-out joint evaluation set.The pseudo sequences are recomputed at every meta-epoch, allowing simulated experiences to evolve with the representation.
  • Bi-level Meta-Refinement: Sequential inner-loop updates deliberately simulate acquisition–interference dynamics, while joint evaluation tests retention across the discovered pseudo-semantic space.The held-out joint objective evaluates the encoder after sequential perturbation rather than only on the latest pseudo task.
  • Bi-level Meta-Refinement: The outer meta-update moves the initialization toward parameters that remain effective after a sequence of continual updates.Repeating this procedure yields the refined encoder θ∗ used to initialize downstream GCL.
  • Stable Representation Prior: MetaPrep extracts pseudo-class prototypes and their between-prototype second-order geometry to form the stable prior Σpre.This prior summarizes global organization among refined pseudo-semantic centers and is computed before downstream adaptation.

C. StreamAlign: Stream-Robust Representation Reconciliation

StreamAlign addresses unreliable online alignment by mapping transient batch geometry toward a stable upstream reference while retaining stream-specific information. Semantic reconciliation then preserves discriminability between the plastic and reference-aligned views.

  • Transient-to-Reference Reconstruction: StreamAlign uses MetaPrep’s stable upstream geometry to reconcile each transient online representation during continual adaptation.This reference reduces reliance on statistics estimated from sparse, blurry online streams.
  • Transient-to-Reference Reconstruction: A covariance-based transformation reconstructs reference-aligned features, while interpolation retains both plastic online information and stable geometric structure.The alignment strength α controls the balance between the current feature and its reconstructed counterpart.
  • Semantic Representation Reconciliation: Semantic reconciliation treats original and reconstructed features as complementary views and uses contrastive learning to preserve class-level agreement.The objective encourages same-class consistency across views while separating representations from different classes.
  • Online Learning Objective: The online objective combines prediction with semantic reconciliation, keeping StreamAlign compatible with different downstream continual-learning algorithms.The prediction loss preserves responsiveness to incoming data, while the contrastive term constrains semantic compatibility with stable geometry.

D. Overall Learning Procedure

MePo++ operates as a two-stage procedure: MetaPrep prepares the pretrained representation before deployment, and StreamAlign maintains geometric stability and semantic discriminability during downstream adaptation.

  • MetaPrep: MetaPrep refines the pretrained encoder on unlabeled upstream data and outputs a GCL-oriented initialization θ∗ together with a stable geometry prior Σpre.These outputs are transferred to downstream continual adaptation.
  • StreamAlign: StreamAlign continuously reconciles transient online representations with pretrained geometry while optimizing the prediction objective.The two stages therefore prepare and maintain the representation at different points in the continual-learning process.

V. EXPERIMENTS

The evaluation covers standard, few-shot, and CLIP-based continual-learning settings, with ablation and representation analyses examining effectiveness and mechanisms under evolving streams.

  • Evaluation Scope: MePo++ is evaluated across standard, few-shot, and CLIP-based continual-learning settings, followed by ablation and representation analyses.The analyses target effectiveness and underlying mechanisms under evolving data streams.

A. Experimental Setting

The experiments evaluate MePo++ under Si-Blurry GCL using three datasets, multiple pretrained backbones and learners, standard metrics, and matched implementation settings. The study also specifies MetaPrep and StreamAlign training details, baselines, and computational conditions.

  • Experiments use CIFAR-100, ImageNet-R, and CUB-200 under the Si-Blurry GCL protocol, covering low-resolution, diverse-rendering, and fine-grained recognition settings.
  • Continual performance is measured by Final Average Accuracy (ALast) and Average Anytime Accuracy (AAUC), with higher values indicating better performance.ALast measures final stream performance, whereas AAUC averages performance throughout learning.
  • Baselines include sequential fine-tuning, linear probing, slow-learner fine-tuning, PTM-based methods, and preliminary MePo, using identical pretrained checkpoints and downstream learners.PTM-based methods use length-5 prefixes in the first five transformer layers for controlled comparisons.
  • MetaPrep uses unlabeled clustered features for pseudo supervision, while StreamAlign uses fixed alignment settings and downstream models train with single-pass Adam.The reported default StreamAlign settings are α = 0.3, λ = 0.002, and τ = 0.05.

B. Comparisons with the State-of-the-Art

MePo++ consistently improves standard, few-shot, and CLIP-based continual learning across datasets, pretrained models, and learners, while its ablations support complementary contributions from representation preparation and reconciliation. The method adds marginal runtime and no learnable parameters over MePo.

  • Standard GCL: MePo++ improves standard GCL across pretrained backbones, datasets, and downstream learners, including 7.92/7.15 points over L2P on CIFAR-100 AAUC/ALast.On ImageNet-R with Sup-21K, the corresponding gains over L2P are 7.63/9.94 points.
  • Few-Shot GCL: Under few-shot GCL with 20% training data, MePo++ improves DualPrompt by 9.14/16.02 points on CIFAR-100 and 4.48/4.77 points on ImageNet-R in AAUC/ALast.The gains remain consistent across checkpoints and learners.
  • CLIP-Based Continual Learning: MePo++ transfers to CLIP-based continual learning, improving both evaluated prompt learners across CIFAR-100 and ImageNet-R.For example, on ImageNet-R it improves DualPrompt by 2.92/3.34 points in AAUC/ALast.
  • Computational Comparison: On CUB-200, MePo++ raises DualPrompt accuracy from 55.73 to 59.05 while increasing batch time from 4.87 s to 5.09 s.It adds no learnable parameters over MePo and incurs marginal runtime.
  • Ablation Study: MetaPrep and StreamAlign are complementary: on ImageNet-R, their combination improves DualPrompt by 8.71/10.06 points in AAUC/ALast, exceeding either component alone.The paper associates MetaPrep with plasticity and StreamAlign with stability.
  • Ablation Study: MetaPrep outperforms alternative representation-preparation strategies, while StreamAlign outperforms no, mask-based, and covariance-only reconciliation.StreamAlign’s additional gains support jointly geometric and semantic reconciliation.
  • Hyperparameter Analysis: Moderate alignment weights perform best: α is effective around 0.3–0.5, while α = 1 suppresses useful plastic information and excessive λ harms adaptation.Performance remains stable across reasonable hyperparameter ranges.

D. Qualitative and Quantitative Analysis

Analyses show that MePo++ produces more discriminative and sparse representations, preserves class structure during reconciliation, and yields a flatter optimization landscape and more confident predictions. The paper also identifies added preparation cost and possible weakness of fixed geometry priors under severe long-term shifts.

  • Representation Quality: MePo++ improves separation scores over MISA across T0–T4, with values increasing from 0.6786–0.5594 to 1.0013–0.8567.The visualizations show more compact class clusters and clearer inter-class boundaries.
  • Representation Sparsity: MePo++ reduces active feature rates across T0–T4 from MISA’s 58.07%–43.62% to 20.96%–13.80%.The analysis characterizes this as concentrating activation on fewer informative features.
  • Representation Reconciliation: StreamAlign preserves the global semantic layout while increasing separation scores by 0.0032–0.0095 across T0–T4.The largest improvements occur at T0 and T4.
  • Optimization Geometry: Compared with DualPrompt, MePo++ has a broader low-loss region around its converged solution, indicating reduced sensitivity to parameter perturbations.The paper links this geometry to improved stability during continual adaptation.
  • Prediction Confidence: On ImageNet-R, MePo++ reaches 48.3% accuracy versus MISA’s 45.6% and exclusively corrects 10.3% of test samples versus 7.6% for MISA.Among samples misclassified by both, MePo++ assigns higher ground-truth probability in 22.5% of all test samples.
  • Limitations: MePo++ requires an additional pre-deployment stage and unlabeled upstream data, while its fixed geometry prior may become less representative under severe or long-term distribution shifts.Future work is proposed to reduce refinement cost and adapt geometry priors using reliable downstream evidence.

APPENDIX A THEORETICAL INTERPRETATION AND ANALYSIS OF MEPO++

The appendix interprets MePo++ as a sequential representation-adaptation framework: MetaPrep prepares initialization through pseudo-sequential evaluation, while StreamAlign operates during downstream adaptation. Its local analyses connect sequential compatibility and gradient interactions to continual learnability without claiming Bayesian inference or global convergence.

  • Lifecycle-Wise Decomposition: MePo++ addresses plasticity and stability at different representation-lifecycle stages rather than explicitly performing Bayesian inference.The Bayesian decomposition is conceptual; sparse mixed batches and unavailable task boundaries motivate separate upstream preparation and downstream alignment.
  • MetaPrep: MetaPrep clusters unlabeled features into pseudo-classes, simulates sequential adaptation, and optimizes held-out cross-sequence performance.This produces an initialization intended to support new-concept acquisition while reducing destructive interference from sparse, mixed updates.
  • MetaPrep: The pseudo-sequential trajectory perturbs initialization according to aggregate pseudo-task adaptation directions, exposing sequential interference beyond shuffled post-training.MetaPrep evaluates whether representations remain effective after successive pseudo-concept updates rather than only minimizing static upstream loss.
  • Local Analysis: Proposition 1 links post-sequence joint improvement to compatibility between sequential adaptation directions and the broader pseudo-semantic objective.A positive local interaction supports joint descent, whereas a negative inner product indicates conflict; the result does not imply universal gradient agreement or elimination of forgetting.
  • Local Analysis: Under an additional approximation, diagonal gradient terms describe individual pseudo-task descent while off-diagonal terms capture local interactions and destructive conflict.For approximately balanced pseudo-tasks, better post-sequence joint performance is associated with less destructive local interaction.
  • Meta-Update: Repeated first-order meta-updates move the encoder toward a state evaluated after sequential perturbation and joint assessment, yielding a GCL-oriented initialization.This differs from ordinary joint post-training because the update incorporates exposure to sequential interference.

C. StreamAlign as Reference-Guided Geometry Transport

StreamAlign transports transient online representations toward a fixed upstream geometry using regularized covariance factors. The reference is operationally stable because it is estimated once, but it can become less representative under severe or prolonged distribution shift.

  • Geometry Transport: StreamAlign compares current-batch covariance with MetaPrep’s fixed upstream covariance to construct reference-guided second-order transport.Regularization makes the covariance matrices positive definite and supports numerical transport when empirical covariances are rank deficient.
  • Stable Geometry Prior: The fixed covariance encodes global between-prototype organization, whereas current covariance reflects sparse, temporally mixed online observations.The prototype-based reference emphasizes variation among pseudo-semantic centers while suppressing within-cluster sample variation.
  • Stable Geometry Prior: The upstream covariance is stable operationally because it is estimated once and held fixed, while the current covariance changes with every incoming batch.The fixed matrix is a target geometry, not an unbiased estimate of instantaneous downstream covariance.

D. Controlled Representation Reconciliation

Controlled reconciliation interpolates between plastic online features and reference-aligned reconstructions, while supervised semantic reconciliation preserves class-level organization that covariance alone cannot determine.

  • Controlled Reconciliation: StreamAlign interpolates each online representation with its reference-aligned counterpart instead of replacing the stream-specific feature outright.This preserves a direct contribution from the plastic representation while introducing structural correction.
  • Controlled Reconciliation: Proposition 2 shows that α controls the correction magnitude: α = 0 leaves the online feature unchanged, α = 1 fully aligns it, and intermediate values apply a proportional displacement.The result establishes controlled interpolation, not an information-preservation guarantee.
  • Trade-off: Aggressive alignment can suppress useful online adaptation because reference-guided transport does not guarantee preservation of all stream-specific discriminative information.Intermediate reconciliation weights retain a direct contribution from the plastic representation.
  • Semantic Reconciliation: Covariance matching alone cannot ensure class compactness or separation because different semantic organizations can share the same global covariance.The semantic objective therefore supplies class-level supervision absent from geometric matching alone.
  • Semantic Reconciliation: Semantic contrastive reconciliation encourages same-label consistency between plastic and reference-aligned views while discouraging collapse across different classes.Each sample contributes two views, allowing a same-label positive pair even when its class appears only once in the current mini-batch.

E. Unified Interpretation

The unified interpretation assigns complementary roles to MePo++’s modules: MetaPrep changes the starting representation, while StreamAlign controls the evolving representation during deployment. Together they prepare future adaptation and constrain transient observations from freely reshaping global geometry.

  • Unified Interpretation: MetaPrep’s local analysis connects post-sequence performance with compatibility between sequential adaptation directions and broader pseudo-semantic descent.Under an additional gradient approximation, the analysis exposes local interactions among pseudo-task updates.
  • Unified Interpretation: StreamAlign combines reference-guided geometry transport with α-controlled influence and semantic reconciliation to support class-level organization during adaptation.The geometric and semantic components address complementary structural and discriminative requirements.
  • Unified Interpretation: MetaPrep changes the starting representation, whereas StreamAlign constrains the trajectory of the evolving representation after deployment.Their roles are complementary rather than redundant: preparation supports future updates, while alignment limits unconstrained geometry changes from transient observations.
Loading 2609.05075v1…