Task-Anchored Representation Shaping for Pre-Trained Model-Based Continual Learning

Zhiming Xu; Huiyu Yi; Zhen-Hao Xie; Baile Xu; Furao Shen; Jian Zhao; Suorong Yang

Source-linked AI summary

Task-Anchored Representation Shaping for Pre-Trained Model-Based Continual Learning

Zhiming Xu, Huiyu Yi, Zhen-Hao Xie, Baile Xu, Furao Shen, Jian Zhao, Suorong Yang

arXiv:2608.16345v1Aug 17, 2026cs.LG

TL;DR

Cross-task ambiguity remains a bottleneck in continual learning when task boundaries are semantically entangled. TAILS reshapes representations using soft, task-anchored latent recall, improving diverse baselines by up to 5.05 percentage points with modest storage overhead.

Problem
Semantically entangled task boundaries make inputs overlap across tasks, leaving task inference ambiguous as continual learning accumulates knowledge.
Method
TAILS composes soft, input-conditioned evidence from fixed task anchors to correct features before prediction without changing the classifier or prediction mechanism.
Results
Up to 5.05 percentage points improve diverse baselines, while TAILS adds as little as 3.0% storage overhead and marginal inference overhead.
Takeaways & Limitations
TAILS provides an efficient representation-level approach that consistently improves evaluated PTM-based continual learners across diverse settings.
Takeaways & Limitations
The setting assumes task IDs are unavailable at inference and previous task images become inaccessible after moving to later tasks.

Abstract

from arXiv · show

Pre-trained models (PTMs) provide a strong foundation for continual learning by offering stable representations that facilitate lightweight adaptation to new tasks. However, adapting well to each task does not ensure reliable inference over all learned tasks. Since task boundaries are often artificial and semantically entangled, an input from an unknown task can remain ambiguous even with strong PTM features, making cross-task prediction a key bottleneck. We propose Task-Anchored Inference Latent Shaping (TAILS), a lightweight post-PTM module that can be integrated into diverse continual learners and optimized through a decoupled step. TAILS uses fixed task anchors as persistent references to accumulated knowledge. It interprets each sample's feature representation relative to these references, then composes relevant evidence across tasks into latent recall. Rather than selecting a task-specific path or adjusting classifier outputs, TAILS uses latent recall to directly correct the feature representation before prediction. It therefore resolves cross-task ambiguity at the representation level, while leaving the original PTM, method-specific modules, and classifier unchanged. Extensive experiments across multiple PTM-based continual learning paradigms show that TAILS can improve classification and task-inference performance with modest parameter overhead and negligible inference cost.

1 Introduction

Continual learning must preserve prior knowledge while adapting to evolving data, but task-wise adaptation does not resolve ambiguous inference across semantically entangled tasks. TAILS addresses this bottleneck by shaping representations with input-conditioned recall from accumulated task structures, improving performance with modest overhead.

Motivation: Continual learning targets sequential knowledge integration without erasing previously acquired knowledge, yet incremental task-specific data can limit feature-space stability.Pre-trained models provide a stronger foundation through representations learned from large-scale data.
Cross-task inference bottleneck: Task-wise adaptation reduces intratask errors, shifting the main challenge toward identifying and exploiting relevant knowledge across tasks without task identity.Inference must cover all accumulated tasks, unlike training data organized into distinct tasks.
Cross-task inference bottleneck: Random task assignments can create semantically overlapping feature distributions, making task inference increasingly ambiguous as more tasks accumulate.The learner must distinguish among a growing number of semantically entangled task distributions.
TAILS: TAILS is a lightweight plug-and-play module inserted after the PTM encoder and before the classifier that composes task evidence into latent recall for feature correction.It avoids routing samples to a discrete task-specific component and instead uses a soft, input-conditioned summary across accumulated tasks.
Effectiveness and efficiency: 5.05 percentage points is the maximum improvement TAILS delivers across diverse state-of-the-art baselines, including TUNA and CL-LoRA.3.0% additional storage overhead is the reported minimum when TAILS is added to storage-heavy base learners.

2 Related Work

Related work spans replay and adaptation strategies for continual learning, increasingly leveraging pre-trained models through prompts and lightweight adapters. Existing retained statistics primarily support retrieval and calibration, whereas TAILS uses task anchors for sample-specific latent recall before prediction.

Continual learning: Continual learning methods preserve prior capabilities over non-stationary streams through complementary mechanisms, including replay-based retention of representative samples or reconstructed historical distributions.The passage frames replay as one broad family among diverse continual-learning approaches.
Pre-trained models in continual learning: Prompt-based methods adapt frozen pre-trained models, while adapter-based methods insert lightweight modules for efficient task adaptation.Examples include L2P, DualPrompt, CODA-Prompt, APT, and EASE.
TAILS: TAILS retains task anchors and composes query-dependent latent recall to shape representations before prediction by the original classifier.Anchors comprise retained class prototypes and features accumulated after each task is learned.
Adapter-based methods: Modular adapter methods address distribution shifts, specialization, and knowledge sharing through dynamic expansion or combinations of task-specific and universal adapters.SEMA dynamically expands adapters, whereas TUNA combines task-specific and universal adapters.
Limitations of retained statistics: Retained class-alignment statistics mainly support adapter retrieval and classifier calibration, rather than directly resolving sample-specific ambiguity at inference.MOS and TUNA preserve these statistics as a form of non-sample replay.

3 Preliminaries

Continual learning trains on sequential data streams while requiring predictions over all accumulated classes without task IDs at inference. In PTM-based continual learning, a frozen pre-trained encoder supports lightweight adaptation using only current-task data, with prior images inaccessible.

Continual learning formulation: A continual learning model processes evolving streams D_1, D_2, · · ·, D_t, treating each current stream D_t as a new task with label space Y_t.The accumulated data and label spaces are D_1:t = D_1 ∪ D_2 ∪ · · · ∪ D_t and Y_1:t = Y_1 ∪ Y_2 ∪ · · · ∪ Y_t.
Continual learning formulation: The learned model f_t: X → Y_1:t is evaluated by classification error on the accumulated test set.Its objective is to perform well across all learned data rather than only the current stream.
Class-incremental setting: In class-incremental continual learning, task IDs are unavailable at inference, so predictions must use only the sample among all classes.This setting makes cross-task class selection part of inference.
PTM-based continual learning: PTM-based continual learners extract representations with a pre-trained encoder ϕ_ptm and predict labels using a classifier G.The pre-trained encoder provides a stronger foundation for continual adaptation.
PTM-based continual learning: These methods typically freeze the PTM backbone, add lightweight fine-tuning modules, and train only on current-task data D_t because previous images become inaccessible.The learner adapts sequentially as it moves from one task to the next.

4 Method

TAILS is a unified post-PTM feature-space module that uses task anchors to retrieve and compose historical evidence, then corrects ambiguous representations before prediction. It is incrementally optimized while preserving the base learner, classifier, and original prediction mechanisms.

Core module: TAILS repositions ambiguous inputs in the accumulated class space through auxiliary feature-space inference, rather than routing tasks or calibrating classifier outputs.The base learner and its original prediction mechanism remain unchanged.
Anchor construction: Task anchors retain class prototypes and balanced support features, providing persistent references for interpreting current representations against accumulated task structures.If a class has fewer than M samples, all its features are retained.
Latent recall: TAILS projects anchors into a shared query space, summarizes each task into learnable slots, and uses soft relevance to form input-conditioned latent recall.Because task-slot relevance is soft, latent recall can combine evidence from multiple tasks whose semantic neighborhoods cross procedural boundaries.
Decoupled optimization: After each task, TAILS alone is optimized using current representations and fixed historical anchors while the representation learner and classifier remain frozen.Historical anchors provide stable references without retaining previous training samples.
Training objectives: Three constraints jointly preserve compatibility with the frozen classifier, recall-address consistency, and continuity of pre-trained semantics across tasks.The objective trains feature corrections rather than a new classifier and does not require task IDs at inference.
Integration: TAILS integrates across heterogeneous PTM-based learners and classifier types by reshaping representations supplied to the original prediction procedure without structural modification.For prototype-based classifiers, class prototypes are transformed by the same mapping to preserve sample–prototype consistency.

5 Experiments

Experiments show that TAILS broadly improves cross-task inference and final classification across PTM-based continual learners, including long-sequence and strong-baseline settings. Ablations further attribute its gains to complementary objectives and mechanism design, while showing modest inference, storage, and hyperparameter costs.

5.2 Comparison with PTM-based CL methods: TAILS improves cross-task prediction and final classification across diverse PTM-based continual learning paradigms, including the 40-task ImageNet-R benchmark and strong baselines such as TUNA and CL-LoRA.High-performing baselines often have strong within-task prediction, making cross-task inference the main performance bottleneck.
5.3 Ablation studies: Lcls, Ltask, and Lstab all contribute to final performance, with the relative importance of Lcls and Ltask varying across base learners.Lcls matters more for SimpleCIL, Ltask is critical for TUNA, and Lstab limits excessive representation drift.
5.3 Ablation studies: TAILS consistently outperforms Anchor-MLP, Anchor Retrieval, Single-Center, and Logit Calibration under matched anchor and optimization budgets.The variants respectively alter the architecture, retrieval operation, task representation, or prediction space, supporting feature-level latent recall shaping.
5.3 Efficiency and storage: 3.0% overhead for MOS, 3.2% for TUNA, and 8.6% for ACMap are incurred when adding TAILS relative to the corresponding base learners.TAILS adds only marginal inference overhead, and its plug-in storage overhead remains below the learner-specific storage of every compared method.
5.4 Hyperparameter sensitivity: TAILS improves the base learner even without support anchors, while increasing support-anchor counts generally improves final classification accuracy by capturing within-class variation.Class prototypes can serve as task anchors, whereas additional support anchors enable richer task-slot representations for latent-recall composition.

6 Conclusion

TAILS is a task-anchored latent-recall inference module for PTM-based continual learning that corrects current representations using accumulated task-level information. It compares representations with fixed task structures, composes input-conditioned recall, and updates shared knowledge incrementally through a decoupled process.

6 Conclusion: TAILS compares each current representation against fixed structures accumulated from learned tasks.These task anchors provide persistent references for interpreting new representations.
6 Conclusion: TAILS composes input-conditioned latent recall and internalizes the recalled evidence as a recall-conditioned feature correction.The module therefore refines representations before downstream inference.
6 Conclusion: TAILS is shared across learned tasks and incrementally updated through a decoupled process.This design allows accumulated task-level information to refine current representations.