Source-linked AI summary
A Shared Encoder Is Not a Shared Task: Conditional Comparison for Deep Expert Pools
Kentaro Oda
TL;DR
Task-comparison scores guide head reuse, spawning, and merging in systems with shared encoders, but sharing alone does not remove their confound. The paper applies a conditional two-discriminator discrepancy to embeddings and uses its functional and covariate axes for comparison. Across the reported tests, the functional axis tracks mechanism changes without responding to rotations, and separates semantic novelty from photometric shift.
Problem
Deep continual-learning systems use comparison scores to decide whether to reuse, spawn, or merge heads, but sharing an encoder alone does not resolve the score confound.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
In the tested GCD setting, the chunk-level functional axis distinguishes when to open a new class from when to adjust normalization.
Source: §4 · p. 3
Abstract
from arXiv · showhide
Sharing a deep encoder does not, by itself, fix the central confound of task-comparison scores. We show that cross-evaluated heads on a frozen shared representation inherit the extrapolation confound of shallow exchange scores: pure input rotations with fixed labels inflate a deep exchange score from about 0 to 0.80, while representation-novelty scores are blind in the complementary direction (flat under label permutations that change the task completely). Transplanting a conditional two-discriminator discrepancy into the embedding space resolves both blind spots: the functional axis stays within +-0.001 under rotations and tracks label-permutation drift mass monotonically. Built into a mixture-of-heads lifecycle, the two-axis gate attains better decision quality with fewer heads than exchange or novelty triggers at a matched training budget. On generalized category discovery, the same chunk-level functional axis separates semantic novelty from photometric shift with AUROC 0.98-0.99 where per-input OOD scores (MSP, Energy, Mahalanobis, KNN) sit near chance for that distinction. All findings replicate across frozen ImageNet-21k ViT-B/16 and self-supervised DINOv2 backbones on CIFAR-100, and extend to residual adapter pools with recurrence, where a null-calibrated novelty trigger never fires on mechanism changes while the two-axis gate handles them with full recurrence reuse. We state explicitly the common-factoring condition under which embedding-space conclusions transfer to the original mechanism.
1 Introduction
Deep continual-learning systems share encoders while task-comparison scores guide whether to reuse, spawn, or merge heads. Sharing an encoder alone does not remove the confound: exchange and representation-novelty scores miss shifts in opposite directions.
1 Introduction
- In these systems, task-comparison scores determine whether to reuse a head, spawn one, or merge two.Source: §1 · p. 1
- Under pure input rotation with labels fixed, the deep exchange score reaches 0.80 while embedding CJSD stays at zero and the covariate axis absorbs the shift.Under shared support, both deep exchange and embedding CJSD track label-permutation drift mass.Source: Figure 1 · p. 2
- Cross-evaluated heads and representation novelty are one-axis instruments in opposite directions, and sharing an encoder alone need not repair the confound.A frozen encoder can amplify covariate shift for off-distribution inputs.Source: §1 · p. 2
2 The deep confound, measured
With a frozen MNIST encoder, rotations inflate deep exchange scores while label permutations leave Mahalanobis novelty flat; embedding CJSD separates the covariate and functional axes. Transfer of the covariate-null interpretation to the original mechanism requires common factoring.
2 The deep confound, measured
- Under pure rotation with the mechanism untouched, exchange score is 0.15 at 30° and 0.80 at 90°, while embedding CJSD stays within [−0.013, 0.012] across rotations and tracks class-permutation drift.CJSD tracking is attenuated by discriminator capacity, with shrinkage downward in all runs.Source: §2 · p. 2
- Common factoring transfers the covariate-null interpretation only when both tasks’ label conditionals share one function of the representation on the union support.Task-wise sufficiency alone can leave different representation-level conditionals, so the original mechanism need not remain common after collapse.Source: §2 · p. 2
- Predictiveness alone does not establish common factoring under a shifted task, and the experiments support it only approximately for this encoder.The paper states the condition as an assumption, not a theorem-free property.Source: §2 · pp. 2–3
- On the CIFAR-10 GCD split, every OOD score detects novel categories against clean chunks, but only the embedding functional axis distinguishes novelty from photometric shifts near perfectly.Source: Figure 2 · p. 3
3 Two-axis gating for mixtures of heads
The mixture-of-heads system routes by recent loss and compares six spawn policies on a stream of known concept changes. The two-axis CJSD rule has the best joint decision profile, while loss-jump performance depends on the stream regime.
3 Two-axis gating for mixtures of heads
- The comparison uses a frozen encoder, expert heads, recent-loss routing, and six spawn policies across plain, permuted, and rotated phases.Source: §3 · p. 3
- Representation novelty misses permuted concepts at a 0.5 missed-concept rate, while deep exchange over-spawns on rotation with false-spawn rate 0.246 and 11.3 heads.Loss-jump reaches 0.807 accuracy when every real switch is large and abrupt.Source: §3 · p. 3
- The two-axis CJSD rule achieves the best joint profile with false-spawn rate 0.044 and 2.7 heads, at a 0.07 accuracy cost from conservative deferral.This fast-switching stream rewards loss-jump, but gradual and recurring regimes favor two-axis gating.Source: §3 · p. 3
4 Application: semantic novelty vs. photometric shift
On the CIFAR-10 GCD split, the chunk-level functional axis distinguishes semantic novelty from photometric shift, a distinction standard OOD scores largely miss. The monitor uses chunk labels, and pseudo-labels derived from the same inputs cannot replace that information.
4 Application: semantic novelty vs. photometric shift
- The embedding functional axis achieves AUROC 0.99 for novelty versus photometric shift, while standard OOD scores score 0.50–0.57 on that distinction.Standard OOD scores achieve AUROC 0.94–1.00 for novelty versus clean chunks.Source: §4 · p. 3
- The monitor operates with chunk labels available and folds novel labels into an “other” bucket; its comparison with label-free scores is not a claim that they fail at their own task.Source: §4 · p. 3
- Pseudo-labels that are deterministic functions of the same inputs carry no information beyond x, so a label-free functional axis is impossible in this setting.Unlabeled novelty detection therefore reduces to the covariate axis; labels from side information are outside this impossibility claim.Source: §4 · p. 4
5 Foundation-encoder scale
On frozen foundation encoders, the two-axis functional and covariate comparison separates mechanism changes from input shifts and supports head-lifecycle decisions. Its findings extend across ViT and DINOv2 backbones, including adapter pools with recurrence.
Foundation-encoder pair comparisons
- On ViT-B/16, rotations raise exchange scores from 0.0005 to 0.065–0.179 while the functional axis stays at |DCJS| ≤0.001; label permutations yield DCJS values 0.149/0.371/0.780 for drift masses 0.2/0.5/1.0.For 100-class GCD, novel chunks score DCJS = 0.063 ± 0.020 versus −0.003 ± 0.016 for photometric chunks, with chunk-level AUROC 0.98.Source: §5 · p. 4
Adapters and recurrence
- With residual adapters and recurrence, the two-axis gate reaches 0.847 accuracy with 3 adapters, zero false expansions or missed concepts, and full reuse of the recurring concept’s original adapter.The novelty trigger never fires on the 8-phase stream and misses both mechanism changes (0.686); all policies use 40 training invocations.Source: §5 · p. 4
Second backbone
- In DINOv2, rotations inflate exchange scores to 0.068–0.272 from null −0.001 while the functional axis remains within ±0.002; label permutations track drift mass, and novel-class chunks separate from photometric chunks in 6/6 repetitions.Source: §5 · p. 4
Head lifecycle at scale
- On the ViT lifecycle stream, the two-axis gate spawns on the first chunk of permutation and novel phases, reuses photometric and recurring clean chunks, and uses exactly 3 heads.Its functional axis jumps to 0.69 and 0.34 against a ≤0.01 clean baseline.Source: §5 · p. 5
- On the 40-chunk ViT lifecycle stream, the two-axis gate achieves 0.849 prequential accuracy with 3 heads, zero false spawns, and zero missed concepts; Mahalanobis misses both mechanism changes (0.709).The loss-jump trigger matches on this stream because switches are large and abrupt.Source: §5 · pp. 4–5
6 Practical notes
Discriminator training and comparison design constrain reliable measurement of the functional axis.
7 Practical notes
- Under-training shrinks the functional axis toward zero, tiny per-chunk discriminators need ∼25 epochs, and duplicated inputs across comparison sides must be avoided to prevent twin-copy memorization leakage.Appending encoder softmax supplies a label-likelihood statistic that low-capacity discriminators otherwise fail to learn.Source: §6 · p. 5
7 Related work
The paper distinguishes its embedding-space decomposition and expert-pool decision layer from related one-axis comparison and scoring approaches.
7 Related work
- SEMA and MoE-adapter triggers use representation novelty, CLS-style scores cross-evaluate swapped predictors, and OOD scores rank single inputs; the paper contributes embedding-space decomposition for deep expert-pool decisions.Source: §7 · p. 5
8 Limitations
The evidence is bounded by encoder, stream, and system-evaluation choices.
8 Limitations
- Because the encoder is frozen after warm-up and lacks invariance to the studied transformations, augmentation-invariant or foundation encoders may shrink the exchange confound; the claim is existence, not universality.Online encoder coadaptation would couple the axes and require representation-version alignment.Source: §8 · p. 5
- The ViT lifecycle stream is shorter than the MNIST streams, and modern GCD systems are not run end-to-end; comparisons target their underlying score families, not full systems.Source: §8 · p. 5