PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Routes to the Middle: Placement Search and Brain Readouts Converge on Where Continual Learners Should Specialize

Yuan Huang, Zihan Chen, Runbin Zhang, Hongwei Ding, Changzeng Fu, Shiqi Zhao

arXiv:2610.01590v1cs.LGcs.CV

TL;DR

Continual learners can reduce adapter storage by specializing only some blocks, but the best placement is unclear and internal criteria may miss it. This paper uses early-task fMRI readouts to select blocks, finding overlap with intermediate placements and strong accuracy at reduced storage. The result is a one-time allocation signal from outside the learner, evaluated on Split ImageNet-R.

  • Problem

    Task-specific adapters in every block make storage grow with the task sequence, while choosing only a few blocks raises the unresolved question of where to place them.

    Source: §1 · p. 1

  • Method

    LS-B uses a frozen fMRI encoding model to read the first tasks’ unlabeled images and commits task-specific capacity to blocks with the greatest readout variation relative to stable structure.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    On Split ImageNet-R, LS-B retains 60% of full-BiLoRA adapter storage while remaining within 1.5 pp of its accuracy.

    Source: §6 · p. 9

  • Takeaways & Limitations

    External brain readouts can serve as priors for layer-allocation decisions otherwise made by placement search or learner-derived statistics.

    Source: §6 · p. 9

    All main-text results use one benchmark and class order, one host adapter, one architecture, and one fMRI subject.

    Source: A LIMITATIONS · p. 13

Abstract

from arXiv · show

Continual learners that keep a task-specific adapter in every block of a pre-trained vision transformer accumulate storage linearly with the number of tasks; keeping task-specific adapters in only a few blocks curbs this growth but raises the question of where to place them. We investigate this question from two perspectives. Algorithmically, training all contiguous four-block placements yields an inverted U: final accuracy peaks at intermediate depth and varies by up to 3.5 percentage points (pp), while inexpensive criteria based on weight spectra or activation statistics favor the deepest blocks. From neuroscience, the hierarchical organization and intermediate-stage plasticity of the visual cortex motivate us to ask whether a measurement taken outside the learner can guide layer specialization without placement search. LS-B observes the first tasks through a frozen fMRI encoding model of twelve human visual areas and commits task-specific capacity once to the blocks whose readouts vary most across tasks relative to their stable structure. Across three ViT-B/16 backbones, LS-B yields stable, backbone-specific allocations. On the two backbones with placement search, AugReg and iBOT, the selected blocks overlap the intermediate-depth region identified by search. Under matched storage and observation budgets, the selected blocks outperform the shallowest and deepest four-block configurations. On Split ImageNet-R, LS-B uses 60% of full-BiLoRA adapter storage while remaining within 1.5 pp of its final accuracy. The allocation requires no labels or backpropagation, adds under 0.6% runtime, and exhibits backbone-specific cortical signatures.

1 INTRODUCTION

Task-specific capacity can reduce continual learners’ storage growth, but deciding where to place it matters: placement search finds an intermediate optimum that inexpensive internal criteria miss. LS-B uses frozen fMRI readouts to select blocks, and its allocations overlap the intermediate region found by search.

1 INTRODUCTION

  • Moving a four-block task-specific window changes final accuracy by up to 2.8 and 3.5 pp across two backbones at identical storage.Keeping task-specific capacity in only some blocks curbs storage growth, but raises the question of which blocks should retain it.Source: §1 · p. 1
  • Placement search traces an inverted U with an intermediate optimum, while weight-spectrum and activation criteria favor the deepest blocks and miss that optimum.Search requires a full continual-learning run per candidate and seed; the cheaper criteria use parameter, activation, or gradient signals.Source: §1 · p. 1
  • Visual-cortex hierarchy and intermediate-stage plasticity motivate testing whether an independent neural measurement can locate task-specific capacity without labels or placement search.The cited neuroscience findings describe early and later visual areas, perceptual learning in V4, and constraints on early-visual-cortex plasticity.Source: §1 · p. 2
  • LS-B observes the first tasks with a frozen fMRI encoding model of twelve visual areas, then commits capacity to blocks whose readouts vary most relative to their stable structure.The remaining blocks share one continually updated adapter.Source: §1 · p. 2
  • The readout-selected blocks overlap the search-best window by one to three blocks out of four, locating the intermediate band without recovering the search optimum.The readout’s ranking of candidate placements shows no detectable correlation with their accuracy; the reported contributions also include gains over both extremes at matched storage and observation and 60% of full-adapter storage within 1.5 pp of its accuracy.Source: §1 · p. 2

2 RELATED WORK

Prior continual-learning work allocates adaptation capacity through fixed rules, search, or signals derived from the learner. LS-B instead uses an independently fitted neural encoding model as its allocation signal.

2 RELATED WORK

  • Continual-learning methods adapt pre-trained backbones with adapters, prompts, or low-rank updates, often separating shared from task-specific knowledge [Wang et al., 2024] [Wang et al., 2022b;a] [Wang et al., 2022a].Early adaptation is used as a strong baseline and as RanPAC’s first phase; related allocation methods include adaptive-rank fine-tuning, LoRA-drop, and D-MoLE.Source: §2 · pp. 2–3
  • Layer-wise allocation has been set by fixed rules, validation or search [Wang et al., 2022a], empirical placement comparisons [Yin et al., 2024], and learner-derived signals [Li et al., 2025].Those allocation signals are read from the learning system; LS-B obtains its signal through an encoding model fitted independently of continual learning.Source: §2 · p. 3
  • Layer-resolved encoding models map cortical locations to network layers, and LS-B uses such a model directly in its layer-allocation rule.Neural data have also been used to regularize models, inform fine-tuning of language and speech models [Moussa et al., 2025] [Vattikonda et al., 2025] [Moussa & Toneva, 2025], or guide intervention-layer choice.Source: §2 · p. 3

3 METHOD

LS-B uses a frozen neural encoding model to score blocks from early-task readouts, then commits a fixed shared/task-specific partition. The rule depends on unlabeled images, and the selection step adds little runtime.

3.1 SETTING AND HOST ADAPTER

  • A task–layer slot is a task’s retained adaptation in one block; full BiLoRA retains 120 such slots after ten tasks.In the released implementation, BiLoRA adapts the key and value projections of each of the 12 ViT-B/16 blocks, with 6,000 parameters per slot.Source: §3.1 · p. 3

3.2 BRAIN READOUT

  • The frozen FactorTopy model maps block features to responses in twelve visual regions, and each task’s readout summarizes task-image modulation weighted by layer-selection weights and vertex fit.The model is fitted once per backbone on NSD subject subj01 and uses no fMRI data during continual learning; its adapter-free copy excludes adaptation from earlier tasks from each readout.Source: §3.2 · pp. 3–4

3.3 LAYER SPECIALIZATION SCORE

  • Two-way centering removes region and block main effects; LS-B then scores each block by task-to-task residual energy relative to its stable brain-aligned structure.A block with strong but consistent readout receives a low score and is treated as a candidate for sharing; the score requires at least two observed tasks.Source: §3.3 · pp. 4–5

3.4 OBSERVE, THEN COMMIT

  • After observing K tasks, LS-B commits the k highest-scoring blocks to task-specific slots and keeps the other blocks shared for subsequent tasks.The partition is fixed once: committed blocks add a slot for each later task, while other blocks retain their observation-stage slots and add one persistent shared slot.Source: §3.4 · p. 5

3.5 IMPLEMENTATION DETAILS AND COST

  • Selection uses only the first K tasks’ unlabeled training images, a frozen backbone copy, and the frozen encoding model; it uses no labels, gradients, held-out data, or trained learner parameters.Three readouts and one ranking take 40–46 seconds of a 2.1–2.3-hour ten-task run; no readout is taken after commitment.Source: §3.5 · p. 5

4 EXPERIMENTAL SETUP

Experiments use class-incremental Split ImageNet-R with three pre-trained ViT-B/16 backbones and compare placement search, internal criteria, and matched-budget controls. Comparisons are paired by seed, with significance claims limited to specified controls.

4 EXPERIMENTAL SETUP

  • The benchmark is 200 Split ImageNet-R classes divided into ten disjoint 20-class tasks, evaluated by final accuracy of one 200-way classifier without task identity.Backbones are AugReg, iBOT, and DINO; all main-text experiments use the same class order and the host adapter’s recipe, with k = 4 and K = 3 by default.Source: §4 · p. 5
  • Placement search compares nine contiguous four-block windows at 48 slots; controls include internal criteria, random or equal-partition blocks, matched-observation extremes, and storage references.The matched-observation controls retain 72 slots and commit either blocks 0–3 or 8–11; the figure also shows window accuracy and readout window scores.Source: §4 · p. 6, Figure 3 · p. 6
  • Seed-paired differences are reported, but no significance claims are made at n = 3; the 0.42 pp run-to-run gap is used only as a magnitude yardstick.Registered paired tests cover random allocation and activation criteria; matched-observation comparisons are descriptive, while sweeps and decompositions are exploratory.Source: §4 · p. 6

5 RESULTS AND DISCUSSION

On Split ImageNet-R, LS-B retains task-specific capacity in selected intermediate blocks, with accuracy close to full BiLoRA at lower storage. Placement search and brain readouts both point to the intermediate-depth region, although the readout does not reproduce the search ranking.

5.1 THE ALGORITHMIC ROUTE: SEARCH FINDS THE MIDDLE, INTERNAL CRITERIA MISS IT

  • Across contiguous four-block windows, accuracy formed an inverted U, peaking at AugReg blocks 4–7 (71.20%) and iBOT 3–6 (68.58%), with ranges of 2.37 and 3.48 pp.This search covered contiguous windows; the best of all 495 possible four-block allocations is unknown, and the search cost 27 runs per backbone.Source: §5.1 · pp. 6–7
  • Weight-spectrum criteria nearly rank blocks by depth and activation criteria favor deep blocks, where search finds the lowest accuracy; gradient norm is the exception.The weight-spectrum criterion’s Spearman correlation with depth is +0.87 to +1.00; activation-selected sets have mean depth 8.0–9.5.Source: §5.1 · pp. 6–7

5.2 THE NEUROSCIENCE ROUTE: THE READOUT LANDS IN THE INTERMEDIATE BAND

  • The readout-selected sets fall in the intermediate band near search optima, but their placement ranking does not track accuracy.Selected sets overlap the best-performing window by one to three blocks; correlations with accuracy were nonsignificant.Source: §5.2 · p. 7

5.3 WHERE THE ROUTES MEET: MATCHED STORAGE AND OBSERVATION

  • At matched 72-slot storage and K=3, selected blocks beat shallowest/deepest by 0.85/0.68 pp on AugReg and 0.98/0.75 pp on iBOT.The margins over the deepest blocks are smaller against LS-B’s 16-seed mean, especially on iBOT.Source: §5.3 · p. 7

5.4 STORAGE–ACCURACY TRADE-OFF

  • The observation stage supplies most of LS-B’s gain over a static allocation, while selection costs 0.53–0.57% of runtime and storage savings do not shorten training.The controls separate the observation gain from the added benefit of choosing where task-specific capacity goes.Source: §5.3 · p. 7, §5.4 · p. 8
  • At k=4, LS-B retains 60% of full BiLoRA adapter storage and stays within 1.5 pp of full BiLoRA across AugReg, iBOT, and DINO.The default uses 72 of 120 slots; Figure 4 compares storage and accuracy across allocations.Source: §5.4 · p. 7, Figure 4 · p. 8

5.5 MAPPING THE COMMITTED BLOCKS ONTO VISUAL CORTEX

  • Across backbones, LS-B’s cortical footprint peaks between areas with below-uniform commitment, while the task-residual signal’s cortical composition differs by backbone.V1–V4 carry 31–79% of the task residual of the top-ranked blocks, against an equal share of 33%; the authors describe this as a coarse correspondence, not recovery of the search optimum.Source: §5.5 · pp. 8–9, Figure 7 · p. 9

6 CONCLUSION

LS-B uses external brain readouts to select intermediate blocks for task-specific capacity, outperforming the shallowest and deepest configurations at matched capacity. On Split ImageNet-R, it retains 60% of full-BiLoRA adapter storage while remaining within 1.5 pp of its accuracy.

6 CONCLUSION

  • 60% of full-BiLoRA adapter storage retains accuracy within 1.5 pp on Split ImageNet-R.At matched capacity, LS-B also outperforms the shallowest and deepest four-block configurations.Source: §6 · p. 9
  • The authors report that generative AI assisted with language editing, figure preparation, and software tasks, while authors conducted the research and analyses.The authors carried out the research questions, literature selection, experimental design and deployment, analyses, and interpretation.Source: AI USE STATEMENT · p. 10
  • The study uses released, de-identified data from one NSD subject and collects no new human data.The authors foresee no negative societal impacts beyond those of image classification generally.Source: ETHICS STATEMENT · p. 10
  • The code and a deployment guide are available in the reproducibility repository.The paper specifies the benchmark, training recipe, configurations, seeds, and statistics protocol.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS.

LS-B approaches but does not recover the best placement window or rank candidate placements by accuracy. The reported evidence is bounded by the evaluation setting, seed count, and the information the readout can encode.

A LIMITATIONS.

  • LS-B identifies the right depth region but does not recover the search optimum or rank placements by accuracy.The authors frame it as reaching the right region at the cost of three forward-only readouts, not as finding the optimum.Source: A LIMITATIONS · p. 13
  • Main-text results cover one benchmark and class order, host adapter, architecture, and fMRI subject; the placement test uses three seeds per configuration.One iBOT comparison against the deepest blocks depends on the seed reference.Source: A LIMITATIONS · p. 13
  • At k = 4, the readout can inject at most approximately 8.95 bits into the allocation.This bounds how much information the readout can encode in the selected allocation.Source: A LIMITATIONS · p. 13

B SETUP DETAILS

The setup fixes the Split ImageNet-R partition and training recipe, uses unlabeled training images to estimate readout statistics, and compares allocations with matched slot budgets. LS-B’s selected blocks vary across seeds, so there is no single allocation shared by every run.

B SETUP DETAILS

  • The train/test split is stratified per class at 80/20 (24,002/5,998 images), and classes are divided by dataset index into ten consecutive groups of twenty.The reported variability reflects training noise, not sensitivity to task order; final ACC is the test-accuracy average weighted by test-image count.Source: B SETUP DETAILS · p. 13
  • The training recipe holds the host adapter’s configuration fixed and uses no validation split or benchmark-selected hyperparameters, epoch budget, or checkpoint.LS-B estimates only readout statistics from unlabeled training images of the observation tasks.Source: B SETUP DETAILS · p. 14
  • A ten-task run takes 2.26, 2.27, and 2.12 h on AugReg, iBOT, and DINO; selection takes about 46, 46, and 40 s, respectively.Storage savings do not shorten training because the implementation sums all retained slots in each forward pass.Source: B SETUP DETAILS · p. 14

C ALLOCATION INVENTORY

  • Configurations include four-block placement windows, internal criteria, same-width controls, and matched-observation controls; one retained slot holds 6,000 parameters.The table indexes blocks 0–11 and reports task–layer slots after ten tasks.Source: Table 1 · p. 14
  • LS-B allocations vary across seeds: AugReg uses {2, 3, 7, 8} in 9 seeds and {2, 6, 7, 8} in 7; iBOT and DINO also have multiple allocations.The equal-partition control shares the AugReg allocation with 9 of 16 LS-B seeds, so those runs differ only in the observation stage.Source: C ALLOCATION INVENTORY · p. 14

D THE FMRI READOUT IN DETAIL

The fMRI readout weights twelve visual regions and their layer profiles, with weighted preferred depth generally rising from V1 toward V4. The retained regional vertices overlap, making some readout profiles partly redundant.

D THE FMRI READOUT IN DETAIL

  • 5,246, 5,214, and 5,255 retained positions correspond to distinct vertices on AugReg, iBOT, and DINO, respectively, because regional masks overlap.FBA and FFA rows are close to duplicates, so the effective number of independent regions is below twelve.Source: D THE FMRI READOUT IN DETAIL · p. 15
  • The weighted mean preferred block rises from V1 to V4 on all three backbones and is deepest in EBA, FBA, and FFA.This weighted mean differs from the per-vertex mode in Figure 7a, where V2–V4 are not monotonic.Source: D THE FMRI READOUT IN DETAIL · p. 15, Figure 8 · p. 15
  • Figure 8 shows the twelve visual regions, retained vertices colored by validation R2, and region-aggregation weights across blocks.The dots mark weighted mean block; they rise from V1 to V4 and are largest in EBA, FBA, and FFA on every backbone.Source: Figure 8 · p. 15

E ADDITIONAL RESULTS

The additional analyses examine LS-B’s performance against storage-matched controls and its readout-based block selection. They also qualify the comparisons by showing that some margins depend on seed coverage and that attribution to observation is descriptive.

E ADDITIONAL RESULTS

  • Against static shallow-first hulls plus the k = 2 point, the default was higher by 0.23, 0.30, and 0.16 pp on AugReg, iBOT, and DINO; against all tested configurations, margins were +0.03/+0.04 pp on seeds 0–2 and +0.10/−0.68 pp over all seeds on AugReg/iBOT.The default point was below the hull of all tested configurations for iBOT over all seeds.Source: E ADDITIONAL RESULTS · pp. 15–16
  • The diagnostic readout’s task residual is 5–7% of its centered readout, while block 11 has the largest shared energy on every backbone and is not selected by the ratio.The task maps are mostly shared, with pairwise correlations of 0.93–0.998.Source: Figure 9 · p. 16, E ADDITIONAL RESULTS · p. 16
  • After the last task, LS-B with k = 4 was 1.46 and 0.47 pp below full BiLoRA, while static blocks 0–3 were 5.25 and 3.53 pp below.For k = 2, the final gaps from full BiLoRA were 2.25 and 1.05 pp.Source: Figure 11 · p. 17

E.1 SAME-WIDTH CONTROLS AND PREFIX ANALYSIS

  • LS-B (72 slots) exceeded same-width controls on AugReg and iBOT: random allocation by +2.32 and +1.83 pp, gradient norm by +2.00 and +2.29 pp, PCA activation by +2.16 and +2.10 pp, random-12 activation by +3.26 and +2.05 pp, and equal partition by +2.15 and +1.60 pp.The random-12 activation result on AugReg was identical to the deepest-four result.Source: §E.1 · p. 17
  • Against random allocation, carry, late, and final differences were 3.66, −1.34, and 2.32 pp on AugReg and 2.91, −1.09, and 1.83 pp on iBOT; late change is descriptive, not a variance decomposition.The compared task sets differ in size: three observed tasks versus ten total tasks.Source: §E.1 · pp. 17–18

F ADDITIONAL DATASETS AND A LONGER TASK SEQUENCE

The supplementary experiments test LS-B across additional datasets and a longer task sequence, with configurations and operating points set before these runs. Results vary by dataset and backbone, while the twenty-task ImageNet-R condition retains a storage advantage over full BiLoRA.

F ADDITIONAL DATASETS AND A LONGER TASK SEQUENCE

  • The supplementary runs covered Split CIFAR-100, Split CUB-200, and Split ImageNet-R at T = 20, using three seeds per configuration; the registered iBOT-by-CIFAR-100 condition was not run.The operating points were fixed on ImageNet-R before these runs and were not re-selected.Source: F ADDITIONAL DATASETS AND A LONGER… · p. 18, Figure 12 · p. 19
  • On CUB-200 with iBOT and DINO, LS-B k = 4 exceeded the other configurations’ upper convex hull by 0.56 and 0.29 pp and static-prefix interpolation by 1.20 and 0.75 pp, while remaining 2.52 and 1.70 pp below full BiLoRA.On CIFAR-100 and CUB-200 with AugReg, configurations were within 2.00 and 1.81 pp of one another, against seed ranges of 1.3–2.4 pp.Source: F ADDITIONAL DATASETS AND A LONGER… · p. 18, Table 3 · p. 18
  • Across sixteen registered hull tests, twelve LS-B points were on or above the hull and four were below; the ImageNet-R iBOT point was above it on seeds 0–2 but below it over all 16 seeds.The supplementary conditions used three seeds per configuration and made no significance claims.Source: Table 3 · p. 18, F ADDITIONAL DATASETS AND A LONGER… · p. 19
  • At T = 20 on Split ImageNet-R, LS-B k = 4 retained 112 of 240 slots (47%) and reached 66.02%, 2.20 pp above static-prefix interpolation, 0.42 pp above the other-configurations hull, and 2.41 pp below full BiLoRA.At T = 10, the corresponding full-BiLoRA gap was 1.46 pp; no comparison across T was registered.Source: F ADDITIONAL DATASETS AND A LONGER… · pp. 19–20, Table 4 · p. 19
  • The predicted dataset ordering of observation-only gains and gains over interpolation was unsupported: three of six primary comparisons held and three were reversed, although observation-only gains were positive in all 21 seed pairs.Observation-only gains ranged from +0.47 to +1.95 pp.Source: F ADDITIONAL DATASETS AND A LONGER… · p. 20, Table 5 · p. 20
  • Committed blocks depended on dataset and backbone, and sets varied across seeds in five of eight conditions; the predicted CUB-200 V1–V4 residual ordering iBOT > DINO > AugReg held at 75%, 59%, and 27%.The corresponding ordering was 79%, 42%, and 31% on ImageNet-R and 53%, 37%, and 22% on CIFAR-100.Source: F ADDITIONAL DATASETS AND A LONGER… · p. 20

G STATISTICS, REGISTRATION AND DEVIATIONS

This section records the registered comparison rules, statistical constraints, and implementation deviations. It also reports a seed imbalance relevant to interpreting iBOT results and bitwise reproducibility for specified reruns.

G STATISTICS, REGISTRATION AND DEVIATIONS

  • The confirmatory criteria held for gains over random allocation and activation controls, but both criteria for matching the best static allocation and backbone-specific placement failed; full BiLoRA was 1.46 and 0.47 pp above LS-B on AugReg and iBOT.The supplementary conditions were registered as exploratory analyses with two directional predictions.Source: G STATISTICS, REGISTRATION AND… · p. 20
  • When the true effect equals the threshold, with n = 3 paired differences, the smallest one-sided sign-test p-value is 0.125, above the corrected level; non-parametric tests therefore serve only as robustness checks.Source: G STATISTICS, REGISTRATION AND… · p. 21
  • The iBOT mean for LS-B seeds 0–2 was 69.60%, equal to its three best seeds among 16, whose full mean was 68.88%.Seeds were fixed by index before the runs, and both values were reported where relevant.Source: G STATISTICS, REGISTRATION AND… · pp. 20–21
  • Four DINO runs used a working tree with uncommitted changes, several comparisons span code versions, and the registered iBOT-by-CIFAR-100 condition was not run.The condition’s run-completion gate was not met by its stated deadline.Source: G STATISTICS, REGISTRATION AND… · p. 21
  • Seven default-configuration reruns and six random-allocation reruns through specified sweep code paths reproduced the original final accuracies bitwise.The reruns covered seeds 0–2 on AugReg and iBOT, plus DINO seed 0 for the default configuration.Source: G STATISTICS, REGISTRATION AND… · p. 21

Related papers

Loading 2610.01590v1…