CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork

• A fixed-size hypernetwork generates each concept’s low-rank adaptation from a compact task embedding, while output-space regularization mitigates interference with earlier adaptations.The approach avoids storing a separate adapter per concept.Source: §1 · p. 2
• The same hypernetwork generates placement tokens so a user-provided bounding box guides a personalized concept to a desired region without additional per-concept parameters.Placement tokens are learned from synthetic composites with controlled object locations.Source: §1 · p. 2
• On SD-1.5, CLASP outperforms CIDM in image alignment at matched text alignment, with approximately 4× lower forgetting than CIDM and 23× lower forgetting than fine-tuning.The method also retains the first ten concepts better than CIDM at every sequence length CIDM can reach and scales to 100 concepts.Source: §1 · p. 2
• The framework places seven personalized concepts in their requested quadrants in every evaluated case.Source: §1 · p. 2

• Continual personalization has used concept-specific isolation, replay or distillation, and parameter merging or allocation; CIDM stores separate low-rank parameters per concept, so storage grows with the stream.Source: §2 · p. 2
• CNS allocates part of the backbone to each concept, bounding the number of concepts by backbone capacity, whereas CLASP keeps the backbone fixed and generates adaptations with one hypernetwork.Source: §2 · p. 3
• Earlier hypernetwork work covered fast personalization and concept removal, while several adapter methods separate concepts into subspaces; CLASP shares one network and distinguishes concepts by task embeddings.Source: §2 · p. 3
• Most spatial-control methods target concepts named directly in prompts, while Mix-of-Show supports personalized subjects using concept-specific adapters; CLASP generates adaptation and spatial conditioning from one compact representation.Source: §2 · p. 3

• A latent diffusion model denoises a latent in a pretrained autoencoder’s space; its UNet predicts noise conditioned on text through cross-attention projections.The method adapts the query, key, value, and output projections.Source: §3 · p. 3
• LoRA adapts a pretrained weight matrix with a rank-r update, training only matrices A and B to reduce parameter count; these updates are applied as a separate branch at selected token positions.Source: §3 · p. 3
• A shared hypernetwork maps each concept’s task embedding to its low-rank update matrices on demand, avoiding storage of individual concept updates; sequential changes to the hypernetwork can alter earlier adaptations.The paper addresses this interference with a regularizer.Source: §3 · p. 3
• After each sequential task, the model should generate any previously learned concept from its natural-language prompt, and a normalized bounding box can specify that concept’s desired image location.Training accesses only the current concept’s small reference set, and concept identity is assumed known at generation time.Source: §3 · p. 3
• The overview separates a content module that generates LoRA updates from a placement module that generates spatial tokens conditioned on the task embedding and bounding box.The content module uses Gram–Schmidt orthogonalization and output-space regularization; the placement module uses a box mask and per-block gate.Source: Figure 2 · p. 4

4 METHOD

• The shared hypernetwork generates both concept adaptations and bounding-box-conditioned placement tokens from each concept’s task embedding.Its content and placement components are trained jointly, without adding concept-specific placement modules.Source: §4 · p. 4, §4.2 · pp. 6–7

4.1 CONTENT MODULE

• Each task embedding generates low-rank updates for adapted cross-attention projections, while all other UNet weights remain frozen.The generated updates replace the corresponding pretrained weights when evaluating the UNet for that concept.Source: §4.1 · pp. 4–5
• Updates to text-token projections are restricted to the concept’s class word, while image-feature projections can be updated at every spatial position.This limits the concept adaptation’s effect on prompt context and uses the class word as the concept identifier.Source: §4.1 · p. 5
• Orthogonal task embeddings and output-space regularization address interference from shared hypernetwork parameters and help preserve earlier concepts’ generated updates.For T ≤ dv, each concept receives a direction orthogonal to all previously assigned ones; without orthogonalization, earlier updates drift 7% more over a 49-concept sequence.Source: §4.1 · p. 5

4.2 PLACEMENT MODULE

• Placement tokens encode the task embedding and normalized bounding box, and a parallel attention branch adds their spatial signal to the unchanged text-attention pathway.Attention lets different latent positions read different placement tokens; a soft box mask and learnable per-block gate control the placement contribution.Source: §4.2 · p. 6
• The placement branch is included in joint reconstruction training, while output-space regularization remains applied only to generated low-rank updates.The placement branch uses its own per-block projections and gates.Source: §4.2 · p. 7

• 79.9 ± 0.8 IA at CIDM’s text alignment versus 78.0 for CIDM on SD-1.5; Figure 4 plots IA against TA.Source: §5 · p. 7, Figure 4 · p. 8
• 0.0060 ± 0.0031 forgetting, nearly four times lower than CIDM and 23 times lower than sequential fine-tuning at comparable TA.Source: §5 · p. 7, Figure 4 · p. 8
• Removing the output-space regularizer raises forgetting from 0.0094 to 0.0213; without it, forgetting remains six times lower than sequential fine-tuning.Source: §5 · p. 8
• Retention remains higher than CIDM at every reported shared stream length; the first ten concepts retain 0.531 DINO similarity at T = 100.Source: §5 · p. 8, Figure 6 · p. 9
• The hypernetwork stays fixed-size; generated LoRAs add 14% of backbone size versus CIDM’s 45% at T = 10 and 154% at T = 37.Source: §5 · p. 8
• With placement conditioning, concepts appear in the requested quadrant in every generation, and 96% of detected generations exceed 0.5 IoU; subject-mask DINO rises from 0.599 to 0.717.Source: §5 · pp. 8–9, Figure 7 · p. 9

• A single fixed-size hypernetwork generates concept-specific adaptations for a frozen diffusion model, with output-space regularization intended to preserve earlier adaptations as it is updated.Source: §6 · p. 9
• The same framework places learned concepts within user-specified bounding boxes, while keeping model-side storage constant as concepts are added.Source: §6 · p. 9
• Image quality depends on reference-image count, finite hypernetwork capacity may limit substantially larger and more diverse streams, and transfer to another backbone may require adjustment.Source: §6 · p. 9

• The benchmarks include photographs of people, but the authors use the datasets only as released, learn no concepts outside them, and release no images of people with the code.Source: ETHICS STATEMENT · p. 10
• Personalization and placement can support benign re-rendering and composition as well as non-consensual likeness synthesis and fabricated photographs.Source: ETHICS STATEMENT · p. 10

• The public repository provides code, configurations for reported runs, evaluation scripts, and a placement test.Source: REPRODUCIBILITY STATEMENT · p. 10
• The released hundred-concept sequence includes concepts, their order, captions, and prompts, while the original images are not redistributed.Source: REPRODUCIBILITY STATEMENT · p. 10
• The paper locates the experimental protocol and seeds in Section 5, implementation details in Appendix C, and the output-space regularizer’s closed form in Appendix D.Source: REPRODUCIBILITY STATEMENT · p. 10

USE OF LARGE LANGUAGE MODELS

• The authors disclose LLM assistance with prose and figure-producing code, while retaining responsibility for the research questions, methods, experiments, analysis, and scientific claims.Captions for ninety added concepts were also generated by an LLM from images and verified under the Appendix E protocol.Source: USE OF LARGE LANGUAGE MODELS · p. 10

Appendix

• The appendices cover related work, diffusion background, training and regularizer computation, added-concept captions, metric definitions, ablations, placement, and concept composition.They also report content-module trade-offs, independently trained adapter comparisons, a second backbone, and task-embedding analyses.Source: Appendix · p. 15

A EXTENDED RELATED WORK

• Prior continual-learning methods use regularization, rehearsal, or parameter-efficient adaptation; continual personalization must retain visual concepts under novel prompts, while CIDM storage grows with sequence length and CNS is capacity-bounded.The paper positions its fixed-backbone, shared-hypernetwork approach as generating adaptations for increasing numbers of concepts.Source: A EXTENDED RELATED WORK · pp. 15–16

B BACKGROUND: LATENT DIFFUSION MODELS

• The background describes latent diffusion as denoising in an autoencoder’s latent space and explains that text conditions the UNet through cross-attention projections, all four of which the method adapts.Classifier-free guidance combines conditional and unconditional noise predictions.Source: B BACKGROUND: LATENT DIFFUSION MODELS · p. 16

C TRAINING AND GENERATION DETAILS

• The frozen backbones train only the hypernetwork, which generates rank-4 updates and placement tokens; placement training uses synthetic composites, and inference applies box-conditioned generation.The hypernetwork has 26.0M parameters on SD-1.5 and 191.1M on SDXL; training a ten-concept sequence takes about an hour and 5.5–7 hours, respectively.Source: C TRAINING AND GENERATION DETAILS · pp. 16–17

D EFFICIENT COMPUTATION OF THE OUTPUT-SPACE REGULARIZER

• The output-space regularizer compares generated low-rank updates without materializing their full matrices, reducing computation to O(r^2d) per layer and normalizing by layer size.The normalization puts the term on a per-entry mean-squared-error scale, allowing one β across layers of different widths.Source: D EFFICIENT COMPUTATION OF THE… · pp. 17–18

E.1 CAPTION ATTRIBUTES

• Stripping concept attributes from captions raises identity for the duck toy and backpack and leaves the cat and seven concepts without named attributes unchanged; across ten concepts, DINO rises from 0.601 to 0.620.Captions for the ninety added concepts describe shared scene context while excluding attributes that should be learned by the adapter.Source: §E.1 · p. 18, Table 1 · p. 18, §E.2 · p. 18

F METRICS

• Evaluation uses text alignment and image alignment to compare with published work, DINO for within-paper analyses, and a forgetting measure based on the best earlier score versus the final score.The paper treats DINO differences below 0.0091 as within seed-to-seed variation at ten concepts, with spread reaching 0.015 after fifty tasks.Source: F METRICS · pp. 18–19

F.1 WHAT THE METRICS CANNOT RESOLVE

• Reference-set ceilings range from 0.42 for painting to 0.91 for dog2, and correlate strongly with the model’s DINO scores across concepts.The reported correlation is 0.93 Spearman and 0.97 Pearson; dog2 is the only concept out of rank.Source: §F.1 · p. 19
• At slora = 0.45, model DINO reaches 0.82–0.98 of the reference ceiling on objects and 0.75–0.80 on styles; at 0.9, it reaches 0.90–1.01 across concepts.The table places each concept’s score beside its mean pairwise reference-image DINO similarity.Source: §F.1 · p. 19, Table 2 · p. 19
• Containment scores can rise despite subject amputation or duplication, so placement numbers need to be read alongside image grids.The authors report that amputation, canvas clipping, and cut-out halos appeared in grids but not in the tables.Source: §F.1 · p. 19

G FULL ADAPTER-SCALE RESULTS

• From slora = 0.9 to 1.05, IA gains only 0.27 versus 0.91 in the preceding step, while DINO falls from 0.663 to 0.655; after fifty tasks, IA falls by four to six points and DINO by 0.05–0.08 as TA shifts by less than a point.Table 3 reports mean and standard deviation over three seeds after ten tasks and after the same model learns fifty.Source: G FULL ADAPTER-SCALE RESULTS · pp. 19–20, Table 3 · p. 20

H COMPARISON WITH INDEPENDENTLY TRAINED ADAPTERS

• At common TA 75.6, generated adapters reach IA 79.1 ± 0.3 and DINO 0.617, versus IA 78.5 and DINO 0.594 for independently trained adapters.Doubling the independent adapters’ training to 1600 steps leaves their IA at 78.4 and DINO at 0.593.Source: H COMPARISON WITH INDEPENDENTLY… · p. 20
• The independent-adapter control stores 0.40M parameters per concept and cannot forget, whereas the generated adapters come from a network whose size does not depend on the concept count.The control uses a separate adapter for each of ten concepts.Source: H COMPARISON WITH INDEPENDENTLY… · p. 20

I RESULTS ON SDXL

• Retuning β from 7120 to 3000 changes SDXL results at slora = 0.40 from TA 72.1, IA 76.4, DINO 0.536 to TA 75.5, IA 80.1, DINO 0.593.The authors state that the unchanged SD-1.5 value is more than twice as strong as SDXL needs.Source: I RESULTS ON SDXL · p. 20
• At CIDM’s TA of 80.0, SDXL IA is 77.5 on the first seed and 77.1 ± 2.1 over three seeds, versus 79.5 for CIDM; at their own TA, the three-seed mean exceeds five of six shared baselines.The IA margins over C-LoRA, L2DM, EWC, LwF, and sequential fine-tuning are 0.3, 0.6, 0.8, 1.6, and 7.6 points, respectively; the first three lie within seed spread.Source: I RESULTS ON SDXL · p. 20, Figure 8 · p. 21
• The SDXL result is bounded by seed instability, final-model-only evaluation without a 55-cell matrix, and forgetting measurements limited to SD-1.5.The authors report no claim that the recipe is stable on SDXL.Source: I RESULTS ON SDXL · p. 21
• With out.0 fixed at 0.50 and the other three projection scales swept, IA at CIDM’s TA rises to 78.7 from 77.5 and DINO to 0.596 from 0.545; the gap to CIDM shrinks to 0.8 points.The per-group variant is above the single-scale curve at all five measured points, but its gain is smaller than seed-to-seed spread.Source: I RESULTS ON SDXL · p. 21, Figure 8 · p. 21
• Pinning out.0 at 0.12 while sweeping the caption-reading projections changes DINO only from 0.475 to 0.479 across 4.9 TA points, supporting the reported role of the output projection as an identity gate.The corresponding single-scale sweep changes IA by 3.1 points.Source: I RESULTS ON SDXL · p. 21

J ABLATIONS

• Only removing the regularizer exceeds the 0.0091 seed-to-seed DINO threshold; two random sequence permutations change DINO by at most 0.005 and forgetting by 0.0063.Tying the task embedding to appearance preserves identity but nearly triples forgetting; removing Gram–Schmidt has no measurable metric cost in these tests.Source: J ABLATIONS · p. 22
• Without composites, placement IoU falls from 0.766 to 0.270, and the requested quadrant is correct in 62% rather than 100% of generations.The ablation also reports that photographs alone do not teach placement.Source: J ABLATIONS · p. 22
• DINO falls from 0.621 to 0.347 when the output projection is removed; raising the LoRA scale does not recover the loss.Image alignment drops from 79.5 to 68.9 and text alignment from 75.3 to 68.7; the same ablation is costly on SDXL-base-1.0.Source: J ABLATIONS · p. 22
• Removing the output projection leaves too little residual stream for the generated update to write into, because the caption-reading projections act on few token-masked positions.The reported explanation distinguishes a hypernetwork-generated adapter from a per-concept adapter such as Custom Diffusion, which adapts only to k and v.Source: J ABLATIONS · p. 23

K CONCEPTS ALONG THE SEQUENCE

• At full adapter strength, the first concept’s DINO falls by 0.16 by task 50, and the reported LoRA scale compromises between identity and prompt fidelity across concepts.The fixed-size network and CIDM storage are equal at T = 62; CIDM’s implementation breaks beyond 37 concepts.Source: K CONCEPTS ALONG THE SEQUENCE · p. 23, Figure 9 · p. 23

L ANALYSIS OF THE TASK EMBEDDINGS

• Tying half the task embedding to reference-image appearance leaves identity unchanged (0.618 versus 0.617) but raises forgetting to 0.016.The authors therefore keep the task embedding arbitrary.Source: L ANALYSIS OF THE TASK EMBEDDINGS · p. 23
• Generated updates have mean absolute cosine 0.056 after ten tasks and 0.044 after fifty, and remain near-orthogonal without Gram–Schmidt.The orthogonalized and non-orthogonalized runs have similar update cosine values; the mean falls rather than rises as concepts are added.Source: L ANALYSIS OF THE TASK EMBEDDINGS · p. 24
• Generated-update drift averages 1.1% of its norm after one task and reaches 8.9% for the first concept after 49 tasks; Gram–Schmidt’s stability effect grows with lag.An independent run reproduces the drift curve within 4% at every lag; the 49-task comparison averages only one pair.Source: L ANALYSIS OF THE TASK EMBEDDINGS · pp. 25–26
• Embedding-space probes show continuous identity transitions between same-class concepts, class choice controlled by the prompt word, and breakdown when an embedding is scaled beyond 1.5×.Random vectors orthogonal to the learned basis produce clean generic class instances in these inference-only tests with localization disabled.Source: L ANALYSIS OF THE TASK EMBEDDINGS · p. 26

M.1 FOLLOWING THE REQUESTED BOX

• Bootstrapping restores text alignment to 0.828 from 0.795 while leaving IoU and containment unchanged; smaller boxes and strips are followed, but placement is less precise than for quadrants.On the nine-cell grid, the concept lands in the requested cell in 86% of generations; for strips, it lands in the requested strip in 94%.Source: §M.1 · p. 27

M.2 PLACEMENT ALONG THE SEQUENCE

• The first concept’s placement IoU rises from 0.60 to 0.75 by task 10, then placement IoU on seven objects falls from 0.71 to 0.55 by task 50.After 50 tasks, the subject still lands in the requested quadrant in 73% of generations; regularizing placement tokens does not preserve placement and weakens the content module.Source: §M.2 · p. 29

M.3 REJECTED ALTERNATIVES TO THE PLACEMENT MODULE

• The placement design keeps the no-box protocol within sampling noise; alternatives include a whole causal-tail penalty that raises containment but costs text alignment, and pre-rendered backgrounds that place subjects worse.Feathering the bootstrapping mask reduces detections to 74 of 84 generations; isolating self-attention inside the box drops placement from 75% to 4%.Source: §M.3 · p. 30

N MULTI-CONCEPT COMPOSITION

• In multi-concept SDXL scenes, the requested concept is the best match in 121 of 124 regions, with mean regional DINO of 0.716.The authors report that concepts share scene lighting and style and appear at plausible depth rather than as a collage.Source: N MULTI-CONCEPT COMPOSITION · p. 30