Localizing Concepts in Visual Autoregressive Models

• Concept localization supports erasing unwanted content while preserving generation utility, as well as personalizing models and injecting novel subjects from minimal reference data.Source: §1 · pp. 1–2
• VAR concepts do not reside in one layer; they anchor at coarse scales and emerge at sparse spatial positions, making diffusion-style layer-wise localization suboptimal.The paper frames joint localization across multiple dimensions as necessary to capture this emergence.Source: §1 · p. 2
• LoCo analyzes attention responses at spatial query positions to identify critical Layer–Scale–Position routes, then intervenes only on those routes.This leaves the vast majority of model parameters untouched.Source: §1 · p. 2
• Top-5% localized positions outperform full-parameter fine-tuning across downstream applications, with 38.5% less memory and 82.9% less training time.The method is also reported to achieve state-of-the-art task performance.Source: §1 · pp. 2–3
• LoCoBench provides 10 categories, 1,467 concept entities, and over 60K paired prompts, with prompt shuffling to reduce context bias.Source: §1 · p. 3

2.1 Visual Autoregressive Generation Models

• VAR models predict successive scales rather than flattening images into a 1D token sequence, supporting coarse-to-fine visual generation.The paper selects Infinity, HunyuanImage-3.0, and InfinityStar to represent image and video next-scale autoregressive models.Source: §2.1 · p. 3

2.2 Interpretability of Generative Models

• Interpretability work localizes concepts in U-Net diffusion models, DiTs, and language models, but next-scale visual autoregressive models remain largely underexplored.This work tracks concepts across layers, scales, and local positions.Source: §2.2 · p. 3

2.3 Concept Erasure, Personalization, and Adversarial Injection

• Prior concept-control work addresses erasure, personalization, and adversarial injection, while this paper unifies these tasks through autoregressive concept routes.The related work notes that discrete-generation error accumulation can make autoregressive erasure challenging.Source: §2.3 · p. 3

3 Method

• LoCo identifies concept routes across layer, scale, and spatial-temporal query position in VAR models’ coarse-to-fine generation.At each scale, the model processes query sites simultaneously while moving from global layout at early scales to fine conceptual details at later scales.Source: §3 · pp. 3–4

3.1 LoCoBench: A Concept Probe Dataset

• LoCoBench covers 10 categories and 1,467 entities with 28,230 training and 37,800 evaluation prompts.Matched neutral prompts replace the target concept span with a semantically related generic placeholder to reduce prompt-specific noise.Source: §3.1 · p. 4

3.2 Localization Method

• LoCo averages concept-token attention contribution scores across prompts to map and rank the top-K layer-scale-position routes.The score aggregates concept-token value vectors weighted by attention across heads; high scores mark positions that attend to concept tokens and extract large value magnitude.Source: §3.2 · p. 5
• At the localized top-K positions, LoCo swaps cross-attention inputs for matched neutral-prompt embeddings, suppressing the concept while preserving scene structure and unrelated semantics.The paper presents this substitution as a training-free intervention.Source: Figure 2 · p. 4, §3.2 · p. 5

Concept Erasure

• For erasure, LoCo uses neutral-condition inference swaps or top-K-restricted LoRA fine-tuning for permanent removal.The qualitative comparison reports that EAR often fails to erase, S-VARE introduces visual artifacts, and CAT can over-purify into generic outputs that do not align with prompts.Source: §4 · p. 5, Figure 3 · p. 6

Model Personalization

• For personalization, LoCo restricts DreamBooth updates to routes identified from the new subject’s broad semantic class.The paper reports largely preserved prior knowledge, reduced computational costs, and superior prompt alignment in complex scenarios.Source: §4 · p. 6

Adversarial Concept Injection

• For adversarial injection, LoCo focuses data-poisoning manipulation on highly responsive routes to enable data-efficient injections.The authors describe concept routes as structural bottlenecks that can be neutralized for safety or exploited for red-teaming.Source: §4 · p. 6

5.1 Implementation Details

• Experiments use Infinity and HunyuanImage-3.0 for images and InfinityStar for video, with LoRA interventions on the top 5% of localized positions by default.The setup generates 512 × 512 images across 10 scales and 720p videos with 81 frames; baselines include VAR erasure methods and adapted diffusion/DiT methods.Source: §5.1 · p. 6

5.2 Results

• LoCo balances NSFW erasure with utility preservation, ranking first in violence erasure on Infinity and HunyuanImage-3.0 while retaining FID and CLIP scores near the base models.For nudity erasure, it has the second-lowest detection rate, behind CAT, whose over-purification often produces generic clothed figures.Source: §5.2 · p. 7, Table 1 · p. 7
• Localized fine-tuning outperforms full-model updates across personalization metrics, improving prompt fidelity while largely preserving irrelevant subject identities.The evaluation uses Infinity for image generation and InfinityStar for video generation, following a DreamBooth setup.Source: §5.2 · pp. 7–8, Figure 5 · p. 8
• Restricting poisoning updates to top-K concept routes yields higher Logo Inclusion Rates and earlier attacks than full-model poisoning, while preserving image quality.The evaluation uses trigger-free training data and measures attack effectiveness with LIR and FAE.Source: Table 2 · p. 8, §5.2 · p. 8

5.3 Ablation Study

• At K = 5%, localized fine-tuning achieves superior concept suppression while reducing VRAM usage by 38.5% and training time by 82.9% versus full-model fine-tuning.K = 1% better preserves benign utility but leaks concepts; K = 10% suppresses targets but damages prompt alignment.Source: §5.3 · p. 8, Table 3 · p. 9
• Layer, Scale, and Position are all needed for precise concept isolation; omitting Position leaves Layer-Scale routing without spatial precision.Layer- or Scale-only updates can cause concept leakage or utility collapse, while the combined dimensions align with coarse-to-fine generation.Source: §5.3 · p. 9, Table 3 · p. 9
• For nudity, 94.7% of targeted concept signals are intercepted in critical layers at the first two coarse scales, before fine-grained spatial layouts form.Later high-resolution positions act as safety refinements guided by the model’s visual priors, rather than evidence of pixel-level overfitting.Source: §5.3 · p. 9, Figure 6 · p. 9

6 Conclusion

• LoCo is a model-agnostic framework for next-scale autoregressive models that maps concept routes across Layer, Scale, and Position.The conclusion presents LoCoBench as the accompanying dataset and reports applications to erasure, personalization, and adversarial injection.Source: §6 · pp. 9–10
• The paper reports that targeted concept-route interventions improve task effectiveness while preserving generation quality and reducing computation.These applications are concept erasure, model personalization, and adversarial injection.Source: §6 · p. 10

A Next-Scale Autoregressive Generation

• Next-scale VAR models predict each residual map in parallel, conditioned on coarser scales and the text prompt, rather than predicting one token at a time.Early scales establish broad scene structure, while later scales add local evidence that makes concepts recognizable.Source: A Next-Scale Autoregressive Generation · p. 16

B LoCoBench Dataset

• LoCoBench covers ten categories, including animals, artists, brands, celebrities, copyrighted characters, places, safety, styles, substances, and weapons.These categories map to use cases such as unlearning harmful or copyrighted content, personalization, and adversarial interventions.Source: B LoCoBench Dataset · p. 16
• Prompt augmentation varies backgrounds, lighting, camera angles, and contextual interactions to evaluate concepts across diverse scenarios rather than fixed templates.Artists are sampled from WikiArt, while entities in the other categories are generated with language-model assistance.Source: B LoCoBench Dataset · p. 16
• Compared with prior datasets used for knowledge localization and model editing, LoCoBench is substantially larger in scale, totaling 66,030 text prompts, and offers greater semantic diversity for evaluating concept routing in next-scale autoregressive generation.Source: B LoCoBench Dataset · p. 17

C.1 Concept Erasure

• Localized fine-tuning on top-K routes erases the target concept while preserving background structure and image quality, unlike full-model updates that introduce visual shifts.Naive fine-tuning accumulates errors; conditional alignment stabilizes generation but shifts visuals when applied across the full model.Source: Figure 7 · p. 17, §C.1 · p. 19
• At inference, swapping cross-attention conditions at the target’s top-K positions for a neutral anchor suppresses the concept without training.For permanent erasure, the method combines target erasure with a preservation objective on unrelated concepts, while restricting gradients to localized routes.Source: §C.1 · pp. 18–19

C.2 Model Personalization

• Localized personalization updates only top-K routes for the subject’s class, reducing VRAM use while preserving prior knowledge and improving subject fidelity in novel contexts.Full-architecture updates on small datasets can disrupt unrelated semantic pathways and degrade prompt alignment in complex scenarios.Source: §C.2 · p. 20

C.3 Adversarial Concept Injection

• For adversarial injection, LoCo restricts poisoning updates to routes responsive to the target context, achieving near-perfect attack success with minimal poisoned data.The paper reports preserved image quality and hidden backdoors during benign generation; full-model updates can instead cause artifacts and stylistic shifts.Source: §C.3 · pp. 20–21

D.1 Implementation Details

• Baselines designed for continuous noise or vector-field prediction are reformulated as next-scale probability alignment objectives for VAR.The adapted baselines also receive frozen neutral tokens to reduce error accumulation and generation collapse during autoregressive decoding.Source: §D.1 · p. 21
• ESD applies classifier-free-guidance-based regularization to predicted probabilities, while MACE combines closed-form cross-attention refinement with global LoRA fine-tuning.The VAR adaptations replace diffusion noise matching with next-scale probability alignment.Source: §D.1 · p. 21
• EraseAnything uses global full-model LoRA and an attention-map regularizer, whereas LoCo updates only top-K localized routes without auxiliary regularization.Both approaches use a bi-level erasure and preservation objective, but differ in optimization scope.Source: §D.1 · p. 21

D.2 Additional Results

• 3D targeted fine-tuning addresses temporal flickering and motion degradation attributed to global updates across spatial queries and generation scales.A binary routing mask marks the Top-K localized layer-scale-position triplets for the subject’s base class.Source: §D.2 · p. 22
• The method freezes base weights and gates trainable LoRA activations and gradients with the routing mask, confining subject identity updates to selected pathways.The paper says this leaves most spatiotemporal representations untouched.Source: §D.2 · p. 22
• On InfinityStar video personalization, the 3D localized intervention yields superior temporal consistency and subject fidelity in qualitative evaluations.The cited qualitative results appear in Figures 8, 9, and 10.Source: §D.2 · p. 22

D.3 Additional Ablation Study

• Targeted Top-5% intervention lowers Target CLIP to 0.185 while maintaining Background CLIP at 0.311; random positions reduce Background CLIP from 0.312 to 0.264, and bottom-ranked positions yield no suppression.Table 6 reports the counterfactual intervention comparison averaged over 500 LoCoBench prompts.Source: §D.3 · p. 22, Table 6 · p. 24
• Localized routes show 89.4% average Cross-Seed IoU and 83.4% Cross-Prompt IoU, while Cross-Concept IoU is below 5%.Table 7 reports overlap of Top-5% triplets across seeds, prompts, and different concepts.Source: §D.3 · p. 23, Table 7 · p. 25
• A single forward pass provides O(L×S)× speedup over evaluating every layer-scale block, while the Target CLIP drop is 0.099 versus 0.102 for brute-force search.Both drops are from 0.284, and the localized method maintains the same background preservation as the brute-force solution.Source: §D.3 · p. 24

E Limitations

• LoCo uses an empirically chosen sparsity threshold, such as the top 5% of routed positions, and does not automatically determine each concept’s minimum route capacity.The authors suggest estimating K from attention-map entropy or peak sharpness as future work.Source: E Limitations · p. 25
• Highly correlated concepts may share neural pathways, making perfectly disjoint localization difficult for highly compositional prompts.The authors suggest synthetic architectures with known structural ground truths to strengthen future validation.Source: E Limitations · p. 25

F Ethics Statement

• The authors describe the work as academic research intended to understand concept routing and support more transparent, controllable, and safety-aligned generative systems.They state that LoCo targets only a small, relevant part of generation to reduce unnecessary changes to benign behavior.Source: F Ethics Statement · p. 25

NeurIPS Paper Checklist

• The authors say LoCoBench categories study model behavior rather than encourage harmful use, and characterize the results as technical insights for future work.The checklist states that the abstract and introduction’s claims align with the paper’s contributions and scope.Source: F Ethics Statement · p. 25, NeurIPS Paper Checklist · p. 26

Reproducibility and assets

• The authors report that implementation details for reproducing main results are provided in Appendix D.1 and say code and data will be publicly released after preparation.The checklist also says new-dataset documentation will accompany the asset.Source: §2 · pp. 27, 28, 31

Experimental details and compute

• The checklist says experimental settings and details are provided in Section 5.1 and the appendix, and compute-resource information is in Section 5.3.It answers Yes to both checklist questions.Source: §2 · pp. 28–29

Statistical significance

• The checklist answers No to whether statistical-significance information is reported, while its justification says experiments were run precisely under given conditions.The checklist guidance describes error bars, confidence intervals, or significance tests as appropriate reporting options.Source: §2 · pp. 28–29

Ethics and societal impact

• The authors state that the research conforms to the NeurIPS Code of Ethics and that potential societal impacts are briefly discussed in the conclusion.The checklist answers Yes to both questions.Source: §2 · p. 29

Assets and research practices

• The checklist reports that external assets are credited and their applicable licenses and terms respected, and that the work involves no crowdsourcing or human subjects.It also states that an LLM was used for dataset construction and described in the paper.Source: §2 · pp. 30–32