PaperlayerView exact revision on arXiv

Source-linked AI summary

V-Engram: Trigger-Indexed External Memory for Modular Text-to-Image Personalization

Haoran He, Runyuan Cai, Yiming Wang, Lin Yu, Xiaodong Zeng

arXiv:2609.37198v1cs.AI

TL;DR

Few-shot personalization must acquire or refine a visual identity from limited references while retaining compositional control. V-Engram uses explicit triggers to retrieve external memory for a frozen Stable Diffusion 3.5 model. It is broadly competitive with LoRA, with better identity preservation in held-out contexts and complementary prompt-alignment strengths.

  • Problem

    Few-shot personalization must bind limited references to a textual handle, recover fine-grained appearance, follow new contexts, and support selective composition.

    Source: §1 · p. 2

  • Method

    V-Engram stores concept-specific gated residual directions in trigger-indexed external memory and injects them into frozen text-encoder and MMDiT context states.

    Source: §1 · p. 2

  • Results

    V-Engram and LoRA show broadly comparable fidelity, with V-Engram preserving identity better in held-out contexts while LoRA favors prompt alignment.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    Exact n-gram routing leaves unmatched prompts on the frozen base path and supports selective activation, paired-trigger composition, and same-class separation.

    Source: §1 · p. 2, §5 · p. 12

Abstract

from arXiv · show

Pretrained text-to-image models contain broad visual knowledge, yet they cannot reliably acquire or refine a specific visual identity from only a few references while preserving compositional control. Token-embedding methods are compact but often underfit identity, whereas adapter-based methods improve fidelity through persistent weight updates that can be costly to store and interfere when concepts are composed. We introduce V-Engram, a trigger-indexed external memory mechanism for Stable Diffusion 3.5. Each concept is assigned an explicit trigger that retrieves concept-specific memory, whose gated directions enter frozen text-encoder and MMDiT context states as relative residuals. Separating this memory from backbone adaptation enables prompt-selective and multi-concept access without merging model updates. Experiments show that V-Engram broadly matches DreamBooth-LoRA in overall subject fidelity while showing advantages in settings such as contextual subject preservation. Prompt-matched loading retrieves only matched entries, reducing most additional adaptation-state loading for a single-concept query. Qualitative results further demonstrate paired-trigger composition and same-class separation, while prompts without registered entries retain the frozen model's base behavior. Together, these results establish trigger-indexed memory as a modular interface for adding targeted visual evidence without rewriting the generator.

1 Introduction

Few-shot personalization must acquire or refine visual identities from limited references while preserving contextual control and selective composition. V-Engram addresses this with trigger-indexed external memory on a frozen Stable Diffusion 3.5 backbone, and experiments show broadly comparable fidelity to DreamBooth-LoRA with complementary strengths.

1 Introduction

  • Few-shot personalization requires binding a concept to a textual handle, recovering fine-grained appearance, following new contexts, and allowing inactive or composable evidence.This covers both acquiring new concepts and selectively refining imprecise prior knowledge.Source: §1 · p. 2
  • Existing approaches expose a capacity–modularity trade-off: token embeddings can lack identity detail, while generator-path adaptation can create concept-coupled state and composition interference.Encoder-based systems instead condition on references at inference time.Source: §1 · p. 2
  • V-Engram stores concept-specific entries outside the backbone and injects their gated directions as relative residuals into frozen text-encoder and MMDiT context states.Exact trigger matches activate registered entries; the overview depicts tokenizer-specific, layer-wise entries.Source: §1 · p. 2, Figure 2 · p. 3
  • V-Engram and LoRA show broadly comparable fidelity, with V-Engram preserving identity better in held-out contexts while LoRA favors prompt alignment.Qualitative studies also show paired-trigger composition, inactive-route preservation, and same-class separation.Source: §1 · p. 2
  • The paper frames few-shot acquisition and refinement as exact trigger-indexed retrieval, with explicit prompt-selective memory access in SD3.5.Its contributions characterize trade-offs against embedding-only learning and joint LoRA across fidelity, addressable loading, and compositional access.Source: §1 · pp. 2–3

2 Related Work

Related work spans external-memory addressing, optimization-based personalization, reference-conditioned methods, multi-concept control, and diffusion-transformer architectures. V-Engram combines deterministic lexical addressing with reference-trained, text-addressed memory injected into SD3.5 context states.

Conditional Memory

  • External-memory methods use structured content lookup, approximate kNN retrieval, or deterministic token n-gram mapping; V-Engram instead learns visual-reference-trained residual directions and gates for a frozen diffusion model.It uses deterministic lexical addressing without semantic or approximate retrieval.Source: §2 · p. 3

Optimization-Based Personalization

  • Optimization-based methods learn concepts in token space or model updates, whereas V-Engram keeps the backbone frozen but stores layer-wise external memory beyond one input embedding.The cited approaches include Textual Inversion [Gal et al., 2023], DreamBooth [Ruiz et al., 2023], and low-rank updates [Hu et al., 2022].Source: §2 · p. 3

Reference-Guided Methods

Multi-Concept Methods

  • Multi-concept methods jointly learn concepts, extract handles, fuse adapters, or reduce interference; V-Engram uses exact-match residuals for selective activation of persistent concept memory.Prior work includes TARA [Peng et al., 2026], which reduces interference in LoRA composition through rare-token masking and token–region alignment.Source: §2 · p. 4

Diffusion Transformers and MMDiT

  • Because SD3.5 combines CLIP/OpenCLIP and T5 streams in MMDiT, where text and image tokens interact jointly, V-Engram targets concept-specific text and context states.This goes beyond an initial embedding or U-Net cross-attention interface.Source: §2 · p. 4

3 Method

V-Engram trains external concept memories while keeping the pretrained Stable Diffusion 3.5 components frozen. Exact trigger matches select memories for gated residual injection; unmatched prompts retain the frozen conditioning path.

3.1 Overview

  • Each concept’s memory stores trainable directions and tanh gates at mounted sites, while the backbone parameters remain unchanged.A mounted site identifies a tokenizer/conditioning stream and hidden-state location.Source: §3.1 · p. 4

3.2 Trigger-Indexed Memory Registry

  • Exact tokenizer-specific trigger matches retrieve registered entries, while unmatched prompts follow the frozen path.Each tokenizer stream has its own registry; all exact matches, including overlaps, are active and summed.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.3 Gated Relative Residual Injection

  • Each matched entry adds a gated relative residual calibrated to the hidden-state scale; overlapping matches contribute across covered token positions.The direction sets residual direction, the gate modulates signed per-channel strength, and the detached norm calibrates activation scale.Source: §3.3 · p. 5

3.4 Memory Mounting in Stable Diffusion 3.5

  • Memory is mounted in all three text-encoder streams and selected MMDiT text/context blocks, with tokenizer-specific offsets preserving the corresponding context regions.The pooled projection path is unchanged; the default MMDiT configuration uses seven evenly distributed blocks.Source: §3.4 · p. 5

3.5 Training Objective and Storage

  • Training freezes pretrained components and updates only Engram directions and gates; external state stores these banks and trigger configuration, and scales with registered entries and mounted sites.At inference, only prompt-matched entries activate, and prompts without active triggers use the unmodified frozen conditioning path.Source: §3.5 · p. 6

4 Experiments

Experiments compare V-Engram with embedding learning and LoRA across subject fidelity, memory mounting, contextual prompts, composition, routing, and familiar-identity refinement. Results show broadly comparable overall fidelity to LoRA, with advantages in contextual subject preservation and selective loading, alongside qualitative evidence of compositional access and base-behavior preservation.

4.1 Experimental Setup

  • The main benchmark uses 15 subjects; learned methods share references and joint training, while evaluation uses five generated images per subject and macro-averages metrics across subjects.The familiar-identity refinement split uses five training and five image-disjoint test images per identity.Source: §4.1 · pp. 6–7

4.2 Main Comparison

  • V-Engram and rank-8 LoRA achieve broadly comparable subject fidelity under the simple prompt, with neither dominating across CLIP-I and DINOv2 representations.The comparison covers 15 subjects; rank-8 LoRA is higher in DINOv2, while V-Engram essentially matches it in CLIP-I.Source: §4.2 · pp. 7–8, Table 1 · p. 7
  • For a single-trigger query to a 15-trigger checkpoint, matched loading activates 1.061M rather than 10.477M Engram parameters, avoiding 89.87% of full-registry adaptation-state loading.In the controlled profile, V-Engram uses 0.940–1.004 GiB less peak training memory but takes 14.9–29.0% longer per step.Source: §4.2 · p. 8

4.3 Ablation Study

  • All mounted variants outperform encoder-only on both metrics, but density is non-monotonic; seven MMDiT blocks perform best on the validation subset, not necessarily across architectures.The matched residual ablation raises DINOv2 from 0.7065 to 0.8113 and CLIP-I from 0.8262 to 0.8892.Source: Table 2 · p. 8, Figure 4 · p. 8, §4.3 · p. 9

4.4 Additional Analysis

  • Across five shared contextual templates, V-Engram better preserves personalized subjects while rank-8 LoRA aligns text better; increasing LoRA rank improves CLIP-T but reduces both CLIP-I scores.V-Engram leads rank 8 in best-reference and all-pairs CLIP-I for 14/15 subjects; rank 8 leads CLIP-T for 13/15.Source: §4.4 · p. 9, Table 3 · p. 9, Figure 5 · p. 9
  • In paired-trigger prompts, V-Engram more consistently depicts both concepts, while LoRA sometimes omits or blends one; both methods still exhibit failures.The comparison is qualitative because full-image similarity cannot attribute retention to each subject.Source: §4.4 · p. 10, Figure 6 · p. 10
  • Without registered triggers, exact-match logs show no active entries and V-Engram reduces exactly to the frozen base; indexed and arbitrary-name triggers both preserve dog-specific cues in qualitative tests.LoRA remains globally active and can shift outputs toward training instances; with indexed dog triggers it tends toward similar dogs, and with names can invoke human priors.Source: §4.4 · p. 10, Figure 7 · p. 11
  • On 15 familiar identities with image-disjoint tests, V-Engram exceeds rank-4 LoRA by 0.1466 best-reference and 0.1401 all-pairs similarity, winning 13/15 identities.Both adaptations beat the base on all 15 identities, and all 75 generations per method are valid detections.Source: §4.4 · p. 11, Table 4 · p. 11, Figure 8 · p. 12

5 Conclusion

V-Engram is broadly competitive with LoRA, with different strengths in simple prompts and held-out contexts rather than universal superiority. Exact token matching provides selective memory access while leaving unmatched prompts on the frozen base path.

5 Conclusion

  • V-Engram has comparable CLIP-I but lower DINOv2 than LoRA on simple subject prompts; in held-out contexts, it preserves identity better while LoRA favors prompt alignment.Source: §5 · p. 12
  • Exact n-gram routing leaves unmatched prompts on the frozen base path and supports selective memory activation and loading.Source: §5 · p. 12
  • Analyses support familiar-identity refinement, paired-trigger composition, and same-class separation, but show a different balance—not universal superiority—over weight adaptation.Source: §5 · p. 12
  • Exact token matching may address external memory in other token-conditioned generative or multimodal architectures, but cross-architecture validation remains future work.Source: §5 · p. 12

Related papers

Loading 2609.37198v1…