Source-linked AI summary
RefDiT: Local Attribute Guidance in Reference-Based Image Generation
Rameshwar Mishra, Srikrishna Karanam, A V Subramanyam
TL;DR
Existing reference-guided generation methods struggle to use local attributes in complex multi-object scenes because global conditioning and single identifier tokens limit local control. RefDiT uses triplet-based local guidance and triplet-conditioned LoRA training for DiT models, and experiments report improved reference guidance across diverse references. Its attribute extraction remains bounded by errors in object detection, attribute labeling, and relation parsing.
Problem
Existing methods struggle to generate images guided by multiple local attributes because global conditioning and single identifier tokens limit local and attribute-level control.
Method
RefDiT extracts object triplets and uses local triplet selection plus triplet-conditioned LoRA training to learn correspondences between identifier tokens and local reference regions.
Results
RefDiT improves reference guidance across a diverse set of references, with extensive comparisons showing consistent improvements over personalization, editing, and commercial models.
Takeaways & Limitations
RefDiT provides users with greater control over generation by enabling effective local, attribute-level guidance from reference images.
Takeaways & Limitations
Errors in object detection, attribute labeling, or relation parsing can propagate into the conditioning signals.
Abstract
from arXiv · showhide
Personalization models generate new images guided by a few subject references, while style transfer methods aim to produce images aligned with a global style derived from a reference image. Recent approaches perform well when the reference image contains a single object, effectively capturing a global style that encompasses all implicit attributes. However, when applied to complex real-world scenes containing multiple objects with distinct attribute characteristics, these methods, due to their global-level guidance, fail to localize relevant elements in the reference image. The global guidance restricts their ability to generate new images based on the local attributes in the reference image. Moreover, existing methods typically employ a single identifier token to capture all details from the reference, resulting in a lack of individual, attribute-level control. Motivated by these limitations, we propose RefDiT, a novel framework for reference-guided image generation. RefDiT takes as input a reference image, a text prompt, and an optional user-provided guidance context. RefDiT employs local region guidance using the attributes of local elements. It constructs an attribute-aware conditioning signal from the reference image by performing attribute-level decomposition of the identifier token and performs context adjustment in the inference prompt to train low-rank adapter (LoRA) blocks of a diffusion transformer (DiT)-based generative model. RefDiT learns the correspondence between identifier tokens and local regions in the reference image, enabling more effective local guidance.
1 Introduction
Existing reference-guided methods use global conditioning and a single identifier, limiting local and attribute-level control in complex multi-object scenes. RefDiT addresses this gap by learning region-level correspondences for local attribute guidance.
- Motivation: Complex scenes require generation guided by attributes of specific local elements, not only global style and content.Local regions can provide critical guidance when generating novel images containing similar elements.
- Limitations: Existing methods often fail on multi-object scenes because global conditioning and a single identifier provide neither local nor attribute-level control.Their global-level guidance cannot effectively localize relevant elements with distinct attribute properties.
- Limitations: In a cube, cat, and mug scene, prior methods fail to localize the mug or guide generation by its color and material for a “cup and saucer” prompt.They also provide no user control over which local elements and attributes should guide generation.
- RefDiT: RefDiT extracts structured object triplets that represent object attributes and interobject relationships without requiring bounding boxes or segmentation masks.These triplets provide semantic information for constructing global and local guidance signals.
- RefDiT: Triplet-conditioned LoRA training for DiT models establishes correspondences between identifier tokens and their associated local reference regions.The method uses these correspondences to support local guidance during generation.
- Results: Extensive qualitative and quantitative comparisons show consistent improvements in reference-guided image generation across diverse references.The comparisons include LoRA-based personalization, DiT-based editing, and commercial image generation models.
2 Related Work
Related work includes diffusion-model customization methods that learn reference-specific visual concepts and editing methods that modify local attributes. However, local editing does not directly address generating new images guided by locally present reference elements.
- Diffusion fine-tuning for customization: Diffusion customization methods capture reference details through text embeddings, full-model fine-tuning, or optimization of selected diffusion-model components.The related methods include Textual Inversion, DreamBooth, and approaches that learn visual concepts by adapting specific model parts.
- Image editing: DiT-based editing methods can modify attributes of locally present elements but are not designed to guide generation of a new image from those elements.The paper distinguishes local manipulation of an existing image from reference-guided synthesis.
3 Method
RefDiT builds attribute-aware global and local conditioning from reference-image triplets, then trains DiT LoRA blocks so identifier tokens correspond to local regions. Its triplet-conditioned streams and shared attention weights provide localized guidance while retaining global reconstruction.
- DiT preliminaries: RefDiT uses rectified-flow conditioning and multimodal DiT attention to jointly process image patches and text embeddings.The model combines visual and textual projections through unified attention, adaptive normalization, and residual connections.
- Problem setup and conditioning: Given a reference image, prompt, and optional control, the method selects relevant local elements through control-constrained or category-based matching.Selected triplets and their attributes are added to the inference prompt as identifier tokens.
- Structural Conditioning Module: SCM extracts object-interaction triplets and decomposes object attributes into identifiers that form a global attribute-aware conditioning signal.Identifiers encode attribute types and values and are appended to objects before triplets are concatenated.
- Triplet-conditioned LoRA training: Global conditioning alone causes identifier tokens to attend broadly, whereas triplet-conditioned LoRA training makes them attend more dominantly to corresponding local objects.Triplet candidates are used only during training and do not modify the inference-time architecture.
- Local correspondence: Triplet candidates initialized from reference patch features are updated across DiT layers using shared attention weights and triplet-level text conditioning.This auxiliary stream progressively guides attention toward regions corresponding to matched local triplets.
4 Experiments
Experiments evaluate RefDiT on multi-object references against personalization, editing, and commercial models using qualitative comparisons, attribute-aware metrics, ablations, and user studies. RefDiT provides local attribute guidance while maintaining prompt and reference alignment.
- The evaluation uses more than 500 generated images from 60 diverse references, with 3–5 prompts and three samples per prompt.
- Qualitative Comparisons: RefDiT selectively transfers attributes from relevant local elements while preserving prompt consistency in complex multi-object scenes.
- Qualitative Comparisons: Existing personalization and editing methods struggle to localize relevant regions or preserve local attributes when generating novel objects.
- Comparative Analysis: Attr-Match measures matched attributes across corresponding objects, while Attr-SIM compares free-form attribute descriptions for semantically similar local regions.
- Comparative Analysis: RefDiT improves Attr-Match and Attr-SIM over open-source baselines and remains competitive with GPT-5 and Gemini-Banana.
- Ablation Analysis: SCM ablations show that structurally attribute-aware conditioning improves guidance, while InternVL achieves Attr-Match 0.80 and Attr-SIM 0.63.
5 Discussion
The discussion validates SCM extraction but identifies conditioning and evaluation boundaries that affect interpretation of local attribute correspondence.
- SCM achieves 90.5% attribute accuracy across five attributes and 94% object-classification accuracy on a manually validated subset of 50 images.
- Extraction errors can propagate into conditioning, especially for rare objects, ambiguous attributes, or heavy occlusion.
- CLIP-I and CLIP-T measure global alignment but are insufficient for evaluating region-level guidance, motivating Attr-Match and Attr-SIM.
6 Conclusion
RefDiT addresses local attribute guidance in reference-constrained generation through a LoRA-based DiT strategy. Experiments report improved guidance, greater user control, and perceptual support from multiple metrics and a user study.
- RefDiT enables effective local, attribute-level guidance for reference-constrained image generation.
- The method uses a local attribute-guided LoRA-based training strategy for DiT-based image generation models.
- Experiments report improved reference guidance across diverse references and greater user control over generation.
- Multiple evaluation metrics and a user study support the perceptual quality of the generated outputs.