Source-linked AI summary
In-Context LoRA for Diffusion Transformers
Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, Jingren Zhou
TL;DR
Adapting text-to-image models to coherent image-set generation across diverse tasks remains challenging, especially when existing approaches produce lower-fidelity results. This paper introduces In-Context LoRA, which concatenates images, jointly captions them, and applies small-data task-specific LoRA tuning without architectural changes, achieving high-quality results across diverse tasks while retaining a task-agnostic framework.
Problem
Existing image-generation approaches often rely on task-specific architectures, while prior in-context adaptation can underperform pretrained text-to-image models in generation fidelity.
Method
In-Context LoRA concatenates images, jointly captions image sets, and applies LoRA fine-tuning on 20–100 image sets without modifying the original DiT architecture.
Results
The pipeline adapts text-to-image models to diverse tasks with high-quality outputs while preserving a task-agnostic framework and requiring minimal data and computation.
Takeaways & Limitations
Text-to-image models can be reused for diverse in-context generation tasks through data changes and lightweight tuning rather than architectural modification.
Takeaways & Limitations
The approach assumes that base text-to-image models possess in-context generation capabilities, whose quality can vary across tasks.
Abstract
from arXiv · showhide
Recent research arXiv:2410.15027 has explored the use of diffusion transformers (DiTs) for task-agnostic image generation by simply concatenating attention tokens across images. However, despite substantial computational resources, the fidelity of the generated images remains suboptimal. In this study, we reevaluate and streamline this framework by hypothesizing that text-to-image DiTs inherently possess in-context generation capabilities, requiring only minimal tuning to activate them. Through diverse task experiments, we qualitatively demonstrate that existing text-to-image DiTs can effectively perform in-context generation without any tuning. Building on this insight, we propose a remarkably simple pipeline to leverage the in-context abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint captioning of multiple images, and (3) apply task-specific LoRA tuning using small datasets (e.g., 20~100 samples) instead of full-parameter tuning with large datasets. We name our models In-Context LoRA (IC-LoRA). This approach requires no modifications to the original DiT models, only changes to the training data. Remarkably, our pipeline generates high-fidelity image sets that better adhere to prompts. While task-specific in terms of tuning data, our framework remains task-agnostic in architecture and pipeline, offering a powerful tool for the community and providing valuable insights for further research on product-level task-agnostic generation systems. We release our code, data, and models at https://github.com/ali-vilab/In-Context-LoRA
1 Introduction
The paper argues that text-to-image DiTs already possess in-context generation abilities and introduces IC-LoRA, a simple task-agnostic framework that activates them without architectural changes. It combines image and prompt concatenation with LoRA tuning on only 20–100 image sets to produce high-quality outputs across diverse tasks.
- Motivation: GDT generates related image sets by concatenating attention tokens, but its fidelity often falls below that of original pretrained text-to-image models.This limitation motivates re-examining how text-to-image models are adapted for complex generative tasks.
- Key insight: Existing text-to-image models inherently support in-context generation, maintaining attributes such as subject identity, style, lighting, and font across multi-image outputs despite imperfections.The authors validate this assumption by applying existing models to diverse tasks requiring sets of images with varied relationships.
- Pipeline: 20 ∼100 image sets suffice for LoRA fine-tuning, reducing computation while largely preserving the base model’s knowledge and in-context capabilities.The approach uses task-specific tuning data rather than large-scale training on hundreds of thousands of samples.
- Pipeline: IC-LoRA concatenates images into one large image and merges per-image captions into one prompt, replacing GDT’s attention-token concatenation and exclusive per-image text cross-attention.Image concatenation is approximately equivalent to token concatenation in DiTs aside from VAE-related differences.
- Results and scope: The framework requires no architectural modifications and adapts to diverse tasks with high-quality outputs, while remaining task-agnostic in its overall architecture and pipeline.For image-conditional generation, it masks images in the concatenated image and uses SDEdit to inpaint them from the remaining images.
2 Related Work
Related work has advanced high-fidelity text-to-image generation and task-agnostic controllable frameworks, but task-specific architectures still limit flexibility and generalizability. The paper argues that existing text-to-image architectures already provide in-context capabilities, avoiding new architectures while enabling efficient, high-quality generation.
- Text-to-image models generate high-fidelity images from complex textual prompts but often lack fine-grained control over specific attributes.
- Task-specific architectures and pipelines limit flexibility and generalizability because capabilities developed for one task are not easily composable or extendable to arbitrary new tasks.
- Recent work has pursued task-agnostic frameworks supporting multiple controllable image-generation tasks within a single architecture, including Emu Edit, Emu2, Emu3, and TransFusion.
- The paper proposes that existing text-to-image architectures inherently possess in-context capabilities, eliminating new architectures while requiring minimal additional data and computational resources.
- The proposed approach is presented as improving efficiency and delivering superior generation quality across a wide array of tasks.
3 Method
The method reframes multi-image generation around consolidated image sets and prompts, preserving compatibility with existing text-to-image architectures. It activates their in-context capabilities with curated training data, concatenated images, and lightweight adaptation rather than extensive full-model training.
- Task formulation: The framework represents tasks as generating n ≥1 images conditioned on m ≥0 images and n + m text prompts.This formulation covers image translation, style transfer, pose transfer, subject-driven generation, picture books, font design, and storyboards.
- Consolidated prompting: Instead of separate prompts for each image, the method uses one consolidated prompt with an overall set description followed by individual image prompts.The unified design matches existing text-to-image models and communicates task intent through the overall description.
- In-context generation: Unlike GDT’s cross-image attention-token concatenation, the method exploits base text-to-image models’ inherent in-context generation capabilities.This assumption motivates replacing extensive large-dataset training with carefully curated, high-quality image sets.
- Image-set construction: The pipeline concatenates images into one large training image, merges their captions into a single guided prompt, and splits generated panels afterward.The merged prompt contains an overarching description and clear guidance for each panel.
- Lightweight adaptation: The approach reuses the original text-to-image architecture without structural modifications and applies low-rank adaptation instead of fine-tuning the entire model.For additional image conditioning, SDEdit inpaints concatenated images based on an unmasked image set without training.
4 Experiments
Experiments show that In-Context LoRA produces high-quality image sets across diverse practical tasks in text-conditioned generation, while image-conditioned generation is effective but can reduce cross-image consistency. The evaluation is qualitative, with unified quantitative benchmarking deferred to future work.
- Experimental Setup: The model is built on FLUX.1-dev and trained with 20 to 100 high-quality image sets per task across diverse applications.Tasks include storyboard generation, font design, portrait photography, visual identity, home decoration, visual effects, portrait illustration, and PowerPoint templates.
- Evaluation: The study presents qualitative results and defers a unified quantitative benchmark and evaluation to future work.This limitation reflects the diversity of the evaluated tasks.
- Text-Conditioned Generation: Text-conditioned generation achieves high-quality image-set results across a broad spectrum of tasks using only text prompts.Figures 4–12 present examples without additional image input.
- Image-Conditioned Generation: Image-conditioned generation combines text prompts, reference image sets, and training-free SDEdit, but visual consistency is sometimes lower than in text-conditioned generation.The method remains effective across multiple tasks, while Figure 13 reports inconsistencies such as changes in identity and attire.
- Text-Conditioned Generation: Across examples, simultaneously generated image sets preserve task-specific consistency in identity, style, presentation, and visual effects.The reported examples include storyboards, portraits, home decoration, font design, PowerPoint templates, visual identities, sandstorm effects, and portrait illustrations.
A Prompts of Figure 3
Figure 3 uses detailed multi-panel prompts spanning portraiture, product design, typography, visual effects, identity design, and portrait illustration. The prompts specify compositions, subjects, transformations, and intended visual qualities for each subfigure.
- Portrait Photography: Portrait Photography depicts a boy’s woodland adventure across four panels, including frogs by a stream, balancing on a branch, and inspecting a yellow mushroom.The prompt emphasizes curiosity, wonder, excitement, and fascination.
- Product Design: Product Design presents a multifunctional baby walker through views emphasizing its form, interactive toys, built-in lighting, style, and safety.The design is framed around play, growth, versatility, and attention to detail.
- Font Design: Font Design demonstrates a minimalist sans-serif font across luxury perfume, serene water, lavender candle, and other elegant visual settings.Visible text examples include “Essence,” “Pure Serenity,” and “Breathe Deep.”
- Sandstorm Visual Effect: Sandstorm Visual Effect contrasts a biker riding through a clear desert landscape with a second panel showing a violent sandstorm obscuring the scene.The transformation conveys a shift from freedom and adventure to swirling sand and reduced visibility.
- Visual Identity Design: Visual Identity Design pairs a playful gray koala holding cheese on a vibrant yellow background with its adaptable practical derivative application.The original graphic uses bold black outlines to emphasize simplicity and playfulness.
- Portrait Illustration: Portrait Illustration transforms a photographic portrait of a seated man in an urban park into a cartoon-like character with exaggerated features and a vibrant blue suit.The illustration includes smooth lines, oversized shoes, and a minimalist park background.