Source-linked AI summary
Spanning the Visual Analogy Space with a Weight Basis of LoRAs
Hila Manor, Rinon Gal, Haggai Maron, Tomer Michaeli, Gal Chechik
TL;DR
Visual analogy learning addresses the difficulty of expressing complex image transformations through text, but single-LoRA adaptation may not generalize across diverse transformations. LoRWeB learns a basis of LoRAs and dynamically weights it from the analogy input, achieving state-of-the-art results and improved generalization to unseen transformations. The method remains constrained by tasks far outside its training corpus and by the need for suitable reference image pairs.
Problem
Text is insufficiently precise for many complex visual transformations, while single-LoRA adaptation attempts to represent diverse transformations within one module.
Method
LoRWeB jointly trains a learnable basis of LoRA adapters with an encoder that predicts combination coefficients conditioned on each input analogy pair.
Results
LoRWeB achieves state-of-the-art results and improves generalization to unseen transformations over competing single-LoRA approaches.
Takeaways & Limitations
LoRA basis decompositions support flexible visual analogy completion by enabling layer-specific adaptation to diverse and unseen transformations.
Takeaways & Limitations
LoRWeB may struggle with tasks significantly different from its training corpus, and image analogies assume access to a reference pair isolating the transformation.
Abstract
from arXiv · showhide
Visual analogy learning enables image editing via demonstration rather than textual description, allowing users to specify complex transformations difficult to articulate in words. Given a triplet $\{\mathbf{a}$, $\mathbf{a}'$, $\mathbf{b}\}$, the goal is to generate $\mathbf{b}'$ such that $\mathbf{a} : \mathbf{a}' :: \mathbf{b} : \mathbf{b}'$. Recent methods adapt text-to-image models with a single Low-Rank Adaptation (LoRA) module, but they face a fundamental limitation: attempting to capture the diverse space of visual transformations within a fixed module constrains generalization. Inspired by recent work showing that LoRAs in constrained domains span meaningful, interpolatable semantic spaces, we propose LoRWeB, which specializes the model for each analogy task in a single inference pass. LoRWeB dynamically composes learned transformation primitives, informally, choosing a point in a "space of LoRAs". We introduce two key components: (1) a learnable basis of LoRAs to span the space of different visual transformations, and (2) a lightweight encoder that dynamically weighs these basis LoRAs given the input analogy pair. Comprehensive evaluations demonstrate state-of-the-art performance and significantly improved generalization to unseen transformations. Our findings suggest LoRA basis decompositions are a promising direction for flexible visual manipulation tasks. See https://research.nvidia.com/labs/par/lorweb for code.
1 Introduction
Visual analogy learning enables image editing through demonstrated transformations, addressing changes that are difficult to specify precisely in text. LoRWeB uses a learnable LoRA basis and analogy-conditioned weighting to improve adaptation and generalization.
- Motivation: Text descriptions struggle to articulate transformations such as specific painting styles or exact target poses.These limitations motivate alternative image-editing paradigms.
- Motivation: Visual analogy learning generates b′ by applying the transformation from a to a′ analogously to b.This lets users specify nuanced visual changes through examples rather than explicit descriptions.
- Limitations of prior work: Single-LoRA methods attempt to capture diverse visual transformations within one adaptation module, potentially limiting generalization.Earlier stand-alone analogy models also had limited task diversity or image quality, or required extensive compute.
- LoRWeB: LoRWeB jointly learns a basis of LoRA modules and a lightweight encoder that dynamically combines them from the input analogy pair.The system specializes the model to each analogy task at inference time and composes transformations for unseen analogies.
- LoRWeB: LoRWeB supplies the full analogy triplet through extended attention while using CLIP-based encodings specifically for LoRA selection.This separates fine-detail preservation from the higher-level semantics used to select transformations.
- Results: LoRWeB achieves state-of-the-art results and improved generalization to unseen transformations compared with single-LoRA approaches.The evaluation compares the method against established baselines.
2 Related Work
Visual analogy research evolved from simple per-pair methods toward in-context models and adaptations of text-to-image foundations. Related work also studies diffusion-based editing controls and semantic composition of LoRA weights.
- Visual Analogies: Early visual analogy methods used explicit filters, per-pair optimization, embeddings, or NeRF representations, but struggled with complex real-world images.Many early methods primarily addressed simple objects or style-transfer analogies.
- Visual Analogies: Newer visual analogy approaches condition models directly on exemplar pairs and reference images, framing analogy learning as in-context learning.Some recent methods adapt pretrained text-to-image foundation models beyond simple style transfer.
- Diffusion-Based Image Editing: Diffusion-based editing progressed from prompt-conditioned denoising toward feature, trajectory, control, and attention-sharing mechanisms for greater precision.Text-free editing methods support modifications that cannot be described textually, though without direct control.
- LoRA and Weight Bases: LoRA research combines multiple low-rank adapters through post-tuning composition or mixture-of-experts formulations.LoRA is a parameter-efficient fine-tuning method that adds learned low-rank matrices to existing model weights.
- LoRA and Weight Bases: Independently trained LoRAs can form a semantic basis whose interpolations represent meaningful blended concepts, but prior practical use required extensive training and test-time optimization.Related language-processing work also combines LoRAs for new tasks from reference outputs.
3 Method
LoRWeB addresses the limited transformation diversity of single-LoRA analogy models by dynamically composing a learned LoRA basis for each input analogy. An encoder selects the mixture, which is injected into a conditional flow model operating on a composite analogy image.
- Preliminaries: LoRA adapts frozen model weights through a low-rank update ΔW = BA, with final weights W = W0 + αBA.The factors have shapes B ∈ R^m×r and A ∈ R^r×n, with r typically much smaller than the original dimensions.
- Our Approach: The conditional flow model receives the composite analogy image and a guiding edit prompt, producing b′ in the bottom-right quadrant.The full analogy triplet is supplied to the diffusion model through extended attention, while CLIP-based encodings select the LoRA mixture.
- Naive Solutions and Limitations: Single-LoRA analogy methods condition a model on a 2 × 2 composite image, but struggle to generalize beyond transformations seen during training.The composite input contains a, a′, and b, while the target output replaces the bottom-right b with b′.
- Our Approach: LoRWeB jointly trains N rank-r LoRAs, learnable keys, and an encoder that predicts combination coefficients from the analogy triplet.A frozen ViT encoder and small projection module produce a query vector used to compare the input analogy with the learned keys.
- Our Approach: The encoder computes coefficients through key similarity, then combines the basis updates as ΔW = Σ e_i(B_iA_i) to form a mixed LoRA.This mixed LoRA is injected into a conditional flow model for the analogy edit.
- Our Approach: A constant router reduces LoRWeB to a static combination equivalent to a single higher-rank LoRA, whereas input-dependent mixing spans more of the rank-r′ LoRA space.The distinction is that LoRWeB combines LoRAs differently for different analogy inputs.
4 Experiments
LoRWeB is evaluated on diverse unseen visual analogy tasks against established baselines, using qualitative, quantitative, human, and ablation studies. The results show broader generalization, stronger edit-preservation tradeoffs, and greater reliance on analogy images than competing approaches.
- Qualitative Evaluations: Compared with RelationAdapter, VisualCloze, EditTransfer, and a standard Flux.1-Kontext LoRA, LoRWeB better preserves subject identity and transformation details across diverse tasks.
- Qualitative Evaluations: LoRWeB generalizes to unseen tasks including style transfer, background replacement, object insertion, and object displacement.
- Quantitative Evaluations: LoRWeB achieves higher edit accuracy while strongly preserving the input image, pushing the Pareto front of edit accuracy-preservation.
- Quantitative Evaluations: Users favor LoRWeB over all baselines in a two-alternative forced-choice study covering 33 users and 45 image pairs, aligning with automated metrics.
- Ablations: Reducing basis capacity or increasing rank naïvely can reduce performance, indicating that parameter count alone does not guarantee better generalization.
- Ablations: Under detail-reduced prompts, LoRWeB is preferred over RelationAdapter, EditTransfer, and VisualCloze at 59.4%, 70.4%, and 66.8%, respectively.
5 Discussion
LoRWeB uses a learned basis of LoRA adapters and a shared encoder to compose task-specific adaptations for diverse visual analogies. It generalizes better than naive LoRA-based methods, but remains limited by out-of-distribution tasks and the availability of suitable reference pairs.
- LoRWeB learns a basis of LoRA adapters and dynamically composes them with a shared encoder conditioned on the input analogy.The framework enables layer-specific adaptations for diverse and unseen transformations.
- LoRWeB outperforms and generalizes better than competing naive LoRA-based methods across various visual analogy tasks.
- LoRWeB may still struggle with tasks that differ substantially from its training corpus.
- The method assumes a reference pair whose noticeable change isolates the transformation while preserving other image details.The paper reports that non-identical pairs can work with textual guidance, within limits.
- A future direction is decomposing transformations into components such as pose and background for interactive selection.
- The LoRA-basis approach may also apply to other tasks where generalization is needed.The paper identifies this broader applicability as future work.
A.1 Implementation Details
The experiments use a fixed 10K-step training setup on one H100 GPU with specified optimization, precision, batching, and encoder configurations.
- All experiments train for 10K steps on 1 H100 GPU using 8-bit AdamW with a 10^-3 learning rate.The setup also specifies β1 = 0.9, β2 = 0.99, weight decay 0.05, and bfloat16 mixed precision.
- Experiments use gradient checkpointing and batch size 6, except r = 16, N = 32, which uses batch size 4.
- The main encoder uses the openai/clip-vit-large-patch14 CLIP checkpoint.
A.2 Efficiency Analysis
LoRWeB adds multiple LoRA modules at inference but incurs only a small measured runtime increase over a single standard LoRA, with further caching optimization possible.
- +3.1% runtime is the measured inference increase over a single r = 128 LoRA.Averaged over 100 images on an A100 GPU, LoRWeB takes 33.4 ± 0.4 seconds versus 32.4 ± 0.3 seconds.
- LoRWeB passes the input triplet through CLIP once and combines the LoRA basis per layer using efficient Einsum multiplication.The paper describes the resulting inference overhead as negligible relative to Flux.1-Kontext.
- Caching the fixed LoRA mixture from the first timestep could further reduce inference cost.
A.3 Custom Inference Dataset
The custom inference dataset extends evaluation with Unsplash images, generated prompts, pretrained LoRA adapters, and aspect-ratio-matched preprocessing.
- Unsplash images are used for the inference dataset extension under the Unsplash license.
- GPT-4o and Claude Sonnet 4 summarize Relation252k training prompts to generate novel prompts for animals, objects, and persons.The dataset uses 15 randomly selected prompts per concept category.
- The evaluation uses 18 pretrained LoRA adapters sourced from HuggingFace and covering diverse transformations.
- Images are selected within 0.15 aspect-ratio distance and cropped to a’s aspect ratio before resizing to a maximum long edge of 512.
A.4 VLM Based Evaluation
The evaluation uses VLM-based scoring and human preference checks to assess editing accuracy, preservation, and pairwise analogy quality. LoRWeB outperforms the compared diffusion analogy and per-sample optimization baselines across the reported evaluations.
- VLM-based metrics: Gemma-3 evaluates each result for editing accuracy relative to the reference transformation and consistency with the original input image.The protocol provides {a, a′, b, b′} and scores how precisely b′ applies the transformation while preserving unchanged content.
- VLM-based metrics: Gemma-3 also performs 2AFC comparisons between LoRWeB and baseline outputs for the same analogy triplet.The VLM selects which of two b′ images better applies the analogy.
- Human alignment: 66.7% average user–VLM agreement is below the 74.2% average user–user agreement, corresponding to 89.9% evaluation consistency with humans.The reported vote variability is 0.3423 for users and 0.4649 for VLM predictions relative to users’ mean.
- Baseline comparison: LoRWeB outperforms DIA, DIA-Kontext, and PairEdit across the reported comparisons and metrics.DIA-Kontext preserves object identity but struggles with the transformation, while PairEdit lacks a strong balance between edit adherence and content preservation.
B.2 Additional Quantitative Results
Additional capacity experiments compare larger LoRWeB configurations with a higher-capacity single LoRA. The results show that increasing parameter count alone does not guarantee better performance and may cause overfitting.
- Capacity ablation: Naïve parameter addition does not strictly correlate with better performance and can cause the methods to overfit.The ablation compares LoRWeB with r = 4, N = 64 against a single LoRA with r = 256.
B.3 Additional Qualitative Results
Additional qualitative evaluations examine generalization, prompt and image-pair alignment, and reference-image source sensitivity. LoRWeB generalizes across diverse transformations, but its performance degrades for out-of-distribution, subtle, or multi-component edits.
- Unseen transformations: The LoRA basis supports qualitative generalization to styles, backgrounds, and clothing transformations not seen in the displayed examples.Examples include clay toys, bronze sculptures, background changes, and clothing edits.
- Unseen transformations: LoRWeB better preserves subject details and applies analogies across diverse unseen tasks than the compared baseline methods.The comparison focuses on both transformation generalization and preservation of visual details.
- Reference-pair sensitivity: Non-identical reference pairs can work with textual guidance, but changing the reference subject may distort the output and reduce identity and background preservation.Changing the dog’s position did not harm the result, whereas replacing it with a different dog did.
- Prompt and image alignment: Mismatched textual prompts and image pairs can produce a combined-editing effect containing elements from both inputs.The output may reflect both the prompt-described edit and the transformation shown by the analogy pair.
- Reference-image source: Reference pairs from TEdBench, including images edited by Imagic, do not hamper LoRWeB’s performance despite not being generated by Flux.This tests sensitivity to the source model used to create the reference pair.
- Failure cases: LoRWeB may struggle with tasks far from training, small-detail changes, or transformations involving multiple components.Reported examples include cubist style transfer, closing an animal’s eyes, and applying only one part of a collar-and-bell transformation.