Source-linked AI summary
VIBE: Visual Instruction Based Editor
Grigorii Alekseenko, Aleksandr Gordeev, Irina Tolstykh, Bulat Suleimanov, Vladimir Dokholyan, Georgii Fedorov, Sergey Yakubson, Aleksandra Tsybina, Mikhail Chernyshov, Maksim Kuprashevich
TL;DR
Open-source instruction-based image editing still often depends on large, costly diffusion models and may fall short of real-world quality. This paper develops a compact pipeline pairing a 2B vision-language model with a 1.6B diffusion model, using efficient conditioning, realistic data, and staged alignment. It matches or surpasses substantially larger models on core tasks, especially edits requiring strict preservation, while fitting within 24 GB of GPU memory and generating up to 2K images in approximately 4 seconds.
Problem
Open-source image editing often trails proprietary quality and usability while relying on 6B to 20B models that raise training and inference costs.
Method
The paper builds a compact editing pipeline using Qwen3-VL-2B-Instruct, Sana1.5-1.6B, realistic instruction data, and a four-stage training recipe.
Results
The compact system matches or surpasses significantly larger models on core tasks, particularly those requiring strict preservation of input content.
Takeaways & Limitations
Careful data curation and training design can support practical, high-quality instruction-based editing with relatively small models.
Takeaways & Limitations
Very complex operations and hard aesthetic requests can fail or remain unstable, while diverse in-the-wild photographs are more challenging than generated images.
Abstract
from arXiv · showhide
Instruction-based image editing is among the fastest developing areas in generative AI. Over the past year, the field has reached a new level, with dozens of open-source models released alongside highly capable commercial systems. However, only a limited number of open-source approaches currently achieve real-world quality. In addition, diffusion backbones, the dominant choice for these pipelines, are often large and computationally expensive for many deployments and research settings, with widely used variants typically containing 6B to 20B parameters. This paper presents a compact, high-throughput instruction-based image editing pipeline that uses a modern 2B-parameter Qwen3-VL model to guide the editing process and the 1.6B-parameter diffusion model Sana1.5 for image generation. Our design decisions across architecture, data processing, training configuration, and evaluation target low-cost inference and strict source consistency while maintaining high quality across the major edit categories feasible at this scale. Evaluated on the ImgEdit and GEdit benchmarks, the proposed method matches or exceeds the performance of substantially heavier baselines, including models with several times as many parameters and higher inference cost, and is particularly strong on edits that require preserving the input image, such as an attribute adjustment, object removal, background edits, and targeted replacement. The model fits within 24 GB of GPU memory and generates edited images at up to 2K resolution in approximately 4 seconds on an NVIDIA H100 in BF16, without additional inference optimizations or distillation.
1. Introduction
The paper targets accessible, real-world instruction-based image editing with strict source consistency, using compact models and a staged training pipeline. Its architecture connects a vision-language model to an efficient diffusion backbone while emphasizing realistic data and flexible deployment.
- Motivation: Open-source image editors generally trail proprietary systems in quality and usability, while typical open models use 6B to 20B parameters.These scale and cost constraints slow training, iteration, and accessibility.
- Architecture and contribution: The approach uses channel-wise reference guidance, learnable meta-tokens, and a connector that maps VLM representations into the diffusion model’s conditioning space.These choices address source-image conditioning, instruction interpretation, and interface alignment.
- Training: Training proceeds through alignment, image-to-image pre-training, supervised fine-tuning, and Direct Preference Optimization on real-world instructions.The stages progressively establish the VLM–diffusion interface, learn editing behavior, refine curated triplets, and align preferences.
- Data: The data pipeline combines diverse sources, extensive filtering, and triplet inversion and bootstrapping to reduce compute and annotation costs.Filtering includes learned triplet scoring, identity checks, and image-quality scoring.
- Consistency and data: The method explicitly treats unintended changes as errors, targeting strict preservation of pixel appearance, composition, identity, and other attributes.The paper also prioritizes real user phrasing over templated or purely synthetic instructions.
- Architecture and contribution: The system combines Qwen3-VL-2B-Instruct with the Sana1.5-1.6B diffusion model and trains on approximately 15 million triplets.The design is presented as open-source, compact, and ultra-fast.
2. Related Works
Prior work spans training-free editing, larger unified foundation editors, multimodal conditioning, and increasingly sophisticated training pipelines. The paper positions its approach around efficient source conditioning, interface alignment, multi-stage training, and realistic instruction distributions.
- Evolution of editing methods: Early training-free methods are cost-efficient but often struggle to achieve high-quality editing, motivating training-based approaches.Later systems fine-tune diffusion backbones using larger datasets and more elaborate objectives.
- Efficiency constraints: Recent open foundation editors commonly use 6B to 20B diffusion backbones, increasing training and inference costs and limiting interactive refinement.The paper frames compactness as a production-oriented efficiency constraint.
- Reference-image conditioning: Channel-wise concatenation provides lightweight source-image conditioning, whereas token-wise multimodal conditioning enables richer interactions with greater architectural and computational overhead.The paper adopts the efficient channel-wise family for latency-sensitive editing.
- Instruction grounding: Instruction representation ranges from native diffusion text conditioning to external vision-language models that interpret edits in the context of the source image.A connector is needed when VLM representations must be mapped into the diffusion backbone’s conditioning space.
- Training and alignment: Training pipelines increasingly combine multi-stage learning, auxiliary objectives, supervised fine-tuning, and preference-based alignment, while interface alignment and catastrophic forgetting remain practical challenges.The proposed four-stage setup uses alignment, image-to-image pre-training, supervised fine-tuning, and preference-based post-training.
- Consistency and real-world instructions: The paper addresses gaps in synthetic instruction distributions by grounding prompts in real user queries and aggressively filtering for consistency.This focus covers ambiguity, phrasing, and intent found in realistic requests while targeting preservation of identity, backgrounds, lighting, and appearance.
3. Method
The method combines a VLM with a diffusion transformer through learnable meta-tokens and a trainable connector, then stabilizes and improves editing through staged, mixed-data training. It targets high-resolution, source-consistent editing while retaining text-to-image capability.
- Architecture: The architecture uses a VLM to interpret instructions with the input image and a diffusion transformer to synthesize edits, linked by a connector.The pipeline is illustrated in Figure 4.
- Reference Image Guidance: The reference image is encoded into latent space and concatenated channel-wise with noise latents, avoiding the sequence-length increase of sequence-wise concatenation.A widened input convolution restores the original channel dimensionality.
- Textual Guidance: The VLM processes the reference image and instruction together with learnable meta-tokens, producing contextualized hidden states for diffusion conditioning.The meta-tokens are concatenated with instruction tokens and propagated through the transformer layers.
- Connector Alignment: The connector is pretrained while the VLM and diffusion model are frozen, using text-to-image generation before training on the primary editing task.This alignment stage improves generated-image quality and instruction following.
- Training Data: Mixed training combines editing triplets with high-quality text-to-image pairs to reduce overfitting and preserve the model’s generative prior.The two data types are mixed within each batch, with task-specific text templates.
- Multi-stage Training: The four-stage training pipeline comprises alignment, image-to-image pre-training, supervised fine-tuning, and DPO preference alignment.During supervised stages, meta-tokens, the connector, and diffusion model are updated while the VLM remains frozen.
- Resolution Strategy: Simultaneous training across resolutions from 384^2 to 2048^2 with diverse aspect ratios accelerates convergence and improves generation quality over iterative resolution increases.Adaptive batch sizing increases batch size for lower-resolution inputs to improve GPU utilization.
- Preference Alignment: Strict-dominance preference pairs require the preferred image to outperform the rejected image across both reward criteria, reducing reward over-optimization.The strategy produced more balanced gains than scalarized objectives and matched or outperformed more involved multi-preference sampling.
4. Assessor
The assessor addresses the weak correlation between standard image-editing metrics and human perception by using task-specific vision-language fine-tuning. Its fine-tuned versions correlate substantially better with human judgments than vanilla baselines.
- Motivation: Standard image-editing metrics often correlate poorly with human perception, motivating a robust automated assessor for training-data filtering.
- Construction: The assessor was first built by fine-tuning Gemini 2.0 Flash on 4,350 examples, then expanded to 12,335 examples for LoRA training of Qwen-2.5-VL-7B.Validation used a held-out set of 2,994 samples.
- Evaluation: Fine-tuned assessor models showed significantly higher correlation with human judgments than their vanilla counterparts.The comparison supports task-specific fine-tuning for a reliable filtering tool.
5. Datasets
The datasets combine large-scale synthetic and remastered sources with carefully filtered real and semi-synthetic triplets to improve diversity, quality, and physical realism. The pipeline uses multiple mining, validation, and augmentation strategies across pretraining and supervised fine-tuning.
- Pretraining data: ≈7.7 million higher-quality triplets replaced a noisy mixed corpus that caused downstream quality degradation.The earlier corpus contained up to 21 million triplets, but noisy pretraining introduced artifacts and failure modes that SFT did not fully remove.
- Pretraining data: 48 million aesthetically curated text-to-image images supplemented pretraining and SFT to improve visual appeal.
- UltraEdit Remake: 6 420 724 remastered UltraEdit triplets were generated by producing candidate edits and retaining those that passed fidelity checks.The remastering addressed low resolution, square-only images, and substantial dataset noise.
- Real Tripod Photos: Real tripod-captured and static-camera video triplets were added because controlled capture better preserves pixel alignment, lighting, shadows, reflections, and transparency.The manually collected tripod set contained 4139 triplets including augmentations, while RORD contributed indoor static-camera frames.
5.3. Generation Augmented Retrieval based Dataset
The Generation Augmented Retrieval-based dataset grounds synthetic edit intents in real user phrasing, validates instruction-image applicability, and generates filtered target edits. Inversion and composite transitions expand each source image into bidirectional and directed training relationships.
- Data format and motivation: Real-world editing requests were cleaned, deduplicated, and organized into a taxonomy of 50 large semantic clusters.Qwen3-VL interpreted the clusters, producing a practical taxonomy aligned with the distribution of user requests.
- Retrieval-based grounding: FRIDA retrieval selected user-like instructions from the top K=20 nearest candidates using stochastic softmax sampling.A frequency cap limited each instruction to at most three appearances, preserving lexical diversity and reducing prompt concentration.
- Instruction validation: Gemini 3 Flash checked whether each retrieved instruction applied to the image and minimally edited inapplicable instructions when possible.
- Target generation: ∼10k images received 4 to 8 valid in-the-wild instructions before target images were generated by multiple proprietary editing models.An in-house Qwen2.5-VL assessor filtered the resulting edits.
- Dataset expansion: 176 532 final triplets were produced, with inversion and composite transitions expanding the relationships among edits sharing one source image.For N edits of a fixed source, the number of directed transitions is N(N −1).
5.4. Issues and Filtering
The filtering pipeline targets artifacts, spatial shifts, and instruction misalignment that undermine strict source consistency. It combines learned quality scoring with geometric checks and pairwise spatial correction.
- Quality filtering: Approximately 15% of SFT data was removed using a Gemini-based filtering threshold of 3.5.Visual inspection indicated that the metric preserved high-quality instruction alignment among retained samples.
- Geometric filtering: Face IoU below 0.9 triggered pair removal, eliminating approximately 35% of data to suppress artifacts correlated with face repositioning.The artifacts included checkerboard patterns and JPEG-like noise, especially near outpainting borders, faces, and uniform regions.
- Spatial correction: Homography-based alignment corrected small shifts, unintended crops, and stretching between input and output images.
5.5. Synthetic Augmentation Pipeline
Synthetic augmentation broadens robustness across reversible, structural, typographic, and degraded-input editing scenarios. Preference data then aligns the model toward higher aesthetic quality, instruction adherence, and fewer model-specific artifacts.
- Synthetic augmentation: Just-in-Time augmentation dynamically creates new triplets during training, multiplying dataset size while enforcing consistency across editing scenarios.
- Photometric and restoration operations: Bidirectional transformations train the model to apply and reverse blur, noise, sepia, and grayscale/colorization effects.
- Instruction adherence and invariance: Identity-mapping triplets with passive instructions teach preservation when no edit is requested, while mirror augmentation increases visual diversity.
- Structural and typographic editing: Geometric overlays and rendered text simulate structural and typographic edits requiring precise spatial control.
- Real-world quality adaptation: Synchronized JPEG compression on source and target images adapts the model to low-quality uploads without editing degradation.
- Preference alignment: The composite DDPO preference dataset uses self-generated successes and failures to suppress artifacts, hallucinations, and distortions from SFT.It also includes symmetric preference optimization for fine-grained instruction differences and teacher-quality outputs as a distillation signal.
6. Results
Ablations compare reference-image and textual-guidance designs under efficiency constraints, selecting channel-wise guidance and meta-token conditioning for the final system. Benchmark results show strong overall performance, particularly on instruction following and preservation-oriented edits.
- Reference Image Guidance: Sequence-wise concatenation consistently outperformed channel-wise concatenation, but its incremental practical gains did not justify the latency cost.The final configuration therefore uses channel-wise guidance for high throughput.
- Reference Image Guidance: Channel-wise guidance substantially reduced generation latency while often achieving similar outcomes to sequence-wise guidance through a few additional resampling steps.
- Textual Guidance: Native text-only conditioning cannot interpret instructions in the context of the reference image, limiting ambiguous and compositional edits.
- Textual Guidance: Meta-token conditioning drastically improved instruction following compared with Q-Former and native-encoder baselines.
- Connector Design: Four Transformer blocks were optimal for both connector configurations, while ELLA produced only minor, inconsistent gains over a standard encoder.
- Benchmark Results: 3.85 was VIBE’s overall score on ImgEdit, ranking second among compared methods and leading Adjust (4.22), Remove (4.42), and Background (4.22).
- Benchmark Results: 6.81 was VIBE’s overall score on GEdit-Bench-EN, with second-highest semantic consistency at 7.91 despite lower perceptual quality of 6.33.The results indicate faithful, minimally invasive edits rather than aggressive scene redrawing.
7. Conclusions
The paper concludes that compact models can deliver high-quality instruction-based editing when architecture, data, and training are carefully designed. VIBE combines a 2B VLM, a 1.6B diffusion backbone, channel-wise guidance, and a four-stage recipe to achieve efficient, source-consistent editing.
- Conclusions: A 2B VLM can interpret complex user requests in image context and provide stable guidance through learnable meta-tokens and a lightweight connector.
- Conclusions: A 1.6B diffusion backbone can produce high-quality edits while the pipeline fits within 24 GB of GPU memory and generates 2K images in about 4 seconds on an NVIDIA H100 in BF16.
- Conclusions: Clean data and a disciplined training recipe allow the system to match or surpass significantly larger models on core tasks, especially edits requiring strict input preservation.
8. Limitations
The compact model remains constrained on highly complex edits and diverse real photographs, while its pretrained and automatically generated components introduce broader reliability and bias considerations. Strict source consistency is intrinsically difficult for some edit types, and full end-to-end VLM adaptation was not studied.
- Model and Data Boundaries: Very complex operations can fail, hard aesthetic requests remain unstable, and diverse in-the-wild photographs are more difficult than generated images.
- Model and Data Boundaries: The pipeline is more robust on generated images because their distribution is closer to the training data, while generative signals can dominate real-photo signals despite filtering.
- Deployment Scope: The research-purpose system does not guarantee correct or safe behavior and may produce incorrect, misleading, or undesirable outputs.
- Deployment Scope: Bias and fairness were not systematically evaluated, and pretrained components, open data, and automatically generated samples may transmit biases.
- Unstudied Adaptation: Full end-to-end VLM adaptation was not studied because the VLM remained frozen throughout the pipeline.
9. Future works
Future work targets lower inference cost and stronger robustness on real photographs. The paper also identifies VLM adaptation as an open direction for balancing retained general knowledge against editing-specific behavior.
- Efficiency: Distillation to fewer diffusion steps, removal of CFG, and quantization could improve throughput and enable faster inference on lower-end hardware.
- Data: Increasing real-world signal in training and validation data could improve robustness on real photos.
- Adaptation: Partial or full VLM fine-tuning remains an open direction for studying the trade-off between preserving general knowledge and improving editing-specific behavior.