Source-linked AI summary
How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing
Huanyu Zhang, Xuehai Bai, Chengzu Li, Chen Liang, Haochen Tian, Haodong Li, Ruichuan An, Yifan Zhang, Anna Korhonen, Zhang Zhang, Liang Wang, Tieniu Tan
TL;DR
Existing image-editing systems and benchmarks largely rely on text, making precise spatial and structural intent burdensome to express and evaluate. VIBE introduces a hierarchical benchmark with task-specific LMM judging, finding that proprietary models show early visual instruction-following ability yet performance declines sharply as task complexity increases.
Problem
Existing image-editing systems and benchmarks remain predominantly text-guided, limiting evaluation of multimodal visual instruction-following.
Method
VIBE benchmarks visual instruction-guided editing across deictic, morphological, and causal levels using task-specific metrics and an LMM-as-a-judge framework.
Results
Proprietary models consistently outperform open-source models, but performance degrades from the Deictic level to the Causal level, with even the strongest models averaging below 50% on Causal tasks.
Takeaways & Limitations
Visual instruction-following is emerging for explicit localized edits, while structural abstraction and causal reasoning remain difficult for current systems.
Abstract
from arXiv · showhide
Recent generative models have achieved remarkable progress in image editing. However, existing systems and benchmarks remain largely text-guided. In contrast, human communication is inherently multimodal, where visual instructions such as sketches efficiently convey spatial and structural intent. To address this gap, we introduce VIBE, the Visual Instruction Benchmark for Image Editing with a three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning. Across these levels, we curate high-quality and diverse test cases that reflect progressively increasing complexity in visual instruction following. We further propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment. Through a comprehensive evaluation of 17 representative open-source and proprietary image editing models, we find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models. However, performance degrades markedly with increasing task difficulty even for the strongest systems, highlighting promising directions for future research.
1. Introduction
VIBE addresses the limits of text-guided image-editing benchmarks by evaluating visual instructions across progressively complex interaction levels. Its experiments show emerging proprietary-model capability, persistent open-source gaps, and marked degradation on causal tasks.
- Text-only image editing burdens users and models with verbose descriptions and ambiguous reconstruction of spatial or structural intent.Visual cues such as sketches, arrows, and region annotations provide more explicit grounding for editing.
- VIBE systematically benchmarks visual instruction-guided image editing using spatially anchored cues that resolve ambiguities in text-only instructions.The benchmark captures deictic grounding, morphological manipulation, and causal reasoning across 10 diverse subtasks and 1,034 human-verified samples.
- VIBE combines a three-level interaction hierarchy with task-specific metrics and an LMM-as-a-judge framework for scalable assessment.The framework is presented as a comprehensive approach to evaluating multimodal instruction-following.
- 17-model evaluation finds emerging visual instruction-following capabilities in proprietary systems, which consistently outperform open-source models across interaction levels.The evaluated set includes 10 open-source and 7 proprietary models.
- Even the strongest models average below 50% on the Causal level, while performance declines from Deictic to Causal tasks.This degradation indicates that complex causal reasoning remains a significant challenge.
2. VIBE
VIBE formalizes visual instructions as spatially explicit inputs to image editing and organizes tasks by increasing abstraction and reasoning complexity. Its benchmark spans three interaction levels and uses task-specific LMM-based evaluation criteria.
- Visual Instruction Formulation: VIBE models editing as Iout = ϕ(Iin, T, V), where a model transforms a source image using textual and visual instructions.Visual instructions may be separate images or overlaid annotations that provide direct geometric grounding.
- Deictic Level: The Deictic Level uses visual selectors to identify localized regions for Addition, Removal, Replacement, and Translation.These tasks assess spatial grounding and basic visual awareness while preserving non-target content where required.
- Morphological Level: The Morphological Level uses sketches or sparse blueprints to specify structural constraints for Pose Control, Reorientation, and Draft Instantiation.Models must realize abstract forms as coherent, style-consistent geometries and appearances.
- Causal Level: The Causal Level uses force vectors or motion arrows to encode dynamics for Light Control, Flow Simulation, and Billiards.These tasks require predicting logical physical outcomes rather than directly depicting final states.
- Evaluation Pipeline: An LMM judge receives the input image, textual instruction, visual instruction, and generated output, then assesses task-specific instruction fulfillment.For Deictic tasks, criteria cover instruction adherence, contextual preservation, and visual coherence.
3. Experiments
VIBE evaluates 17 proprietary and open-source models across progressively demanding visual-instruction tasks using a 100-point scoring framework. Proprietary models show early competence, outperform open-source systems, and decline from Deictic to Causal tasks.
- Main Results: Proprietary models achieve early visual instruction-following competence, with nearly all scoring above 60 on Deictic Addition, Removal, and Replacement.They also perform well on Pose Control and Draft Instantiation, while more abstract directional or reference-based tasks are comparatively moderate.
- Style Analysis: Deictic performance varies by image style, with models showing distinct preferences across real-world, animation, and sketch inputs.Figure 4 compares average level scores and metric-level heatmaps for four proprietary models, including Instruction Adherence, Contextual Preservation, and Visual Coherence.
- Main Results: Proprietary systems achieve substantially higher overall scores than open-source models across all interaction levels.The reported gap is associated with differences in model scale and training-data diversity, while motivating further open-source work on multimodal alignment and reasoning.
- Main Results: Performance on proprietary models degrades from the Deictic Level through the Morphological Level to the Causal Level.The paper attributes this pattern to increasing demands for localized editing, structural abstraction, and causal reasoning.
4. Discussion and Analysis
The analysis finds that visual-instruction performance depends on image style, task composition, and the division of labor between visual and textual guidance. Multi-task composition is harder than isolated instruction following, while LMM scores closely track human judgments.
- Style-wise Performance: Nano Banana Pro and Wan 2.5 perform relatively evenly across styles, while Seedream 4.5 declines on sketches and GPT-Image-1 is strongest on real-world images.Seedream 4.5’s sketch degradation is linked primarily to lower Instruction Adherence and Visual Coherence; GPT-Image-1 leads on real-world inputs across metrics.
- Style-wise Performance: Most models perform notably better on animation-style images than on other styles for Draft Instantiation.The paper suggests cleaner contours and more explicit structural cues in animated images may align better with sketch-based instructions.
- Multi-task Visual Instruction Following: Models often execute individual visual instructions correctly but fail when multiple instructions are composed in one query.The qualitative examples include failures after adding a third instruction, indicating a gap between isolated competence and compositional understanding.
- Evaluation Reliability: LMM-based evaluations correlate strongly with human judgments, reaching an overall Pearson correlation of r = 0.9602.Model-specific correlations are r = 0.9673 for Nano Banana Pro and r = 0.9531 for GPT-Image-1, based on 100 samples per model assessed by four experts.
- Text–Visual Synergy: Visual and textual instructions provide complementary capabilities: visual cues ground target localization, while detailed text conveys complex semantic constraints.A simple visual cue can reduce the textual detail needed for localization, but complex edits may require detailed text paired with explicit visual guidance.
5. Conclusion
The paper concludes that VIBE offers a structured benchmark for visual instruction following in image editing, spanning deictic, morphological, and causal interaction. Current systems show early competence on localized cues but weaken as complexity, reasoning, and coordination demands increase.
- Conclusion: VIBE systematically evaluates visual instruction following in image editing through a three-level Deictic, Morphological, and Causal hierarchy.The hierarchy is intended to assess increasingly complex visual–linguistic interactions.
- Conclusion: Current systems demonstrate early-stage competence on explicit, localized visual instructions, but performance degrades as interaction complexity increases.The conclusion also reports differences across models, visual styles, and task compositions.
- Conclusion: VIBE highlights continuing challenges in reasoning, compositional instruction execution, and multi-task coordination.The paper presents the benchmark as a testbed for developing improved visual instruction following and coordination with textual guidance.
- Conclusion: Existing image-editing benchmarks largely rely on textual prompts and do not explicitly model visual prompts.The related-work discussion identifies limited primitive-operation coverage and task diversity in earlier benchmarks.
B. Data Collection and Annotation
VIBE constructs diverse visual-instruction datasets across deictic, morphological, and causal tasks using manually curated images, schematic annotations, and procedural simulation. The collection spans multiple styles and task-specific visual cues.
- Deictic Data: The four Deictic tasks share 100 source images: 34 real-world, 33 animation-style, and 33 sketch-style images.Images are collected from public online resources and manually filtered for visual clarity and localized-editing suitability.
- Deictic Annotation: Deictic annotations use red task-specific primitives, including bounding boxes for localized edits and box-plus-arrow annotations for translation.Addition marks where content should be introduced, while Removal and Replacement enclose the target region.
- Morphological Data: Pose Control uses 27 human-subject images and 26 schematic pose-reference images to evaluate pose transfer while preserving identity.The source images contain single, clearly visible people with minimal occlusion.
- Morphological Data: Reorientation annotates objects with viewing frustums that specify target yaw, pitch, and roll directions.The dataset includes objects such as vehicles, chairs, cameras, humans, and shoes with unambiguous facing directions.
- Morphological Data: Draft Instantiation contains 100 images—40 real-world, 34 animation-style, and 26 sketch-style—with sparse schematic drafts indicating structural modifications.The drafts function as abstract blueprints rather than detailed renderings.
- Causal Data: Causal tasks use directional visual cues and procedural simulation to represent lighting, airflow, and reflected billiards trajectories.Billiards varies difficulty by controlling two to seven boundary reflections before collision with a target ball.
C.1. Full Experimental Results
The VIBE quantitative comparison reports results as Mean ± Standard Deviation over three independent runs, supporting a fair and robust evaluation.
- Results on VIBE are reported as Mean ± Standard Deviation over 3 independent runs.
C.2. Style-wise Performance Analysis
The style-wise analysis examines Draft Instantiation performance across models and qualitatively studies Seedream 4.5, which struggles to preserve the original sketch style after editing.
- Tables 4 and 5 report complete style-wise performance results for all evaluated models on Draft Instantiation.
- Seedream 4.5 often fails to preserve the original sketch style after applying draft-based visual instructions.Edited regions frequently deviate from the input sketch style.
C.3. Error Analysis
The error analysis identifies failures across deictic, morphological, and causal interaction levels, while qualitative studies show that frontier models can interpret visually embedded instructions in some cases.
- Deictic Level: Deictic-level failures include inaccurate localization, unintended removal beyond marked regions, and stylistic inconsistency in removal and replacement.
- Morphological Level: Morphological-level failures include loss of character identity, incomplete orientation alignment, and incorrect or incomplete realization of draft-specified entities or structures.
- Causal Level: Causal-level failures include directionally inconsistent lighting or wind effects, incorrect billiards trajectories, and unintended changes to static scene elements.
- Overall, the error cases reveal persistent challenges in precise spatial grounding, stylistic consistency, and causal reasoning under visual instruction guidance.
- Qualitative Case Studies: Nano Banana Pro successfully interprets visually embedded instructions across single-task, multi-task, and some causal light-direction scenarios.The model can parse image-embedded instructions, associate them with operations, and execute the requested edits.
D.1. Evaluation Metrics
VIBE uses task-specific metrics to evaluate whether edits follow visual instructions while preserving identity, context, structure, and visual integrity across morphological and causal tasks.
- Pose Control: Pose Control evaluates target-pose realization alongside character integrity and contextual consistency using pose, body, identity, and preservation criteria.
- Reorientation: Reorientation measures alignment with a reference orientation across yaw, pitch, and roll while preserving object identity and visual integrity.
- Draft Instantiation: Draft Instantiation uses Instruction Adherence, Contextual Preservation, and Visual Coherence, matching the metric design used for deictic tasks.
- Light Control: Light Control evaluates lighting-direction consistency and preservation of non-lighting content, combining LDC and CP through a geometric mean.
- Flow Simulation: Flow Simulation evaluates wind-direction consistency and preservation of wind-affected subjects’ identity and placement, combining WDC and CP through a geometric mean.
- Billiards: Billiards evaluates multi-step physical interactions through trajectory structure, target-ball identity, and contextual preservation.