PaperlayerView exact revision on arXiv

Source-linked AI summary

VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation

Yuta Oshima, Masakazu Yoshimura, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta

arXiv:2609.37709v1cs.CVcs.AI

TL;DR

Existing benchmarks do not test whether generators can compose multiple references under multiple heterogeneous visual instructions. VIF-Bench evaluates that joint setting across 1,241 tasks, finding an adherence–artifact tension, lower adherence when references carry salient controlled attributes, and benefits from direct visual constraints or moderately detailed text.

  • Problem

    Existing benchmarks do not evaluate multiple references composed under multiple heterogeneous visual-instruction images, including cases where references may interfere with instructions.

    Source: §1 · p. 2

  • Method

    VIF-Bench evaluates the joint setting with tasks covering multiple references, heterogeneous visual instructions, potential reference conflicts, and text descriptions at different specificity levels.

    Source: §1 · p. 2

  • Results

    Models show an adherence–artifact tension, lower visual-instruction adherence when references carry salient controlled attributes, and better recovery from medium than dense text descriptions.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The best constraint representation depends on a model’s visual-instruction capability: direct visual instructions benefit strong followers, while intermediate textual specificity outperforms exhaustive wording.

    Source: §5 · p. 10

    Image-generation benchmarks sit within an ecosystem carrying misuse risks including deepfakes, non-consensual intimate imagery, identity fraud, and visual misinformation.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such as layouts, arrows, and pose cues. However, existing benchmarks do not evaluate the joint setting in which multiple references must be composed under multiple and heterogeneous visual-instruction images. To address this gap, we introduce VIF-Bench, a benchmark of 1,241 tasks designed to assess the edge of model capabilities in this joint setting by covering: (i) multi-reference generation (up to 7) under multiple heterogeneous visual instructions (up to 6), (ii) cases where reference images can potentially compete with visual instructions (e.g., a strongly posed subject vs. a target pose), and (iii) controlled comparison of visual instructions with text descriptions at different levels of specificity. Using these capabilities, we uncover three findings: (1) models face an adherence-artifact trade-off: once models reach stronger visual instruction adherence, stronger adherence tends to coincide with more instruction artifacts in generated images, (2) visual instruction adherence tends to be lower on tasks whose reference images carry a salient state of the controlled attribute (e.g., a neon-lit subject under a light-direction instruction), most consistently for light and wind, and (3) for models that can understand visual instructions, it is often better to provide visual constraints directly rather than describe them in text; when using text, a moderate level of detail works better than an exhaustive description. VIF-Bench is released as an open benchmark to establish a basis for fair comparison in controllable multi-reference image generation.

1 INTRODUCTION

Recent multimodal generators can use multiple references and visual control marks, but existing benchmarks largely test these capabilities separately. VIF-Bench evaluates their joint use, including reference–instruction conflicts and the trade-offs observed across models.

1 INTRODUCTION

  • The benchmark targets practical workflows in which users specify both what to generate and how to compose it using references, layouts, orientation cues, and global wind or lighting effects.The paper identifies advertising, virtual try-on, and content creation as relevant applications.Source: §1 · p. 1
  • Existing benchmarks do not capture how multiple subject references interact with multiple visual instructions or the associated failures of residual instruction marks and missed constraints.Recent generators interpret layouts, arrows, and orientation cues directly as image inputs.Source: §1 · p. 2
  • VIF-Bench evaluates multiple references composed under heterogeneous visual instructions, covering up to 7 references and 6 instruction images per task.It also includes cases where reference content may compete with the requested visual constraint and compares visual instructions with text at different specificity levels.Source: §1 · p. 2

2 RELATED WORKS

Prior work has expanded text-to-image generation with spatial controls, reference conditioning, and unified text-image inputs. Existing benchmarks nevertheless do not cover the joint setting of multiple references with several heterogeneous visual-instruction images.

2.1 CONTROLLABLE TEXT-TO-IMAGE GENERATION

  • Controllable text-to-image work adds spatial maps and identity-preserving references to diffusion models, while newer systems process text and images together.VIF-Bench asks whether these systems follow visual instructions when multiple reference subjects must be composed together.Source: §2.1 · pp. 2–3

2.2 BENCHMARKS FOR REFERENCE-BASED GENERATION

  • Table 1 compares benchmarks by maximum reference images, maximum explicit visual-instruction images, and whether reference–instruction conflicts are identified.Its caption describes VIF-Bench as jointly evaluating multiple references and heterogeneous visual instructions while explicitly identifying potential conflicts.Source: Table 1 · p. 3
  • MultiRef uses at most one visual-instruction image per task, VIBE combines operations within one annotated image, and DreamOmni3 supports up to two instruction images but focuses on scribble control.These benchmarks do not evaluate multiple references composed under several heterogeneous visual-instruction images.Source: §2.2 · p. 3

3 VIF-BENCH

VIF-Bench combines hierarchically organized reference images with procedurally constructed visual instructions, including potential conflicts between reference-implied attributes and requested controls. Its evaluated tasks span varied reference and instruction counts, and images are scored on six criteria.

3.1 CONSTRUCTION OF VIF-BENCH

  • The reference hierarchy separates Main Reference subjects, Sub Reference attributes, and Scene Context, preventing semantically inconsistent pairings such as assigning facial-expression references to objects.The image pool combines real and synthetic images, including attribute-specific references such as hairstyle, makeup, facial expression, and object surface.Source: §3.1 · pp. 3–4
  • Tasks combine per-subject and global conditions, render layouts, orientations, and wind/light directions as visual instructions, and provide templated text for controlled comparisons.Tasks undergo automatic consistency checks and verification by Gemini and human annotators; three text-converted variants are built for a stratified 200-task subset.Source: §3.1 · p. 4
  • A conflict task has a salient reference attribute that competes with the corresponding visual instruction; the benchmark flags orientation, light, wind, and pose cases.Orientation is judged against the target, while light, wind, and pose cases are flagged when the controlled attribute is salient in the reference and must be re-rendered.Source: §3.1 · p. 5

3.2 STATISTICS IN VIF-BENCH

  • The evaluated set contains 1,241 tasks, with 1–7 reference images and 1–6 visual-instruction images per task; layout appears in every task.Orientation, wind, light, and pose occur in 315, 250, 233, and 141 tasks, respectively.Source: §3.2 · p. 5, Figure 3 · p. 6
  • 438 of 1,241 tasks contain at least one reference–instruction conflict, and conflict rates vary by instruction type, reaching 73.8% for light.The reported rates are 33.7% for orientation, 58.2% for pose, 61.2% for wind, and 73.8% for light; categories may overlap within a task.Source: §3.2 · p. 5

3.3 EVALUATION SETTING

  • Two VLM judges independently score each generated image on six 10-point criteria, including separate measures of visual-instruction adherence and cleanliness.Adherence measures whether constraints are satisfied; cleanliness measures whether instruction elements such as boxes and arrows remain in the output.Source: §3.3 · p. 6

4 EXPERIMENTS

Across the 1,241-task evaluation, closed-source models outperform open-source models overall, but visual-instruction adherence remains difficult and trades off against cleanliness. The experiments also examine how reference and instruction counts, reference conflicts, constraint representation, task decomposition, and judge reliability shape evaluation outcomes.

4 EXPERIMENTS

  • The evaluation applies six criteria to a representative set of state-of-the-art generators on the full 1,241-task benchmark.Generated images are independently judged by Gemini 2.5 Flash and GPT-5, and the reported score averages the two judges.Source: §4 · p. 6

4.1 OVERALL AND PER-VISUAL-INSTRUCTION EVALUATION

  • Closed-source models outperform open-source models overall, yet the best Visual Instruction Adherence score is only 5.67 and stronger adherence among closed models tends to coincide with lower cleanliness.GPT-Image models have higher cleanliness but lower adherence, while Nano Banana models show stronger adherence with more residual instruction marks.Source: §4.1 · pp. 6–7, Table 2 · p. 7, Figure 4 · p. 7
  • Across visual-instruction types, VIF-Bench captures both broad capability gaps across model families and incremental improvements across model versions.The reported improvements include FLUX.1 Kontext to DreamOmni2, Qwen-Image-Edit-2509 to 2511, and GPT-Image-1 to 1.5.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2 EFFECT OF THE NUMBER OF REFERENCES AND VISUAL INSTRUCTIONS

  • Adding references degrades all three criteria; with five or more, open-weight models approach the floor in Reference Consistency and Visual Instruction Adherence, while closed models largely retain Reference Consistency.Nano Banana Pro retains much of its adherence as references increase, whereas GPT-Image-1.5 drops sharply; increasing visual instructions from one to three changes adherence much less but mainly lowers Text Instruction Following.Source: §4.2 · p. 7, Figure 5 · p. 8

4.3 REFERENCE–VISUAL-INSTRUCTION CONFLICT

  • Reference conflict generally lowers adherence, with the largest gaps for Light and Wind; for Pose, reductions appear only in generators with non-trivial adherence.Open-weight models already score near the floor on pose tasks, leaving little room for a further drop.Source: §4.3 · p. 8, Figure 6 · p. 8

4.4 HOW SHOULD VISUAL CONSTRAINTS BE SPECIFIED?

  • For Nano Banana Pro, visual instructions achieve higher adherence than dense text conveying nearly the same information, while GPT-Image-1.5 benefits from text conversion.TI Medium recovers the original constraint better than TI Dense despite omitting fine-grained values; TI Sparse removes information needed to convey the intent.Source: §4.4 · pp. 8–9, Figure 6 · p. 8

4.5 AGENTIC MULTI-STEP GENERATION

  • Decomposing tasks raises Nano Banana Pro’s average score from 6.96 to 7.40 but lowers GPT-Image-1.5’s from 7.60 to 7.53.For both models, cleanliness rises while adherence falls, and splitting references across steps further reduces Reference Consistency; decomposition benefits therefore depend on generator and criterion.Source: §4.5 · p. 9

4.6 RELIABILITY OF VLM JUDGES

  • On average scores, GPT-5 and Gemini 2.5 Flash correlate positively with human ratings, at 0.78/0.75 and 0.74/0.71, respectively.The correlations are Pearson/Spearman values measured on a randomly sampled subset of 168 generated images.Source: §4.6 · p. 9

5 CONCLUSION

VIF-Bench evaluates multi-reference generation under heterogeneous visual constraints, including reference conflicts and comparisons between visual and textual instructions. Its findings identify adherence–artifact tension, reduced adherence when references contain salient target attributes, and model-dependent benefits from visual instructions or moderately detailed text.

5 CONCLUSION

  • VIF-Bench evaluates multi-reference generation under heterogeneous visual constraints, potential reference conflicts, and visual-versus-text instruction comparisons.The benchmark is presented as a basis for systematic evaluation of this joint setting.Source: §5 · pp. 9–10
  • The results identify an adherence–artifact tension, lower adherence when references carry a salient state of the controlled attribute, and model-dependent advantages for visual instructions or moderately detailed text.Direct visual constraints benefit stronger visual-instruction-following models, while intermediate textual specificity can outperform exhaustive verbalization.Source: §5 · p. 10
  • The authors report using generative AI to create synthetic data, propose instructions, judge outputs, and assist with figures, code, artifacts, and manuscript drafting, with human review and validation described.They report validating VLM judging against human ratings on a 168-image subset.Source: AI USE STATEMENT · p. 10
  • The benchmark is released as an evaluation-only resource without model weights or new generative tooling, and uses references from existing public datasets.The stated safeguards also avoid introducing new identifiable persons.Source: ETHICS STATEMENT · p. 10
  • The released code and benchmark support reproducibility, and an open-source judge is provided for evaluation without closed-source API access.The paper also reports collecting results with stable-version API endpoints for VLM judges.Source: REPRODUCIBILITY STATEMENT · p. 10

A RESULTS OF QWEN3-VL JUDGE

The open-weight Qwen3-VL-32B-Instruct evaluator largely preserves the generator ranking and reproduces the adherence–artifact trade-off. Its scores support reproducible overall comparisons, while showing that judge choice can affect some distinctions.

A RESULTS OF QWEN3-VL JUDGE

  • Qwen3-VL-32B-Instruct supports reproducible overall comparison, and the main conclusions do not hinge on judge choice.It is a fixed-version open-weight evaluator that can provide a baseline without proprietary VLM API access.Source: A RESULTS OF QWEN3-VL JUDGE · p. 19
  • Qwen3-VL is more lenient on Adherence, compressing the four closed models into a 0.5-point band, while only GPT-5 reaches human–human agreement on that criterion.GPT-5’s human–human agreement is 0.76/0.72 versus 0.75/0.74; the cross-family gap is independently agreed on by GPT-5 and Gemini.Source: A RESULTS OF QWEN3-VL JUDGE · p. 19
  • The generator ranking is preserved under Qwen3-VL-32B-Instruct except for DreamOmni2 and FLUX.1 Kontext, which differ by less than 0.1 under every judge.The adherence–artifact trade-off is also reproduced within each generator.Source: Table 4 · p. 19, A RESULTS OF QWEN3-VL JUDGE · p. 19

B IMPLEMENTATION DETAILS

The implementation fixes API versions for reproducibility, standardizes generated image size, and reports release locations and judging costs.

B IMPLEMENTATION DETAILS

  • The code and benchmark are released on GitHub and Hugging Face, respectively.The listed API endpoints include Gemini and GPT image generators, with fixed endpoint versions for VLM judges.Source: B IMPLEMENTATION DETAILS · p. 20
  • Each task uses a forced output size of 1024×1024.The generation endpoints include gemini-3-pro-image-preview, gemini-2.5-flash-image, gpt-image-1.5, and gpt-image-1.Source: B IMPLEMENTATION DETAILS · p. 20
  • A full two-judge pass over the 1,241-task set costs approximately $47 for one generator.GPT-5 accounts for approximately $35 and Gemini 2.5 Flash for approximately $12.Source: B IMPLEMENTATION DETAILS · p. 20

C DETAILS OF TASK CONSTRUCTION

These sections detail how VIF-Bench filters and organizes references, constructs visual and textual controls, labels conflict tasks, and reports reference adoption statistics.

C.1 LAION-5B IMAGE FILTERING

  • LAION-5B references are filtered for composition, semantic alignment, and inappropriate content, leaving about 48% after the first stage and excluding about 3% of the remainder.Images without detected objects are retained for background or style use; near-duplicate synthetic images are also manually removed.Source: §C.1 · p. 20

C.2 LANGUAGE COVERAGE OF TEXT REFERENCES

  • Text references cover English, Chinese, and Japanese to diversify language and typography across primary-text and secondary-typography tasks.The language coverage follows MultiBanana’s design.Source: §C.2 · p. 20

C.3 VISUAL INSTRUCTION CONSTRUCTION

  • Visual controls include indexed layout boxes, yaw/pitch pyramids, directional wind/light arrows, and person-specific pose references.Layouts avoid collage-like tiling and keep subject correspondence tied to canonical image indices; orientation is kept separate from layout placement, while arrows use distinct colors and start/end semantics.Source: §C.3 · pp. 20–21

C.4 CONFLICT DETECTION

  • Conflict tasks are labeled by matching reference attributes to task instructions; orientation conflict requires reliable facing estimates and a large angular difference or left/right sign mismatch.The procedure first pre-labels reference attributes, then links task references to those labels and checks each included visual-instruction type.Source: §C.4 · pp. 21–22
  • Light, wind, and pose conflicts apply when references carry distinctive lighting, wind-responsive elements, or a clear pose controlled by the corresponding instruction.Layout is excluded because it specifies whole-scene arrangement rather than an intrinsic reference-image attribute.Source: §C.4 · p. 22

C.5 CONVERTING VISUAL INSTRUCTIONS INTO TEXT INSTRUCTIONS

  • The text-conversion study compares visual instructions with Dense, Medium, and Sparse text variants on a fixed, stratified 200-task subset, while judges see the original visual task.Structured annotations generate layout, orientation, and arrow variants deterministically; pose descriptions are produced together in one GPT-5 vision call per unique pose image.Source: §C.5 · p. 23
  • Dense text preserves fine-grained coordinates and descriptions, Medium discretizes positions and directions, and Sparse retains only coarse spatial or directional information.Sparse variants omit details such as box size, vertical position, fine-grained angles, and individual joint configurations.Source: §C.5 · p. 23

D FURTHER STATISTICS

  • Across 1,241 tasks, main-reference adoptions are comparable for person, animal, and text, while the aggregated object total is larger because it combines three environment-conditioned types.Sub-reference counts are conditioned on compatible main-subject types; scene-context adoptions include 252 style and 149 background references.Source: D FURTHER STATISTICS · p. 24, Figure 9 · p. 24

E COMPARISON WITH PRIOR WORKS

VIF-Bench evaluates multiple independent references together with multiple heterogeneous visual-instruction images, a joint setting not covered by the compared benchmarks. On MultiRef, GPT-Image-1.5 scores above the ground-truth images under that benchmark’s Overall Assessment protocol.

E COMPARISON WITH PRIOR WORKS

  • VIF-Bench jointly tests multiple independent reference images and multiple heterogeneous visual-instruction images, including layout, orientation, pose, wind, and lighting.MultiRef supports multiple references but at most one visual-instruction image per task; DreamOmni3 uses up to two scribble images but a single instruction modality.Source: E COMPARISON WITH PRIOR WORKS · p. 25
  • GPT-Image-1.5 scores 0.848 on MultiRef’s Overall Assessment, exceeding the ground-truth images’ 0.771 and indicating limited headroom for distinguishing strong recent generators.The assessment combines Image Quality, Instruction Following, and Source Fidelity.Source: Table 5 · p. 25, E COMPARISON WITH PRIOR WORKS · p. 25
  • The MultiRef example tasks are described as almost fully solvable by advanced models, and each uses at most one visual-instruction image.Figure 10 illustrates the qualitative comparison.Source: Figure 10 · p. 25

F.1 FURTHER RESULTS FOR REFERENCE–VISUAL-INSTRUCTION CONFLICT

Reference conflict consistently reduces orientation, light, and wind adherence, with the largest effects for light and wind; pose effects vary by model. The conflict analysis covers 438 of 1,241 tasks, with type-specific prevalence differing substantially.

F.1 FURTHER RESULTS FOR REFERENCE–VISUAL-INSTRUCTION CONFLICT

  • 438 of 1,241 tasks contain at least one reference–instruction conflict, including 73.8% of Light tasks and 61.2% of Wind tasks.Conflicts occur in 33.7% of Orientation tasks and 58.2% of Pose tasks; categories can overlap within a task.Source: §F.1 · p. 26
  • For all eight generators, conflicts lower Light and Wind adherence, with larger gaps than Orientation; Pose effects are more model-dependent.The conflict–no-conflict average is 3.46 versus 4.67 for Light and 2.99 versus 4.43 for Wind, compared with 2.77 versus 3.08 for Orientation.Source: §F.1 · p. 26, Figure 11 · p. 27
  • Pose-conflict effects differ by generator: several open-weight models show little change or a small opposite-direction change, but score near the floor in both subsets.Across all eight generators, average Pose adherence is 3.11 versus 3.17 for conflict and no-conflict tasks.Source: §F.1 · p. 26, Figure 11 · p. 27

F.2 FURTHER RESULTS FOR THE NUMBER OF REFERENCES AND VISUAL INSTRUCTIONS

More reference images generally make generation harder, with particularly sharp consistency and adherence declines for open-weight models, while adding visual instructions has a weaker and less uniform effect. Visual-versus-text results vary by generator, and the appendix also supplies further qualitative examples.

F.2 FURTHER RESULTS FOR THE NUMBER OF REFERENCES AND VISUAL INSTRUCTIONS

  • With five or more references, every generator scores lower than with one on all criteria except Visual Instruction Cleanliness.Open-weight Reference Consistency and Visual Instruction Adherence approach the floor; closed-generator Visual Quality stays high except for Qwen-Image-Edit-2509.Source: §F.2 · p. 28, Figure 12 · p. 29
  • As references accumulate, closed models divide between higher adherence with more instruction marks for Nano Banana and cleaner outputs with less adherence for GPT-Image.For open-weight models, higher cleanliness with five or more references coincides with near-floor adherence, suggesting neither following nor reproducing the instructions.Source: §F.2 · p. 28
  • Increasing visual instructions from one to three lowers every generator’s Average score, but less than the same increase in references.Adherence changes only slightly; the 4+ bin contains few tasks and shows large model-dependent deviations.Source: §F.2 · p. 28, Figure 12 · p. 29

F.3 FURTHER RESULTS FOR VISUAL VS. TEXT-CONVERTED INSTRUCTIONS

  • Nano Banana Pro has its highest adherence with original visual instructions, whereas GPT-Image-1.5 performs best with TI Medium.For GPT-Image-1.5, moderately abstracted text outperforms both visual instructions and exhaustive TI Dense; other criteria generally improve as constraints are relaxed.Source: §F.3 · p. 28, Figure 13 · p. 29
  • Text-converted conditions are evaluated against the original task and visual instructions, so the comparison tests recovery of the visual constraint rather than matching a coarsened text version.Removing visual-instruction images also increases Visual Instruction Cleanliness.Source: §F.3 · p. 28, Figure 13 · p. 29

F.4 ADDITIONAL QUALITATIVE RESULTS

  • Figures 14 and 15 provide additional qualitative examples of VIF-Bench.The appendix identifies these as further qualitative results without specifying a particular outcome in the supplied passage.Source: §F.4 · p. 30, Figure 14 · p. 30, Figure 15 · p. 31

G.1 ROBUSTNESS TO JUDGE-SPECIFIC PREFERENCES

The appendix tests evaluation robustness across judges and human ratings, checks visual-quality and spatial judgments against other measures, and examines diversity across repeated generations. It also situates VIF-Bench among related editing, instruction-following, and multimodal-generation work.

G.1 ROBUSTNESS TO JUDGE-SPECIFIC PREFERENCES

  • GPT-5 and Gemini 2.5 Flash give somewhat different absolute scores, but their Average Scores produce the same generator ranking.This supports robustness of the benchmark’s overall conclusions to the choice between the two primary judges.Source: §G.1 · p. 32, Table 6 · p. 32

G.2 CROSS JUDGE CORRELATION

  • On 168 images, GPT-5 and Gemini correlate positively with human Average Scores at PLCC/SRCC 0.78/0.75 and 0.74/0.71, respectively.GPT-5’s Visual Instruction Adherence correlation is 0.76/0.72, comparable to human–human agreement at 0.75/0.74; Qwen3-VL-32B reaches 0.73/0.70 on Average Score.Source: §G.2 · p. 32, Table 7 · p. 32

G.3 CONSISTENCY WITH AESTHETIC PREDICTORS

  • The VLM judges’ Visual Quality scores show moderate positive correlations with LAION Aesthetic Predictor v1/v2 and human judgments.GPT-5’s Visual Quality scores correlate with human ratings at PLCC/SRCC 0.57/0.63.Source: §G.3 · pp. 32–33, Table 8 · p. 33

G.4 CONSISTENCY WITH DETECTOR-BASED SPATIAL JUDGMENTS

  • GPT-5’s adherence ratings correlate with human judgments at 0.76/0.72, close to human–human agreement of 0.75/0.74, and are competitive with detector-based spatial metrics.The comparison uses Grounding DINO layout boxes, bounding-box IoU, and center-in-region rate; the VLM approach also applies beyond bounding-box layouts.Source: §G.4 · p. 33, Table 9 · p. 33

G.5 DIVERSITY UNDER REPEATED GENERATION

  • Across repeated generations, variation tends to rise as visual constraints increase while Average Score falls.For Nano Banana Pro, CLIP diversity rises from 0.078 at NVI = 1 to 0.216 at NVI = 4, while Average Score falls from 7.74 to 6.48; the authors caution that diversity may reflect instability rather than useful creative variation.Source: §G.5 · pp. 33–34, Table 10 · p. 34

H EXTENDED RELATED WORK

  • Prior editing benchmarks commonly assume one reference image and a textual edit, while visual editing research also explores richer visual controls and interaction.The related-work discussion covers exemplar and spatial conditioning, point- or drag-based manipulation, realistic editing benchmarks, test-time scaling and agents, and instruction-following evaluation in LLMs.Source: H EXTENDED RELATED WORK · pp. 34–35

I LIMITATIONS AND FUTURE WORK

The authors identify limited synthetic-reference source coverage and a narrow set of visual-instruction types, while describing video generation as a possible extension. The appendix provides prompts for proposing visual instructions and for judging generated images.

I LIMITATIONS AND FUTURE WORK

  • The reference pool’s synthetic images come from Nano Banana and GPT-Image; future versions could broaden coverage with generators such as Qwen-Image and FLUX.The authors say prior work found low statistical bias in this kind of curated mixed-source pool, so they do not consider it a blocking risk.Source: I LIMITATIONS AND FUTURE WORK · p. 35
  • VIF-Bench covers five visual-instruction kinds, while future extensions could add structured cues such as skeletons or part-segmentation maps and freeform scribbles or sketches.The stated aim is to test generalization across a broader range of compositional cues.Source: I LIMITATIONS AND FUTURE WORK · p. 35
  • The authors propose extending the evaluation philosophy to video, where multiple references and instructions must remain coherent across frames.Suggested evaluations include frame-level instruction adherence and spatial and temporal scene coherence, as well as memory–instruction interactions.Source: I LIMITATIONS AND FUTURE WORK · p. 35

J.1 VISUAL INSTRUCTIONS PROPOSAL PROMPTS

  • A VLM receives sampled references and canvas dimensions, proposes visual-instruction data, and returns a JSON record that is rendered into an instruction image.The proposal prompts cover layouts, directional arrows, and 3D orientation pyramids.Source: §J.1 · p. 36
  • Layout prompts assign normalized boxes to reference subjects and require a cohesive scene rather than a tight grid or collage.Arrow prompts encode wind or light with a tail at the source and a head toward the direction of flow or light; orientation prompts specify yaw and pitch.Source: §J.1 · pp. 36, 37, 39

J.2 JUDGE PROMPT

  • The judge receives references, visual instructions, the generator’s text instruction, and the output, then scores six criteria independently on a 1–10 scale.Adherence is capped by the worst violation among visual instructions, and recognizable instruction marks cap Visual Instruction Cleanliness at 6 or below.Source: §J.2 · pp. 40–41
Loading 2609.37709v1…