MultiRef: Controllable Image Generation with Multiple Visual References

• The work falls within artificial intelligence and evaluation.Source: CCS Concepts · p. 1

• Key topics include multi-images-to-image generation, unified models, benchmarks, and datasets.Source: Keywords · p. 1
• The paper is published in the Proceedings of the 33rd ACM International Conference on Multimedia.Source: Keywords · p. 1

1 Introduction

• Multiple visual references are motivated by creative workflows that blend elements such as palettes, architectural forms, textures, and styles.The paper frames this as more controllable than relying on a single source of inspiration.Source: §1 · p. 1, Figure 1 · p. 2
• Current generators mainly support text or one reference image, while combining inconsistent viewpoints, styles, or semantics can confuse models and degrade outputs.Existing benchmarks likewise emphasize text-to-image alignment or single-image editing rather than truly multi-reference integration.Source: §1 · pp. 1–2
• MultiRef-bench contains 1,000 real-world and 990 synthetic samples spanning straightforward and complex multi-reference alignment tasks.The real-world samples use requests, references, and ground-truth images from Reddit, while synthetic samples are programmatically generated.Source: §1 · p. 2
• RefBlend combines extracted visual conditions through compatibility rules to create diverse multi-reference image-to-image training samples with automatic input-output pairings.Its references can include depth maps, Canny edges, object masks, semantic maps, poses, and captions.Source: §1 · p. 2
• 10 reference types and 33 reference combinations make the synthetic benchmark broader and more complex than existing collections.Source: §1 · p. 2
• The benchmark evaluates precise conditions with rule- or model-based metrics and semantic conditions with fine-tuned MLLM-as-a-Judge assessments.Source: §1 · p. 2
• 0.496 alignment is achieved by OmniGen on multi-reference tasks, despite near-perfect performance on single-reference inputs.The evaluation covers three interleaved image-text models and six agentic frameworks.Source: §1 · p. 2

2 MultiRef-bench

• MultiRef-bench contains 1,990 examples: 1,000 real-world tasks and 990 synthetic examples from 38,076 RefBlend samples.The real-world portion uses Reddit’s r/PhotoshopRequest, while the synthetic portion supplies diverse guidance signals such as depth maps, bounding boxes, and art styles.Source: §2 · p. 2

2.2 RefBlend: The Synthetic Data Engine

• RefBlend forms samples by extracting references, selecting compatible combinations, creating basic and enhanced prompts, and integrating the prompts with references.Figure 2 separates reference generation from instruction-prompt generation and depicts the final integration step.Source: Figure 2 · p. 3, §2.2 · p. 4

2 MultiRef-bench

• Real-world examples include 2–6 input images, one structured instruction, and one golden output image after quality-controlled selection.The collection began with 2,300 multi-image queries, of which 45% met the benchmark criteria.Source: §2.1 · pp. 2–3

2.2 RefBlend: The Synthetic Data Engine

• RefBlend generates 38,076 samples through reference-condition generation, compatibility-based combination, prompt alignment, and low-quality filtering.Its extracted conditions include bounding boxes, depth maps, edges, semantic maps, sketches, masks, poses, and art-related references.Source: §2.2 · pp. 3–4

2.3 Evaluation

• Evaluation combines rule-based metrics, model-based metrics, and fine-tuned MLLM judgments for reference following and overall quality.Rule-based measures target precise conditions such as depth, masks, and boxes, while MLLM evaluation addresses semantic references and human-aligned judgments.Source: Table 2 · p. 4, Table 3 · p. 5, Figure 4 · p. 5

2.2 RefBlend: The Synthetic Data Engine

• Compatibility rules govern valid combinations, including mutual exclusions for overlapping global references and attachment of spatial references to specific objects or concepts.The dataset generates combinations of 2, 3, or 4 references while respecting these constraints.Source: §2.2 · p. 4

Compositional Frameworks

• 0.774 image quality, but only 0.589 instruction following and 0.462 source fidelity, were achieved by Claude + SD3.5 versus ground-truth scores of 0.767 and 0.706, respectively.Compositional frameworks can exceed ground truth in image quality while underperforming on instruction following and source fidelity.Source: §3.2 · p. 5

Unified Models

• Unified models still underperform the professional-software golden answer on real-world image generation, despite OmniGen balancing quality and instruction adherence against some compositional systems.The reported results indicate substantial room for improvement in fidelity preservation and generation quality.Source: §3.2 · p. 5

Overall Comparison

• ACE reaches 0.219 on Bbox and 0.090 on Pose, yet both unified and compositional models struggle with complex reference combinations or caption-free generation.The findings identify multi-reference conditioning and generalization as persistent challenges.Source: §3.2 · p. 5

Reference-Format Ablation

• No reference format is universally best: ACE and ChatDiT are more robust on depth and mask formats, while models show distinct preferences across BBox, depth, and mask inputs.For Depth MSE, ACE favors ori depth, whereas OmniGen and ChatDiT perform slightly better or comparably with color depth.Source: §3.2 · p. 5, Figure 6 · p. 6

Input-Order Ablation

• Switching image order causes minor or no FID changes but can substantially alter condition-specific fidelity, including dramatic increases in ACE’s depth and sketch errors.Input sequence therefore matters more for controlling particular visual conditions than for global image quality.Source: §3.2 · p. 5, Table 5 · p. 6

Caption Ablation

• Removing captions improves depth fidelity and aesthetic quality across models, while semantic-map and sketch fidelity can improve for selected models or conditions.ACE’s depth error increases without captions, whereas OmniGen and ACE perform better on semantic-map fidelity and all three models improve sketch fidelity.Source: §3.2 · p. 6

4 Conclusion

• MultiRef provides a large-scale dataset and benchmark for image generation conditioned on multiple visual references.The dataset contains 38,076 samples, including 990 test samples, and the benchmark evaluates multi-reference generation.Source: §4 · p. 6, B Details of Collecting MultiRef · p. 9
• The paper positions MultiRef against prior controllable-generation datasets and benchmarks that mainly address single or dual conditions, editing, or text alignment.Existing resources are described as insufficient for complex multi-reference combinations.Source: A Related Work · p. 9
• The benchmark and dataset combine diverse reference modalities, including edges, semantic maps, sketches, depth, boxes, masks, poses, styles, subjects, and captions.These references are standardized from prior controllable-generation guidance and adapted to flexible input and output formats.Source: §B.1 · p. 10
• Reference generation uses visual extraction and metadata construction across multiple source datasets, including DreamBooth and Subjects200K.The supplied materials describe subject references from these datasets and note that they do not include art-style or pose references.Source: §B.1 · p. 10, §B.2 · p. 10

B.4 Data Filtering

• References scoring below 3 are filtered out after MLLM-based evaluation of alignment and reference quality.The evaluation uses a 1–5 scale, while pose, subject, and art-style references receive manual verification.Source: §B.4 · p. 12
• Qwen2-7B-VL-FT achieves the best overall alignment with human preferences among the evaluated judging models.The fine-tuned judge performs well across many conditions but remains challenged by nuanced pose and bounding-box accuracy.Source: §B.4 · p. 13
• The filtering evaluation covers condition-specific alignment and quality rubrics for captions, sketches, semantic maps, masks, and Canny edge maps.These rubrics define ordered scores from poor or unaligned outputs to high-quality, well-aligned references.Source: §B.4 · p. 14
• The resulting MultiRef-bench contains 1,990 examples: 990 synthetic samples and 1,000 real-world tasks from r/PhotoshopRequest.The synthetic portion is held out from MultiRef, while the real-world portion represents user-submitted editing tasks.Source: C Details of MultiRef-bench · p. 15

C.3 Evaluation

• GPT-4o-mini is selected for overall assessment because it aligns better with human judgments than the compared models.Correlation is validated on 300 samples from each of the realistic and synthetic datasets.Source: §C.3 · p. 17

D Details of Experiment

• All experiments run on a server equipped with two A800 GPUs and two 4090 GPUs.Source: §D.1 · p. 17
• The experiments use four open-source unified models, including OmniGen, ChatDiT, ACE, and Show-o, with model-specific generation settings.The settings vary in resolution, guidance scale, sampler, timesteps, temperature, and dialogue or image-to-image usage.Source: §D.1 · pp. 17–18
• Emu2-Gen is omitted because it reportedly requires at least 75GB of memory.The exclusion is attributed to computational limitations.Source: §D.1 · p. 18
• Three proprietary multimodal generators and three diffusion image generators are also evaluated with specified sampling and temperature settings.The proprietary models are GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro-latest; the image generators are Flux1-dev, Stable Diffusion 3, and Stable Diffusion 2.1.Source: §D.1 · p. 18

• Table 11 reports the full results of model generation.Source: §E.1 · p. 18

Qualitative comparisons

• Figures 11 and 12 provide qualitative comparisons for combinations of two and three references, with ground-truth images included for comparison.Source: §E.2 · p. 18, Figure 11 · p. 19, Figure 12 · p. 19

Two-reference conditioning

• Two-reference generation exposes weak spatial adherence: most models poorly maintain bounding boxes, while compositional frameworks struggle with depth references.Source: §E.2 · p. 18

Three-reference conditioning

• Three-reference conditioning increases complexity, with ACE strongest for spatial scene composition and OmniGen and ACE preserving architectural details in one Canny + Style case.Source: §E.2 · p. 18

Reference-format ablations

• Conditioning formats produce model-specific preferences: ACE handles depth plausibly, ChatDiT is robust to mask colors, and no format is universally best.Source: §E.2 · p. 18, Figure 13 · p. 20
• Bounding-box conditioning yields variation in spatial accuracy and semantic fidelity, while input order changes the balance between depth-cue and style adherence.Source: §E.2 · p. 18, Figure 15 · p. 21

Caption ablation

• Caption removal markedly degrades fidelity and relevance, producing distorted, abstract, or unrelated outputs across tested models.Source: §E.2 · p. 18, Figure 14 · p. 21