Source-linked AI summary
InstanceControl: Controllable Complex Image Generation without Instance Labeling
Xiaoyu Liu, Huan Wang, Fan Li, Zhixin Wang, Jiaqi Xu, Ming Liu, Wangmeng Zuo
TL;DR
Complex multi-instance controllable image generation often confuses attributes or requires labor-intensive instance labeling. InstanceControl uses a VLM for instance-level text–visual correspondence and adaptive mask refinement, outperforming FLUX ControlNet with 87.97% versus 77.78% average accuracy.
Problem
Existing controllable methods struggle with attribute control in complex multi-instance scenes and often require time-consuming manual instance labeling during inference.
Method
InstanceControl uses a VLM to associate text descriptions with visual regions, predict instance masks, and refine them during generation for instance-aware control.
Results
87.97% vs. 77.78% average accuracy: InstanceControl outperforms FLUX ControlNet even using predicted masks without mask refinement.
Takeaways & Limitations
InstanceControl achieves superior fidelity and precise instance-level control across multiple benchmarks and visual conditions without manual instance labeling.
Abstract
from arXiv · showhide
Controllable image generation methods, such as ControlNet, have demonstrated a remarkable capacity to introduce visual conditions(e.g., depth maps) to guide image generation. However, these methods often struggle with complex multi-instance scenes, frequently leading to attribute confusion among instances. While recent approaches attempt to mitigate this via manual instance labeling, such requirements are labor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for instance labeling. We identify the primary bottleneck in existing methods as the inability to accurately associate instance descriptions with their corresponding regions within visual conditions. To address this, we leverage the Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions. Specifically, the VLM automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. Furthermore, since the predicted masks may contain noise, we introduce an adaptive mask refinement strategy that dynamically refines these instance masks during the generation process. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.
1 Introduction
InstanceControl targets attribute confusion in complex multi-instance controllable generation without requiring manual instance labeling. It uses VLM-based text–visual association and adaptive mask refinement to improve instance-level fidelity across visual conditions.
- Motivation: Existing controllable methods often confuse attributes across instances as scene complexity increases, while manual labels add costly inference-time annotation.The annotations pair instance descriptions with regions such as masks or bounding boxes.
- Approach: InstanceControl automatically establishes instance-level correspondences between text prompts and visual conditions without instance labeling.The method is designed for fine-grained generation in complex multi-instance scenes.
- Results: InstanceControl performs favorably against state-of-the-art methods across canny edges, depth maps, and HED conditions.The experiments focus on complex multi-instance scenarios and fine-grained control.
- Approach: VLMs parse instance descriptions from text prompts while predicting corresponding instance masks in visual conditions.This two-part process forms the instance-level text–visual association stage.
- Approach: Adaptive mask refinement dynamically reduces noise or inaccuracies in predicted masks during generation.The refinement strategy is intended to preserve more accurate multi-instance correspondences.
2 Related Work
Related work develops controllable generation, instance-aware conditioning, and VLM-based segmentation. These lines of research motivate InstanceControl’s focus on precise multi-instance correspondence without manual labeling.
- Controllable Generation: ControlNet-style methods introduce visual conditions such as depth maps to guide image layout, with later work improving efficiency and control strength.Examples include UniControl, ControlNeXt, and SmartControl.
- Instance-Aware Control: Recent instance-aware methods use descriptions paired with bounding boxes or segmentation masks to reduce attribute confusion in multi-instance scenes.FineControlNet processes instance descriptions and human pose in parallel before merging them in latent space.
- VLM-Based Segmentation: VLMs support holistic reasoning over text and images, but early grounding models primarily provide bounding-box localization rather than pixel-level precision.This limitation matters for complex visual scenes requiring accurate masks.
- VLM-Based Segmentation: Reasoning Segmentation methods such as LISA generate precise masks from complex or implicit queries using a specialized <SEG> token and SAM mask decoder.Later methods improve robustness, mask quality, and spatial sensitivity.
3 Methodology
InstanceControl links each textual instance description to its visual region before generation, then uses those correspondences to enforce instance-aware control. Its VLM parses descriptions and predicts masks, while adaptive refinement limits errors from noisy masks.
- Motivation: The method targets attribute confusion caused by missing instance-level correspondences between text prompts and visual conditions.Existing controllable methods often struggle as the number of instances increases.
- Instance-level association: A VLM parses instance descriptions from complex prompts and predicts the corresponding instance masks in the visual condition.The correspondence set pairs each extracted description with an instance mask.
- Instance-level association: Predicted descriptions use phrase, SEG-token, and instance-identity markers so multiple phrases can be grouped into one entity.A shared SEG-token strategy aggregates representations for phrases describing the same instance.
- Mask prediction: SAM decodes aggregated instance queries and visual-condition features into predicted masks and confidence scores.The SAM image encoder produces dense features, and its decoder predicts each mask together with a confidence score.
- Adaptive refinement: The refinement module combines predicted masks, attention-based masks, confidence scores, and image latents to produce rectified masks at each denoising timestep.This allows the model to follow reliable predicted masks while relaxing unreliable constraints.
- Instance-aware generation: A correspondence mask restricts image tokens for each instance to attend to its associated text tokens, reducing inter-instance interference.The mask is defined over image and text token dimensions and uses refined instance masks to assign correspondence regions.
4 Experiments
InstanceControl is evaluated against controllable and multi-instance generation baselines across visual conditions, with ablations isolating instance association and mask refinement. The results show stronger instance-level attribute control and measurable accuracy gains.
- Experimental setup: The evaluation compares FLUX ControlNet, DreamRenderer, EliGen, CreatiLayout, and Seg2Any across canny edges, depth maps, and HED maps.The comparison includes methods with and without instance labeling.
- Qualitative comparison: InstanceControl achieves significantly finer attribute control for each instance than existing methods in qualitative multi-instance comparisons.The comparisons span varied image prompts and visual conditions.
- Qualitative comparison: FLUX ControlNet frequently confuses attributes, generating red and blue fish despite a prompt specifying a bright blue tang with a yellow tail and a vibrant orange fish.InstanceControl instead maintains precise attribute binding aligned with the text.
- Ablation study: Accuracy rises from 87.97% to 90.10% with the Mask Refinement Module and reaches 93.19% with interactive correction.The refinement adaptively integrates multiple cues to produce more robust masks for complex scenarios.
- Ablation study: 87.97% average accuracy exceeds FLUX ControlNet’s 77.78% even when InstanceControl uses only predicted masks without mask refinement.This isolates the contribution of instance-level association.
- Ablation study: The Shared SEG Token improves controllable image generation by maintaining semantic consistency across multiple descriptions of the same instance.Table 4 confirms its positive effect in ablation experiments.