PaperlayerView exact revision on arXiv

Source-linked AI summary

Appearance Pointers -- Multimodal Region Control of Diffusion Transformers

Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch, Srinath Sridhar, Matheus Gadelha

arXiv:2607.19344v1cs.CVcs.AIcs.GR

TL;DR

Creative professionals need more precise regional control than text prompting reliably provides, while DiTs lack mechanisms for assigning heterogeneous cues to locations. AppearancePointers aligns text or image inputs with masks through compact pointer tokens and evaluates them as a unified multimodal control interface. The model reaches or surpasses modality-specific state-of-the-art methods across the reported evaluation, while supporting multiple regional descriptions in one denoising process.

  • Problem

    Creative professionals require precise regional control over materials, object identities, and spatial arrangements, but text prompting provides indirect and unpredictable control and DiTs lack mechanisms for localizing heterogeneous tokens.

  • Method

    AppearancePointers uses a region correspondence network and spatial aggregation to align text or image inputs with masks and provide compact pointer tokens to a DiT.

  • Results

    The model reaches or surpasses modality-specific state-of-the-art methods across reported metrics while supporting multiple regional descriptions in a single denoising process.

  • Takeaways & Limitations

    AppearancePointers provides a lightweight, modality-agnostic interface for precise multimodal control of where regional appearance cues are applied.

  • Takeaways & Limitations

    Fine-grained details can be smoothed by the token-reduction design, and future work identifies real-time efficiency as an open direction.

Abstract

from arXiv · show

Controllable image generation remains challenging for creative professionals, who often require precise regional control over materials, object identities, and spatial arrangements that cannot be reliably achieved through text prompting alone. Diffusion Transformers (DiTs) can natively ingest heterogeneous tokens stemming from texts and images, but they lack mechanisms for determining where and how these tokens should influence the output. We introduce appearance pointers, compact tokens that guide DiTs toward the correct appearance cues at the correct spatial locations by aligning text or image inputs with user-specified masks. Appearance pointers are produced by a region correspondence network and refined through a spatial aggregation mechanism, enabling the model to handle multiple regional descriptions without significantly increasing token load. Our approach introduces the first modality-agnostic interface for localized multimodal control in a DiT without retraining the base model from scratch. Across a range of metrics, our single model reaches or surpasses the performance of modality-specific state of the art methods, offering a simple and extensible path toward precise, region-aware, multimodal guidance in generative image synthesis.

1 Introduction

AppearancePointers addresses the difficulty of spatially controlling heterogeneous text and image cues in Diffusion Transformers. It routes regional appearance information to specified locations while supporting multiple editing and multimodal generation capabilities.

  • Motivation: Text prompting offers indirect, unpredictable control over the precise materials, layouts, and stylistic details required by creative professionals.This mismatch can require extensive trial and error in practical creative workflows.
  • Motivation: DiTs ingest text and image tokens but lack a mechanism to determine where and how those modality-specific cues should influence the output.Additional tokens alone do not specify spatial intent or region-specific appearance usage.
  • Method: AppearancePointers are compact tokens that route text, image, and mask information to the appropriate spatial regions while preserving DiT architectural flexibility.A region correspondence network fuses prompts or reference images with associated masks to produce the pointers.
  • Results: Across the evaluation, the method is best or second best on all six metrics for text-described regions and surpasses MSDiffusion and DreamRenderer on image-described region adherence and identity preservation.The evaluation uses a synthetic dataset containing multiple elements with image and text descriptions within each image.
  • Capabilities: AppearancePointers supports image and text regional descriptions, insertion, generation, and multimodal conditioning simultaneously in a unified framework.The framework is presented as covering these capabilities together rather than requiring separate modality-specific systems.
  • Capabilities: The framework supports region-controlled generation, insertion, pose control, and multimodal region control for multiple regions in one denoising process.The contribution list also introduces the AppearancePointers-37K dataset with regional text descriptions and novel-view appearance images.

2 Related Works

Prior localized generation methods commonly rely on specialized architectures or inference-time manipulation and often support only one conditioning modality or region. AppearancePointers instead routes multimodal appearance cues to spatial locations through compact pointer tokens in a single diffusion pass.

  • Localized conditioning: Earlier localized conditioning methods used specialized architectures or general conditioning frameworks for segmentation maps and bounding-box layouts.These approaches include methods operating on segmentation maps and extensions to bounding-box control.
  • Localized conditioning: Bounding-box methods often lack precise shape control, while training-free approaches manipulate noise, cross-attention, self-attention, or gradients during inference.These strategies target spatial or semantic guidance but differ in how they impose region constraints.
  • Localized conditioning: Semantic-layout methods strengthen regional concept influence through attention manipulation, latent updates, independent object generation, or per-step image optimization.The related approaches vary between attention-based, gradient-based, and iterative optimization strategies.
  • Reference image guidance: Reference-image methods commonly use specialized feature injection, dual U-Nets, channel concatenation, or text-embedding transformations for localized appearance control.The cited methods generally process a single region at a time, unlike AppearancePointers.
  • AppearancePointers: AppearancePointers converts text, images, and spatial masks into pointer tokens that guide a DiT toward the correct appearance cues and locations during generation.The framework uses joint noisy, text/image, and pointer inputs to synthesize spatially aligned outputs.
  • AppearancePointers: Unlike prior approaches that entangle conditioning globally or process regions sequentially, AppearancePointers supports multiple image- or text-described regions in one diffusion pass.This provides a modular multimodal alternative to region-by-region processing.

3 Method

The method links masked regional prompts to DiT conditioning streams, then aggregates region information into compact appearance pointers for multimodal, spatially controlled generation. Boundary guidance supplements these pointers, while computation is reduced through downsampling and one-time pointer construction.

  • Overview: The method represents each region as a mask–prompt pair, where prompts may contain text, reference images, or both, alongside a global text condition.Each region is generated according to its local prompt while the remaining image follows the global prompt.
  • Appearance Pointers: Appearance pointers link regional masks with local text and image information, guiding the DiT on where and how those cues should influence generation.They are provided alongside local prompts and region contour maps for fine-grained appearance and boundary alignment.
  • Appearance Pointers: The Region Correspondence Transformer jointly processes masks, image tokens, and text tokens into feature maps targeting the DiT’s image and text streams.Mask transformers first align regions with their corresponding conditions before multimodal correspondence processing.
  • Region Aggregation: The Region Aggregation Transformer performs patch-local region-wise attention and uses learnable [CLS] tokens to collapse multiple regional representations into one spatial semantic canvas.Separate aggregation blocks process text and image streams, and their updated [CLS] tokens become the final appearance pointers.
  • Base DiT Conditioning: Appearance pointers and local prompt tokens condition the DiT’s image and text streams, after which the model performs the denoising update.Local image or text prompts may be empty, while pointers remain available for both conditioning streams.
  • Base DiT Conditioning: Boundary maps improve precision because token reduction and aggregation can smooth fine-grained details; pointers are computed once per inference procedure.The added modules contain approximately 400M parameters, a 3.33% increase over the base model, and spatial downsampling reduces attention complexity.

4 AppearancePointers-37K Data Generation

The dataset pipeline creates varied, region-grounded scenes with language descriptions and edited object appearances, then filters the resulting examples for quality. Qualitative evaluation spans diverse region-conditioned text-generation scenarios and compares against regional guidance baselines.

  • Scene and Description Generation: The pipeline samples objects at varying scales and uses an LLM to generate scene-level and object-level descriptions.These descriptions provide structured semantic guidance for subsequent image synthesis and object grounding.
  • Synthesis and Grounding: Images are synthesized with Flux.1 Dev and grounded with Grounded SAM to associate described objects with image regions.Grounding supplies the regional structure used by the dataset.
  • Editing and Filtering: Each object receives pose and material edits, which are filtered using a vision-language model before inclusion.The edits expand variation in object appearance and configuration while the filter removes unsuitable examples.
  • Qualitative Evaluation: Qualitative text-conditioned comparisons cover indoor scenes, object–person interactions, retail shelves, and multi-object arrangements against Seg2Any, DreamRenderer*, and InstanceDiffusion.The caption reports stronger regional fidelity, object appearance accuracy, and alignment with global and local textual cues for the proposed model.

5 Experiments & Results

Experiments evaluate AppearancePointers on text- and image-conditioned regional generation, editing, and multimodal benchmarks. The method achieves strong region fidelity and image quality, outperforming key baselines in reported image-region comparisons.

  • Evaluation setup: The benchmark covers region, text, and image-conditioned generation using the AppearancePointers-37K dataset.Evaluation includes 500 images under text-only and image-only conditions, with global prompts and an average of five regions per image.
  • Editing: The model also supports an editing variant that preserves background regions while applying targeted modifications.This setting provides region-specific edits to existing image content.
  • Text-conditioned generation: On text-conditioned region generation, AppearancePointers records CLIP-IQA 95.02, CLIP-I 90.40, and DINO-I 56.09, the highest reported scores for these metrics.Its MIoU is 40.35, indicating strong adherence to region masks.
  • Image-conditioned generation: On image-region control, AppearancePointers outperforms MS-Diffusion and DreamRenderer* with CLIP-I 93.29, MIoU 40.97, and DINO-I 69.31.The comparison measures semantic alignment and adherence to specified region boundaries.

6 Conclusion

The paper presents appearance pointers as a lightweight, modality-agnostic interface for region-specific control in Diffusion Transformers. Its single model supports multiple regional descriptions in one denoising process and reaches or surpasses modality-specific state-of-the-art performance on most reported metrics.

  • Conclusion: Appearance pointers route text- or image-based appearance cues to their intended spatial locations in a Diffusion Transformer.The mechanism is supported by a region correspondence network and spatial aggregation module.
  • Conclusion: The framework combines multiple regional descriptions within a single denoising process and uses a synthetic dataset with text and image descriptions of multiple regions.The paper describes the resulting interface as simple, extensible, and effective for multimodal guidance.

A.1 Training Setting

The training setting samples multimodal regional prompts, assigns region-aware positional information, and aggregates masked regional features into appearance pointers. The modular design also extends to background-preserving multi-subject insertion.

  • Training Setting: Training uses up to seven regions, including background, and replaces the global prompt with a background-only prompt in 50% of iterations.This encourages regional modules to attend to region prompts and supports cases without a global prompt.
  • Training Setting: Regional prompts can combine same-view or novel-view subject images, material references, and text according to sampled probability ranges.Image prompts are denoted by I, text prompts by T, and combined prompts by I+T.
  • Conditions’ RoPE Positional IDs: The model assigns distinct non-overlapping positional embeddings to conditioning images and text-prompt tokens, with region identifiers incorporated into their positional IDs.The region identifier R(i), grid height H, and token coordinates h,w determine the region-aware positions.
  • Appearance Pointer Mask: Region masks are applied before aggregation so regional conditions contribute non-zero features only within their designated spatial regions.The masked features are passed through modality-specific Region Aggregation Transformer networks to produce image and text appearance pointers.
  • Region Correspondence: The region correspondence block is a six-layer multimodal transformer with separate projections for text, image, and mask tokens, followed by modulation and an MLP skip connection.Its hidden dimension is 768 and it uses a bottleneck operation.
  • Multi-Subject Insertion: Background image and global text prompt can be treated as region conditions, enabling multi-subject insertion with simultaneous regional conditioning.The method is benchmarked against an iterative InsertAnything baseline that inserts one object at a time.

C Ablations

Ablations show that aggregation, masking, positional-ID resampling, and contour guidance each affect identity preservation or region adherence. Performance remains stable in some dimensions under more regions and coarser masks, while spatial metrics decline in crowded or sparse-mask settings.

  • Component ablations: Removing appearance pointer aggregation lowers DINO-I to 54.47 and CLIP-I to 89.93, while increasing MIoU to 41.49.The ablation trades identity preservation for finer region control because the model does not aggregate regions.
  • Component ablations: Removing the appearance pointer mask yields MIoU 69.31 for the masked configuration, compared with the lower-MIoU unmasked baseline.The mask ensures regional condition tokens attend only to target-region tokens.
  • Component ablations: Without position-ID resampling, DINO-I falls to 67.46 and MIoU to 40.75.The resampling strategy improves identity and region adherence for larger numbers of regions.
  • Component ablations: Removing Region Contour Guidance reduces MIoU from 40.97 to 35.81.The reported ablation indicates that contour guidance improves region metrics.
  • Number of regions: As regions increase from 1 to 9, CLIP-I remains stable while DINO-I and MIoU gradually decline because of spatial occlusions and overlapping boundaries.This analysis concerns image-plus-region-to-image generation.
  • Mask precision: Identity preservation and aesthetic quality remain stable with fine, bounding-box, or ellipse masks, while DINO-I and MIoU decrease for coarse masks.The decrease reflects synthesis of novel subject orientations within coarse spatial regions.

D Qualitative Results

The qualitative results demonstrate AppearancePointers for multimodal regional generation, editing, insertion, and layout-guided synthesis, including real-world images and mixed conditioning signals.

  • Image Editing and Layout Guidance: The method performs in-the-wild image editing and layout-guided generation with region-specific conditions.The reported examples include Google-image editing and layouts derived from image segmentation.
  • Generation Pipeline: The generation pipeline links region masks and prompts through a region correspondence network before denoising with concatenated image and text streams.For editing, a masked global image prompt and global text prompt are added before region-prompt linking.
  • Multimodal Region Conditioned Generation: AppearancePointers support multimodal regional generation using material, image, and text conditions in a single scene.Regions are linked to their corresponding conditioning signals, while the background can be generated from a global description.
  • Real-World Evaluation: AppearancePointers are the only evaluated model supporting both image and text as regional guidance with the same weights on SACap-eval.The comparison reports competitive performance in zero-shot evaluation and improved region-text adherence after fine-tuning with 2.3% real data.

F Limitation & Discussion with Future Works

The discussion identifies limitations in facial identity preservation, dataset verification, and scaling to many conditioned regions, while proposing broader modalities and faster editing as future directions.

  • Limitations: FLUX Kontext’s difficulty preserving intricate human facial identities is inherited by AppearancePointers.Identity is preserved more reliably when faces occupy larger spatial regions than smaller ones.
  • Limitations: Large-scale VLM-based dataset verification may leave residual errors that degrade appearance-transfer performance.
  • Limitations: Conditioning on about 10 regions can produce leaking artefacts or degrade regions.The paper illustrates this scaling failure in its failure-case figure.
  • Future Work: Future work includes extending conditioning to audio, video, 3D, and 4D, and improving efficiency for real-time editing and scene control.

G Appearance Pointer Attention Map Analysis

Attention-map analysis shows that appearance pointer tokens attend to the intended regions during multimodal generation, while increasing the number of regions can reduce adherence and quality.

  • Attention Map Analysis: Appearance pointer attention maps highlight the correct regions responsible for conditioning generated segments.Examples include a chair pattern, a bench foot, a lamp, and a crane body.
  • Scaling Behavior: Increasing text-conditioned regions to about 10 can degrade region adherence and image quality.The reported failure case includes an inaccurately captured sofa.

H.1 Dataset VLM Check

The dataset pipeline uses VLMs to generate structured object, material, and scene descriptions and to verify novel-view and material consistency through standardized JSON evaluations.

  • VLM Verification: InternVL rates edit quality and novel-view consistency using overall, geometric, sharpness, and pose-related checks.The novel-view prompt assumes the inputs depict the same 3D model and material.
  • Material Verification: The material verification prompt scores perceptual similarity, albedo color fidelity, roughness-gloss fidelity, and texture-detail fidelity.An example reports scores of 0.7, 0.8, 0.6, and 0.8 for these four criteria, respectively.
  • Dataset Description Generation: Qwen 3 generates object descriptions, succinct Grounding DINO captions, material captions, and scene captions for the dataset.The prompts emphasize object geometry or function, sensory material properties, and scene-level placement and composition.
  • Dataset Output Structure: The structured output stores names and captions for multiple objects alongside scene and background captions.Object records include visual, detection-oriented, and material-specific fields.
  • Scoring Protocol: The verification instructions define consistency scores from 0.0 to 1.0 and categorical checks for geometry, sharpness, and novel pose.They also specify score interpretations ranging from poor to excellent matches.
Loading 2607.19344v1…