PaperlayerView exact revision on arXiv

Source-linked AI summary

MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation

Bharath Krishnamurthy, Ajita Rattani

arXiv:2603.29029v1cs.CVcs.AI

TL;DR

Existing multimodal face-generation systems remain limited by shallow fusion, duplicated or rigid architectures, and failures under conflicting modalities, making spatial–semantic alignment difficult. MMFace-DiT introduces a unified dual-stream diffusion transformer that processes text and spatial tokens in parallel and fuses them with shared RoPE Attention, while dynamically adapting to masks or sketches. The model reports state-of-the-art performance, including a relative 40.5% FID reduction to 16.63 for its flow-matching variant and a 40% improvement in visual fidelity and prompt alignment over six multimodal models.

  • Problem

    Existing multimodal face-generation approaches use auxiliary modules or composed uni-modal networks that remain constrained by shallow fusion, duplicated parameters, conflicting modalities, and mismatched latent spaces.

  • Method

    MMFace-DiT is a unified end-to-end diffusion transformer with parallel semantic and spatial streams, shared RoPE Attention for deep fusion, and a Modality Embedder for masks or sketches.

  • Results

    MMFace-DiT attains state-of-the-art performance, while its flow-matching variant reduces FID by a relative 40.5% to 16.63.

  • Takeaways & Limitations

    The model provides a unified paradigm for high-fidelity, controllable face synthesis from text, masks, and sketches.

Abstract

from arXiv · show

Recent multimodal face generation models address the spatial control limitations of text-to-image diffusion models by augmenting text-based conditioning with spatial priors such as segmentation masks, sketches, or edge maps. This multimodal fusion enables controllable synthesis aligned with both high-level semantic intent and low-level structural layout. However, most existing approaches typically extend pre-trained text-to-image pipelines by appending auxiliary control modules or stitching together separate uni-modal networks. These ad hoc designs inherit architectural constraints, duplicate parameters, and often fail under conflicting modalities or mismatched latent spaces, limiting their ability to perform synergistic fusion across semantic and spatial domains. We introduce MMFace-DiT, a unified dual-stream diffusion transformer engineered for synergistic multimodal face synthesis. Its core novelty lies in a dual-stream transformer block that processes spatial (mask/sketch) and semantic (text) tokens in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism. This design prevents modal dominance and ensures strong adherence to both text and structural priors to achieve unprecedented spatial-semantic consistency for controllable face generation. Furthermore, a novel Modality Embedder enables a single cohesive model to dynamically adapt to varying spatial conditions without retraining. MMFace-DiT achieves a 40% improvement in visual fidelity and prompt alignment over six state-of-the-art multimodal face generation models, establishing a flexible new paradigm for end-to-end controllable generative modeling. The code and dataset are available on our project page: https://vcbsl.github.io/MMFace-DiT/

1. Introduction

Existing diffusion and multimodal face-generation approaches struggle to align precise spatial structure with semantic and attribute consistency. MMFace-DiT addresses this with unified dual-stream processing, shared cross-modal fusion, dynamic modality conditioning, and an enriched annotated dataset.

  • Current diffusion models lack precise spatial control for structured synthesis tasks such as controllable face generation.
  • Existing multimodal approaches trade spatial fidelity against semantic consistency and can fail with conflicting modalities or rigid latent-space requirements.GANs also struggle with fine-grained attributes, while adapters limit deep fusion and compositional systems impose architectural constraints.
  • MMFace-DiT jointly processes semantic text and spatial masks or sketches in a unified, end-to-end architecture.The model avoids separate models and inference-time composition.
  • Shared RoPE Attention fuses text and spatial streams at every block while treating both modalities as co-equals.The design targets improved cross-modal alignment and prompt adherence.
  • A VLM-based multi-prompt annotation pipeline produces a large-scale, semantically rich face dataset for multimodal face-generation research.
  • A Modality Embedder lets one model dynamically interpret different spatial conditions without retraining.Supported conditions include masks and sketches.

2. Related Work

Related work spans latent-space GAN methods, conditioning adapters, and inference-time composition, while diffusion transformers provide scalable backbones but require additional spatial inputs for precise control. These alternatives impose representation, fusion, or compatibility constraints.

  • Diffusion models evolved from DDPMs and latent diffusion models to scalable transformer-based DiT backbones.
  • Precise spatial control in modern DiTs requires additional conditioning inputs such as masks or sketches.
  • GAN-based methods using StyleGAN latent manipulation suffer from entangled representations that limit fine-grained facial attributes and photorealism.DDGI inherits similar limitations.
  • ControlNet adds trainable spatial-conditioning modules to pretrained diffusion models, but its frozen backbone limits bidirectional semantic–spatial fusion.
  • Inference-time compositional frameworks combine pretrained single-purpose models but can be bottlenecked by the weakest model and require compatible latent dimensions.They may also fail when modalities provide conflicting information.

3. Proposed Methodology

MMFace-DiT is a unified latent-diffusion transformer that jointly processes text with dynamically selected masks or sketches. Its dual-stream architecture combines global conditioning, shared attention, and gated updates to support multimodal face synthesis.

  • Unified multimodal model: MMFace-DiT natively processes textual descriptions with dynamically selected masks or sketches in a unified end-to-end diffusion transformer.The model operates in a VAE latent space and uses co-equal spatial and semantic streams.
  • Data and representations: The data pipeline enriches FFHQ and CelebA-HQ with VLM-generated captions, using ten prompts per image plus filtering and language-model post-processing.The resulting dataset contains 1M captions for 100K images.
  • Data and representations: Noisy image and spatial-condition latents are concatenated and projected into image tokens, while CLIP sequence embeddings form text tokens.Image tokens represent the combined latent input; text tokens retain sequence-level contextual information.
  • Unified conditioning and dynamic modality adaptation: A unified global conditioning vector combines timestep, pooled text, and modality embeddings, allowing one model to adapt between masks and sketches without retraining.The modality flag is mapped to a dense vector and injected into global context.
  • The dual-stream MMFace-DiT block: Parallel image and text streams are deeply fused through shared RoPE Attention, with AdaLN conditioning and gated residual connections controlling each stream.Combined attention enables bidirectional image-text information flow, while gates selectively regulate attention and MLP outputs.

4. Experiments

MMFace-DiT is evaluated for mask- and sketch-conditioned face generation against established multimodal baselines using realism, spatial-fidelity, and text-alignment metrics. Across qualitative and quantitative comparisons, it preserves structural detail while rendering semantically specified facial attributes.

  • Experimental Setup: The evaluation compares mask conditioning against six leading approaches and sketch conditioning against the same baselines except CD, which lacks pretrained task weights.Baselines include TediGAN, ControlNet, UaC, CD, and DDGI; the passage describes six leading approaches while listing five names.
  • Experimental Setup: Performance is measured with FID, LPIPS, mask ACC and mIoU, SSIM, CLIP Score and Distance, and LLM semantic-consistency scores.These metrics jointly assess realism, perceptual similarity, structural integrity, spatial fidelity, text-image alignment, and semantic consistency.
  • Mask-Conditioned Generation: MMFace-DiT renders complex mask-specified attributes such as wavy blonde hair, blue eyes, high buns, and gold earrings while maintaining structural fidelity and identity coherence.The reported qualitative comparison emphasizes photorealism, accurate attribute geometry, and material realism.
  • Sketch-Conditioned Generation: Sketch-conditioned MMFace-DiT preserves detailed geometry and natural skin texture while rendering textual attributes such as warm smiles and dark blue shirts with realistic shading.The qualitative comparison reports that baselines often produce oversmoothed or semantically inconsistent outputs.
  • Text and Mask Conditioning: 27.95 FID is achieved by diffusion-trained MMFace-DiT, a 42.8% reduction relative to UaC; it also records 24.8% higher CLIP Score than ControlNet and 24.4% lower LPIPS than DDGI.The flow-matching variant further reaches 16.63 FID, a relative 40.5% reduction.
  • Text and Sketch Conditioning: 27.67 FID is achieved by the diffusion model for text-and-sketch conditioning, improving 32.4% over MM2Latent; LPIPS is 44.2% lower than DDGI and LLM consistency is 56.8% higher than ControlNet.The flow-matching variant reduces FID by 66.9% relative to the diffusion model.

5. Ablation Study

The ablation study isolates the effects of the Modality Embedder, Dual-Stream design, RoPE Attention, and VAE choice. The results associate these components with improved multimodal and spatial performance, while Flux provides the strongest perceptual balance.

  • Model Components: The baseline DiT reaches 44.52 FID and 44.86 mIoU but requires separate training for each spatial modality.This establishes the reference configuration for the component ablation.
  • Model Components: Adding the Modality Embedder enables shared spatial learning, improving FID by 9.1% and mIoU by 3.3% over the baseline.The Modality Embedder is evaluated as Model-2 in the incremental ablation.
  • VAE Choice: VAE selection trades statistical fidelity against perceptual quality: SD3 can appear oversaturated, SDXL desaturated, while Flux yields more accurate colors and natural texture.The qualitative comparison identifies Flux as the most perceptually faithful backbone.
  • VAE Choice: Flux attains 0.239 LPIPS on sketches and is described as the most balanced, artifact-free VAE backbone for preserving skin texture and color fidelity.Its latent space is associated with training on curated photorealistic data.

6. Conclusion

MMFace-DiT is presented as a unified architecture for high-fidelity face generation from text, masks, and sketches. Its dual-stream fusion and dynamic modality conditioning support flexible, end-to-end multimodal synthesis.

  • MMFace-DiT generates high-fidelity faces from text, masks, and sketches within a unified architecture.
  • Dual-stream blocks with shared RoPE attention enable deep cross-modal fusion.
  • The dynamic modality embedder supports flexible conditioning within a single model.
  • MMFace-DiT is reported to outperform five leading methods and establish a benchmark for unified multimodal face synthesis.

1. Implementation Details

The model uses a large DiT-XL configuration with shared positional attention and dynamic modality conditioning. Its architecture combines latent spatial inputs with global and sequence-level conditioning.

  • The DiT-XL configuration contains 1.345 billion parameters, 28 transformer blocks, hidden size 1152, and 16 attention heads.
  • The model uses 2D RoPE in shared attention to encode positional information in both streams’ query and key tensors.
  • A Modality Embedder converts mask or sketch conditions into a dense vector for dynamic conditioning.
  • AdaLN derives twelve parameter vectors per block to control attention and MLP shifts, scales, and gates.
  • Training progressively starts from scratch at 256 × 256 resolution for 300 epochs with batch size 32 per GPU.

2. Training Objectives and Inference

MMFace-DiT supports both DDPM and RFM training paradigms using the same core model and conditioning mechanisms. The two objectives differ in their prediction targets and sampling procedures.

  • MMFace-DiT supports DDPM and RFM training with the same core model and conditioning mechanisms.
  • DDPM: Under DDPM, the model predicts noise added to a clean latent at a sampled timestep.
  • DDPM: DDPM conditioning combines noised and spatial latents with timestep, pooled-text, modality, and sequence-text signals.
  • RFM: RFM trains the model to predict the constant velocity connecting a noise sample to a data sample along an interpolated latent path.
  • RFM: RFM inference integrates the predicted velocity from t = 0 to t = 1 with an ODE solver such as Euler’s method and CFG.

3. VLM-Powered Data Enrichment Pipeline

The data-enrichment pipeline creates rich textual annotations for FFHQ and CelebA-HQ to address missing captions and limited semantic detail. It combines multi-prompt VLM generation with rule-based cleaning and LLM refinement, producing one million public captions.

  • Motivation: The pipeline addresses missing FFHQ captions and limited semantic richness in existing CelebA-HQ captions.
  • Multi-Prompt Strategy: InternVL3 generates ten caption variants per image through prompts targeting different information styles and attributes.
  • Multi-Prompt Strategy: The prompts include few-shot, keyword-focused, and detail-oriented formats for broad and fine-grained descriptions.
  • Two-Stage Refinement: Qwen3 then refines cleaned captions and mitigates hallucinations through grammatical and coherence improvements.
  • Two-Stage Refinement: A rule-based cleaning stage removes instructional artifacts, template placeholders, and inconsistent formatting.
  • Output Dataset: The released dataset contains 10 captions for each of 100,000 FFHQ and CelebA-HQ images, totaling 1 million captions.

4. Qualitative Visualizations

The qualitative results examine fine-grained attribute control, training paradigms, and VAE choices through visual comparisons. They show how outputs change when text attributes vary, how Diffusion and Flow objectives compare, and how VAE selection affects perceptual quality and artifacts.

  • Figures 1 and 2 vary one text-prompt word while keeping the spatial condition fixed to assess fine-grained, disentangled attribute control.
  • Figures 3 and 4 compare output quality under Diffusion and Flow training paradigms.
  • Figures 5 and 6 visualize the VAE ablation, focusing on perceptual quality and artifact reduction.
Loading 2603.29029v1…