Source-linked AI summary
X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesis
Sonali Godavarthy, Matthias Neuwirth-Trapp, Tim-Felix Faasch, Maarten Bieshaar, Michael Moeller, Kristof Van Laerhoven, Danda Pani Paudel
TL;DR
Imaging factor disentanglement lacks direct supervision for unseen factor combinations, and FAA can misrepresent disentanglement because of cross-factor correlation leakage. X-MULTI adds zero-shot VLM supervision for novel combinations and I-FAA uses factor-specific augmentation to reduce leakage; on novel combinations, X-MULTI raises average I-FAA from 0.47 to 0.53 over MULTI.
Problem
MULTI trains reconstruction only on observed factor combinations, while FAA can exploit cross-factor correlations rather than independently measuring factor correctness.
Method
X-MULTI supervises synthetically generated unseen combinations with a zero-shot VLM, and I-FAA uses class-balanced training and factor-specific augmentations to reduce shortcut learning.
Results
0.47 to 0.53: X-MULTI improves average I-FAA over MULTI on novel factor combinations, while I-FAA substantially reduces cross-factor correlations.
Takeaways & Limitations
X-MULTI achieves stronger factor disentanglement on novel combinations, and I-FAA provides a more robust assessment of factor alignment.
Abstract
from arXiv · showhide
Imaging factor disentanglement in text-to-image generation aims to independently control image acquisition properties such as types of camera lenses, sensor types, viewpoints, and domains to enable combinatorial generalization. This should let the model synthesize novel factor combinations unobserved in the training data, such as pairing a fisheye lens with an event sensor never observed in training data. Recent work, MULTI, introduced learnable, factor-specific embeddings to disentangle imaging factors, along with the Factor Alignment Accuracy (FAA) metric to evaluate disentanglement quality. We identify and address two independent limitations. First, MULTI's pixel-level reconstruction objective supervises the model only on observed imaging factor combinations, providing no direct training signal for novel combinations. We therefore propose X-MULTI, which uses a pretrained vision-language model (VLM) to supervise novel factor combinations synthesized during training. Second, we show the FAA metric exhibits severe cross-factor correlation leakage, misrepresenting true disentanglement quality. We therefore propose Improved-FAA (I-FAA), which employs factor-specific augmentation strategies to break these correlations and enables more rigorous evaluation. Experiments demonstrate that X-MULTI achieves improved factor alignment on novel combinations compared to MULTI. Moreover, we show that correlation leakage in FAA distorts the evaluation of true factor disentanglement and I-FAA reduces this leakage and therefore provides a more robust assessment of factor alignment.
1 Introduction
Imaging factor disentanglement seeks independent control of acquisition factors and synthesis of unseen combinations. X-MULTI adds VLM supervision for novel combinations, while I-FAA addresses correlation leakage in FAA evaluation.
- Imaging factor disentanglement targets independent control of lenses, sensors, viewpoints, and domains, including novel combinations absent from training data.The goal matters because collecting data for every acquisition condition is expensive, while vision systems must remain robust across such conditions.
- MULTI’s reconstruction objective trains only on observed factor combinations, leaving no direct supervision for unseen combinations and causing factor confusion.The paper gives thermal-versus-RGB sensor identity as an example of semantically distinct factors that may be confused at inference.
- FAA can exploit severe correlations between factor classifiers, misrepresenting true disentanglement quality instead of evaluating factors independently.The authors identify cross-factor leakage as a critical limitation of the original metric.
- I-FAA reduces shortcut learning through class-balanced training and factor-specific augmentations, enabling more robust assessment of imaging factor disentanglement.Its design aims to break correlations between factors and dataset-specific cues.
- X-MULTI introduces zero-shot VLM supervision for synthetically generated unseen imaging-factor combinations, adding explicit factor-level supervision beyond reconstruction.The approach augments MULTI with VLM-based guidance for factor embeddings.
2 Related Work
Prior work has adapted text-to-image models for concept and attribute disentanglement, but imaging-factor control has received less attention. X-MULTI extends MULTI with VLM guidance and improves evaluation through factor-specific augmentation.
- Disentanglement in Generative Models: Textual Inversion learns token embeddings for target concepts while keeping the generative model frozen, complementing LoRA-based approaches to separating style and content.These methods typically address object attributes such as color or texture rather than image-acquisition factors.
- Imaging-Factor Disentanglement: MULTI applies Textual Inversion to learn imaging-factor embeddings and improves novel-combination generation, but its factor alignment remains limited.X-MULTI builds on this setup with VLM-guided supervision.
- Discriminator-Based Guidance and VLM Supervision: VLMs have been repurposed as frozen discriminators that guide generative models toward high-level conceptual constraints.X-MULTI uses this paradigm to enforce imaging-factor identity.
- Disentanglement Evaluation Metrics: FAA evaluates imaging-factor disentanglement with classifiers trained on DF-RICO, but correlated dataset shortcuts can prevent isolation of individual attributes.I-FAA addresses this issue with factor-specific augmentations that break such correlations.
3 Background
Imaging factor disentanglement represents image formation using discrete factor categories and learnable prompt embeddings. MULTI optimizes these embeddings with a frozen diffusion backbone, while FAA measures factor-label agreement in generated images.
- Imaging Factor Disentanglement: The framework decomposes image formation into K factors, including lens, sensor, viewpoint, and domain, with each image assigned a discrete factor tuple.The full factor space is the Cartesian product of the category sets for all factors.
- Learnable Embeddings: Each factor value is represented by learnable token embeddings inserted into a structured prompt with an image caption.The embeddings are optimized while the text-to-image diffusion model remains frozen.
- Learnable Embeddings: The diffusion objective minimizes the squared difference between sampled noise and the denoising network’s prediction from the noisy latent and text-conditioning embedding.The denoising network, noisy latent, sampled noise, and conditioning embedding are defined in the background formulation.
- Factor Alignment Accuracy: FAA trains one classifier per factor category and measures the fraction of predictions matching the factor labels used to condition generation.The metric estimates whether generated images visually contain the requested imaging factors.
4 Methodology
X-MULTI extends MULTI with VLM supervision for synthetic novel factor tuples while retaining its two-stage factor-embedding structure. I-FAA evaluates factor alignment with balanced, factor-specific classifiers designed to reduce shortcut learning.
- X-MULTI: VLM-based Factor Disentanglement: X-MULTI retains MULTI’s two-stage structure, learning general factor embeddings before adapting them to target-dataset factor values.VLM supervision is added only during Stage 1.
- X-MULTI: VLM-based Factor Disentanglement: The VLM factor-alignment loss sums cross-entropy losses between predicted and generated factor labels across categories.The supervision strength is controlled by λ in the combined diffusion and VLM objective.
- X-MULTI: VLM-based Factor Disentanglement: Factor-wise VLM reliability is estimated by comparing zero-shot predictions on ground-truth images with their true labels using factor-specific prompts.This assesses whether the VLM provides meaningful supervision for each imaging factor.
- X-MULTI: VLM-based Factor Disentanglement: Synthetic factor tuples are created by resampling factor values across categories, and the diffusion model generates images used for additional supervision.This training branch operates after an initial diffusion-only warm-up.
- Improved Factor Alignment Accuracy (I-FAA): I-FAA uses separate classifiers for each factor category, with per-category alignment accuracy computed from predictions matched against conditioning labels and averaged across categories.Each classifier uses a shared frozen visual backbone and a factor-specific trainable linear head.
- Improved Factor Alignment Accuracy (I-FAA): I-FAA reduces classifier shortcuts through class-balanced under-sampling and dedicated factor-specific augmentations.Augmentations are sampled from category-specific operation sets and applied during classifier training.
5 Experimental Results
Experiments evaluate X-MULTI on unseen imaging-factor combinations and assess both its VLM supervision and the reliability of FAA versus I-FAA. X-MULTI improves novel-combination factor alignment, while I-FAA reduces cross-factor leakage but retains residual correlations.
- Novel Factor Combinations: X-MULTI shows clearer adherence to novel viewpoint, lens, and domain factors than MULTI, DreamBooth, and SDXL Zeroshot, but is less proficient with thermal sensors.The qualitative comparison covers unseen factor combinations.
- ControlNet Guidance: With ControlNet guidance, X-MULTI cleanly transforms event-sensor inputs to gated sensors and removes real-world traces when shifting domains from real to video-game.DreamBooth retained event artifacts, while MULTI and DreamBooth did not eliminate real-world traces in the domain shift.
- Quantitative Results: Under I-FAA, X-MULTI obtains the best overall factor disentanglement, factor CLIP alignment, and Diversity Score, while other metrics remain mixed.Table 5 compares methods with and without ControlNet structural guidance.
- Novel Factor Combinations: X-MULTI achieves an average I-FAA of 0.53 versus 0.47 for MULTI on novel factor combinations, a 0.06 or 11% relative improvement.Relative improvements are 21% for lens and 18% for viewpoint; sensor and domain each improve by 0.01 relatively.
- VLM Reliability: VLM supervision is disabled for rgb-thermal sensors and viewpoints other than front because these categories receive unreliable predictions.Rgb-thermal is strongly confused with rgb, and several non-front viewpoints are inconsistently predicted.
- VLM Supervision: Qwen2-VL-7B-Instruct reaches 0.78 average zero-shot classification accuracy versus 0.51 for LLaVA-1.6, supporting its use for VLM supervision.Detailed factor-specific prompts with per-class descriptions are also evaluated.
- Ablations: Moderate 10^-6 and weak 10^-8 supervision both reach an I-FAA of 0.53, whereas strong 10^-4 supervision lowers performance to 0.47.The authors interpret strong supervision as over-constraining factor adaptation.
- Metric Reliability: I-FAA reduces most FAA cross-factor correlations, although residual correlations remain for strongly coupled or sparsely represented factor pairs.The correlation analysis uses Cramer’s V; the metric is designed to reduce shortcut learning through factor-specific augmentations.
6 Discussion
The discussion identifies benefits and dependencies of X-MULTI and I-FAA. VLM supervision improves disentanglement, but its reliability depends on prompts and factor recognition, while I-FAA reduces rather than eliminates leakage.
- VLM-Guided Supervision: Zero-shot VLM supervision adds factor-level signals from synthetic images and significantly improves alignment, particularly for geometric attributes.The supervision predicts factors during training rather than relying only on reconstruction.
- VLM Reliability: X-MULTI’s performance depends on VLM choice and factor-specific prompt design because some factor values are recognized less reliably.Richer visual context in prompts may further improve supervision.
- I-FAA Reliability: FAA exhibits correlations consistent with shared representations entangling factor predictions, whereas factor-specific augmentations reduce this leakage in I-FAA.Residual correlations remain and may require stronger augmentations or correlation-reduction methods.
7 Conclusion
The paper proposes X-MULTI to add zero-shot VLM supervision for unseen factor combinations and I-FAA to reduce cross-factor correlation in disentanglement evaluation. Experiments report stronger novel-combination disentanglement and a more robust evaluation metric.
- X-MULTI augments MULTI with zero-shot VLM supervision for unseen factor combinations, enforcing semantic factor identity.
- I-FAA uses targeted augmentation strategies to reduce cross-factor correlations in FAA and improve disentanglement assessment.
- X-MULTI achieves stronger factor disentanglement than baselines, particularly for novel factor combinations.
- I-FAA substantially reduces correlations and provides a more robust metric for factor disentanglement evaluation.
- Imaging factor disentanglement remains challenging despite the reported improvements.
S.1 Experimental Results on X-MULTI: Existing Factor Combinations
The extended evaluation tests whether X-MULTI’s semantic supervision preserves performance on factor combinations present in training. X-MULTI reportedly represents existing imaging modalities faithfully and achieves the strongest qualitative matching among the compared methods.
- The evaluation examines existing factor combinations to test whether semantic supervision compromises performance on training-distribution combinations.
- X-MULTI most faithfully matches source datasets among the compared methods on existing factor combinations.
- X-MULTI captures specialized gated, rgb-thermal, and event sensors well, with representations reported as superior.
- SDXL Zeroshot struggles to represent imaging factors accurately, while MULTI, DreamBooth, and Inspiration Tree show competence.
S.2 Augmentations of Factor Classifiers for I-FAA
The I-FAA factor classifiers use non-overlapping transformation boundaries to reduce reliance on spurious benchmark correlations. The classifier-specific restrictions separate relevant physical and geometric cues across heads.
- I-FAA enforces strict, non-overlapping transformation boundaries across individual factor-classifier heads.
- Color deviations are excluded from the Sensor classifier so it focuses on physical noise distributions and exposure characteristics.
S.3 Additional Correlation Analysis on FAA and I-FAA
Additional analyses diagnose cross-factor leakage in FAA using conditioned accuracy and conditional error co-occurrence. FAA shows substantial instability and shared failures, whereas I-FAA maintains stable accuracy and nearly isolated errors.
- The analysis evaluates classifier independence through cross-factor accuracy variance and prediction-error co-occurrence patterns.
- FAA lens accuracy conditioned on viewpoint ranges from ∼0% to 75%, while I-FAA remains at 100%; FAA standard deviations reach 37.0%.
- FAA errors are strongly shared: P(view wrong | lens wrong) = 0.98, P(sensor wrong | lens wrong) = 0.91, and P(lens wrong | domain wrong) = 0.99.
- I-FAA errors are nearly fully isolated, with conditional error probabilities at or near 0.
- On existing factor combinations, X-MULTI achieves the highest I-FAA metric while maintaining highly competitive image-generation quality scores.
- I-FAA has less cross-factor accuracy variance and low error co-occurrence in the supplementary analyses.