PaperlayerView exact revision on arXiv

Source-linked AI summary

InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators

Xuanlang Dai, Jiazi Bu, Yujie Zhou, Yuhong Liu, Beichen Zhang, Penghui Yang, Ziyu Liu, Yuhang Zang

arXiv:2610.13944v1cs.CV

TL;DR

Heterogeneous generators differ in architecture and representation, while cross-model latent alignment and denoising-dynamics compatibility remain underexplored. InstantFusion uses model-specific LAEs to map intermediate states into a shared space, aligning matched-noise states and their local denoising velocities for cross-model transfer. InstantFusion achieves 1.8–3.3× generation speedup while maintaining strong perceptual quality; composing Qwen-Image with SD3 or FLUX improves GenEval-v2 by 13 points and CVTG-2K by 0.18–0.28 over corresponding base models.

  • Problem

    Heterogeneous generators differ in architecture and representation, while cross-model latent alignment and denoising-dynamics compatibility remain underexplored.

    Source: §1 · p. 1

  • Method

    InstantFusion uses model-specific LAEs to map intermediate states into a shared space, aligning matched-noise states and their local denoising velocities for cross-model transfer.

    Source: §1 · p. 2

  • Results

    InstantFusion achieves 1.8–3.3× generation speedup while maintaining strong perceptual quality; composing Qwen-Image with SD3 or FLUX improves GenEval-v2 by 13 points and CVTG-2K by 0.18–0.28 over corresponding base models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Results across diverse downstream tasks demonstrate InstantFusion’s flexibility and broad applicability as a shared interface for heterogeneous image generators.

    Source: §6 · p. 11

    The formulation relies on a shared autoencoder and compatible time parameterization, while hybrid inference requires hosting multiple frozen backbones simultaneously.

    Source: §5 · p. 11

Abstract

from arXiv · show

Diffusion/flow models have emerged as the de facto paradigms for image generation, giving rise to a diverse family of modern generators with increasingly heterogeneous architectures and representations. Despite their remarkable generation capabilities, the cross-model alignment of latent representations and the consistency of their underlying denoising dynamics remain underexplored. In this work, we uncover an intriguing cross-model compatibility: independently trained generators exhibit similar latent spatial organization, while directly handing off intermediate states between unaligned models can still preserve coarse visual content. Motivated by these observations, we propose InstantFusion, a lightweight shared latent interface that connects the denoising trajectories of heterogeneous image generators. Specifically, to enable flexible cross-model state translation, model-specific Latent AutoEncoders (LAEs) are introduced to project intermediate states from different generators into a shared latent space and decode them into the latent space of a target generator. This shared space is constrained by two complementary objectives: aligning intermediate states derived from the same image at matched noise levels and synchronizing their corresponding local denoising velocities. Once trained, InstantFusion allows denoising to seamlessly transition across frozen generators without requiring a dedicated mapping for every model pair. The versatility of InstantFusion is demonstrated across acceleration, multi-preference composition, cross-architecture on-policy distillation, and cross-model composition, achieving competitive performance across these diverse settings.

1. Introduction

Heterogeneous image generators are usually studied in isolation, leaving cross-model latent alignment and denoising-dynamics compatibility underexplored. InstantFusion turns observed partial compatibility into a shared interface, with applications spanning cross-model composition and other downstream tasks.

1. Introduction

  • Unaligned generators show similar intermediate-state spatial organization, while direct handoffs can preserve the main subject and coarse layout despite artifacts.The figure describes similar patterns for Qwen-Image, SD3, and FLUX.1, with related-model handoffs preserving quality and cross-family handoffs retaining coarse content.Source: §1 · p. 1, Figure 2 · p. 3
  • Prior latent-alignment methods mainly map model pairs without a shared multi-generator space or explicit dynamics modeling, while trajectory stitching generally assumes a common latent space.These constraints leave trajectory transfer across heterogeneous latent spaces open.Source: §1 · p. 2
  • InstantFusion uses model-specific LAEs to encode generator states into a shared space and decode them into a receiver’s latent space, aligning matched-noise states and local denoising velocities.At inference, a sender state is encoded and decoded by the receiver’s LAE; new generators can be added without dedicated pairwise mappings.Source: §1 · p. 2, Figure 1 · p. 2
  • InstantFusion supports generation acceleration, multi-preference composition, cross-architecture on-policy distillation, and cross-model composition, with reported speedups of 1.8–3.3× while maintaining strong perceptual quality.For Qwen-Image composed with SD3 or FLUX, it improves GenEval-v2 by 13 points and CVTG-2K by 0.18–0.28 over the corresponding base models.Source: §1 · p. 2
  • The paper’s contributions combine evidence of partial generator compatibility, a shared interface for transferring states and dynamics, and demonstrations across downstream applications.The reported applications include acceleration, multi-preference composition, and cross-architecture on-policy distillation.Source: §1 · p. 3

2. Related Work

Modern text-to-image generators differ in latent parameterizations, denoising architectures, and training objectives, yet are commonly studied in isolation. Related work aligns latent states or stitches trajectories, but leaves the compatibility of denoising dynamics across models unresolved.

Text-to-Image Diffusion Models

  • Text-to-image diffusion and flow models commonly denoise in learned latent spaces, but modern generators vary in latent parameterization, architecture, and training objective.Examples named include SD3, FLUX.1, Qwen-Image, and Z-Image.Source: §2 · p. 3

Cross-Model Latent Alignment

  • Cross-model alignment studies use linear mappings, relative representations, closed-form transformations, or learned bridges to connect latent spaces.These approaches primarily establish correspondences between isolated latent states.Source: §2 · p. 3
  • Prior alignment work does not model how corresponding states evolve along denoising trajectories, leaving dynamic compatibility unclear.The passage distinguishes static latent correspondence from compatibility between underlying generative dynamics.Source: §2 · p. 3

3. Methods

InstantFusion connects frozen heterogeneous generators through model-specific LAEs that align and reconstruct intermediate states in a shared space, then synchronize denoising dynamics. The methods motivate this interface from partial compatibility across models and use predicted-clean-state consistency to choose handoff timing online.

3.1. Preliminaries

  • Flow matching defines a path from clean latent data to noise, and generators learn a prompt-conditioned vector field that predicts its velocity for denoising.Generation integrates the learned field from t=1 to t=0.Source: §3.1 · pp. 3–4

3.2. Observation: Shared Structure Across Models

  • Across generators, intermediate states show similar spatial organization, while unaligned cross-family handoffs introduce artifacts but preserve the central cup and coarse layout.The contrast suggests shared coarse semantic structure, not interchangeable native latent spaces.Source: §3.2 · p. 4

3.3. Latent Alignment and Reconstruction

  • Each frozen generator receives a time-conditioned LAE that encodes matched noisy states into a shared representation and reconstructs them in its native latent space.Pairwise alignment encourages correspondence across generators; reconstruction preserves input-dependent information needed for denoising, while norm regularization encourages comparable representation magnitudes.Source: §3.3 · p. 4, Figure 3 · p. 5

3.4. Denoising Dynamics Alignment

  • State alignment alone leaves cross-model denoising displacements poorly aligned, so the LAEs also align one-step displacements over a common sampling interval.The generators remain frozen and only the LAEs are optimized.Source: §3.4 · pp. 5–6, Figure 4 · p. 6

3.5. Adaptive Model Handoff

  • Because optimal handoff timing varies across model pairs and early switching is ineffective, the method selects a handoff online using temporal consistency of predicted clean states.It chooses the first step meeting a similarity threshold, after which the receiver continues denoising from the translated state.Source: §3.5 · pp. 6–7

4. Experiments

Experiments train LAEs across four frozen generators, then assess reconstruction, state transfer, and downstream applications. Results cover acceleration, composition, distillation, and multi-stage rollout stability.

4.2. Performance of LAE

  • Qwen-Image guidance raises SD3 and FLUX.1 GenEval-v2 by 12.92 and 13.68 to 35.06 and 36.54; CVTG-2K improves by 0.2806 and 0.1803.Single-trip reconstruction exceeds 31 dB PSNR and 0.94 SSIM, though fidelity degrades as latent round trips increase.Source: §4.2 · pp. 7–8, Table 2 · p. 8, Table 1 · p. 8

4.4. Ablation and Analysis

  • Replacing time-conditioned residual blocks with standard convolutions reduces GenEval-v2 by 7.20 for Qwen-Image–SD3 and 19.17 for Qwen-Image–FLUX.1.This comparison uses the same layer configuration.Source: Table 6 · p. 9, §4.4 · p. 10

4.3. Applications of LAE

  • Qwen-Image hybrids reduce inference time from 18.84s to 4.373s and 2.545s without the quality collapse observed in T-Stitch [27].InstantFusion also works with DiCache: latency falls from 10.64s to 5.70s, while GenEval-v2 changes from 32.70 to 32.46.Source: Table 3 · p. 8, §4.3 · p. 9
  • Cross-model OPD improves SD3 from 26.27 to 26.46 in HPSv2.The teacher computes velocity targets along the student’s trajectory through the shared latent space.Source: Table 1 · p. 8, §4.3 · p. 9
  • InstantFusion scores 30.97 on GenEval-v2 and 30.68 on HPSv2, outperforming velocity fusion and closely matching jointly trained multi-reward GRPO.The composed experts are existing single-reward models, avoiding the additional optimization required by joint multi-reward training.Source: §4.3 · p. 9, Table 4 · p. 9

4.4. Ablation and Analysis

  • Removing L_dyn causes cross-architecture OPD divergence despite aligned latents, while removing L_rec impairs reconstruction; selected handoffs often follow global-structure stabilization.The ablations distinguish the roles of dynamics alignment and reconstruction fidelity; predicted clean images at selected handoff steps already preserve the final output’s coarse layout.Source: §4.4 · pp. 9–10

4.5. Ablation on Three-Stage Rollout Scheduling

  • Generation quality remains highly consistent across all six execution orders of SD3, FLUX.1, and Qwen-Image in a three-stage rollout.The evaluation covers GenEval-v2 and HPSv2.Source: §4.5 · p. 10, Table 7 · p. 11

4.6. Inference Efficiency Analysis

  • Each handoff translates the latent state without another denoiser evaluation, so InstantFusion adds no theoretical NFE.Table 8 reports latency and NFE at 1024 × 1024; measurements average 20 runs after three warm-ups.Source: §4.6 · p. 11, Table 8 · p. 11

5. Limitations

InstantFusion’s formulation and hybrid inference have specific compatibility and resource constraints.

5. Limitations

  • The formulation relies on a shared autoencoder and compatible time parameterization across generators.These are current formulation assumptions.Source: §5 · p. 11
  • Hybrid inference requires hosting multiple frozen backbones simultaneously, increasing memory and storage overhead.Source: §5 · p. 11
  • Extending the interface across heterogeneous autoencoders and optimizing dynamic model loading or offloading remain future directions.Source: §5 · p. 11

6. Conclusion

The conclusion presents InstantFusion as a shared latent interface for connecting heterogeneous image generators and enabling cross-model state transfer. The appendix outlines supplementary implementation, evaluation, and qualitative material.

6. Conclusion

  • InstantFusion connects heterogeneous image generators by aligning latent states and enabling cross-model state transfer without modifying frozen generators.Source: §6 · p. 11
  • Future work may develop the learned shared space into a reusable pretrained interface for incorporating new generators with minimal adaptation.The proposed common starting point is intended for model composition, knowledge transfer, and downstream optimization.Source: §6 · p. 11

A. Appendix

  • The appendix covers implementation and training, related work, extended quantitative results, and additional qualitative examples and handoffs.It also points to a limitations discussion.Source: A. Appendix · p. 15

B. Training details

Training details describe separate LAE-optimization and cross-model OPD stages, with the shared-interface hyperparameters listed in Table 9.

B. Training details

C. Additional Related Work

Prior work addresses acceleration, on-policy distillation, and multi-preference generation, but leaves open how to compose generators with heterogeneous latent spaces and reuse independently trained experts.

Efficient Generation

  • Acceleration methods reduce denoising evaluations through few-step distillation or consistency learning, feature caching, and scheduling pretrained models across denoising stages.Existing scheduling work primarily assumes that the models’ intermediate states are directly interchangeable; heterogeneous latent spaces remain largely unexplored.Source: C. Additional Related Work · p. 15

Diffusion On-Policy Distillation

  • Teacher-based on-policy distillation transfers denoising capabilities along student trajectories, but direct transfer across incompatible latent spaces remains an open question.Existing approaches assume the same model family or compare decoded outputs in a frozen visual representation.Source: C. Additional Related Work · p. 15

Multi-Preference Composition

  • Multi-preference methods jointly optimize one generator or merge preference-specific models, tying composition to training or parameter manipulation.Parameter merging relies on compatible architectures and weight-space interpolation, limiting flexible reuse of independently trained experts.Source: C. Additional Related Work · p. 16

D. Per-Model Reconstruction Analysis Beyond Average Metrics

The per-model reconstruction analysis evaluates fidelity across repeated latent round trips and finds that errors accumulate, revealing stability limits of the shared latent space.

Per-model breakdown

  • Table 11 reports per-model PSNR, SSIM, and LPIPS across 1, 5, and 10 latent round trips.This granular breakdown complements Table 1a’s average reconstruction fidelity across generators.Source: D. Per-Model Reconstruction Analysis… · p. 16

Sequential translations

Stability limits

E. Extended Qualitative Results

Cross-model handoffs are evaluated at arbitrary intermediate denoising timesteps, with transitions reported as reliable virtually throughout the trajectory.

E.1. Handoffs at Different Denoising Steps

  • Cross-model handoffs at arbitrary intermediate timesteps execute reliably at virtually any denoising stage, indicating LAE maintains robust state and dynamics alignment across the full generation trajectory.Figure 11 shows cross-model state handoffs between Qwen-Image and SD3 at different denoising timesteps.Source: §E.1 · p. 16, Figure 11 · p. 17
  • The reported flexibility indicates that the LAE maintains state and dynamics alignment throughout the trajectory, not only within specific timestep windows.The handoff evaluation tests arbitrary intermediate timesteps.Source: §E.1 · p. 16
  • Figure 11 depicts cross-model state handoffs involving Qwen-Image and SD3 at different denoising timesteps.Source: Figure 11 · p. 17

E.2. Additional Results across Applications

Extended qualitative results cover shared-latent reconstruction, cross-architecture handoffs, acceleration, and compositions across different preference alignments.

E.2. Additional Results across Applications

  • Qualitative comparisons show single-trip and multi-trip latent reconstructions across SD3, FLUX.1, and Qwen-Image, with spatial structures consistently preserved across their distinct autoencoder designs.Table 11 compares reconstruction fidelity across shared-space configurations.Source: §E.2 · p. 17, Figure 12 · p. 18, Table 11 · p. 19

SD3–FLUX.1–Qwen

Loading 2610.13944v1…