PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Learning in Diffusion and Flow-based Model: An Application Aspect

Yanchen Xu, Sida Huang, Zhenyu Gu, Ruishu Zhu, Yilan Gao, Hongyuan Zhang

arXiv:2608.24068v1cs.CV

TL;DR

Generative models learn rich representations, but their use beyond generation lacks a unified taxonomy and broader evaluation. This survey organizes diffusion- and flow-based research through a three-tier framework spanning generative improvement, perception, and general-purpose applications, while identifying challenges for future work.

  • Problem

    Existing research lacks a unified taxonomy of representation learning across diffusion and flow-based models, limiting systematic organization of developments.

  • Method

    The survey reviews diffusion and flow-based methods through a three-tier framework covering generative enhancement, representation extraction for perception, and general-purpose unified applications.

  • Results

    The survey systematically categorizes representative approaches across generative capabilities, perception tasks, and broader applications.

  • Takeaways & Limitations

    The framework clarifies research logic and evolutionary trends while connecting representation learning with applications beyond generation.

  • Takeaways & Limitations

    Representation quality is usually evaluated through indirect downstream metrics, without a unified benchmark for intrinsic properties such as separability, disentanglement, and interpretability.

Abstract

from arXiv · show

Diffusion models and flow-based models have recently become the dominant paradigms in generative modeling, largely due to their ability to learn rich, multi-level visual representations through large-scale training. This creates a bidirectional relationship between generative models and representation learning: improving representation learning enhances generation quality, while the learned representations can be leveraged for broader understanding tasks. This survey systematically explores this interplay with a focus on applications. We propose a three-tier progressive framework that organizes existing works from three perspectives: using representation learning to improve generative capabilities, exploiting generative models to extract representations for perception tasks, and ultimately moving toward general-purpose unified applications. We systematically categorize representative methods across a wide range of downstream tasks, including image classification, dense visual prediction, instance-level perception, and annotation-scarce scenarios. By providing a unified taxonomy and identifying key challenges, this survey aims to clarify the underlying logic of current research and suggest promising directions for future exploration. We hope this work can serve as a valuable reference for researchers interested in harnessing the representation power of generative models for applications beyond generation.

1 Introduction

Diffusion and flow-based models learn rich, multi-level visual representations that support both stronger generation and broader visual understanding. This survey organizes the field through a three-tier framework covering generation enhancement, representation extraction, and general-purpose unification.

  • Diffusion models have become dominant in generative modeling across image, video, audio, and molecular-design domains.
  • Their large-scale training learns rich visual representations at multiple abstraction levels, motivating their use beyond generation.
  • Generative modeling and representation learning have a bidirectional relationship: better representations can improve generation, while learned representations support classification, segmentation, and detection.
  • Existing literature lacks a unified taxonomy, with prior survey coverage focused mainly on diffusion models or treating representation learning only briefly.
  • The survey introduces three progressive perspectives: improving generative capabilities, extracting representations for perception, and pursuing general-purpose unified applications.
  • It reviews diffusion and flow-based models in parallel and discusses their interplay with representation learning, challenges, and future directions.

2 Background

The background introduces diffusion and flow-matching models as stochastic or deterministic routes from simple source distributions to data, along with their continuous formulations and common backbones. It also situates representative application methods across diffusion and flow-based paradigms.

  • Diffusion Models: Diffusion models progressively add Gaussian noise to data and generate samples by reversing the denoising process.
  • Diffusion Models: The diffusion forward process is a Markov chain governed by timestep-dependent variance schedules, with direct noisy-sample sampling enabled by reparameterization.
  • Diffusion Models: A neural network is trained to predict noise from a noisy sample and its diffusion timestep.
  • Diffusion Models: As diffusion steps approach infinity, the discrete process generalizes to a continuous-time SDE with drift, diffusion, and score-function components.
  • Flow Matching Models: Flow matching learns a time-dependent vector field whose ODE deterministically transports a source distribution, typically Gaussian, to the data distribution.
  • Flow Matching Models: Conditional Flow Matching addresses unavailable probability paths and vector fields by constructing conditional paths from source noise toward data samples.
  • Backbones and paradigms: Rectified Flow learns a linear path between source and data distributions, while common generative backbones include U-Nets, latent-space U-Nets, and Transformers.
  • Application taxonomy: The survey catalogs representation-learning methods across diffusion and flow-based models, alongside broad diffusion applications such as image editing and control.

3 Applications

The survey organizes applications into three progressive tiers, moving from generation-centric improvements to perception-oriented uses and ultimately general-purpose unified applications.

  • 3 Applications: Existing applications are organized across three tiers spanning generation-centric, perception-oriented, and general-purpose unified tasks.The generation-centric tier examines training efficiency, generation quality, and controllability.

3.1.1 Image and Video Generating

Representation learning improves generative models through external alignment, internal self-organization, latent-space redesign, and workflow changes. The surveyed approaches report faster training, improved FID, and new challenges around alignment, teacher capacity, and joint transport.

  • External Representation Alignment: External alignment injects pretrained visual or language representations into diffusion-model intermediate layers to address semantic deficiencies and accelerate training.REPA aligns DiT representations with DINOv2 using patch-wise cosine similarity, accelerating SiT training by over 17 times and reaching an FID of 1.42 on ImageNet.
  • External Representation Alignment: REPA adaptations extend alignment to U-Net, end-to-end VAE training, text-image modalities, and video generation.U-REPA reaches an FID of 1.41 at 400 epochs, while REPA-E reaches an FID of 1.12 with 17× and 45× speedups over REPA and standard training, respectively.
  • External Representation Alignment: External alignment has documented structural and training-stage limitations because patch-wise alignment can distort representation distributions and fixed teachers cannot provide later high-frequency details.SARA reports 82.6% versus 63.9% energy in the top 50 singular values, while HASTE finds REPA may erase details after over 400K iterations.
  • Self-Organization of Internal Representation: Internal self-organization improves representations without external supervision through regularization, self-distillation, long residual connections, or diversity losses.Dispersive Loss reaches an FID of 1.97 for SiT-XL/2, DiverseDiT reaches 1.52 at 200 epochs, and SRA reaches 1.58 at 800 epochs.
  • Improving Generative Paradigms with Representations: Representation-driven flow-model methods treat stochasticity or joint transport as design variables, but must manage differing dimensions, distributions, uncertainty, and evolution rates.Rectified Noise reduces ImageNet-1K FID from 10.16 to 9.05 with only 0.39% extra parameters; Flow Matching can schedule semantic generation before image-detail formation.

3.1.2 Image Editing

Image editing methods are organized by the visual component being changed and by how the model is adapted. Content-level editing changes entities or scene context, whereas expression-level editing changes attributes while preserving recognizable semantics.

  • Visual Target Perspective: Image editing modifies an existing image according to user intent while preserving irrelevant source content across insertion, removal, replacement, structure, style, lighting, texture, and composition tasks.The central requirement is to synthesize plausible pixels while deciding which factors to change or preserve.
  • Content-Level Editing: Content-level editing manipulates semantic entities and scene context, with object methods adding, removing, or replacing foreground entities.Adding methods address composition, inpainting, or reference-guided appearance transfer; the core challenge is localizing intended changes and reducing collateral effects.
  • Expression-Level Editing: Expression-level editing keeps main semantic entities recognizable while changing structure, style, lighting, or texture.Structure editing includes pose, shape, layout, and spatial correspondence changes, including point-based local manipulation and region-based geometric control.
  • Model Adaptation Perspective: Existing editing methods are categorized by model adaptation into training-free editing, test-time adaptation, and training-based editing.These strategies trade off editing effectiveness, computational burden, input-specific optimization, and generalizability.
  • Training-Free Editing: Training-free methods freeze the pretrained generator and directly modify inference through perturb-and-denoise, inversion, attention, or feature interventions.Their key challenge is balancing source-image fidelity with sufficient generative flexibility for the desired modification.
  • Test-Time Adaptation: Test-time adaptation customizes a generator or optimizes embeddings and latent or feature representations for an individual image or a few references.These methods capture instance-specific appearance but rely on instance-level adaptation during inference.
  • Training-Based Editing: Training-based editing learns generalizable editing behavior from paired or large-scale multitask data, avoiding input-specific optimization at inference time.Editing coverage, instruction fidelity, and region accuracy remain dependent on training-data diversity.

3.1.3 Trustworthy Generation: Watermarking, Backdoors, and Adversarial Protection

Trustworthy generation organizes watermarking, backdoor injection, and adversarial protection by the functional roles of representations across the generation lifecycle. These roles determine how signals are carried, stored, activated, or disrupted.

  • Representations can function across conditioning, initial latent states, denoising or flow trajectories, generated outputs, and later downstream reuse.The same representation may participate in multiple tasks while serving different functional roles.
  • Trustworthy-generation tasks center on watermark carriers, backdoor injection or storage sites, and downstream targets of adversarial protection.Purification and model mismatch are treated as cross-cutting evaluation challenges rather than standalone representation categories.
  • Watermarking: Watermarking embeds provenance evidence in outputs, decoders, latent states, or initial noise, shifting attention from design to survival under manipulation.Output-level signals face regeneration and editing attacks, while latent carriers can be attacked through public generative components and inversion limitations.
  • Backdoor attacks: Backdoor methods inject malicious associations through triggers or conditioning signals, or store them in model parameters, latent semantics, and denoising trajectories.These approaches are evaluated using attack success, stealth, clean-generation quality, and target specificity, with persistence depending on objective, architecture, and conditioning interface.
  • Adversarial protection: Adversarial protection disrupts representations or learning signals needed for unauthorized personalization, style imitation, editing, or knowledge distillation.Effective representation-level methods must encode task-specific signals and preserve them through generation or downstream optimization.

3.2 Perception-Oriented Applications

Perception-oriented applications use diffusion and flow-based models to support classification and other visual understanding tasks through extracted representations, direct prediction, or synthetic data. The surveyed methods span frozen feature extraction, task-specific generative adaptation, and data augmentation for scarce or imbalanced categories.

  • Diffusion and flow-based models are surveyed across image classification, dense visual prediction, instance-level perception, and annotation-scarce scenarios.
  • Leveraging the Intermediate Representations of the Model: Pretrained diffusion models can serve as frozen feature extractors whose intermediate activations support discriminative downstream tasks without generating samples or changing model architectures.DDAE extracts and pools U-Net activations before training a linear classifier, while related methods extend the approach to segmentation and keypoint detection.
  • Leveraging the Intermediate Representations of the Model: Generative pre-training alone enables diffusion models to learn strongly linearly separable visual representations, and deep features resemble discriminative representations from ResNet and ViT.GDC uses CKA analysis to reveal this representational similarity and improves classification with an attention-based head.
  • Leveraging the Intermediate Representations of the Model: Representation quality depends on network depth and noise level, with Diff-Feat identifying the twelfth DiT layer as its optimal extraction position.
  • Repurposing Generative Models as Direct Predictors: Direct-prediction methods transform image representations into classification or detection outputs through conditional flow or diffusion processes rather than complete image generation.DFM learns time-dependent vector fields from noise toward task targets, while DiDiCM progressively refines noisy or uncertain label states.
  • Synthetic Data for Classification: Synthetic-data approaches address missing poses, backgrounds, styles, and tail categories through sample-level augmentation, distribution-controlled augmentation, and synthetic-data-driven representation learning.MiPO improves minority coverage without additional downstream images or an external reward model, and better minority coverage improves zero-shot classification.

3.2.2 Dense Visual Prediction Task

Dense visual prediction repurposes diffusion and flow-based models for pixel-level semantic or geometric outputs, using extracted features, modified prediction targets, or synthetic annotations. The surveyed methods address the mismatch between generative denoising and deterministic dense prediction through architectural and process changes.

  • Dense prediction requires representations that preserve high-resolution spatial details while encoding global context for per-pixel semantic and geometric labels.
  • Leveraging Intermediate Representations: Diffusion U-Nets provide multilevel feature maps, and methods such as DDPM-Seg, DAAM, and VPD extract or combine these features for dense prediction.DAAM uses cross-attention maps that encode class-to-position bindings, while VPD combines them with multiscale decoder features.
  • Leveraging Intermediate Representations: MDM redesigns diffusion pretraining for perception and shows that representation learning can exist independently of image-generation capacity.This addresses the mismatch between high-frequency texture modeling for realistic synthesis and the requirements of dense prediction.
  • Repurposing Generative Models as Direct Predictors: Other methods directly predict dense labels by replacing image-generation targets with task-specific supervision or by transporting image distributions to mask or depth distributions.FlowDIS applies time-dependent transport to dichotomous segmentation and supports optional text conditioning.
  • Repurposing Generative Models: GenPercept finds that scheduler settings can simplify multistep denoising to single-step deterministic inference without performance loss, while full U-Net fine-tuning outperforms freezing or LoRA-only tuning.
  • Flow-Based Methods: Deterministic flow-matching formulations address the mismatch between stochastic denoising and deterministic dense outputs, enabling reversible mappings or low-step inference for segmentation and depth estimation.DepthFM performs single-step or two-step inference while maintaining competitive accuracy.
  • Synthetic Data for Dense Prediction: Diffusion-synthesized dense annotations can expand training distributions and alleviate annotation bottlenecks.

3.2.3 Instance-Level Perception Task

Instance-level perception requires representations that both distinguish objects within the same category and localize their boundaries. Diffusion and flow-based methods support this through frozen feature extraction, direct prediction, and synthetic instance annotations, but frozen representations remain constrained in complex scenes.

  • Instance-level tasks include detection, instance segmentation, panoptic segmentation, and referring image segmentation, requiring category recognition, instance discrimination, and precise localization.
  • Leveraging the Intermediate Representations of the Model: Frozen text-to-image diffusion features can support open-vocabulary and referring instance segmentation without retraining the encoder.ODISE finds semantic discriminability and spatial localization in internal representations, while LD-ZNet fuses latent and VQGAN features through cross-attention.
  • Leveraging the Intermediate Representations of the Model: Frozen-feature approaches are constrained by how well pretrained models encode instance boundaries and face challenges in complex scenes or heavy object overlap.
  • Repurposing Generative Models as Direct Predictors: Other methods replace image outputs with bounding boxes or instance masks, enabling diffusion or flow-based models to directly regress instance-level predictions.DiffusionDet formulates detection-box generation as denoising from noisy boxes toward ground-truth boxes, while RLFSeg uses deterministic Rectified Flow and SAM refinement.
  • Synthetic Data for Instance-Level Perception: Because instance-level annotation is labor-intensive, diffusion models are also used to synthesize training data with instance annotations.

3.2.4 Annotation-Scarce Task

Annotation-scarce applications use generative models to extract transferable representations, directly predict labels, or synthesize annotated data. The survey identifies conditional data synthesis as the most active of these directions, including approaches for weak supervision, few-shot learning, and long-tailed recognition.

  • High-quality annotations are often prohibitively expensive for semantic segmentation, instance segmentation, and object detection, motivating generative augmentation when labeled data is extremely scarce.
  • Leveraging Intermediate Representations: Representation-distillation methods transfer semantic features from pretrained diffusion models into lightweight backbones for semi-supervised learning, classification, and instance segmentation.DreamTeacher uses MSE and attention-transfer losses for this purpose.
  • Repurposing Generative Models as Direct Predictors: Direct-adaptation methods replace diffusion or flow outputs with prediction labels, including instance masks, and can improve robustness in few-shot instance segmentation.
  • Synthetic Data: Conditional generation can synthesize annotated training data to augment existing datasets, forming the most active technical direction for annotation-scarce scenarios.
  • Synthetic Data: Synthetic-data methods improve segmentation under sparse scribble annotations and support few-shot classification through semantic image-to-image augmentation and semantically diverse generation.
  • Synthetic Data: Other methods target long-tailed recognition by increasing diversity, combining text-to-image and image-to-image synthesis, and improving minority-category instance segmentation.
  • Synthetic Data: Recent approaches reduce synthetic-data failure rates through prompt rethinking, self-correction, validation, and filtering for long-tailed instance segmentation and open-vocabulary detection.

3.3 Generalist and Unified Applications

The survey organizes clustering applications into diffusion-based generative and assignment methods, flow-matching representation learning, and clustering-guided transport, alongside broader world-model applications.

  • Clustering Analysis: Clustering quality depends on embedding spaces that preserve semantic information while removing task-irrelevant variation.
  • Clustering Analysis: Diffusion clustering methods combine cluster discovery with generation, predict assignments, or extract representations from intermediate features.
  • Clustering Analysis: CLUDI represents uncertain samples through stochastic assignment predictions, improving clustering stability at the cost of repeated reverse-diffusion computation.
  • Clustering Analysis: DiFiC refines compact textual conditions with neighborhood similarity and diffusion-based constraints to emphasize object-level semantics in fine-grained clustering.
  • Clustering Analysis: Flow-based approaches introduce cluster structure before or during transport, supporting clustering, disentanglement, downstream prediction, and sub-mode-conditioned generation.
  • World Models: World-model applications extend representation learning toward predictive representations across video, 3D/4D, robotics, and embodied systems, but face inference cost, long-horizon drift, and physical-adherence limits.

3D/4D World Models

The survey describes unified multimodal models that connect understanding and generation through diffusion or flow components, including cascaded, end-to-end, and hybrid designs.

  • 3D/4D World Models: 3D/4D world models use RGB-D, occupancy, 3D Gaussian splatting, and point clouds to improve geometric and temporal consistency over 2D video.
  • 3D/4D World Models: OccSora generates trajectory-conditioned occupancy videos with a 4D scene tokenizer and diffusion transformer, while HY-World 2.0 constructs interactive 3D worlds through reconstruction and rendering.
  • Robotics and Embodied World Models: Embodied applications use world models for robot-video simulation, policy evaluation, VLA post-training, motion priors, and latent visual subgoals.
  • Unified Multimodal Models: Unified multimodal architectures combine modality-specific encoders, fusion backbones, and generative components, with the diffusion or flow insertion point distinguishing model designs.
  • Unified Multimodal Models: The surveyed unified models include cascaded decoders, end-to-end continuous or discrete diffusion, and hybrid architectures combining autoregressive text with diffusion or flow generation.

4 Discussion and Future Directions

The survey identifies open challenges in balancing diffusion and flow research, evaluating representations intrinsically, extending them to intelligent tasks, and reconciling understanding with generation.

  • Research currently favors diffusion models, leaving the efficiency and controllability advantages of flow-based representation learning relatively underexplored.
  • Representation quality is usually assessed through downstream metrics rather than unified measures of linear separability, disentanglement, and interpretability.
  • Future representations must encode semantic abstractions alongside causal structures, physical laws, and temporal dynamics for higher-level intelligent tasks.
  • Unified multimodal models must reconcile semantic abstraction with detail fidelity within shared-parameter architectures.
  • The survey presents bridging generation and understanding as a route toward generative models serving general-purpose intelligent systems.

5 Declarations

The paper reports no competing financial or non-financial interests and describes collaborative authorship of writing, editing, and project administration.

  • The authors declare no competing financial or non-financial interests.
  • All authors contributed to writing and editing the review.
  • Hongyuan Zhang handled project administration.
Loading 2608.24068v1…