Representation Learning in Diffusion and Flow-based Model: An Application Aspect

• Diffusion models dominate generative modeling across image, video, audio, and molecular-design domains, while flow-based models offer efficient and flexible deterministic transport.The two model families are increasingly complementary or alternative generative paradigms.Source: §1 · p. 1
• Their bidirectional relationship with representation learning supports both more realistic generation and downstream classification, segmentation, and detection.Representations are distributed across layers, timesteps, and attention heads rather than explicitly extracted, so extraction methods vary by task.Source: §1 · p. 2
• Because prior reviews lacked a unified treatment—one focused only on diffusion models [1] and others treated representation learning peripherally [2]—the survey organizes work from the early 2020s to early 2026 into a comprehensive taxonomy.Its selection emphasizes novel representation-learning mechanisms, clear application value, or new paradigms.Source: §1 · p. 2
• The proposed three-tier framework progresses from improving generative capabilities, to extracting representations for perception, and finally toward general-purpose applications and unified frameworks.The survey also identifies key challenges, highlights the models’ potential for representation learning, and suggests future research directions.Source: §1 · p. 2

2.1 Preliminaries

• Generative models learn to transform a source distribution into the data distribution, using progressive denoising for diffusion and deterministic transport for flow-based methods.These formulations provide the conceptual foundation for the subsequent model and representation-learning discussions.Source: §2.1 · p. 2

2.1.1 Diffusion Models

• Diffusion models transform data into Gaussian noise through forward noising and generate samples by reversing the denoising process [3].Their continuous-time formulation uses SDEs, whose reverse-time dynamics depend on a learned score function [4].Source: §2.1.1 · p. 3

2.1.2 Flow Matching Models

• Flow-matching models learn a deterministic transport process that maps a source distribution, typically Gaussian, to the data distribution [8].A time-dependent vector field defines an ODE, and Conditional Flow Matching enables training without direct access to the marginal probability path or vector field.Source: §2.1.2 · pp. 3–4

2.1.3 Rectified Flow

• Rectified Flow is a flow-matching variant that learns a linear path from a source distribution to the data distribution [9].The path is parameterized using samples from the source and target distributions, with a neural vector field trained over time.Source: §2.1.3 · p. 4

2.2 Network Architecture

• Backbone architectures determine how generative-model representations are structured and their potential for feature extraction.The section surveys architectures commonly used in diffusion and flow-based models.Source: §2.2 · p. 4

2.2.1 Diffusion Models

• Diffusion models commonly use U-Nets whose encoder-decoder hierarchy and skip connections provide multilevel features from texture to semantic abstraction [10].Latent diffusion reduces computation through VAE latents, adds text-conditioning cross-attention whose maps encode word-to-pixel correspondences [7, 11], and DiT uses scalable ViT-style Transformer blocks [12].Source: §2.2.1 · p. 4

2.2.2 Flow-based Models

• Flow-based models similarly progress from U-Net vector-field parameterizations to Transformer backbones, with SiT [268] applying DiT to flow matching.SiT learns linear interpolation paths from Gaussian noise to data and achieves comparable or better scalability and performance than diffusion models on large-scale ImageNet generation.Source: §2.2.2 · pp. 4–6

Overview

• The survey organizes representative applications into three tiers: generation-centric tasks, perception-oriented tasks, and general-purpose unified applications.These tiers progress from improving generation to extracting representations for perception and transcending the generation–discrimination boundary.Source: §3 · p. 6

Generation-centric applications

• Generation-centric applications use representation learning to improve training efficiency, generation quality, and controllability across image and video generation, image editing, and trustworthy generation.Source: §3 · p. 6

Perception-oriented applications

• Perception-oriented applications extract transferable representations from pretrained generative models for image classification, dense visual prediction, and instance-level perception, especially when annotations are scarce.Source: §3 · p. 6

General-purpose and unified applications

• General-purpose and unified applications extend generative models to clustering analysis, world model construction, and unified multimodal modeling through architectural paradigms that integrate generation and discrimination.Source: §3 · p. 6

Overview

• Figure 1 summarizes the representative works surveyed in this section and their taxonomic relationships.Source: §3 · p. 6

Diffusion Model

• REPA-E[15] reaches FID 1.12 while achieving 17× and 45× speedups over REPA and standard training by back-propagating its alignment loss to the VAE.REPA aligns DiT[12] representations with DINOv2[269], accelerating SiT training by over 17 times and reaching FID 1.42; U-REPA[14] reaches FID 1.41 at 400 epochs.Source: Diffusion Model · p. 6
• Representation-enhanced diffusion extends across modalities and paradigms: SoftREPA[16] aligns text-image features, VideoREPA[17] improves VideoPhy physical commonsense by 24.1%, and RCG[26] restructures unconditional generation.VideoREPA distills intra-frame spatial and inter-frame temporal relations, while RCG first generates representations and then images.Source: Diffusion Model · pp. 7–8
• SARA[18] finds that REPA concentrates 82.6% of representation energy in its top 50 singular values versus 63.9% for DINOv2, while HASTE[19] shows alignment can erase later-stage detail.SARA adds structural and adversarial alignment, whereas HASTE terminates alignment when the fixed-capacity teacher becomes a bottleneck and adds attention alignment.Source: Diffusion Model · p. 7
• Under zero external dependency, Dispersive Loss[20] reaches FID 1.97, DiverseDiT[21] reaches 1.52 at 200 epochs, and SRA[22] reaches 1.58 at 800 epochs versus REPA’s 1.42.These methods respectively separate sample features, promote block-wise diversity, and self-distill late-layer teacher representations; Dispersive Loss also applies to flow-based models.Source: Diffusion Model · pp. 7–8
• Latent-space redesign improves generation by strengthening semantic and fine-detail representations: VA-VAE[23] reaches FID 1.35, while RAE[24] reaches FID 1.13 with guidance and 47× training acceleration.VA-VAE improves the existing VAE with vision-foundation-model guidance; RAE replaces its encoder with DINOv2, and SVG[25] combines DINOv3 with a residual encoder for fine details.Source: Diffusion Model · p. 8
• FID 2.15 on ImageNet 256×256 reduced the previous SOTA of 5.91 by 64%, as RCG generates representations before conditioning an image generator for unconditional synthesis.The lower-dimensional representation space retains high-level semantics, making unconditional generation easier to learn and improving multiple generators, including LDM, ADM, DiT, and MAGE.Source: Diffusion Model · p. 8

Flow-based Model

• Flow-based representation learning uses heterogeneous token corruption to learn cross-region relations, jointly transports visual and semantic states, and aligns transport paths with pretrained representation geometry.Self-Flow[27] assigns different time steps to token groups, while REG[28] generates image and semantic tokens together. DINO-SAE combines a spherical autoencoder with Riemannian Flow Matching to follow representation-space geometry.Source: Flow-based Model · p. 9
• FID on ImageNet-1K falls from 10.16 to 9.05 with only 0.39% extra parameters when Rectified Noise[32] injects representation-conditioned π-noise into a pretrained rectified-flow velocity field.The method treats stochasticity as a beneficial driving force by using a lightweight learnable noise generator conditioned on representations.Source: Flow-based Model · p. 10

3.1.2 Image Editing

• Image editing synthesizes plausible pixels while deciding what to change or preserve across tasks including object manipulation, background replacement, structure editing, style transfer, and multi-reference composition.The survey organizes methods by visual targets and model adaptation.Source: §3.1.2 · p. 10

Visual target perspective

• Visual targets divide into content-level editing of entities and scene context, and expression-level editing of structure, style, lighting, or texture.Content-level editing must localize semantic changes and limit collateral effects, whereas expression-level editing preserves recognizable entities while altering arrangement or appearance.Source: Visual target perspective · pp. 10, 12

Model adaptation perspective

• Training-free editing freezes the pretrained model and alters inference through noise perturbation, inversion, attention or feature intervention, semantic directions, and increasingly flow-based trajectories.It offers flexibility and low training cost but depends heavily on inversion strength, feature selection, and guidance scale.Source: Model adaptation perspective · p. 12
• Test-time adaptation improves input fit through instance-specific parameter, embedding, or drag optimization, while training-based editing learns generalizable behavior offline using full, parameter-efficient, dedicated, unified, or reasoning-enhanced architectures.Test-time optimization slows inference and can restrict large semantic changes; training-based methods depend on data diversity and alignment while preserving identity, structure, and unmodified regions.Source: Model adaptation perspective · pp. 12–13

Scope and task-specific representation taxonomy

• Representations serve as watermark carriers, backdoor injection or storage sites, and intervention targets for adversarial protection across the generation lifecycle.The taxonomy distinguishes task, representation, intervention stage, mechanism, evaluation, diffusion–flow differences, and limitations.Source: §3.1.3 · p. 13, Scope and task-specific representation… · p. 13, Figure 2 · p. 14

Task I: Watermarking and provenance

• Watermarking evolves from decoder-integrated attribution to output-level marking and latent or initial-state encoding, but robustness remains threat-model-dependent under regeneration, inversion, forgery, and model mismatch.Output carriers face regeneration and editing, whereas latent carriers depend on inversion secrecy, public component access, and recovery assumptions.Source: Task I: Watermarking and provenance · pp. 15–16

Task II: Backdoor attacks on generative models

• Backdoors range from trigger- or conditioning-level poisoning to distributed modifications of latent semantics and denoising dynamics, with generated samples able to propagate malicious associations into downstream models.Propagation evidence remains diffusion-centered, leaving transfer to flow-based generators uncertain because velocity prediction and continuous-time trajectories may alter persistence and detectability.Source: Task II: Backdoor attacks on… · pp. 16–17

Task III: Adversarial protection against unauthorized reuse

• Adversarial protection disrupts semantic features, training or denoising dynamics, editing representations, attention pathways, or trajectories to hinder unauthorized personalization, imitation, distillation, and editing.Methods increasingly use surrogate-aware, bottleneck-aware, compression-aware, and system-level defenses, but purification, adaptive preprocessing, pathway bypasses, and model mismatch remain limitations.Source: Task III: Adversarial protection… · pp. 17–19

Representation-level synthesis.

• Across tasks, useful representations must encode and preserve task-specific signals through generation or downstream optimization, yet research remains more mature for diffusion than for flow-based models.A key next step is developing methods tailored to flow architectures rather than directly transferring diffusion-oriented designs.Source: Representation-level synthesis · p. 19

3.2 Perception-Oriented Applications

• Diffusion and flow-based models support perception through intermediate representations, direct task prediction, synthetic-data generation, and annotation-scarce learning, summarized in Tables 3 and 4.These approaches span image classification, dense visual prediction, instance-level perception, and annotation-scarce scenarios.Source: §3.2 · p. 19

3.2.1 Image Classification Task

• Image classification is a direct test of generative representations, encompassing feature extraction, classifier adaptation, synthetic-data strategies, direct generative prediction, and implicit discrimination.Most methods follow three mainstream adaptation paradigms, alongside the independent Denoising-Evidence Classification route.Source: §3.2.1 · p. 19

Leveraging the Intermediate Representations of the Model

• Frozen diffusion features become linearly separable representations without contrastive or additional discriminative objectives, while later methods fuse layers and timesteps or dynamically select distillation timesteps.DDAE uses noised inputs and pooled U-Net activations with a linear classifier; DiffFormer, DiffFeed, and RepFusion extend feature use across tasks and architectures.Source: Leveraging the Intermediate… · pp. 20–22
• Representation quality depends on extraction location and noise level, with Diff-Feat identifying DiT’s twelfth layer as optimal and diffusion features resembling ResNet and ViT representations.These findings support diffusion models as general-purpose representation learners across classification, segmentation, detection, and keypoint tasks.Source: Leveraging the Intermediate… · p. 21

Repurposing Generative Models as Direct Predictors

• Direct predictors recast classification as conditional transport or discrete label diffusion, enabling local flow aggregation or iterative refinement under corrupted inputs, limited data, or uncertainty.DFM transports noise toward class embeddings through local Flow Matching predictors, while DiDiCM progressively denoises class probabilities or discrete labels.Source: Repurposing Generative Models as… · p. 22

Synthetic Data for Classification

• Synthetic classification data progresses from class-preserving augmentation toward distribution-controlled generation and broad representation pretraining, targeting missing modes while limiting semantic drift and label noise.StableRep, SynCLR, and Free-ATM use generated diversity for transferable representations, shifting the bottleneck toward prompt design, filtering, and distribution coverage.Source: Synthetic Data for Classification · pp. 22–23

Denoising-Evidence Classification with Diffusion Models

• Denoising-evidence classification selects labels with the lowest conditional denoising error, supports zero-shot classification without an added head, but requires repeated evaluations for candidate classes.MDC further weights foreground errors using attention-derived masks and structural distances, while MiPO improves minority-concept zero-shot performance through parameter-efficient preference optimization [159].Source: Denoising-Evidence Classification with… · pp. 23–24

3.2.2 Dense Visual Prediction Task

• Dense visual prediction assigns semantic or geometric labels per pixel, requiring high-resolution spatial details for localization and global context for reasoning.Source: §3.2.2 · p. 24

Leveraging the Intermediate Representations of the Model

• Diffusion representations support dense prediction through multi-scale U-Net features, cross-attention guidance, or task-aligned pretraining with dynamic masking and SSIM loss.DDPM-Seg extracts and concatenates decoder activations; DAAM uses cross-attention without training; VPD and ECoDepth add semantic guidance, while MDM redesigns pretraining for structural representations independent of image generation.Source: Leveraging the Intermediate… · pp. 24–25

Repurposing Generative Models as Direct Predictors

• Generative models can directly predict dense labels, while flow matching replaces stochastic denoising with deterministic transport, improving efficiency, controllability, and cross-task generalization.DDP, Marigold, DMP, and GenPercept repurpose diffusion models for dense outputs; GenPercept identifies full U-Net fine-tuning as important and extends the framework to five tasks, while FlowDIS surpasses prior SOTA methods on DIS benchmarks.Source: Repurposing Generative Models as… · pp. 25–26

Synthetic Data for Dense Prediction

• Diffusion-generated data expands dense prediction training distributions by providing diverse, densely annotated examples that improve robustness and alleviate annotation bottlenecks.DatasetDiffusion and FreeMask synthesize segmentation data, while DiffusionForDepth creates challenging scenes that preserve 3D labels for robust depth-model training.Source: Synthetic Data for Dense Prediction · p. 26

3.2.3 Instance-Level Perception Task

• Instance-level tasks include detection, instance segmentation, panoptic segmentation, and referring image segmentation, demanding stronger instance discrimination and localization than dense prediction.The section reviews diffusion- and flow-based methods under three technical paradigms adapted from dense prediction settings.Source: §3.2.3 · pp. 26–27

Leveraging the Intermediate Representations of the Model

• Frozen diffusion representations enable open-vocabulary detection and segmentation, but performance depends on boundary encoding and degrades in complex or heavily overlapping scenes.ODISE[172] extracts Stable Diffusion U-Net features in one pass for open-vocabulary panoptic segmentation, detection, and open-world instance segmentation; LD-ZNet[182] fuses latent and VQGAN features for referring expression segmentation.Source: Leveraging the Intermediate… · p. 27

Repurposing Generative Models as Direct Predictors

• Diffusion and flow models can directly generate bounding boxes or instance masks, extending generative prediction from dense outputs to instance-level tasks.DiffusionDet[183] denoises noisy boxes toward ground-truth boxes, DiffusionInst[184] generates instance mask vectors, OC-DiT[189] performs zero-shot mask generation and refinement, and RLFSeg[191] uses Rectified Flow with SAM[288] for referring expression segmentation.Source: Repurposing Generative Models as… · p. 27

Synthetic Data for Instance-Level Perception

• Diffusion models can synthesize large-scale instance-level training data, reducing reliance on costly manual bounding-box and pixel-mask annotation.DALL-E Detection[185], GroundedDiffusion[190], DiffusionEngine[187], MosaicFusion[186], and ReCon[188] generate annotated detection or segmentation data using composition, grounding, adapters, attention aggregation, or in-process rectification.Source: Synthetic Data for Instance-Level… · p. 28

3.2.4 Annotation-Scarce Task

• Annotation-scarce methods address costly labels in segmentation and detection by leveraging generative capabilities to improve few-shot, long-tailed, weakly supervised, and open-vocabulary performance.The section organizes these works under the same three paradigms introduced earlier.Source: §3.2.4 · p. 28

Leveraging the Intermediate Representations of the Model

• DreamTeacher distills intermediate diffusion features into lightweight backbones such as ResNet[289], supporting semi-supervised learning, classification, instance segmentation, and label-efficient semantic segmentation.It uses MSE and attention-transfer losses so small networks inherit visual priors from large-scale diffusion models under scarce annotations.Source: Leveraging the Intermediate… · p. 28

Repurposing Generative Models as Direct Predictors

• MaskDiff directly generates instance masks with stronger few-shot robustness than traditional discriminative approaches, while iFSS-Diff enables incremental segmentation of new classes through class-specific embeddings and mask generation.Both methods repurpose generative inference as prediction when annotations are scarce.Source: Repurposing Generative Models as… · p. 28

Synthetic Data for Representation Learning

• Synthetic annotated-data generation is the most active direction, augmenting datasets for weakly supervised segmentation, few-shot classification, open-vocabulary panoptic segmentation, and semantically diverse few-shot learning.ScribbleGen [196] targets sparse scribbles, DAFusion performs semantic image-to-image augmentation, DIAGen emphasizes diversity, and DreamMask combines LLM-generated categories, diffusion synthesis, CLIP filtering, and SAM uncertainty filtering.Source: Synthetic Data for Representation… · p. 29
• DiverGen and TMI [199] alleviate long-tailed data scarcity through multi-level diversity enhancement or complementary text-to-image and image-to-image synthesis that preserves scene coherence.Both approaches use generated-data post-processing or high-confidence instance insertion to improve long-tailed instance segmentation.Source: Synthetic Data for Representation… · p. 29
• InstaDA and Gen-n-Val reduce synthetic-data failure rates through prompt rethinking, self-correction, foreground-alpha generation, and VLLM-based validation and filtering.Their applications include long-tailed instance segmentation and open-vocabulary detection.Source: Synthetic Data for Representation… · p. 29

3.3.1 Clustering Analysis

• Clustering applications use diffusion and flow models to integrate semantic representation learning, assignment, latent-group discovery, and generation rather than relying only on deterministic embeddings.Methods span generative clustering, diffusion-based assignment and representation clustering, Flow Matching embedding learning, and clustering-guided transport.Source: §3.3.1 · pp. 29–30
• Diffusion-based clustering improves uncertainty handling and fine-grained semantics through stochastic assignment prediction, teacher–student consistency, and compact textual conditions refined by neighborhood and semantic constraints.CLUDI improves clustering stability but incurs repeated reverse-diffusion cost, while DiFiC emphasizes object-level distinctions obscured by background, pose, or texture.Source: §3.3.1 · p. 30
• Flow Matching methods embed cluster structure into transport through structured latent variables, bidirectional data–embedding flows, incomplete-view alignment, or sub-mode conditioning for more consistent generation.SCFM supports clustering, disentanglement, and prediction; paired flow paths preserve cluster identity across missing views; SubFlow reduces mode averaging and improves generation diversity.Source: §3.3.1 · p. 31

3.3.2 World Model

• World models compress observations into predictive latent states and learn environment dynamics for future prediction, planning, decision making, and interactive control across video, 3D/4D, and embodied representations.The review organizes these applications into video world models, 3D/4D world models, and robotics and embodied world models.Source: §3.3.2 · p. 32

Video World Models

• Video world models generate future frames or latent representations conditioned on actions, trajectories, language, or goals, combining pixel-space diffusion, foundation-model embeddings, and flow matching for efficient world prediction.Pixel-space methods offer visual fidelity but suffer high inference cost, long-horizon error accumulation, and insufficient physical adherence; latent methods improve efficiency, while useful generative manifolds require reconstruction fidelity, spatial coherence, continuity, and global semantics.Source: Video World Models · pp. 32–33

3D/4D World Models

• 3D/4D world models use RGB-D, occupancy, 3D Gaussian splatting, and point clouds to improve geometric and temporal consistency for verifiable future-state prediction in driving and robotics.Representative systems include UniScene, DrivingSphere [303], X-Scene, COME [305], R2Flow, LidarDM [307], HoloDrive [308], and LOGen.Source: 3D/4D World Models · p. 33
• 4D occupancy generation and interactive 3D reconstruction support rendering- and physics-engine interfaces for safety-critical applications, but require more data, annotation, training, and inference cost.OccSora uses trajectory-conditioned occupancy videos, while HY-World 2.0 combines panorama generation, planning, view generation, reconstruction, and 3DGS rendering [217] [310].Source: 3D/4D World Models · p. 33

Robotics and Embodied World Models

• World models in embodied AI now support action-conditioned simulation, scalable policy evaluation, VLA representation priors, latent subgoals, and motion-aware control rather than only future video generation.IRASim simulates robot videos, WorldEval and dWorldEval evaluate policies [219] [311], World2Act uses video-dynamics latents [312], LaMP supplies 3D scene-flow priors, and LaWAM predicts latent visual subgoals [313].Source: Robotics and Embodied World Models · pp. 33–34

3.3.3 Unified Model

• Unified multimodal models combine modality-specific encoders, a cross-modal fusion backbone, and modality-specific decoders, with diffusion or flow placement distinguishing their architectures.The survey focuses on models whose generation pathway includes diffusion or flow and groups them into cascaded, end-to-end, and fused autoregressive–diffusion/flow paradigms.Source: §3.3.3 · p. 34

LLM/VLM-Driven Diffusion/Flow Decoders.

• Cascaded models retain LLM/VLM understanding backbones while delegating pixel synthesis to diffusion or flow decoders, evolving from CLIP–Stable Diffusion systems toward decoupled encoders, flow matching, FLUX-family decoders, and DiT designs.Examples progress from DreamLLM and SEED [223] [225] to Emu2, PUMA [228], Unifluid [229], BLIP3-o [230], OmniGen2 [231], Nexus-Gen [258], X-Omni [259], Qwen-Image [234], and UniPic-2.0 [235].Source: LLM/VLM-Driven Diffusion/Flow Decoders · pp. 34–35

End-to-End Diffusion/Flow Models

• End-to-end models jointly represent text and vision within continuous or discrete diffusion/flow processes, progressing from shared denoising backbones toward shared representation spaces, token spaces, and generation mechanisms.Continuous methods include VersatileDiffusion, UniDiffuser, and UniModel [242]; discrete systems include LLaDA [319], LLaDA-V, Dimple [328], MMaDA [245], Lumina-DiMOO [246], ViewMask-1-to-3, FUDOKI [260], and NExT-OMNI.Source: End-to-End Diffusion/Flow Models · pp. 35–36

Hybrid Autoregressive and Diffusion/Flow Modeling

• Hybrid models combine autoregressive text prediction with diffusion or flow image generation in one Transformer, with recent work improving efficiency, modality decoupling, and unified image–video representations.Transfusion, MonoFormer [253], JanusFlow, BAGEL [263], Mogao [264], LightFusion [265], and Show-o instantiate continuous, flow-based, and discrete variants; unified versus decoupled spaces trade semantic abstraction against visual detail preservation.Source: Hybrid Autoregressive and… · p. 36

Model diversity

• Flow-based representation learning remains underexplored despite potential advantages in inference efficiency and controllability, motivating use of temporal asymmetry and vector field geometry.Current research heavily favors diffusion models, while flow-based models’ distinctive characteristics remain insufficiently incorporated into representation learning.Source: §4 · pp. 36–37

Evaluation

• A standardized benchmark is needed to evaluate intrinsic representation properties, including linear separability, disentanglement, and interpretability, beyond indirect downstream-task metrics.Such a framework could support fairer comparisons across methods and guide model design.Source: §4 · p. 37

Higher-level intelligence

• Representations must progress from perception toward decision-making, planning, and interaction while encoding causal structures, physical laws, and temporal dynamics for embodied intelligence.Semantic abstraction alone is insufficient for general-purpose embodied intelligence.Source: §4 · p. 37

Unified multimodal models

• Unified multimodal models must reconcile understanding and generation within shared parameters, especially the tension between semantic abstraction and detail fidelity.As diffusion and flow-based models become widespread in unified multimodal frameworks, representation learning becomes central to integrating understanding with generation.Source: §4 · p. 37

5.2 Conflicts of Interest

• The authors declare no competing financial or non-financial interests.Source: §5.2 · p. 37

5.5 Authors’ Contributions

• All authors contributed to writing and editing the review, while Hongyuan Zhang handled project administration.Source: §5.5 · p. 37