PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis

Hila Chefer, Patrick Esser, Dominik Lorenz, Dustin Podell, Vikash Raja, Vinh Tong, Antonio Torralba, Robin Rombach

arXiv:2603.06507v1cs.CV

TL;DR

Existing generative-model approaches rely on external representations that show scaling and cross-modal limitations, motivating a method that learns semantics within generation. Self-Flow adds self-supervised representation learning to flow matching through heterogeneous token noise, and it improves generation across image, video, audio, and multimodal settings. The approach also incurs extra training overhead from an additional teacher forward pass.

  • Problem

    External alignment has unexpected scaling behavior, can harm video and audio generation, and makes encoder effectiveness difficult to predict.

  • Method

    Self-Flow integrates self-supervised representation learning into flow matching using Dual-Timestep Scheduling and predictions from a cleaner-input representation.

  • Results

    Self-Flow outperforms leading external alignment methods across image, video, audio, and multimodal generation, with improvements reported across several autoencoder choices.

  • Takeaways & Limitations

    Joint optimization of generation and representations offers a robust, scalable, and general path for multimodal generative modeling.

  • Takeaways & Limitations

    The additional teacher forward pass increases training overhead, although the paper reports that accelerated convergence and improved performance justify this cost.

Abstract

from arXiv · show

Strong semantic representations improve the convergence and generation quality of diffusion and flow models. Existing approaches largely rely on external models, which require separate training, operate on misaligned objectives, and exhibit unexpected scaling behavior. We argue that this dependence arises from the model's training objective, which poses a denoising task with little incentive to learn semantic representations. We introduce Self-Flow: a self-supervised flow matching paradigm that integrates representation learning within the generative framework. Our key mechanism, Dual-Timestep Scheduling, applies heterogeneous noise levels across tokens, creating an information asymmetry that forces the model to infer missing information from corrupted inputs. This drives learning strong representations alongside generative capabilities without external supervision. Our method generalizes across modalities and enables multi-modal training while following expected scaling laws, achieving superior image, video, and audio generation.

1. Introduction

External representation alignment improves generative models but has scaling, modality, and encoder-selection limitations. Self-Flow integrates self-supervised representation learning into flow matching through heterogeneous token corruption, supporting multimodal generation and broad improvements.

  • Motivation: External alignment improves generative models but can violate expected scaling laws and provide diminished or negative returns from stronger encoders.Scaling the generative model also fails to yield proportional improvements under external alignment.
  • Motivation: External alignment can harm video and audio generation, limiting its suitability for multimodal models.The paper reports that alignment with most external encoders harms performance in these modalities.
  • Motivation: Encoder effectiveness is difficult to predict because SigLIP 2 performs worse than DINOv2 for text-to-image generation despite seemingly suitable training properties.
  • Approach: Self-Flow integrates self-supervised learning directly into flow matching to strengthen representations beyond those learned by the generative objective alone.
  • Approach: Dual-Timestep Scheduling applies distinct noise levels to token subsets, making the model predict cleaner-input representations from heterogeneously corrupted inputs alongside flow reconstruction.Two forward passes provide the mixed noisy input and the cleaner reference, enabling semantic feature prediction within the generative framework.
  • Results: Self-Flow extends to joint image, video, and audio training and improves structural coherence, text rendering, and video temporal consistency across multiple autoencoders.The method operates on internal representations without external encoders and shows consistent improvements across SD, FLUX.2, Wan2.2, Songbloom, and representation autoencoders.

2. Related Work

Representation-learning methods for generative models include external encoder alignment and approaches that avoid external models. Self-Flow combines self-supervision with flow matching and reports surpassing external alignment without external models.

  • Representation Learning: Representation learning uses diverse pretraining objectives, including contrastive image methods and vision-language alignment for zero-shot transfer.
  • Representation Alignment for Generation: Generative-model alignment methods commonly match diffusion or flow features to external pretrained encoders to accelerate training and improve generation quality.Extensions include domain-specific settings such as physics in video and geometry.
  • Representation Alignment for Generation: Methods without external models either add self-supervised objectives that alter training dynamics or preserve the generative framework without explicit self-supervision.Some explicit-objective approaches require an additional pure diffusion fine-tuning stage to close the train-inference gap.
  • Self-Flow: Self-Flow integrates self-supervised learning directly into flow matching and surpasses external alignment methods without requiring external models.

3. Method

Self-Flow integrates representation learning into flow matching by using heterogeneous token noise to create information asymmetry, then reconstructing cleaner teacher features alongside denoising. This design avoids external encoders while improving generation and representation quality across latent-space settings.

  • Flow Matching Preliminaries: Flow matching transports noise to data along continuous-time paths, with a neural network trained to predict the path velocity.Inference solves the learned ODE backward from pure noise to clean data.
  • Motivation: External representation alignment can degrade as encoder strength increases and may behave inconsistently across data distributions, model scales, and modalities.DINOv2-B achieves better FID than stronger DINO variants in the cited scaling experiment.
  • Dual-Timestep Scheduling: Dual-Timestep Scheduling assigns two noise levels to token subsets, encouraging the model to infer heavily corrupted tokens from cleaner contextual tokens while preserving the marginal timestep distribution.It balances the global-relation benefits of heterogeneous noise against the train-inference gap caused by fully heterogeneous or fully masked inputs.
  • Results: Dual-Timestep Scheduling alone slightly improves generation without an explicit self-supervised objective, while the full method improves training and generation with representation autoencoders.The cleaner token context encourages global relations, and the RAE result demonstrates compatibility with semantic latent spaces.
  • Self-Supervised Representation Learning: The student receives heterogeneously noised inputs, while an EMA teacher observes inputs noised at the lower timestep and supplies features for reconstruction.The representation objective is combined with the standard flow loss, with student and teacher layers selected so l < k.

4. Experiments

Self-Flow is evaluated across image, video, audio, scaling, and multimodal generation, consistently outperforming alignment baselines while improving representation quality and complex joint prediction. Ablations identify the self-supervised representation loss and Dual-Timestep Scheduling as central to generation quality.

  • Single-Modality Results: Self-Flow outperforms REPA on ImageNet without external representations and improves RAE results from FID 3.24 to 2.95.
  • Single-Modality Results: On text-to-image generation, Self-Flow achieves the best FID at 3.61, ahead of REPA at 3.92, SigLIP 2 at 3.97, and SRA at 3.70.
  • Single-Modality Results: Self-Flow achieves the best video FVD of 47.81 and per-frame FID of 8.92, while video-specific external alignment methods can harm performance relative to vanilla flow matching.
  • Single-Modality Results: Self-Flow achieves the best audio FAD scores across CLAP variants, whereas MERT alignment provides no benefit over vanilla flow matching.
  • Scaling Behavior: The performance gap over REPA widens with model scale, and a 625M Self-Flow model outperforms a 1B-parameter REPA model while following expected scaling behavior.
  • Multi-Modal Results: Self-Flow improves all three modalities simultaneously under extreme modality weightings and outperforms vanilla flow matching on complex multi-object and sequential robotics tasks.
  • Qualitative Results: Qualitative results show improved text rendering, structural accuracy, visual fidelity, prompt adherence, and temporal coherence, including with a roughly 625M-parameter video model trained on 6M samples.
  • Ablations: Removing the representation loss causes degradation of over 4 points, while altering Dual-Timestep Scheduling substantially harms results and replacing cosine similarity with ℓ1 causes later-training FID increases.

5. Limitations and Future Work

The paper identifies timestep scheduling as a tunable factor and outlines experiments spanning datasets, autoencoders, and modalities. It also notes training overhead from the teacher forward pass and a future direction toward world models.

  • Limitations: The additional forward pass through the teacher increases training overhead, but accelerated convergence and improved performance justify this cost.The paper identifies this as a trade-off of the approach.
  • Limitations: The noise scheduler p(t) requires tuning because it determines masking behavior.A uniform scheduler outperformed a logit-normal scheduler with shift α = 1.78 in the cited text-to-image setup.
  • Future work: Future work targets world models that combine generative-model scalability and perceptual grounding with semantic abstraction for planning and understanding.The paper presents this as a direction motivated by bridging representation learning and generative modeling.
  • Experimental scope: Experiments use ImageNet-1K, a 200M-image research dataset, a 6M-video dataset, and 1M ten-second audio samples.The research image dataset supports text-to-image and multimodal experiments, while the video and audio datasets support modality-specific evaluations.
  • Experimental scope: The method is evaluated with modality-specific autoencoders, including SD-VAE, FLUX.2 AE, WAN2.2 AE, and Songbloom AE.The setup follows latent diffusion and flow-based generation in modality-specific latent spaces.
  • Timestep scheduling: Timestep distributions use uniform, shifted uniform, logit-normal, or plateau-logit-normal variants for training and evaluation.The timeshift parameter α is called trainshift for training distributions and sampleshift for evaluation grids.

A.3. Architecture

The architecture extends timestep conditioning to token-specific values and uses modality-appropriate evaluation and sampling configurations. Comparisons include vanilla flow matching, external alignment, and non-external alignment baselines.

  • Architecture: Dual-Timestep Scheduling extends timestep conditioning from one scalar t ∈ R^1 to a vector t ∈ R^N, assigning each token its corresponding noising timestep.This enables heterogeneous token-level noise conditioning.
  • Architecture: ImageNet models use a roughly 675M-parameter SiT-XL backbone, while other experiments use a roughly 625M-parameter FLUX-based design.The FLUX-based configuration includes shared modulation layers, SwiGLU, and 3D RoPE.
  • Architecture: The method and REPA variants add lightweight projection layers and use EMA weights with decay factor 0.9999 for the teacher.The projection layers add around 10M parameters.
  • Evaluation: Evaluation uses 250 SDE steps for SD-VAE ImageNet models, 50 steps for RAE models, and 50 ODE steps for non-ImageNet experiments.Sampling shifts are adapted to the autoencoder for non-ImageNet experiments.
  • Baselines: The comparisons cover vanilla flow matching, REPA, SRA, and LayerSync across methods with and without external encoders.SRA and LayerSync are used for selecting the leading non-external baseline.
  • Baselines: SRA outperforms LayerSync after 400K training steps in the text-to-image baseline-selection experiment.The caption attributes this result to SRA’s use of an EMA teacher.

B. Additional Details on Audio Experiments

The audio experiments select hyperparameters across timestep shifts, sampling shifts, and masking ratios using rankings over three FAD metrics. The best runs are compared across variants, with the approach performing favorably across hyperparameters.

  • Hyperparameter selection: Audio experiments sweep training shifts α ∈ {0.75, 1.0, 1.78}, sampling shifts {4.62, 6.93}, and masking ratios R_M ∈ {0.05, 0.1, 0.25, 0.5}.The masking-ratio sweep applies to the proposed approach.
  • Hyperparameter selection: Hyperparameters are selected by minimizing median rank across FAD (CLAP), FAD (CLAP-M), and FAD (CLAP-A).Rankings are computed within each approach before choosing the best set.
  • Results: The best hyperparameter runs are bolded in the early-stopping comparison across all audio variants.The figure caption defines bold runs as those selected by the described process.
  • Results: The approach compares favorably against all other variants across the evaluated audio hyperparameters.This is the reported figure-level comparison rather than a single metric value.

C. Additional Details on Video Experiments

Video experiments tune the WAN2.2 training shift and test a low-SNR modification. The reported comparisons examine timestep shifts and methods with or without the modification, including multimodal results where all metrics improve over vanilla flow matching.

  • Timestep selection: α = 2.95 is selected as the WAN2.2 training shift because it achieves the best FVD and framewise FID among the tested shifts.The search evaluates α ∈ {1.0, 1.78, 2.95, 4.62, 6.93}.
  • Low-SNR modification: The low-SNR modification samples 5% of training timesteps uniformly from the high-noise interval [0.95, 1.00].It is introduced to address potential issues from insufficient training in low-SNR regimes.
  • Low-SNR modification: The low-SNR modification leaves SRA largely unchanged but consistently improves the proposed approach and REPA DA3.REPA DINOv2 improves in FVD under the modification according to the cited text.
  • Experimental comparisons: Figure 15 compares video performance across shift parameters and approaches with or without the low-SNR modification.The shift experiments use logit-normal sampling with a Vanilla Flow Matching model trained on WAN2.2 representations.
  • Multimodal results: All metrics improve for the proposed method relative to vanilla flow matching across multimodal weightings.Table 6 reports FM, Ours, and the change of Ours relative to FM for each metric.

D. Aditional Details on Multi-Modal Experiments

The multimodal experiments share most model weights while using modality-specific input and output layers, sampling ratios, and loss weights. Self-Flow improves performance across modalities for all tested weightings throughout training.

  • Model setup: Multimodal training uses separate input and output layers per modality while sharing all other model weights.This supports simultaneous image, video, and audio training with minimal architectural changes.
  • Batching: Image, video, and audio batches use sizes 38, 8, and 16, respectively, to keep training-step durations roughly comparable.Runtime depends mainly on each modality’s autoencoder and latent sequence length.
  • Weighting: Relative modality performance depends on sampling frequencies and modality-specific loss weights because model capacity is shared across modalities.The experiments vary weighting triplets for image, video, and audio losses.
  • Results: Self-Flow improves multimodal performance consistently for all tested weightings and simultaneously across all involved modalities.The benefits remain present throughout the complete training process.

E. Joint Video-Action Prediction

Self-Flow is evaluated for video-based action prediction and related multimodal generation settings. It learns more efficiently from limited robotics data, with its strongest action-prediction gains on complex multi-object and sequential tasks.

  • Video-action prediction: Self-Flow consistently outperforms vanilla Flow Matching during SIMPLER finetuning, demonstrating more efficient learning from limited data.The evaluation uses 73.5k RT-1 episodes and reports variance across two evaluation runs per checkpoint.
  • Video-action prediction: At 30k steps, Self-Flow succeeds in every task category, whereas the baseline fails entirely on Open and Place tasks.This comparison covers Pick Coke Can, Open/Close Drawer, Move Near, and Open and Place tasks.
  • Video-action prediction: At 100k steps, performance converges on simpler single-object tasks, while Self-Flow retains a significant advantage on complex multi-object and sequential tasks.The complex tasks are Move Near and Open and Place.
  • Joint video-audio prediction: Mixed-modality initialization benefits audio-video prediction, while video-only Self-Flow outperforms mixed-modality vanilla flow matching.The comparison reports FVD under equal amounts of video training samples.
  • ImageNet and semantic latents: Self-Flow outperforms REPA on ImageNet after approximately 3M training steps despite REPA’s DINOv2 alignment.The ImageNet training distribution overlaps heavily with DINOv2’s training data, delaying Self-Flow’s advantage.
  • ImageNet and semantic latents: Self-Flow consistently improves generation over a semantic autoencoder baseline, showing complementarity with already structured latent spaces.The experiments use RAE on ImageNet.
  • Scope: End-to-end joint training of the autoencoder remains a proposed direction for future work.The paper identifies this extension as promising rather than evaluating it here.

H. Layer Selection Ablations

Layer selection is stable near the default student and teacher layers but degrades when layers are substantially too shallow or too deep. Shallow teacher layers weaken semantic supervision, whereas deep student layers interfere with generation.

  • Setup: The ablation uses student layer l = 0.3D and teacher layer k = 0.7D as the default configuration.D denotes the total network depth.
  • Results: Performance remains stable across a range of layer choices near the default selection, indicating limited sensitivity to this hyperparameter.Varying the student layer from 8 to 4 or 12 has little impact on results.
  • Results: Shallow teacher layers hurt performance because semantic representations have not fully emerged, while deeper student layers interfere with generation quality.Performance degrades when layer choices deviate substantially from the stable range.

I. Additional Qualitative Results

Additional qualitative results show stronger image structure, texture, detail, typography, and video coherence for Self-Flow across varied prompts and modalities. These examples include a scaled 4B-parameter model and video training with only 6M samples.

  • Text-to-image: Self-Flow consistently improves structural coherence, texture fidelity, and high-frequency detail preservation in text-to-image generation.The improvements hold across prompts of varying complexity.
  • Text-to-video: Self-Flow produces spatially and temporally coherent text-to-video results, while baselines exhibit structural and temporal artifacts.The video model was trained on 6M video samples.
  • Typography: A scaled 4B-parameter multimodal model produces accurate and legible typography after 100K high-resolution finetuning steps.The comparison is against vanilla flow matching.
Loading 2603.06507v1…