PaperlayerView exact revision on arXiv

Source-linked AI summary

GenRec: Unifying Video Generation and Recognition with Diffusion Models

Zejia Weng, Xitong Yang, Zhen Xing, Zuxuan Wu, Yu-Gang Jiang

arXiv:2408.15241v2cs.CV

TL;DR

Video diffusion models capture spatial-temporal priors, but their potential for recognition and joint generation-recognition training remains under-explored. GenRec builds on Stable Video Diffusion with random-frame conditioning to train both tasks. It reports competitive recognition, class-conditioned video generation, and performance with limited observed frames.

  • Problem

    Video diffusion models’ spatial-temporal representations and their potential for video understanding and joint generation-recognition training remain under-explored.

    Source: §1 · p. 1

  • Method

    GenRec builds on Stable Video Diffusion and jointly trains generation and recognition using randomly selected conditioning frames while masking the rest.

    Source: §1 · p. 2

  • Results

    GenRec reports competitive recognition, class-conditioned image-to-video generation, and limited-frame robustness, including 75.8% accuracy on SSV2 and 87.2% on K400.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GenRec demonstrates strong generation and recognition capabilities across normal or partial-video recognition, video completion, and class-conditioned image-to-video generation.

    Source: §6 · p. 10

    GenRec fine-tunes a pretrained video diffusion model and uses more pretraining data and more parameters than previous methods.

    Source: §A.7 · p. 17

Abstract

from arXiv · show

Video diffusion models are able to generate high-quality videos by learning strong spatial-temporal priors on large-scale datasets. In this paper, we aim to investigate whether such priors derived from a generative process are suitable for video recognition, and eventually joint optimization of generation and recognition. Building upon Stable Video Diffusion, we introduce GenRec, the first unified framework trained with a random-frame conditioning process so as to learn generalized spatial-temporal representations. The resulting framework can naturally supports generation and recognition, and more importantly is robust even when visual inputs contain limited information. Extensive experiments demonstrate the efficacy of GenRec for both recognition and generation. In particular, GenRec achieves competitive recognition performance, offering 75.8% and 87.2% accuracy on SSV2 and K400, respectively. GenRec also performs the best on class-conditioned image-to-video generation, achieving 46.5 and 49.3 FVD scores on SSV2 and EK-100 datasets. Furthermore, GenRec demonstrates extraordinary robustness in scenarios that only limited frames can be observed. Code will be available at https://github.com/wengzejia1/GenRec.

1 Introduction

The paper asks whether video diffusion models’ spatial-temporal priors support recognition and whether generation and recognition can be optimized together. GenRec addresses their differing training and inference processes with random-frame conditioning and reports competitive results across both tasks.

1 Introduction

  • Video diffusion models capture spatial-temporal information, but their potential for downstream video understanding remains under-explored.The paper asks whether generation-trained backbones support semantic recognition, whether joint optimization can retain generation, and whether unification helps when only limited frames are available.Source: §1 · p. 1
  • Recognition typically uses clean, multiple-frame inputs, whereas diffusion generation trains on corrupted inputs and may use a single conditioning frame, creating a training-inference mismatch for unified models.Prior work either treats diffusion models as frozen feature extractors or adapts them for new tasks while sacrificing generation capability.Source: §1 · p. 2
  • GenRec jointly trains noise prediction and classification on randomly selected conditioning frames while masking the rest, bridging video generation and recognition through partially observed content.The framework builds on Stable Video Diffusion; random-frame conditioning supports arbitrary-frame prediction and is intended to produce more discriminative, robust recognition features.Source: §1 · p. 2, Figure 1 · p. 2
  • 75.8% accuracy on SSV2 and 87.2% on K400; GenRec also reports 46.5 and 49.3 FVD scores on SSV2 and EK-100 for class-conditioned image-to-video generation.When only the front half of a video is visible, GenRec achieves 57.7% accuracy versus 75.3% with the full video, retaining 76.6% of full-video accuracy.Source: §1 · p. 2

2 Preliminary

The preliminary section describes diffusion as denoising from noise toward the data distribution. It introduces the score-based formulation and the Stable Video Diffusion foundation used by GenRec.

2 Preliminary

  • Diffusion generation starts from Gaussian noise and iteratively denoises at decreasing noise levels until the result follows the original data distribution.The data distribution is smoothed by adding Gaussian noise with standard deviation σ.Source: §2 · p. 3
  • The probability-flow ODE follows the score function, and training minimizes an L2 denoising loss across noise levels.The noise schedule σ(t) is set as the time step t.Source: §2 · p. 3
  • GenRec builds on Stable Video Diffusion, which uses the EDM framework for large-scale video training and encodes videos frame by frame into latent space.A pretrained VAE produces the latent representation on which the model is built.Source: §2 · p. 3

3 GenRec

GenRec builds on Stable Video Diffusion to jointly support video generation and recognition, using random latent masking to handle arbitrary or limited frame inputs. Its shared spatial-temporal model reconstructs video content and classifies actions from intermediate features.

3 GenRec

  • GenRec uses a Spatial-Temporal UNet to reconstruct clean video latents and classify video categories from intermediate features.It builds on Stable Video Diffusion and feeds noisy and masked latents together to the UNet.Source: §3 · p. 3, §3.1 · p. 4

3.1 Pipeline Overview

  • Random latent masking supports conditioning on arbitrary frame subsets, from only the first frame to no visible frames.Training randomly erases half to all video conditions, retaining about one-fourth of the original video information on average.Source: §3.1 · pp. 3–4

3.2 Optimization

  • GenRec jointly trains generation and classification with a weighted total loss balancing latent reconstruction and cross-entropy classification objectives.The generative loss measures L2 error between original and reconstructed latents; γ controls the relative importance of the two objectives.Source: §3.2 · pp. 4–5

3.3 Inference for Different Downstream Tasks

  • At inference, GenRec iteratively denoises video latents conditioned on provided frames to produce a complete video.The resulting denoised latent is decoded as video.Source: §3.3 · p. 5
  • When visible frames are extremely limited, class information guides video generation toward the desired category.The class-conditioned score uses the classifier branch, and a scaling factor greater than one sharpens attention to the target category.Source: §3.3 · pp. 5–6
  • For recognition, GenRec supports standard video inputs and partially observed frames, including early action prediction and sparse video recognition.Partially visible inputs are handled by masking invisible frames and replacing the noisy latent with Gaussian random noise.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate GenRec across recognition and generation, including recognition with limited frames, and compare it with specialized methods while testing design choices.

4.1 Experimental Setup

  • Across four datasets, experiments assess top-1 recognition accuracy and FVD for recognition, partial recognition, class-conditioned image-to-video generation, and frame completion.Lower FVD indicates higher video fidelity and realism.Source: §4.1 · p. 6

4.2 Main Results

  • 75.8% accuracy on SSV2 and 87.2% on K400; GenRec is competitive with specialized methods and achieves lower FVD than other methods for class-conditioned image-to-video generation.The table reports recognition and generation performance; SEER† predicts 16 frames while other methods predict 12.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • GenRec achieves the highest accuracy and ratio metrics for early prediction at all observed-frame levels, and reaches 55.7% accuracy with 2 sparsely sampled frames.It performs similarly to MVD† when all frames are observed, with its advantage growing as observed frames decrease; it also outperforms TemPr and MVD† more as frames decrease on EK-100 and UCF-101.Source: §4.2 · pp. 8–9, Table 2 · p. 8, Figure 3 · p. 8
  • Removing generation supervision lowers early-prediction accuracy by 0.3% at ρ = 1.0 and 2.3% at ρ = 0.3.The reported degradation across tasks is especially pronounced when fewer frames are visible.Source: §4.2 · p. 8, Table 2 · p. 8
  • With 2 frames, uniform sampling yields 55.7% recognition accuracy and 46.7 FVD, compared with 28.9% and 57.8 for early prediction.Three interpolated frames reach 31.7 FVD, comparable to an eight-frame prefix at 30.3, while achieving higher recognition accuracy.Source: §4.2 · p. 9, Table 3 · p. 9
  • The second UNet up-sampling block gives the best recognition accuracy at 75.8%, compared with 75.2% for the third block.The first block has the lowest accuracy, so the second block is selected for feature extraction.Source: §4.2 · p. 9, Table 4 · p. 9
  • Masking ratios produce similar FVD scores, but an 87.5% masking ratio reduces action-recognition accuracy by 0.9% versus the selected ratio.A larger masking ratio may benefit generation by resembling class-conditioned frame prediction with one or two given frames.Source: §4.2 · p. 9, Table 5 · p. 9

5 Related Work

Related work covers diffusion models for video generation and visual understanding, while identifying joint generation and classification in one model as under-explored.

5 Related Work

  • Diffusion work spans text-to-video, image-and-text-to-video generation, and video editing, with Stable Video Diffusion providing a foundation for high-quality, diverse, temporally consistent videos.Many video adaptations incorporate temporal convolutions and attention mechanisms [52] [18].Source: §5 · p. 10
  • Visual-understanding studies use pretrained diffusion models for segmentation and correspondence [55] [39] or treat diffusion learning as self-supervision for feature representations [8].Source: §5 · p. 10
  • Most prior approaches use diffusion networks as pretrained backbones or sacrifice generative capability, leaving integrated generation and classification under-explored.Source: §5 · p. 10

6 Conclusion

GenRec jointly optimizes video generation and recognition, with experiments demonstrating capabilities across full and partial video recognition, video completion, and class-conditioned image-to-video generation.

6 Conclusion

  • GenRec jointly optimizes video generation and recognition, using temporal modeling in the diffusion model to support mutual reinforcement between the tasks.Source: §6 · p. 10
  • Experiments report generation and recognition capabilities across normal and partial-video recognition, video completion, and class-conditioned image-to-video generation.Source: §6 · p. 10
  • The findings support the potential of combining generation and classification in a unified model for video analysis.Source: §6 · p. 10

A.1 Early Prediction on EK-100 and UCF-101

The appendix points to detailed early-action-prediction evaluations on EK-100 and UCF-101.

A.1 Early Prediction on EK-100 and UCF-101

  • The appendix provides detailed early-action-prediction results for EK-100 and UCF-101.Source: §A.1 · p. 15
  • Table 7 is identified as the early-action-prediction table for EK-100.Source: Table 7 · p. 15
  • Table 8 is identified as the early-action-prediction table for the UCF dataset.Source: Table 8 · p. 15

A.2 Training and Inference Time Cost Compared with Previous Classification Methods

The comparison reports that GenRec and Baseline II take longer to test than MVD-H, while GenRec's generation components add a small training-time cost relative to Baseline II.

A.2 Training and Inference Time Cost Compared with Previous Classification Methods

  • The timing comparison uses the same hardware resource, 4 nodes × 8 V100s, and the same batch size.Source: §A.2 · p. 15
  • Baseline II fully fine-tunes the original SVD model using classification supervision only.Source: §A.2 · p. 15
  • GenRec and Baseline II take longer to test than MVD-H; GenRec matches Baseline II at test time but takes slightly longer to train.The comparison is presented in terms of parameters, training time, and test time.Source: Table 9 · p. 15, §A.2 · p. 15

A.3 Ablation on Loss Balance Ratio

The loss-balance ratio has little effect on SSv2 recognition accuracy, but setting it too low harms generation, with a zero ratio causing significant forgetting.

A.3 Ablation on Loss Balance Ratio

  • A zero loss-balance ratio causes significant forgetting and severely compromises generation, while varying the ratio has only a minor impact on SSv2 accuracy.Table 10 reports SSv2 accuracy and FVD; higher accuracy and lower FVD indicate better performance.Source: §A.3 · p. 15, Table 10 · p. 16

A.4 Comparision with SVD Baseline

The SVD comparison finds it suboptimal to GenRec in frame-prediction tests, while classification training does not degrade frame-prediction generation. Case studies cover category-guided generation, video interpolation, and comparisons with SEER; the paper also notes higher data and parameter requirements.

A.4 Comparision with SVD Baseline

  • SVD achieves competitive scores against prior state-of-the-art methods but is suboptimal to GenRec in frame-prediction tests on SSV2 and Epic-Kitchen.SVD videos were generated at 512x512 and downsampled to 256x256 for FVD calculations; Baseline I fine-tunes SVD on target datasets.Source: §A.4 · p. 16, Table 11 · p. 16

A.5 Impact of Classification Training on Generative Ability

  • Similar frame-prediction scores indicate classification loss does not degrade generation, while more accurate classification guides higher-quality class-conditioned frame predictions.The comparison uses Baseline I, which adapts SVD without classification supervision and freezes its generation backbone while training a classifier head.Source: §A.5 · p. 16, Table 12 · p. 16

A.6 Case Study for Class-Conditioned Image-to-Video Generation and Video Interpolation

  • Case studies visualize category-guided generation from the first frame, interpolation from start and end frames, and comparisons with SEER.The SEER cases were selected from its official website.Source: §A.6 · p. 16, Figure 4 · p. 17, Figure 5 · p. 18

A.7 Limitations and Broader Impacts

  • GenRec is fine-tuned from a pretrained video diffusion model and uses more pretraining data and parameters than previous methods; distillation into a smaller model is proposed for future work.The paper also describes potential impacts in content creation, surveillance, and accessibility.Source: §A.7 · p. 17

Related papers

Loading 2408.15241v2…