PaperlayerView exact revision on arXiv

Source-linked AI summary

UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models

Lan Chen, Yuchao Gu, Qi Mao

arXiv:2509.21760v1cs.CV

TL;DR

Unified sequential vision models require task- and modality-specific pre-training data, motivating a more scalable way to adapt vision tasks. UniVid fine-tunes a pre-trained video diffusion transformer using visual sentences, and reports generalization across modalities and data sources, with understanding and generation switched by sequence order.

  • Problem

    Sequential vision modeling requires task- and modality-specific pre-training data, a labor-intensive burden that limits scalability to new tasks.

    Source: §1 · p. 1

  • Method

    UniVid fine-tunes a pre-trained video diffusion transformer with lightweight SFT, encoding each task and its expected output modality in a visual sentence.

    Source: §1 · p. 1, §3.3 · p. 5

  • Results

    Despite pre-training solely on natural video data, UniVid generalizes to cross-modal contexts and cross-source tasks, while visual-sentence ordering switches understanding and generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight the potential of pre-trained video generation models as a unified backbone for general-purpose visual modeling.

    Source: §1 · p. 2

    The Wan model is limited to 81 frames per sequence, and generative randomness can prevent consistent instance labels in segmentation.

    Source: §4.6 · p. 8

Abstract

from arXiv · show

Large language models, trained on extensive corpora, successfully unify diverse linguistic tasks within a single generative framework. Inspired by this, recent works like Large Vision Model (LVM) extend this paradigm to vision by organizing tasks into sequential visual sentences, where visual prompts serve as the context to guide outputs. However, such modeling requires task-specific pre-training across modalities and sources, which is costly and limits scalability to unseen tasks. Given that pre-trained video generation models inherently capture temporal sequence dependencies, we explore a more unified and scalable alternative: can a pre-trained video generation model adapt to diverse image and video tasks? To answer this, we propose UniVid, a framework that fine-tunes a video diffusion transformer to handle various vision tasks without task-specific modifications. Tasks are represented as visual sentences, where the context sequence defines both the task and the expected output modality. We evaluate the generalization of UniVid from two perspectives: (1) cross-modal inference with contexts composed of both images and videos, extending beyond LVM's uni-modal setting; (2) cross-source tasks from natural to annotated data, without multi-source pre-training. Despite being trained solely on natural video data, UniVid generalizes well in both settings. Notably, understanding and generation tasks can easily switch by simply reversing the visual sentence order in this paradigm. These findings highlight the potential of pre-trained video generation models to serve as a scalable and unified foundation for vision modeling. Our code will be released at https://github.com/CUC-MIPG/UniVid.

1. Introduction

UniVid asks whether video-generation pre-training can reduce the task- and modality-specific data burden of unified vision modeling. It adapts a pre-trained video diffusion transformer through lightweight fine-tuning, representing tasks as visual sentences whose context specifies the task and output modality.

1. Introduction

  • LVM-style sequential vision modeling requires separate task- and modality-specific pre-training data, making data curation labor-intensive and limiting scalability to new tasks.The paper asks whether a video generator pre-trained once for synthesis can instead be adapted efficiently through SFT.Source: §1 · p. 1
  • UniVid adapts a pre-trained video diffusion transformer to image and video tasks through lightweight SFT, without task-specific architectural modifications.Visual sentences encode the task and expected output modality in their context sequence.Source: §1 · pp. 1–2
  • Despite pre-training only on continuous natural videos, UniVid generalizes to cross-modal contexts and cross-source tasks, including natural-to-annotated data.Understanding and generation are distinguished by the ordering of elements in the visual sentence.Source: §1 · p. 2
  • The paper presents UniVid as a potential unified pre-training backbone for general-purpose visual modeling.Its stated contributions include adapting a pre-trained video DiT with minimal paired data and evaluating cross-modal and cross-source generalization.Source: §1 · p. 2

2. Related Works

Related work frames UniVid against image-resembling generation, sequential vision modeling, vision in-context learning, and video-generation architectures.

2. Related Works

  • Vision-task unification has used image-resembling generation, which reformulates tasks as image inpainting, and sequential modeling, which treats visual data as token sequences.The paper characterizes both approaches as relying heavily on large-scale annotated task-specific training samples.Source: §2 · p. 3
  • Vision in-context learning extends example-prompted task performance from language models to vision, with early approaches relying on annotated input-output pairs.More recent DiT-based text-to-image models demonstrate in-context controllable image generation and manipulation.Source: §2 · p. 3
  • Video-generation research includes autoregressive token-sequence models and diffusion models, with recent DiT-based hybrids improving generation quality and temporal coherence.UniVid uses Wan as its video-generation foundation to study adaptation to downstream vision tasks.Source: §2 · p. 3

3. Methodology

The methodology motivates a unified visual-sequence approach, then adapts a naturally pre-trained video DiT with SFT to test cross-modal and cross-source generalization. The reported observations cover mixed-modality contexts, natural-to-annotated tasks, and switching between understanding and generation by reordering the sequence.

3.1. Preliminaries: Limitations of Sequential Vision Models

  • LVM relies on large-scale task- and modality-specific annotated pairs and remains limited to single-modality contexts and cross-source video tasks.These constraints motivate reducing annotation burdens and extending generalization across modalities and sources.Source: §3.1 · pp. 3–4

3.2. Problem Formulation: Unified Visual Sentence Paradigm

  • UniVid formulates each task as an example pair, a query, and a predicted output, with context defining the task and output modality across image and video elements.The study evaluates mixed-modality contexts and tasks spanning data sources, including pixel-level understanding and generation.Source: §3.2 · p. 4

3.3. Video Generative Pre-training with SFT for Unified Vision Modeling

  • The method starts from Wan, pre-trained on continuous natural videos, and uses LoRA-based SFT with visual sentences containing an example pair, query, and target.Context tokens remain clean while noise is added to the target; DiT blocks process the sequence with 3D full attention, and inference generates the target conditioned on context.Source: §3.3 · pp. 4–5
  • Experiments cover six generation and understanding tasks with 20 training samples each, fine-tuned separately for applicable contexts; tasks are also tested with reversed generation ordering.The reported observations summarize adaptation across the tested contexts and tasks.Source: §3.3 · p. 5
  • Despite pre-training solely on continuous natural videos, the fine-tuned model handles mixed-modal contexts, generalizes to depth prediction from natural videos, and switches between understanding and generation by reordering visual sentences.These three observations are illustrated in Figure 3.Source: Figure 3 · p. 4, §3.3 · p. 5

4. Experiments

The experiments evaluate UniVid across tasks and contexts, mixed fine-tuning, shot counts, and comparisons with LVM [1]. They report broad generalization while identifying context length and segmentation consistency as limitations.

4.1. Implementation Details.

  • Experiments use Wan2.1-T2V-1.3B, with clips set to one frame for image modality and 17 frames for video modality.Fine-tuning uses LoRA rank 16, learning rate 1 × 10−4, and batch size 1.Source: §4.1 · p. 6

4.2. Generalization across Tasks and Contexts

  • UniVid adapts across modalities and data sources, generates coherent videos under reversed sentence orders, and changes B′ according to query B.Camera motion transfer uses contexts I and IV; the other tasks use contexts I, II, and III.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3. Mixed Fine-tuning Strategy

  • Mixed-context training enables context-based output modality selection, while joint co-training with 20 examples per task performs better than separate fine-tuning across all tasks.Each task in joint training covers its applicable contexts.Source: §4.3 · p. 7, Figure 6 · p. 7, Table 2 · p. 8

4.4. Impact of Shot Number

  • Depth estimation tends to degrade when test shots exceed fine-tuning shots, while style transfer remains stable; longer contexts improve results but increase inference time.The study adopts a four-shot setting for consistent analysis.Source: §4.4 · p. 7, Table 3 · p. 8

4.5. Quantitative Comparisons

  • With limited training subsets, UniVid consistently outperforms LVM [1] across style transfer, edge prediction, segmentation, depth estimation, and surface normal estimation.The tasks are evaluated on full test splits, while LVM uses the full training split.Source: §4.5 · p. 8, Table 4 · p. 8, Table 5 · p. 8

4.6. Limitations and Future Work

  • Wan's context is limited to 81 frames per sequence, and segmentation label consistency across instance types cannot be guaranteed.The authors propose exploring long-context video generation and mitigating ambiguity in understanding tasks such as segmentation.Source: §4.6 · p. 8

5. Conclusions

UniVid adapts a pre-trained video generation model to diverse vision tasks through lightweight supervised fine-tuning. The conclusion reports generalization across cross-modal and cross-source contexts, and interchangeability of understanding and generation through sentence ordering.

5. Conclusions

  • UniVid reuses a pre-trained video generation model with minimal supervised data, organizing visual inputs as sentences whose context specifies the task and output modality.The framework supports joint adaptation to diverse tasks and context types through a single fine-tuning process.Source: §5 · p. 8
  • Despite pre-training solely on natural continuous videos without annotations, the fine-tuned model generalizes to cross-modal and cross-source contexts.Source: §5 · p. 8
  • Reordering visual sentences distinguishes understanding from generation, allowing the tasks to be interchanged within the unified paradigm.Source: §5 · p. 8

A. Implementation Details

The appendix describes task data and fine-tuning implementation, then compares the training data and model properties of LVM [1] and Wan [28].

A. Implementation Details

  • The appendix details paired data collection for six tasks, the Wan2.1-T2V-1.3B backbone, fine-tuning schedules, context sampling, and text-prompt settings.For co-training across tasks and contexts, training runs for 20 epochs with 1,200 iterations per epoch.Source: A. Implementation Details · p. 11

B. Comparison between LVM and video generation model

  • LVM [1] requires complex training-data construction, while Wan [28] is pretrained on raw images and videos and has fewer parameters than released LVM.Wan uses more total training tokens, an 8× downsampling encoder versus LVM's 16×, and is described as having lower computational costs.Source: B. Comparison between LVM and video… · p. 11

C. Additional Experimental Results

The appendix points to further results across tasks, contexts, and training settings, alongside a text-prompt figure and a model-comparison table.

C. Additional Experimental Results

  • Additional results cover each vision task across contexts, mixed-context fine-tuning, text-prompt effects, and co-training across tasks and contexts.Source: C. Additional Experimental Results · p. 11
  • Figure 7 presents text prompts at multiple levels of granularity.Source: Figure 7 · p. 11
  • Table 7 compares LVM [1] with the video generation model Wan [28].Source: Table 7 · p. 11

C.1. Conditional Generation Tasks

UniVid generates videos conditionally from depth maps or masked videos, and handles image, video, cross-modal, and cross-data-source vision tasks across contexts.

C.1. Conditional Generation Tasks

  • The fine-tuned video generation model performs conditional generation from depth maps or masked videos.The passage identifies these as conditional generation tasks.Source: §C.1 · p. 11

C.2. Performance Across Different Contexts

  • Across contexts, the fine-tuned model handles image and video tasks as well as cross-modal and cross-data-source tasks.Source: §C.2 · p. 11

C.3. Performance under Mixed-Context Fine-Tuning

Under mixed vision contexts, the fine-tuned model adjusts to each context, demonstrating strong generalization.

C.3. Performance under Mixed-Context Fine-Tuning

C.4. Co-training with All Vision Tasks and Contexts

Co-training across all vision tasks and mixed contexts yields consistent performance across tasks and contexts, with limited supervision.

C.4. Co-training with All Vision Tasks and Contexts

  • Co-training on all vision tasks under mixed contexts achieves consistent performance across tasks and contexts with limited supervision.The authors characterize this as robust generalization.Source: §C.4 · p. 12

C.5. Impact of Texts Across Different Vision Contexts

Across different vision contexts, the model learns relationships among four clips without explicit textual guidance.

C.5. Impact of Texts Across Different Vision Contexts

  • The model learns relationships among four clips across vision contexts without explicit text, demonstrating in-context learning in the temporal dimension.The cited figures examine text impact across contexts I–IV.Source: §C.5 · p. 12, Figure 14 · p. 15, Figure 15 · p. 16
Loading 2509.21760v1…