PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation

Cong Wei, Xuanchi Ren, Bryan Chu, Weiming Ren, Huan Ling, Jiahui Huang, Laura Leal-Taixé, Sanja Fidler, Wenhu Chen, Zian Wang, Jay Zhangjie Wu

arXiv:2609.38597v1cs.CV

TL;DR

Unified models face a design gap: visual representations for understanding and generation differ, and video adds distinct temporal conventions. PixelUMM uses pixel-space patches and tubelets with shared multimodal attention and task-specific experts; it achieves performance comparable to state-of-the-art baselines across image and video tasks.

  • Problem

    Unified models often use separate visual representations for understanding and generation, increasing context length and complicating integration with VLM pretraining; video tasks also use different temporal representations.

    Source: §1 · p. 4

  • Method

    PixelUMM connects image patches and video tubelets directly to a shared multimodal backbone, using shared attention with separate understanding and generation parameters.

    Source: §1 · p. 4

  • Results

    PixelUMM achieves performance comparable to state-of-the-art baselines across image and video understanding and generation tasks.

    Source: §1 · p. 4, §5 · p. 25

  • Takeaways & Limitations

    The experiments suggest that less aggressive spatial and temporal token compression generally lowers generation loss, while convolutional heads can suppress patch artifacts.

    Source: §5 · p. 25

    Because training data differ across models, the results cannot establish which architecture is superior or which model converges faster.

    Source: §3.2 · p. 21

Abstract

from arXiv · show

Unified Multimodal Models (UMMs) often rely on separate visual representations for understanding and generation, increasing visual context length and complicating integration with established vision-language pretraining pipelines. Recent advances in pixel-space modeling offer an encoder-free alternative, but extending this paradigm from images to videos is non-trivial: video understanding and generation adopt different temporal representations, leaving the design of a unified visual interface an open question. We present PixelUMM, an encoder-free model for unified image and video understanding and generation directly in pixel space. PixelUMM represents images as spatial patches and videos as spatiotemporal tubelets, connecting raw pixels to a shared multimodal backbone through single-layer linear projections. Its Mixture-of-Transformers architecture combines shared attention with task-specific parameters and extends clean-pixel prediction to video generation, jointly supporting autoregressive text prediction and pixel-space flow matching. Experiments show that PixelUMM achieves competitive performance across image and video understanding and generation tasks. We further conduct empirical studies of key design choices, including decoder design and spatial-temporal patch size, providing insights for future pixel-space unified multimodal models.

1. Introduction

Unified models must reconcile visual representations for understanding and generation, while dual visual streams increase context costs and complicate VLM pretraining integration. PixelUMM extends encoder-free pixel-space modeling to video, with competitive performance across image and video understanding and generation.

1. Introduction

  • Dual ViT/VAE streams approximately double visual context and require changes to established VLM pretraining pipelines.The burden grows in long-context and multi-turn conversations as visual tokens accumulate across images, video frames, and turns.Source: §1 · p. 4
  • Video understanding and generation use different temporal representations, leaving no natural shared interface for both tasks.The paper notes causal 3D VAEs for generation and independently sampled frames with image encoders for understanding.Source: §1 · p. 4
  • PixelUMM represents images as spatial patches and videos as spatiotemporal tubelets, projecting raw pixels into a shared backbone.Its Mixture-of-Transformers combines shared multimodal attention with separate understanding and generation parameters, jointly supporting text prediction and pixel-space flow matching.Source: §1 · p. 4
  • PixelUMM achieves competitive performance across image and video understanding and generation tasks.The paper also studies image and video patch sizes and alternative video decoders, examining training behavior and spatiotemporal patch artifacts.Source: §1 · p. 4

2. Method

PixelUMM extends encoder-free pixel-space modeling to image and video tasks through a shared visual interface. It combines task-specific Transformer experts, shared multimodal attention, task-specific objectives, and staged training.

2.1. Architecture

  • PixelUMM uses spatial image patches and spatiotemporal video tubelets, mapped to the backbone by linear projections without visual encoders or VAE tokenizers.Understanding and generation use separate projection alternatives; clean media enters the understanding projections and corrupted targets the generation projections.Source: Figure 3 · p. 5, §2.1 · p. 5
  • The Mixture-of-Transformers routes text and clean visual tokens to an understanding expert and noisy visual tokens to a generation expert, with shared self-attention.Route-specific normalization, QKV/output projections, and FFNs accompany the shared attention operation.Source: §2.1 · p. 6

2.2. Unified Multimodal Sequence Modeling

  • ChatML serializes image and video understanding and generation as raw-pixel token spans, with clean inputs for understanding and noisy targets for generation.Attention is causal within text, bidirectional within visual blocks, and temporally ordered across sparse video frames; generation targets attend to the causal prompt and the full target block.Source: Figure 4 · p. 6, §2.2 · p. 7
  • Three-axis Native RoPE assigns attention-head dimensions to temporal, height, and width coordinates for text and visual tokens.Text advances only along the temporal axis, while visual tokens also carry spatial grid coordinates.Source: §2.2 · p. 7

2.3. Objectives

  • Training combines assistant-token cross-entropy with image and video pixel-space losses, activating only loss terms relevant to each sample.The pixel objective corrupts clean patches or tubelets and predicts clean pixels while computing loss in velocity space.Source: §2.3 · p. 7

2.4. Training Details

  • Six training stages progressively add image and video tasks and raise spatial resolutions, ending with joint training across all tasks.Training draws image-understanding data from FineVision, video-understanding data from two collections, and public real and synthetic data for generation.Source: §2.4 · pp. 8–9

3.1. Empirical Findings

The patch-size studies find that stronger spatial or spatiotemporal compression generally raises generation training loss, while the selected video configuration also reflects alignment with common video-VAE compression conventions.

3.1. Empirical Findings

  • Experimental identifiers use F1–F8 for families and R01, R02, and so on for configurations within each family.For example, F1-R01 denotes the first configuration in family F1.Source: §3.1 · p. 9

3.1.1. F1: Image Patch Size

  • With the same sequence-length budget, 32×32 patches process four times as many images per step, but 16×16 patches maintain lower late-training T2I MSE.The runs differ only in the generation input projection and output head; all other training settings are identical.Source: Figure 6 · p. 9, §3.1.1 · p. 9

3.1.2. F2: Video Patch Size

  • Less aggressive spatiotemporal compression generally yields lower T2V loss, with p32/t1 attaining the lowest loss; the final model uses p16/t4 to align with common video-VAE compression.The four runs vary only in video input-projection and output-head patch configurations; other training settings are identical.Source: §3.1.2 · p. 10, Figure 8 · p. 11

3.1.3. F3: Patch Artifacts

Convolutional video output heads reduce the linear head’s grid artifacts, but their optimization, visual quality, and compute costs differ across designs.

3.1.3. F3: Patch Artifacts

  • Patch artifacts become more pronounced at high CFG scales, such as around 6, particularly in low-texture regions with the linear decoder.The reported examples show grid-aligned intensity changes in smooth image and video regions.Source: §3.1.3 · p. 11, Figure 10 · p. 12
  • The Wan-style decoder costs 2,274 GFLOPs, T-S PixelShuffle 1,279 GFLOPs, and S-T PixelShuffle 747 GFLOPs per 96 × 176 × 320 clip, versus 133 GFLOPs for the linear head.The convolutional alternatives have 18.3M, 52.9M, and 15.1M parameters, respectively, compared with 12.6M for the linear head.Source: §3.1.3 · p. 12, Figure 11 · p. 13
  • The linear F3-R01 head shows the strongest normalized boundary elevations: 1.078 at spatial patch boundaries and 1.160 at temporal tubelet boundaries.The probe covers 72 prompts, normalizes each prompt by its video-wide adjacent-sample mean, and reports mean ± SEM across prompts.Source: §3.1.3 · p. 13, Figure 13 · p. 14
  • The compared decoder outputs use the same prompt and a fixed crop, with sampled views at frames 0, 32, 64, and 95 across the 96-frame video.The displayed frames were captured before video encoding, with pre-clamp float32 tensors retained separately for numerical analysis.Source: §3.1.3 · p. 14, Figure 14 · p. 15
  • PixelShuffle offers a better trade-off than the Wan-style decoder, with T-S slightly outperforming S-T; the Wan-style head gives the cleanest boundaries but higher loss and blurrier outputs.The PixelShuffle heads converge to similar T2V pixel-flow losses near 0.02, while the Wan-style head remains above 0.05 over the observed interval.Source: §3.1.3 · pp. 13, 15

3.1.4. F4: Pixel-Space and VAE-Space Training Dynamics

The early pixel-space and VAE-space loss magnitudes are not directly comparable as evidence of learning speed or generation quality.

3.1.4. F4: Pixel-Space and VAE-Space Training Dynamics

  • At 10K steps, VAE-space loss is about 4.3× pixel-space loss, but this cross-space gap does not show faster pixel-space learning; gradient norms are nearly equal and pixel-space loss occasionally spikes.The comparison covers the first 10K steps, with the pixel-space and VAE-space runs using different prediction targets.Source: §3.1.4 · pp. 15–16, Figure 15 · p. 16

3.1.5. F5: Model Size

Larger models reach comparable training losses in fewer steps, while increasing GPU count produces a larger step-efficiency gain for text than for generation.

3.1.5. F5: Model Size

  • The 8B model reaches comparable losses in roughly one-third as many training steps as the 1.7B model: MSE 0.060 at about 14K versus 40K steps, and CE 0.50 at about 10K versus 31K steps.These curve-based estimates indicate roughly 3× faster convergence in training steps, not wall-clock time.Source: §3.1.5 · p. 16, Figure 16 · p. 16

3.1.6. F6: Compute Scaling

  • Increasing GPU count from 8 to 128 yields a larger step-efficiency gain for text than generation: CE reaches 1.0 in about 2K versus 17.5K steps, while MSE reaches 0.065 in about 17K versus 23K steps.These curve-based estimates measure training steps to a fixed loss, not wall-clock speedup or compute efficiency.Source: §3.1.6 · p. 17, Figure 17 · p. 17

3.1.7. F7: Multimodal Context Conditioning

Multi-task fine-tuning adds visual conditioning through the understanding expert while preserving generation through the generation expert. Understanding changes are mixed for image benchmarks, while all four video benchmarks improve.

3.1.7. F7: Multimodal Context Conditioning

  • PixelUMM extends SenseNova-U1’s encoder-free image-conditioning design [32] to image-to-video generation and video editing.The examples show image-conditioned video generation and editing through the understanding expert.Source: §3.1.7 · p. 17
  • Clean visual conditions enter the understanding expert and noisy target-video tokens enter the generation expert, with both routes interacting through shared multimodal attention.Text instructions and clean image or video conditions can accompany the target video.Source: §3.1.7 · p. 17, Figure 18 · p. 18
  • Image changes were mixed (BLINK +2.37, CV-Bench +2.27, SEED-I −5.41), while all four video benchmarks improved 0.89–3.49 in dense_mode and 1.00–3.49 in sparse_mode.Table 3 compares performance before and after multi-task fine-tuning.Source: §3.1.7 · p. 18, Table 3 · p. 19

3.1.8. F8: Video Understanding Interfaces

The dense and sparse video-understanding interfaces differ in sampling and frame representation, but the reported benchmarks show no consistent advantage for dense_mode.

3.1.8. F8: Video Understanding Interfaces

  • dense_mode samples at 4 FPS and forms temporal tubelets, whereas sparse_mode samples at 1 FPS and projects frames independently.For eligible videos, dense_mode can use up to 384 frames; sparse_mode uses up to 96.Source: §3.1.8 · p. 20
  • The evaluation protocols differ only for videos no longer than 96 seconds with source frame rates of at least 4 FPS, and both use the same per-frame resolution budget.Outside that setting, the protocols use frame-based sampling; the reported resolution budget is 4482 pixels per frame.Source: §3.1.8 · p. 20
  • 71.22 vs 71.15 on MVBench, but dense_mode scores 57.67 vs 57.89 on Video-MME and 58.71 vs 59.39 on LongVideoBench; LVBench ties at 41.38.Values are dense_mode versus sparse_mode; higher-FPS tubelet input offers no clear overall advantage.Source: §3.1.8 · p. 20, Table 4 · p. 20

3.2. Benchmark Results

PixelUMM’s reported results are comparable to baselines across image and video understanding and generation. Because training data differ across models, these results do not establish architectural superiority or faster convergence.

3.2. Benchmark Results

  • Different training data mean these comparisons cannot establish which architecture is superior or which model converges faster.The authors state this boundary on interpreting the cross-model results.Source: §3.2 · p. 21
  • Across image and video understanding and image and video generation, PixelUMM achieves overall performance comparable to the baselines.The reported benchmark groups include image understanding, video understanding, image generation, and video generation.Source: Table 5 · p. 21, Table 6 · p. 21, §3.2 · p. 21
  • The benchmark suite spans LMMS-Eval image understanding, sparse-mode video understanding, GenEval and DPG-Bench, and VBench Parts 1 and 2.Video understanding is evaluated with up to 96 sampled frames.Source: Table 5 · p. 21, Table 6 · p. 21, Table 7 · p. 22

4. Related Work

Related work spans unified architectures and visual representations, while PixelUMM brings pixel-space modeling together with image-video unification through a shared raw-pixel interface.

4. Related Work

  • Visual interfaces include semantic encoders, discrete tokenizers, VAEs, shared continuous representations, and semantic or reconstruction-based alignment approaches [114] [115] [128] [155] [88] [95] [103] [56] [40] [54] [118] [146] [169] [173] [127] [176] [86] [131] [134] [150].Representation alignment and denoising objectives also appear in the literature.Source: §4 · p. 24
  • Pixel-space work embeds image patches directly, removes latent image compression for generation, and includes unified understanding-and-generation models such as TUNA-2 and SenseNova [27] [28] [29] [31] [81] [13] [19] [66] [135] [168] [30] [32] [77].Related designs also integrate generation and editing through pixel-level architectures [8].Source: §4 · p. 24
  • Video research includes task-specific generators, video-language understanding models, and systems combining video understanding with generation [7] [57] [100] [129] [163] [20] [83] [117] [140]; PixelUMM investigates a shared raw-pixel interface for images and videos.PixelUMM uses spatial image patches and spatiotemporal video tubelets without a pretrained visual encoder or VAE.Source: §4 · p. 24

5. Conclusion

PixelUMM unifies image and video understanding and generation directly in pixel space, without pretrained visual encoders or VAE-based latent tokenizers. Experiments show performance comparable to state-of-the-art baselines across these tasks.

  • PixelUMM achieves performance comparable to state-of-the-art baselines across image and video understanding and generation tasks.Source: §5 · p. 25
  • PixelUMM connects spatial image patches and spatiotemporal video tubelets to a shared multimodal backbone through lightweight linear projections.Source: §5 · p. 25
  • Less aggressive spatial and temporal token compression generally lowers generation loss, while convolutional heads can suppress patch artifacts.Source: §5 · p. 25
  • PixelUMM retains separate understanding and generation Transformer experts with shared attention rather than fully unified parameters.Source: §5 · p. 25
  • Future work could test larger spatial patches (p=64) and longer temporal tubelets (τ=8) without compromising training or generation quality.Source: §5 · p. 25
Loading 2609.38597v1…