PaperlayerView exact revision on arXiv

Source-linked AI summary

HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers

Guozhen Zhang, Xuerui Qiu, Yutao Cui, Tianhui Song, Changlin Li, Junzhe Li, Tao Huang, Xiao Zhang, Yang Li, Jianbing Wu, Miles Yang, Zhao Zhong, Liefeng Bo, Limin Wang

arXiv:2606.13289v1cs.CVcs.AI

TL;DR

Unified multimodal models lack a holistic tokenizer that represents images and videos in one space while supporting reconstruction and semantic understanding. HYDRA-X addresses this with a native ViT tokenizer and reports strong performance across image and video understanding and generation tasks.

  • Problem

    Existing unified multimodal models lack well-supported holistic tokenization that binds images and videos into a single representation space with temporal and semantic structure.

  • Method

    HYDRA-X uses a single-ViT tokenizer with causal temporal attention, hierarchical temporal compression, and a decompressor trained with joint image-video teacher supervision.

  • Results

    HYDRA-X achieves strong performance across image and video understanding and generation, while its design improves editing consistency and accelerates convergence.

  • Takeaways & Limitations

    HYDRA-X establishes a holistic image-and-video interface that unifies five tasks under one shared backbone.

  • Takeaways & Limitations

    Evaluation is limited to a 7B dense LLM and excludes long video generation and video editing because of resource constraints.

Abstract

from arXiv · show

Holistic visual tokenizers are fundamental to unified multimodal models (UMMs) as they map diverse visual inputs into a unified representation space. In this paper, we present HYDRA-X, the first UMM that unifies image and video tokenization within a single Vision Transformer (ViT). Our design is driven by two core challenges: efficiently injecting spatiotemporal reconstruction capability into a native ViT, and embedding image- and video-level semantic awareness into the latent space. To address the first, comprehensive ablations reveal two key findings: (1) frame-level causal temporal attention suffices for visual reconstruction, whereas full spatiotemporal attention degrades it; and (2) hierarchical temporal compression substantially outperforms single-step alternatives. To tackle the second, we propose a lightweight decompressor that upsamples temporally compressed features under joint image-video teacher supervision, thereby enforcing complementary semantic structures within the compact latent space. Building on this holistic tokenizer, we further propose a principled improvement of the editing pipeline: source-target interaction should occur at the latent level inside the tokenizer rather than at the semantic level inside the LLM, substantially improving editing consistency and accelerating convergence. Instantiated at the 7B dense model, HYDRA-X achieves strong performance across image and video understanding and generation tasks, paving the way for future unified-tokenizer UMMs.

1 Introduction

HYDRA-X introduces HYDRA-XTOK, a single-ViT holistic tokenizer that unifies image and video representations for a native unified multimodal model. Its design combines causal temporal attention, hierarchical compression, joint semantic supervision, and latent-level editing to support five visual tasks.

  • Holistic tokenization: HYDRA-X is the first UMM built on a single ViT tokenizer that encodes both images and videos into one representation space.The tokenizer extends HYDRA’s compact-latent reconstruction paradigm from images to joint image-video processing.
  • Temporal reconstruction: Frame-level causal temporal attention outperforms full spatiotemporal attention because global interaction disrupts locality and structure learned during image pretraining.The causal mechanism uses a minimal temporal receptive field, attending only to the immediately preceding frame.
  • Temporal reconstruction: Hierarchical patchify outperforms single-step patchify by distributing temporal compression across multiple stages for progressive, multi-scale folding.The finding indicates that temporal compression benefits from staged rather than single-step processing.
  • Semantic supervision: HYDRA-X addresses mismatched image-video semantic supervision by using a lightweight decompressor with joint image and video teacher supervision.The approach supplies semantic guidance for video latents despite the lack of a video encoder operating at their compressed temporal resolution.
  • Unified tasks and editing: The shared encoder unifies image and video generation, image and video understanding, and image editing, while editing moves source-target interaction into the tokenizer’s latent representation.This preserves fine-grained structural information that semantic-only interaction in the LLM would forfeit.

2 Related Work

Prior work has unified reconstruction and semantics in visual tokenizers, while joint image-and-video tokenization remains under-explored. Related UMM and image-editing systems differ in how they share representations and inject source conditioning.

  • Visual Tokenizers: Unified visual tokenizers co-train reconstruction and understanding in a single ViT, while RAE freezes a semantic encoder and learns a pixel decoder.These approaches primarily address image tokenization.
  • Visual Tokenizers: HYDRA introduces a progressive ViT with a Generation–Semantic Bottleneck, which HYDRA-XTOK inherits for compress-then-restore semantic distillation.Aligning generative latents with semantic features has also been shown to mutually benefit generation and understanding.
  • Visual Tokenizers: Joint image-and-video tokenization remains largely under-explored, with video prior work including 3D-convolutional VAEs.The passage identifies this gap while introducing prior video-tokenization approaches.
  • Unified Multimodal Models: Composite UMMs bridge pretrained understanding and generation models through lightweight adapters or projection layers, preserving each specialized model’s strengths.UMMs target visual understanding and generation within a single backbone, and systems are grouped by parameter and representation sharing.
  • Image Editing: Image-editing pipelines mainly differ in where source conditioning is injected, including dedicated ControlNet-style condition adapters and BAGEL-style reference-token streams.ControlNet-style branches add a parallel encoder for spatially aligned source features, while reference-token streams prepend the source as extra context.

3 Preliminaries: Representation-Harmonized Tokenization

HYDRA-X builds on HYDRA’s single-ViT architecture, which separates generation and semantic perception through a Generation–Semantic Bottleneck. It extends this image-based design to videos using temporal causality, hierarchical patchification, and a Decompressor.

  • HYDRA framework: HYDRA splits one ViT into Gen-ViT and Sem-ViT, linked by a Generation–Semantic Bottleneck for unified generation and semantic perception.Gen-ViT extracts structural features, while the Bottleneck maps them into a compact generation latent and Sem-ViT produces semantic features.
  • HYDRA framework: For an image x∈R^H×W×3, Gen-ViT produces structural features h, the Bottleneck projects h into z∈R^N×C, and Sem-ViT un-projects z into a high-dimensional representation.This establishes separate compact generation and semantic representations within a single backbone.
  • Representation flow: The downstream LLM uses only Sem-ViT output s, while the pixel decoder reconstructs images from z only during tokenizer training.The supplied passage states that this division is retained in the extended design.
  • Video extension: HYDRA-X extends the image architecture to videos through explicit temporal causality, hierarchical patchify, and a Decompressor introduced in Section 4.These mechanisms adapt the overall design from image inputs to video inputs.

4 HYDRA-XTOK: Holistic Visual Tokenization in a Single ViT

HYDRA-XTOK is a single-ViT visual interface that combines compact reconstruction with semantic distillation for image and video latents. Ablations show that local tubelet attention, hierarchical temporal compression, and Decompressor-enabled dual-teacher supervision provide the strongest reconstruction, understanding, and generation behavior.

  • Tokenizer objective: HYDRA-XTOK combines reconstruction and semantic distillation losses so its compact latent remains pixel-faithful while encoding semantic features.Gen-ViT and Sem-ViT are initialized from SigLIP 2, and UMM-side ablations use Qwen2.5-1.5B.
  • Temporal attention: Restricting causal attention to a 2-frame tubelet yields the best reconstruction, while wider temporal receptive fields degrade local detail.Full bidirectional and all-past causal attention both perform worse than Tubelet attention.
  • Temporal compression: Distributing temporal compression across two 2× patchify stages consistently outperforms a single 4× patchify at the same compression ratio.The hierarchical schedule progressively folds the temporal axis across two stages, with the anchor frame zero-padded during each operation.
  • Semantic supervision: The lightweight Decompressor lifts compressed video features to native temporal length, enabling dense alignment with both image and video teachers during tokenizer training.It is discarded afterward, so the LLM still receives the compact Sem-ViT output.
  • Semantic supervision: Dual image- and video-teacher distillation makes semantic supervision indispensable, with video-teacher supervision producing the strongest video understanding while preserving image performance.The same configuration also achieves the best image generation and editing scores, whereas bidirectional Sem-ViT attention uniformly degrades every metric.

5 HYDRA-X: Advancing Unified Multimodal Models with Holistic Tokenizers

HYDRA-X uses one shared holistic tokenizer and a common Gen-ViT-based architecture to support five tasks, with task specialization limited to output decoding heads. For editing, it moves source-target interaction into the tokenizer’s semantic stage, improving reconstruction consistency and benchmark performance without added parameters.

  • Unified architecture: HYDRA-X interleaves text and visual tokens for a shared LLM backbone with autoregressive language and flow-matching vision heads.The model follows a native unified multimodal model template with HYDRA-XTOK producing visual tokens.
  • Unified architecture: The same Gen-ViT serves all five tasks, with only the head decoding the LLM output varying by task.Training is end-to-end with next-token prediction and rectified flow-matching losses, both weighted by λ1 = 1 and λ2 = 1 by default.
  • Latent-level editing: Independent source-target tokenization leaves editing latents disconnected, forcing the LLM to learn cross-image alignment and causing failures on detail-faithful edits.This pipeline can support high-level semantic edits but does not reliably preserve details.
  • Latent-level editing: STI routes editing pairs as length-2 clips through Sem-ViT tubelet causal attention while keeping Gen-ViT encodings independent, improving alignment without added parameters.Only the semantic stage performs cross-image interaction because structural reconstruction benefits from independent encoding.
  • Latent-level editing: Nearly 7 dB Recon-PSNR and 0.4 ImgEdit gains result from STI, while GenEval improves by +1.46 and most non-editing benchmarks also gain.Recon-PSNR measures source reconstruction on ImgEdit and directly probes editing consistency.

6 Main Results

HYDRA-X delivers strong results across image and video understanding, visual generation, and editing with a shared ViT tokenizer and 7B backbone. It matches or exceeds comparable unified baselines, leads 7B-scale generation and editing comparisons, and narrows the gap to stronger video systems.

  • Implementation: HYDRA-X uses Qwen2.5-7B-Instruct as its reported LLM backbone, while a matched 1.5B variant supports methodological ablations.Its tokenizer uses a symmetric ViT encoder/decoder with 3D RoPE and a lightweight 4× temporal upsampler.
  • Image understanding: HYDRA-X matches or exceeds 7B native UMM baselines on most image-understanding metrics, including OCR- and chart-heavy tasks.Evaluation covers AI2D, MME, MMMU, OCRBench, MMBench, RealWorldQA, ChartQA, DocVQA, and InfoVQA.
  • Video understanding: HYDRA-X improves over reported 1.5B and 7B unified baselines on comparable video-understanding benchmarks, while remaining below several strongest dedicated or proprietary video LMMs.The model uses one ViT tokenizer shared across understanding, generation, and editing.
  • Visual generation: HYDRA-X is the strongest 7B-scale unified baseline on every reported GenEval and WISE column and remains competitive with ≥14B unified models on Overall scores.Video generation uses 17-frame outputs at 640 × 384 and reports VBench Quality Score, Semantic Score, and Total score.
  • Visual generation: +2.15 Total on VBench, with additional leads in semantic-heavy dimensions including Object Class, Human Action, and Scene.The results suggest that dual-teacher distillation transfers semantic structure into the latent while preserving its role in visual synthesis.
  • Image editing: HYDRA-X leads 7B-scale unified models on Ext. (4.04, +1.77), Rm. (4.38, +1.14), ImgEdit Over. (4.34, +0.90), and GEdit G-SC/G-Over. (7.80/7.17).It also beats BAGEL-14B on every column, supporting tokenizer-stage source–target interaction for identity-faithful source preservation.

7 Conclusion … A.2 Tokenizer Pre-training

HYDRA-X unifies image and video tokenization in one native ViT through causal temporal attention, hierarchical temporal patchification, and dual-teacher decompression. Its tokenizer training combines reconstruction and semantic distillation objectives with progressive pre-training, decoder refinement, and latent-space harmonization.

  • 7 Conclusion: HYDRA-X is the first native unified multimodal framework to tokenize images and videos within a single ViT.Its holistic tokenizer supports five tasks through one shared visual backbone.
  • 7 Conclusion: Three design choices—frame-level causal tubelet attention, hierarchical temporal patchify, and dual image-video teacher supervision—convert an image tokenizer into a unified video-image tokenizer.These mechanisms provide reconstruction and semantic capabilities within a compact latent representation.
  • 7 Conclusion: Source and target images are processed as length-2 clips, restoring latent-level coupling lost by independent-encoding editing pipelines.The approach moves source-target interaction into the visual tokenizer rather than treating editing as purely an LLM-side problem.
  • A Training Details: HYDRA-XTOK serves as HYDRA-X’s visual interface by producing latents compact enough for generation, faithful enough for reconstruction, and semantic enough for understanding.Gen-ViT and Sem-ViT are initialized from SigLIP 2.
  • A.1 Tokenizer Training Loss: The tokenizer objective combines reconstruction and semantic distillation, with reconstruction integrating pixel recovery, perceptual fidelity, texture realism, and latent regularization.The reconstruction term uses L1, LPIPS, adversarial GAN, and KL-divergence losses, while distillation aligns Sem-ViT with image and video teachers.
  • A.2 Tokenizer Pre-training: HYDRA-XTOK uses three progressive stages: foundation training, decoder refinement, and representation harmonization.The stages balance foundational representation learning with high-fidelity generative quality.
  • A.2 Tokenizer Pre-training: Foundation training starts from SigLIP-2, uses ImageNet-1.2M and mixed-resolution image-video data, and runs for 300k AdamW iterations at a peak learning rate of 2 × 10−4.Training begins at 256 × 256 resolution before incorporating videos at 256 × 256 and images from 256 to 2048 pixels with hybrid SigLIP-2 / InternVideo distillation.
  • A.2 Tokenizer Pre-training: Decoder refinement freezes the encoder while fine-tuning the 27-layer ViT decoder with GAN loss, then harmonization freezes Gen-ViT and the decoder while updating normalized Sem-ViT features.Normalization removes feature heterogeneity between the two heads and establishes a unified semantic-aware latent space.

A.3 Native Unified Multimodal Models Pre-training · A.4 Ablation Study Training Details

HYDRA-X is pretrained through three progressive stages that align visual and linguistic representations, jointly train multimodal capabilities, and refine quality with curated instruction and generation data. Ablation studies separately specify training setups for multimodal understanding, image generation, and image reconstruction.

  • A.3 Native Unified Multimodal Models Pre-training: HYDRA-X uses three progressive training stages to cultivate a harmonized unified multimodal model.The stages are representation alignment, comprehensive multimodal pre-training, and high-quality instruction fine-tuning.
  • A.3 Native Unified Multimodal Models Pre-training: Stage 1 freezes Qwen2.5-7B-Instruct and tunes vision components on 100M image–text pairs to align visual latents with the linguistic domain.Trainable components include the projector, time-step embedding, and flow head.
  • A.3 Native Unified Multimodal Models Pre-training: Stage 2 unlocks all parameters and jointly trains on 30M understanding samples, 30M generative samples, approximately 2M image-editing samples, and 10M video samples.The understanding and generative samples are strategically filtered from Stage 1.
  • A.3 Native Unified Multimodal Models Pre-training: HYDRA-XTOK pre-training requires an additional 24h on 256h GPUs.Table 8 also defines Data Ratio as Text: Image Caption : Image Generation : Video Caption: Video SFT: Image SFT: Edit.
  • A.3 Native Unified Multimodal Models Pre-training: Stage 3 performs high-quality instruction fine-tuning with 6M multimodal-understanding samples, 1.2M video-instruction samples, 10M aesthetic-filtered images, and 6M synthetic images.The datasets are sourced from LLaVA-OneVision, Pixmo, and LLaVA-Video, with generation data emphasizing aesthetic filtering and fidelity.
  • A.4 Ablation Study Training Details: Ablations evaluate three capabilities: multimodal understanding, image generation, and image reconstruction, using distinct training setups.The supplied setup combines LLaVA-1.5 with LLaVA-Video SFT for understanding and uses Qwen2.5-1.5B, 20M image-caption pairs, and ImgEdit for generation and editing.

B Visual Reconstruction · C Evaluation Details of Multi-modal Understanding Benchmarks · D Tokenizer-Stage Source–Target Interaction: Visual Evidence

HYDRA-X’s holistic tokenizer delivers strong reconstruction under matched-data controls, supports broad multimodal understanding evaluation, and improves editing fidelity by routing source–target pairs jointly through the tokenizer. The evidence attributes gains to the holistic ViT design and tokenizer-stage interaction rather than extra parameters or data.

  • B Visual Reconstruction: HYDRA-XTOK† outperforms RAE† and VAVAE† on every ImageNet metric under matched ImageNet-1.2M training data.The controlled variant isolates architecture from training-data effects.
  • B Visual Reconstruction: 32.96 vs. 32.86 ImageNet PSNR and 0.154 vs. 0.176 rFID show HYDRA-XTOK† exceeding FLUX.1 despite twice the compression ratio.This supports the conclusion that the holistic ViT design, rather than data scale, drives the gain.
  • B Visual Reconstruction: +1.59 dB DAVIS PSNR and +2.22 dB UCF PSNR make fully trained HYDRA-XTOK the strongest unified tokenizer on every video metric.It also more than halves rFVD on both datasets.
  • C Evaluation Details of Multi-modal Understanding Benchmarks: Nine benchmarks evaluate HYDRA-X across general understanding, expert knowledge, document and chart comprehension, and fine-grained visual perception.The protocol includes AI2D, MME, MMMU, OCRBench, MMBench, RealWorldQA, and ChartQA among the reported benchmarks.
  • D Tokenizer-Stage Source–Target Interaction: Visual Evidence: Tokenizer-stage source–target interaction jointly routes xc and xt through a shared Sem-ViT with tubelet causal attention for identity-faithful editing.This replaces independent encoding while leaving the rest of the architecture and parameter count untouched.
  • D Tokenizer-Stage Source–Target Interaction: Visual Evidence: Nearly 7 dB of source-reconstruction PSNR is recovered by tokenizer-stage interaction without changing architecture or parameter count.HYDRA-X-STI routes the pair as a length-2 clip, whereas HYDRA-X-Indep uses independent Sem-ViT branches.
  • D Tokenizer-Stage Source–Target Interaction: Visual Evidence: HYDRA-X-STI preserves scene identity in still-life and car examples, while HYDRA-X-Indep produces fragmented mosaics or re-imagines the vehicle.The independent variant hallucinates local fruit details, changes the car, removes occupants, and erases on-screen text.

E Limitations · F Broader Impacts · HYDRA-X-Indep HYDRA-X-STI Input

HYDRA-X remains limited by training scale, resource constraints, and evaluation on only a 7B dense LLM, while its strong text-to-image capabilities introduce misuse risks. The proposed HYDRA-X-STI tokenizer-stage interaction preserves identity-sensitive details that independent encoding can lose, and responsible deployment measures are advocated to manage risks.

  • E Limitations: Training-data and parameter scale remains a bottleneck for capturing the full complexity of high-dimensional video distributions.The paper identifies current model and data scale as a limitation.
  • E Limitations: Resource constraints prevented exploration of long video generation and video editing, despite their status as natural extensions of the holistic encoder.These capabilities are left for future work.
  • E Limitations: For fair comparison, HYDRA-X was instantiated only with a 7B dense LLM, leaving pairings with more advanced backbones unexplored.The passage begins citing MoE as an example of a more advanced backbone, but the supplied text is truncated.
  • F Broader Impacts: HYDRA-X’s strong text-to-image generation capabilities could enable misleading or fabricated visual content, including deepfakes, disinformation, or impersonation.These are identified as potential downstream risks of the unified framework.
  • F Broader Impacts: The paper advocates content watermarking, provenance tracking, gated model access, and usage guidelines to mitigate deployment risks.These measures are presented as responsible release practices.
  • HYDRA-X-Indep HYDRA-X-STI Input: HYDRA-X-STI preserves identity-sensitive details such as object layout, characters, and on-screen text that HYDRA-X-Indep loses.Figure 5 compares independent Sem-ViT encoding with joint encoding through tubelet causal attention; the variants share every other architectural component.
  • F Broader Impacts: Community-driven safety standards are proposed as a way to manage the risks associated with unified multimodal architectures.The paper also emphasizes substantial positive societal value from advancing scientific understanding.

G Additional Main Results · H Qualitative Comparisons · H.1 Image Reconstruction at 512×512

The appendix expands HYDRA-X’s benchmark results into per-category analyses and presents qualitative comparisons across supported tasks, including 512×512 image reconstruction against specialized and unified tokenizer baselines.

  • G Additional Main Results: GenEval is broken down across single-object, two-object, counting, color, position, and color-attribute prompts to identify models’ compositional strengths.Table 10 excludes models with ≥14B parameters from ranking and marks methods using LLM rewriters with †.
  • G Additional Main Results: WISE reports per-category performance across culture, time, space, biology, physics, and chemistry, complementing GenEval’s geometric and compositional probes.Table 11 excludes models with ≥14B parameters from ranking.
  • G Additional Main Results: ImgEdit-Bench expands editing evaluation across nine operations, including addition, removal, background replacement, style transfer, and compositional edits.The dimensions are Add, Adj. (Alter), Ext. (Extract), Rep. (Replace), Rm. (Remove), Bg. (Background), Sty. (Style), Hyb. (Compose), and Act. (Action).
  • G Additional Main Results: VBench expands the main-paper QS/SS/Total summary to fourteen dimensions covering visual quality, motion, dynamics, semantic correctness, and compositional reasoning.The table includes dimensions such as subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, imaging quality, object class, multiple objects, human action, color, spatial relationship, and scene.
  • H Qualitative Comparisons: The qualitative section compares HYDRA-X across five supported tasks with representative unified multimodal and task-specialized baselines, organized by task and resolution.The comparisons are presented as qualitative results rather than additional quantitative benchmark scores.
  • H.1 Image Reconstruction at 512×512: At 512×512, image reconstruction compares HYDRA-X with FLUX, MingTok, AToken, and RAE across texture, fine-edge, and small-text fidelity.These baselines comprise a dedicated image VAE, unified UMM tokenizers, and the recently proposed RAE.
  • H.1 Image Reconstruction at 512×512: The reconstruction comparison presents the input alongside RAE, MingTok, AToken, FLUX, and HYDRA-X outputs for direct visual inspection.The displayed ordering is Input, RAE, MingTok, Atoken, FLUX, and HYDRA-X.

H.2 Image Reconstruction at 1280×768 … H.6 Image Editing

The appendix qualitatively evaluates HYDRA-X across high-resolution image reconstruction, video reconstruction, image and video generation, and instruction-guided image editing. These comparisons examine fine-detail handling, motion-sensitive consistency, diverse synthesis, and editing quality against dedicated or representative baselines.

  • H.2 Image Reconstruction at 1280×768: At 1280×768, HYDRA-X is qualitatively compared with Wan 2.2, AToken, and FLUX to stress-test dense fine-detail reconstruction beyond the training resolution.The evaluation focuses on text, foliage, and small structural elements under a stretched spatial token budget.
  • H.4 Image Generation: HYDRA-X image-generation samples span realistic photography, stylised illustration, compositional scenes, and knowledge-driven prompts to characterise coverage and aesthetic quality.The results are presented qualitatively as text-to-image samples.
  • H.5 Video Generation: HYDRA-X video-generation samples cover varied subjects, scenes, and motion patterns, illustrating temporally coherent synthesis under the shared UMM backbone.The section presents qualitative text-to-video results.
  • H.5 Video Generation: One video-generation example depicts snow-capped mountains with clouds forming “HYDRA-X,” cinematic lighting, realistic cloud movement, and a slow sweeping camera pan.The prompt specifies a photorealistic wide shot and natural blending with surrounding clouds.
  • H.6 Image Editing: Instruction-guided image edits compare HYDRA-X with BAGEL, OmniGen2, Qwen-Image-Edit, and Step1X-Edit to assess identity preservation, instruction adherence, and visual quality.The baselines include unified multimodal models and editing-specialised generators.
Loading 2606.13289v1…