HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization

1. Introduction

• Understanding needs semantic abstractions while generation needs compact structural primitives, and existing tokenizers trade off fidelity, alignment, or shared-representation convergence.The paper identifies decoupled encoders, VAE-stacked representation encoders, and shared encoders as approaches with different limitations.Source: §1 · p. 2
• HYDRA-TOK is a representation-harmonized pure ViT that progresses from Gen-ViT structural primitives to Sem-ViT semantic encoding through a compress-and-reconstruct GSB.The bottleneck filters noise in compact features before restoring their dimensionality for semantic distillation; the approach is presented as avoiding quantization errors.Source: §1 · pp. 2–3
• HYDRA surpasses native UMMs by ∼10.0 points across eight understanding benchmarks and posts rFID 0.08, GenEval 0.86, DPG-Bench 86.4, and WISE 0.53.The paper reports state-of-the-art generation performance on GenEval, DPG-Bench, and WISE.Source: §1 · p. 3
• HYDRA unifies understanding and generation in one parameter space, processing signals as sequences with an autoregressive text head and a rectified-flow image-generation head.The paper reports that joint training consistently outperforms separate training for both generation and understanding.Source: §1 · p. 3

2.1. HYDRA-TOK

• HYDRA-TOK preserves fine-grained spatial covariance in Gen-ViT features, retaining structural primitives needed for high-fidelity synthesis.Its subsequent Sem-ViT maps the bottleneck representation into a high-dimensional semantic space.Source: §2.1 · pp. 3–4
• The Generation-Semantic Bottleneck compresses intermediate features into a compact probabilistic space to filter noise before semantic abstraction.Sem-ViT then maps the bottleneck representation into a high-dimensional semantic space.Source: §2.1 · pp. 3–4
• A lightweight flow-matching decoder recovers high-frequency image details, while LPIPS and adversarial losses refine perceptual fidelity and texture realism.The reconstruction objective combines flow matching, LPIPS, and GAN losses.Source: §2.1 · p. 4

2.2. HYDRA

• HYDRA feeds visual latents and text embeddings as a unified sequence to a shared LLM, with specialized heads for autoregressive text prediction and image-flow prediction.The shared backbone is specialized only at the input and output layers.Source: §2.2 · pp. 4–5

2.3. Training Recipe

• Training progresses from tokenizer reconstruction and semantic distillation to input alignment, joint multimodal pre-training, and instruction fine-tuning.The later stages use balanced understanding and generation samples, followed by curated instruction-tuning and generation data.Source: §2.3 · p. 5

3.1. Main Results

• The 12 + 12 layer allocation is the architectural sweet spot, peaking across evaluated metrics; C = 64 is the representational sweet spot, while 24+0 performs worst.Source: Figure 5 · p. 6, §3.1 · pp. 6–7
• HYDRA reaches GenEval 0.86, DPG-Bench 85.51, and WISE 0.53, while its 1.5B understanding average is 63.1 versus Show-o2’s 53.2.Source: Table 2 · p. 6, Table 3 · p. 7, §3.1 · p. 8
• On ImageNet-1K, HYDRA-TOK achieves PSNR 36.39 and rFID 0.08, exceeding BLIP3-o’s PSNR 14.71 and FLUX-VAE’s PSNR 32.74.Source: §3.1 · p. 7

3.2. Ablation Analysis

• Ablations report that teacher initialization and distillation improve tokenizer synergy, while all three HYDRA training stages contribute to the full recipe’s peak performance.Source: Figure 6 · p. 7, Table 4 · p. 7, §3.2 · p. 8

3.1. Main Results

• Joint training improves generation convergence and final fidelity over generation-only training, while understanding performance overtakes understanding-only training after about 6k steps.Source: Figure 7 · p. 8, §3.1 · p. 8
• At 7B, HYDRA scores 57.7 on OCRBench versus Show-o2’s 32.4, with its continuous, uncompressed ViT flow preserving high-frequency character details.Source: §3.1 · pp. 7–8

• HYDRA-TOK uses the Generation-Semantic Bottleneck to harmonize structural primitives and semantic abstractions, providing HYDRA a cohesive basis for single-parameter-space integration.Source: §4 · p. 8
• The framework is presented as satisfying input-representation unification, information-flow coherence, and learning-process compatibility.Source: §4 · p. 8
• The authors conclude that understanding and generation are complementary forces that drive mutual enhancement through rational unification.Source: §4 · p. 9

Impact Statement

• The work aims to advance machine learning by integrating visual understanding and generation within a single parameter space.Source: Impact Statement · p. 9

A.1. Unified Multimodal Models

• Composite UMMs use specialized pretrained experts and lightweight adapters, whereas native UMMs train holistically in a shared parameter space but face representation trade-offs.Source: §A.1 · p. 13

A.2. Unified Tokenizers and Representations

• Prior tokenizer approaches are described as compromising reconstruction fidelity, single-stream information flow, textural detail, or perception capacity; HYDRA-TOK instead uses a progressive pure ViT.Source: §A.2 · p. 13

B. Limitations and Future Work

• The 300M-parameter HYDRA-TOK may bottleneck fine-grained tasks, while the 7B model’s 100M image-text pairs constrain generalizability and world-knowledge absorption versus state-of-the-art foundation models.Future work will scale both the model architecture and training dataset.Source: B. Limitations and Future Work · p. 13
• Figure 8 tracks Avg QA, Geneval, and rFID as tokenizer data grows from 1.2M to 20M, and CKNNA across 24 HYDRA-TOK layers against teacher ViT representations.Source: Figure 8 · p. 14

C.1. Decoder Size

• A 358.44M decoder improves PSNR from 35.85 to 36.84, SSIM from 0.96 to 0.97, and rFID from 0.17 to 0.14 versus a 144.26M decoder.The largest decoder variant achieves the best performance across the evaluated reconstruction metrics.Source: §C.1 · p. 14, Table 5 · p. 14

C.2. Scaling HYDRA-TOK training data

• At 20M image-text pairs, Geneval exceeds 45 and rFID reaches approximately 0.08, while Avg QA remains above 61 across the 1.2M, 4M, and 20M regimes.Avg QA averages POPE, MMBench, MMMU, AI2D, and RealWorldQA; it is evaluated using the LLaVA-1.5 setting.Source: §C.2 · p. 14, Figure 9 · p. 15

C.3. Visualization of CKNNA

• On 10,000 ImageNet2012 validation images, HYDRA-TOK aligns strongly with teacher InternViT in early layers; generation-understanding CKNNA is 0.10 versus Show-o2’s 0.03, rising to 0.13 during unified training.Source: §C.3 · p. 15

C.4. Visualization of t-SNE

• HYDRA’s generation and understanding features form distinct class clusters, unlike the disparately distributed features of UniFlow and Show-o2.The t-SNE visualization compares both HYDRA branches with those baselines.Source: §C.4 · p. 15, Figure 9 · p. 15

D. Training details

• HYDRA-TOK’s foundation stage trains the full encoder-decoder end-to-end for 300k iterations with reconstruction and distillation objectives on ImageNet-1.2M, CC-12M, and SAM-10M.The global batch size is 256 and the learning rate is 2 × 10^-5.Source: D. Training details · p. 15
• In decoder refinement, the pretrained encoder is frozen and the decoder is fine-tuned for 100k iterations using reconstruction and GAN losses, with distillation removed.Source: D. Training details · p. 16
• HYDRA’s three-stage curriculum keeps resolution at 448 × 448 and batch size at 1024, while data ratios shift from 0:1:3 to 0:2:2 to 1:2:2.The stages run for 150K, 100K, and 20K steps; stage I trains the Vision Head and Sem-ViT, while stages II–III train the LLM and Sem-ViT jointly with a reduced Vision Head learning rate.Source: D. Training details · p. 16, Table 6 · p. 16

D.1. Ablation study training details

• Multimodal understanding follows the LLaVA-1.5 protocol and reports Avg QA across POPE, MMBench, MMMU, AI2D, and RealWorldQA.Source: §D.1 · p. 16
• Image generation replaces Nitro-E’s VAE with HYDRA-TOK, trains for 15k iterations on SAM-1B and JourneyDB, and evaluates on GenEval.Source: §D.1 · p. 16
• Image reconstruction trains on ImageNet-1k (1.2M) for 150k iterations and evaluates quality using rFID.Source: §D.1 · p. 16

D.2. Representation-harmonized co-promotion experiment training details

• Joint training is compared with generation-only training across GenEval stages and with understanding-only training after 20K Stage III steps.Both Show-o2 and HYDRA were trained on the same 100M dataset for 150K steps in the generation comparison.Source: §D.2 · p. 16

D.3.1. IMAGE UNDERSTANDING

• Image understanding uses about 22M captioned images for Stage I, continues with the same sources in Stage II, then uses 3.2M instruction-tuning examples in Stage III.The Stage I sources include ImageNet, DataComp-1B, COYO-700M, SA-1B, and DenseFusion; Stage III data comes from LLaVA-One-Vision.Source: §D.3.1 · pp. 16–17

D.3.2. IMAGE GENERATION

• Image generation training progresses from about 78M Stage I samples to added synthetic data in Stage II, then a filtered 5M-image subset for Stage III fine-tuning.Stage II adds 10M Flux-distilled images; Stage III also uses JourneyDB and DenseFusion-1M.Source: §D.3.2 · p. 17

F.1. Multi-modal Understanding Benchmarks

• Eight benchmarks span general perception, reasoning, expert knowledge, OCR, diagram, and real-world spatial tasks; Figure 10 tracks six benchmarks across training steps.The benchmark suite includes MME, MMBench, MMStar, SEED-Bench, MMMU, AI2D, OCRBench, and RealWorldQA.Source: §F.1 · pp. 17–18, Figure 10 · p. 18

F.2. CKNNA

• CKNNA measures local representational alignment using mutually close neighbors; experiments sample 10,000 ImageNet validation images with k = 10.Unlike global pairwise correlation, CKNNA restricts comparison to pairs close in both representation spaces.Source: §F.2 · pp. 18–20

G.1. Image Reconstruction

• Figure 11 presents qualitative comparisons of HYDRA image reconstruction.The caption identifies RAE as a comparison, while the accompanying passage lists SD1.5, VTP, and FLUX.Source: Figure 11 · p. 21, §G.1 · p. 21

G.2. Image Generation

• Figures 12–13 compare text-to-image generations from HYDRA with Harmon (Wu et al., 2025d), Show-o2 (Xie et al., 2025a), Bagel (Deng et al., 2025a), Janus-pro (Wu et al., 2025b), and Blip-3o (Chen et al., 2025a).Source: Figure 12 · p. 22, Figure 13 · p. 23

G.3. Multimodal Image Understanding

• Qualitative examples show HYDRA producing structured text from complex visuals, including LaTeX and Python for formulas and charts and details from concept maps.Source: Figure 14 · p. 24