PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Vision-Language System for Multimodal Understanding and Generation

Yifan Xu, Baochen Xiong, Xiaoshan Yang, Donglin Di, Yaowei Wang, Changsheng Xu

arXiv:2608.20382v1cs.CLcs.CV

TL;DR

Libra addresses the challenge of integrating vision and language while preserving modality-specific modeling for multimodal understanding and generation. It decouples self-modal modeling from cross-modal interaction through separate vision and language branches, cross-modal bridges, and switch modules, with Libra-1 targeting understanding and Libra-2 unifying understanding and generation. Across understanding and generation evaluations, the Libra series demonstrates strong benchmark performance, including leading GenEval results and over 80% relative FID reductions over a Libra-1 variant on COCO and MJHQ.

  • Problem

    MLLMs need effective multimodal comprehension while preserving vision- and language-specific representations, but unified modeling can create modality imbalance and visual information loss.

  • Method

    Libra uses separate vision and language branches with cross-modal bridges, switch attention, and switch FFN modules to decouple self-modal modeling from cross-modal interaction.

  • Results

    The Libra series shows strong performance across understanding and generation benchmarks; Libra-2 achieves over 80% relative FID reductions over a Libra-1 variant on both COCO and MJHQ.

  • Takeaways & Limitations

    A decoupled vision-language system can improve both multimodal understanding and generation, with Libra-2 combining accurate understanding and high-quality image generation.

Abstract

from arXiv · show

We introduce a new architecture design for multimodal large language models (MLLMs), Libra, capable of both multimodal understanding and generation. Libra architecture contains one vision system and one language system, connected by cross-modal bridges. This design decouples self-modal modeling and cross-modal interaction, enabling each modality to learn its unique representations while maintaining effective cross-modal comprehension. The decoupling is mainly achieved in a switch attention module and a switch FFN module, which dynamically routes the computation flow for self-modal modeling and cross-modal interaction scenarios. We evaluate the effectiveness in two important settings: \textbf{Libra-1} for the understanding-only image-to-text setting, and \textbf{Libra-2} for unified image-to-text understanding and text-to-image generation. In addition to the architecture design, we discuss various improvements on tokenization, positional encoding, and supervision. Experiments demonstrate that the dedicated Libra design enables mutual improvements on multimodal understanding and generation, achieving strong performance on both understanding and generation benchmarks.

I. INTRODUCTION

Existing multimodal architectures trade off modality integration against training stability and balanced learning. Libra addresses this tension by decoupling self-modal modeling from cross-modal interaction while supporting understanding-only and unified generation settings.

  • Understanding-only models support image-to-text comprehension, whereas unified models additionally support text-to-image generation but typically incur greater training complexity and instability.
  • Modality imbalance arises because language is symbolically structured and semantically dense, while images contain fine-grained detail with narrower knowledge coverage in available datasets.
  • Unified architectures offer flexible joint modeling but depend on large-scale, balanced vision-language data, whereas cascade architectures mitigate imbalance through a pretrained visual backend at the cost of scalable multimodal understanding.
  • Libra separates vision and language into self-modal streams connected by cross-modal bridges, combining bottom-up modality learning with modular mitigation of modality imbalance.
  • Switch attention uses standard attention for self-modal modeling and bridged feature transformations for cross-modal interaction, preserving modality-specific representations.
  • Libra-1 targets understanding-only tasks with unified discrete autoregression and hybrid tokenization, while Libra-2 unifies understanding and generation using masked visual generation, Unified RoPE, and continuous visual features.
  • Across more than 10 understanding and generation benchmarks, Libra reports strong performance, with Libra-2 substantially advancing unified multimodal understanding and generation.

III. PRELIMINARIES

MLLMs combine text-token processing with image-derived visual embeddings, while visual tokenization and positional encoding adapt Transformer modeling to multimodal inputs. The preliminaries describe quantization-based visual tokens, RoPE’s relative-position mechanism, and Libra’s architecture overview.

  • MLLMs extend the text-only pipeline by interleaving language embeddings with visual embeddings produced by an image encoder.
  • Understanding-only models commonly supervise language tokens, while unified models may discretize visual features so images and text share a next-token prediction objective.
  • Visual quantization encodes images into spatial features, matches them against a learnable VQ codebook, and uses the matched vectors for reconstruction.
  • Lookup-free Quantization replaces codebook embeddings with an integer set, eliminating embedding lookup from the quantization process.
  • RoPE rotates query and key vectors according to token positions, making attention scores depend on relative position rather than absolute position.
  • The Libra overview centers on switch attention and switch FFN modules that decouple self-modal modeling from cross-modal interaction across its vision and language systems.

D. Masked Image Generation in Continuous Space

Masked image generation predicts randomly masked image tokens from visible context, extending masked modeling to continuous visual representations with iterative generation at inference.

  • A special token x∗ replaces the features at positions selected by the mask set M, while unmasked positions retain their original tokens.The spatial token sequence may contain continuous latent features or discrete codebook indices.
  • Masked modeling predicts each masked token from its unmasked context under a masking distribution.The objective becomes cross-entropy for discrete tokens and reconstruction-style losses for continuous tokens.
  • MAR extends masked image generation to continuous space by attaching a lightweight diffusion head to each spatial token.Images are encoded with a VAE into continuous tokens before masking and Transformer processing.
  • For masked positions, per-token diffusion heads predict reconstructed features and minimize a diffusion loss.The resulting loss is described as reconstruction-style.
  • At inference, MAR begins with a fully masked sequence and iteratively accepts some predictions while re-masking the remainder.Iterations continue until all positions are accepted.

IV. DECOUPLED VISION-LANGUAGE SYSTEM

Libra separates vision and language into self-modal streams connected by cross-modal bridges, using switch attention and switch FFN to route modality-specific and interactive computation.

  • Overall Pipeline: Libra partially decouples vision and language systems while retaining cross-modal interaction through a modular pipeline.Its principles are modularity, interactivity, and end-to-end integrity for both modalities.
  • Overall Pipeline: Libra supervises both image and language sides, using next-token prediction for language and setting-specific visual objectives.The understanding-only setting uses discrete autoregression, whereas unified generation uses continuous-space masked generation.
  • Core Architecture: Switch attention dynamically routes tokens between self-modal modeling and cross-modal interaction pathways.The mechanism is intended to prevent self-modal generation from interfering with cross-modal comprehension during training.
  • Core Architecture: A low-rank query adaptation with rank 8 provides lightweight finetuning of modality-specific query patterns during cross-modal interaction.The design preserves modality-specific feature structures while enhancing cross-modal alignment.
  • Core Architecture: In cross-modal attention, bridge transformations project queries, keys, and values into a new feature space before alignment.Self-modal attention retains the standard operation, while cross-modal bridges avoid directly aligning modality-specific feature spaces.
  • Core Architecture: Removing the cross-modal bridge produces repetitive image-completion patterns and poor language instruction following in text-to-image generation.The bridge is described as an intermediate adapter that enables controlled information exchange while preserving modality-specific characteristics.
  • Core Architecture: Switch FFN routes visual feature subsets to separate understanding and generation experts, then concatenates their outputs.It is used only in the vision system to avoid significantly disturbing pretrained language-model weights.

A. Model Overview

Libra-1 validates the framework for understanding-only image-to-text tasks, combining discrete modeling with continuous visual signals and a CLIP-based LFQ tokenizer.

  • Model Overview: Libra-1 targets understanding-only image-to-text tasks and models both image and text sides end-to-end.Because it does not generate images, Libra-1 uses a standard FFN instead of switch FFN.
  • Model Overview: Libra-1 tokenizes images and language into discrete tokens for a unified next-token-prediction paradigm.Image sequences receive BOI and EOI markers, while a newline token separates images and text.
  • Hybrid Tokenization: Discrete visual modeling can lose fine-grained visual information and fail to exploit pretrained vision-encoder features.The passage identifies these as obstacles to effective vision-language comprehension.
  • Hybrid Tokenization: Hybrid tokenization concatenates continuous encoder signals with discrete token embeddings to form Libra-1’s visual input.Continuous features are extracted, quantized into IDs, embedded through the MLLM table, and concatenated along the channel dimension.
  • Hybrid Tokenization: Lookup-free quantization largely addresses poor reconstruction when a frozen CLIP encoder replaces VQGAN’s original image encoder.LFQ avoids emulating CLIP’s highly semantic input features during quantization.

C. Unified Discrete Supervision

Libra’s supervision and positional design differ by task setting: discrete autoregression supports understanding-only modeling, while UniRoPE addresses joint 2D-image and 1D-text representation.

  • Unified Discrete Supervision: Libra-1 applies unified sequential next-token prediction across the multimodal input sequence.The objective is implemented as a discrete cross-entropy classification loss over image and text tokens.
  • Unified Discrete Supervision: Because Libra-1 requires full continuous visual signals as inputs, its discrete autoregressive vision objective regularizes visual learning rather than enabling image generation.The continuous signal is unavailable before an image is generated.
  • Unified Multimodal Extension: Libra-2 extends the framework to unified multimodal understanding and generation using switch FFN, UniRoPE, continuous visual tokenization, and continuous-space masked generation.These changes address architecture, representation, and supervision requirements for the unified setting.
  • Unified Rotary Position Encoding: UniRoPE is designed to preserve two-dimensional image structure while integrating with one-dimensional text representations.The method is described as offering greater representational capacity than previous encoding strategies and seamless integration with existing LLM architectures.
  • Unified Rotary Position Encoding: UniRoPE splits rotary frequencies into horizontal and vertical groups for image positions.For text, where i = j, it degenerates to standard RoPE and remains compatible with existing LLMs.

C. Continuous-Space Visual Tokenization

Libra-2 combines continuous visual features with multimodal masked prediction to support direct image generation while preserving spatial visual structure. Its training uses causal language modeling and bidirectional continuous visual modeling.

  • Continuous visual representation: Libra-2 encodes images with a VAE and semantically enriches the resulting features using a CLIP-style encoder through cross-attention.The VAE uses downsampling factor 16 and latent channel dimension 16; the CLIP-style encoder is SigLIP-Large/384.
  • Adaptive supervision: Libra-2 trains language with discrete autoregression and vision with masked prediction over continuous visual tokens.The language branch uses cross-entropy on next-token IDs, while the visual branch uses mean squared error on masked continuous tokens.
  • Motivation: Continuous-space visual modeling addresses the need for direct image generation rather than generating prompts for a separate visual backend.The paper positions this as an open challenge for multimodal large language models.
  • Attention structure: Text tokens use causal attention, whereas visual tokens attend bidirectionally to multimodal context, preserving the visual input’s 2D spatial structure.Masked visual tokens are predicted from all visual tokens and preceding textual tokens.
  • Loss balancing: The unified objective combines language and visual modeling losses, with the visual MSE reweighted by α = 0.25 because it is smaller than the text CE loss.The objective is written as L = −α log P(t | v) − log P(v | t).

VII. EXPERIMENTS

The Libra series uses separate training configurations for understanding-only and unified understanding-generation models. Libra-1 follows staged multimodal training, while Libra-2 interleaves understanding and generation data.

  • Initialization: Libra-2 uses officially pretrained VAE and SigLIP encoders without further tokenizer training, while the vision branch and cross-modal bridges are initialized from scratch.The resulting models have 11B parameters for Libra-1 and 3B for Libra-2.
  • Tokenization: Libra-1 uses hybrid vision inputs combining continuous signals with discrete tokenized embeddings.Its image tokenizer enlarges the vision vocabulary to 218 using lookup-free quantization and predicts in two concatenated codebooks of size 29.
  • Training variants: Libra-1 and Libra-2 are trained as distinct variants targeting understanding-only and unified multimodal understanding-generation settings.Their training pipelines are organized as multi-stage pretraining followed by instruction fine-tuning or additional generation-oriented stages.
  • Libra-1 pipeline: Libra-1 pretraining uses 50M image-text pairs from COYO-700M and CC12M, plus 500K COCO pairs, while freezing the language branch.The full model is later fine-tuned on 665K high-quality supervised samples.
  • Libra-2 pipeline: Libra-2 pretraining uses 200M LAION-COCO image-text pairs in both image-to-text and text-to-image formats.The sampling probabilities are 20% for understanding and 80% for generation.

C. Multimodal Understanding

Experiments report strong multimodal understanding and image-generation performance for Libra, while also identifying resolution dependence and generation failures involving detail, text, and spatial relations.

  • Multimodal understanding: Libra-1 surpasses contemporary MLLMs on multimodal understanding, while Libra-2 underperforms on image captioning.The paper attributes Libra-2’s captioning weakness to a mismatch between long training captions and concise COCO captions.
  • General-purpose benchmarks: Libra-2 substantially outperforms Show-O across unified benchmarks, and increasing resolution from 256^2 to 384^2 yields significant gains.The comparison uses a similar data scale and composition, while the resolution change improves multimodal understanding.
  • General-purpose benchmarks: Despite far fewer parameters than Libra-1, Libra-2 achieves comparable results on VQAv2, MME, POPE, and SEED.The authors report this pattern as evidence that multimodal generation can benefit multimodal understanding.
  • Text-to-image generation: Libra-2 ranks among leading GenEval models and surpasses the previous state of the art on counting by 12%.The paper connects counting performance to interpreting numerals and grounding them in visual realizations.
  • Visual quality: Libra-2 achieves the best FID among compared unified models and reduces Libra-1’s relative FID by over 80% on both COCO and MJHQ.COCO emphasizes realism, while MJHQ emphasizes aesthetics; FID primarily reflects vision modeling.
  • Failure cases: Libra-2’s image-generation failures include insufficient visual detail, illegible text, and unresolved spatial relations.The paper links these issues to limited resolution or VAE constraints, sparse textual signals, and difficulty modeling relations such as “to the right of.”
  • Ablations: Removing switch attention causes performance degradation, while adding a vision branch without a cross-modal bridge yields limited gains.The ablations support the contribution of both the decoupled architecture and cross-modal bridging.
  • Ablations: Removing Libra-1’s contiguous visual signal produces a 14.2% degradation on SEED, supporting its importance for visual perception.The ablation retains only discrete embeddings as visual inputs.

F. Analysis on Unified Modeling

Libra-2 combines architectural, positional, and tokenization improvements for unified multimodal understanding and generation. Ablations show that these design choices materially affect both tasks, while understanding and generation can mutually reinforce one another.

  • Impact of Switch FFN: Removing Switch FFN decreases MME by 69 points and GenEval by 2%.The authors attribute this degradation to a granularity mismatch caused by forcing understanding and generation into one shared feature space.
  • Impact of unified RoPE: Replacing UniRoPE with vanilla RoPE substantially degrades Libra-2 performance.UniRoPE reconciles two-dimensional visual layouts with one-dimensional causal language sequences for cross-modal modeling.
  • Impact of continuous image modeling: Hybrid tokenization prevents image generation and reduces multimodal understanding by 59 MME points.The Libra-1 strategy introduces discrete noise into continuous features when applied to Libra-2’s unified setting.
  • Mutual impact of understanding and generation: Adding generation training improves VQAv2 from 76.0% to 77.5% and MMB from 58.2% to 60.0% over the understanding-only variant.Conversely, understanding training improves the unified model’s performance across all three generation benchmarks relative to the generation-only variant.
  • Impact of sampling steps: Increasing masking sampling steps consistently improves GenEval, whereas diffusion-step count has only a modest effect.Masking steps control whole-image generation granularity; diffusion steps operate on individual image tokens.

VIII. CONCLUSION

The Libra series presents a decoupled vision-language system designed to improve multimodal understanding and generation. Its conclusion emphasizes switch-based computation and continuous-space masked image generation as central design elements.

  • Conclusion: Libra uses switch attention and switch FFN modules to decouple vision-language computation.The authors report diverse attention patterns across layers, suggesting reduced learning redundancy.
  • Conclusion: Libra-2 integrates continuous-space masked image generation into a unified multimodal language model.The resulting system is described as providing accurate multimodal understanding and high-quality image generation.
  • Conclusion: The authors conclude that a decoupled vision-language design can improve both multimodal understanding and generation.They advocate integrating vision and language beyond simple modality alignment.
Loading 2608.20382v1…