PaperlayerView exact revision on arXiv

Source-linked AI summary

C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Space

Yujie Shen, Lianlei Shan

arXiv:2608.28603v1cs.AI

TL;DR

Existing unified multimodal models rely largely on statistical correlations and lack structural consistency, leading to semantic drift, weak compositional generalization, and unstable interventions. C3-UniMM introduces shared causal representations, unified decoding, Super Alignment, and cycle consistency; experiments report strong results across understanding, video, and text-to-image tasks.

  • Problem

    Existing unified multimodal models primarily rely on statistical co-occurrence rather than explicit cross-modal structural mechanisms, limiting semantic consistency and causal generalization.

  • Method

    C3-UniMM uses a Structured Latent Causal Graph, structured encoding, a Unified Decoding Space, Super Alignment, causal intervention, and cycle consistency.

  • Results

    C3-UniMM substantially outperforms existing unified multimodal baselines across understanding, video comprehension, and text-to-image generation tasks.

  • Takeaways & Limitations

    The framework provides a shared causal semantic structure for multimodal understanding and generation, with reported gains in consistency, controllability, and reversibility.

  • Takeaways & Limitations

    The method assumes that mechanisms governing object states remain invariant across modalities.

Abstract

from arXiv · show

Unified Multimodal Models aim to achieve any-to-any understanding and generation across arbitrary modalities. However, existing methods primarily rely on modeling implicit statistical correlations and lack cross-modal structural consistency constraints. This deficiency leads to profound issues, including semantic drift, poor compositional generalization, and instability under interventions. In this paper, we propose C3-UniMM, a unified multimodal modeling framework based on Causal Cycle Consistency and Super Alignment. Specifically, we introduce a Structured Latent Causal Graph (SLCG) as a shared cross-modal semantic space and design unified multimodal encoding blocks, enabling understanding and generation to be synergistically optimized within the identical causal semantic structure. Furthermore, we propose a Unified Decoding Space to enforce structural preservation and semantic invertibility during the cross-modal generation process. Theoretical analyses demonstrate that our approach significantly enhances both the invertibility and mechanism invariance of cross-modal mappings. Extensive experimental results across multiple understanding, generation, and compositional generalization tasks indicate that C3-UniMM substantially outperforms existing unified multimodal baselines.

1 Introduction

Unified multimodal models seek joint cross-modal understanding and generation, but existing approaches lack structural and causal consistency. C3-UniMM addresses this gap with a shared causal structure, unified encoding and decoding, Super Alignment, and cycle consistency.

  • Unified models aim to perform cross-modal understanding and generation within one framework, unlike traditional specialized understanding or generative models.
  • Real-world modalities share underlying semantic mechanisms, such as object identities, spatial relations, and action sequences.
  • Statistical co-occurrence in current unified models is associated with semantic drift, weak compositional generalization, and unstable generation under interventions.
  • C3-UniMM introduces a Structured Latent Causal Graph as a shared semantic intermediary, reformulating modality translation as xm → G → xn.
  • Unified structured encoding and a Unified Decoding Space jointly optimize understanding and generation within one semantic structure while supporting consistency and reversibility.
  • Super Alignment operates at representation, structure, and mechanism levels, together with causal consistency learning, intervention constraints, and cycle consistency.

2 Related Works

Related work spans multimodal understanding, multimodal generation, and causal or cycle-consistent frameworks. C3-UniMM combines explicit structured causal graphs with bidirectional cycle-consistent transformations to address their remaining structural limitations.

  • Multimodal understanding models: Multimodal understanding models provide strong semantic understanding and cross-modal alignment but generally have weaker generation than specialized synthesis engines.
  • Multimodal generative models: Multimodal generative models synthesize high-fidelity content across modalities but commonly lack mechanistic invariance and structural control.
  • Causal and cycle-consistent frameworks: Causal multimodal frameworks seek invariant generative mechanisms beyond statistical correlations, while cycle consistency preserves information through cross-modal transformations.
  • C3-UniMM: C3-UniMM integrates structured latent causal graphs with cycle-consistent bidirectional transformations for high-fidelity generation and semantic stability.

3 Method

C3-UniMM maps multimodal observations into structured causal graphs, aligns them across representation, structure, and mechanism, and decodes through a shared latent space. Interventions and cycle consistency support controllability, reversibility, and stable generation.

  • 3.1 Overview: Existing models use direct conditional mappings between modalities, whereas C3-UniMM mediates generation through a shared latent structural representation.
  • 3.1 Overview: The framework extracts composable and intervenable structural semantics before performing multimodal understanding and generation.
  • 3.2 Structured Latent Causal Graph and Unified Decoding: Slot Attention abstracts modality-specific feature sequences into fixed object-level latent units, forming the Structured Latent Causal Graph.
  • 3.2 Structured Latent Causal Graph and Unified Decoding: The graph separates object identities in slots from relational dependencies in its adjacency matrix, supporting object-level structure and compositional reasoning.
  • 3.2 Structured Latent Causal Graph and Unified Decoding: The Unified Decoding Space carries modality-invariant structural semantics before modality-specific decoders produce target observations.
  • 3.3 Super Alignment and Causal Consistency Learning: Super Alignment enforces consistency across representation, structure, and mechanism, while the objective also aligns object relations and generative laws.
  • 3.3 Super Alignment and Causal Consistency Learning: Causal intervention edits latent slots before decoding, and cycle paths target observational reversibility and structural stability.
  • 3.3 Super Alignment and Causal Consistency Learning: Super Alignment, intervention, and cycle consistency jointly enhance semantic consistency, structural controllability, and cross-modal reversibility.

4 Experiments

C3-UniMM is evaluated across multimodal understanding, video comprehension, image and video generation, and structural preservation, including zero-shot and compositional settings. Across these evaluations, it reports strong benchmark performance, while ablations identify the Graph component as especially important for structural understanding.

  • Evaluation setup: C3-UniMM is evaluated across understanding, generation, structural preservation, and compositional generalization benchmarks.The evaluation includes multimodal understanding datasets, video benchmarks, text-to-image and video metrics, and zero-shot compositional measures.
  • Evaluation setup: Unless otherwise specified, results are reported under the zero-shot setting.
  • Understanding results: The 7B model establishes a new state-of-the-art among native unified models and outperforms larger 14B and 17B models across nearly every reported understanding metric.The 1.5B variant also often matches or exceeds 7B and 8B models from other families.
  • Video understanding results: The 7B model achieves a VideoMME score of 60.5/64.2 and surpasses VILA (40B) and LLaVA-OV (7B) on video comprehension.It also reports NExT-QA (81.7) and Perception (64.8).
  • Image generation results: C3-UniMM achieves an Overall score of 0.82 in text-to-image generation and reaches 1.0 on Single and Two-object compositions.The Overall score surpasses specialized Gen-Only models such as SD3-Medium and unified models including Show-o2 and Janus-Pro.

5 Conclusion

C3-UniMM moves unified multimodal modeling beyond statistical correlations by grounding understanding and generation in a shared Structured Latent Causal Graph. Super Alignment, causal cycle consistency, and intervention constraints enforce causal integrity and preserve internal logic under variable shifts.

  • C3-UniMM anchors multimodal reasoning in a Structured Latent Causal Graph rather than statistical correlations.Slot-based abstraction projects disparate data streams into a shared causal latent space for understanding and generation.
  • The shared causal space lets understanding and generation derive outputs from the same structures used to interpret inputs.
  • Super Alignment enforces causal integrity through causal cycle consistency and intervention-based constraints.These constraints are designed to keep the model’s internal logic invariant when specific variables shift.

A Additional Implementation Details

C3-UniMM extends Show-o2’s native unified multimodal design with structured causal representation, decoding, alignment, intervention, and cycle-consistency modules.

  • C3-UniMM is built on Show-o2 while adding SLCG, UDS, Super Alignment, causal intervention, and cycle-consistency learning modules.

A.1 Backbone Architecture

The implementation retains Show-o2’s unified backbone for multimodal encoding and generation, then adds structural slot and graph processing to inject causal information into its heads. Paired modalities are aligned across representations, structures, and mechanisms.

  • Backbone Architecture: The backbone combines text and visual encoders with semantic and low-level visual paths, spatial-temporal fusion, a language model, and separate text and flow-generation heads.Text is autoregressively predicted, while image and video latents are generated through flow matching and decoded by the frozen 3D causal VAE.
  • Model Scales: Two model sizes are instantiated: C3-UniMM-1.5B and C3-UniMM-7B.The models use Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, respectively, with the Show-o2 3D causal VAE frozen unless otherwise specified.
  • Structural Slot Adapter: The structural slot adapter projects modality-specific hidden sequences into a shared structural space and extracts K = 16 object-centric slots.Three refinement steps and eight attention heads support object, event, or semantic-entity representations across image, video, and text inputs.
  • Structured Latent Causal Graph: Directed slot graphs encode entities as nodes and their dependencies as adjacency relationships, with acyclicity and sparsity regularization constraining the structure.The adjacency entry Am,ji represents dependency strength from slot j to slot i; relation supervision is used for structured datasets such as CLEVR and Visual Genome.
  • Unified Decoding Space: Graph representations are transformed into the Unified Decoding Space and expanded into structural prefix tokens that condition understanding and visual generation.The default is Np = 8 prefix tokens, which provide object-relation information to the language head and guide the flow head with structured semantic information.
  • Super Alignment: Paired multimodal samples align latent graphs at representation, structure, and mechanism levels using soft slot matching and a shared mechanism encoder.Mechanism alignment is directly trainable rather than based on comparisons of unspecified abstract functions.

A.6 Cycle Consistency and Intervention Training

C3-UniMM trains cross-modal mappings with structural and observation-level cycle consistency, then applies latent interventions to improve controllability. Re-encoding generated outputs checks whether structures and intended edits are preserved.

  • Cycle Consistency: Cycle consistency re-encodes a generated target-modality sample and compares its inferred graph with the source graph.When reconstruction is available, an observation-level cycle loss is also added; text uses token cross-entropy, while images and videos use VAE-latent ℓ2 distance.
  • Intervention Training: Latent intervention randomly edits a slot or edge before decoding the modified graph into a target modality.The edited output is re-encoded and optimized against the intended intervention, with CLEVR using ground-truth attributes and relations and other data using pretrained evaluators.

A.7 Training Objective

C3-UniMM extends the Show-o2 base objective with structural, alignment, cycle-consistency, and intervention losses. These additions explicitly target structure-aware multimodal learning.

  • A.7 Training Objective: C3-UniMM augments next-token prediction and flow matching with structural graph learning, Super Alignment, cycle consistency, and intervention training.The base objective is Lbase = αNTPLNTP + LFM, while the full objective adds four structure-oriented terms.
  • A.7 Training Objective: The full training objective combines the base loss with weighted graph, Super Alignment, cycle-consistency, and intervention losses.The formulation is L = Lbase + λgraphLgraph + λSALSA + λcycLcyc + λintLint.
  • A.7 Training Objective: Default loss weights are λgraph = 0.1, λSA = 0.5, λcyc = 0.2, and λint = 0.2.These values specify the relative contribution of the added structural and causal objectives.

A.8 Training Schedule

C3-UniMM follows a staged training schedule that progressively adds generation, instruction, and structural-consistency objectives. The recipe preserves a frozen 3D causal VAE while adapting model components across stages.

  • A.8 Training Schedule: C3-UniMM follows Show-o2’s two-stage recipe while adding structure-specific objectives, with the 3D causal VAE frozen throughout training.Initialization uses Show-o2 components alongside newly introduced randomly initialized structural modules.
  • A.8 Training Schedule: Stage 1 warms up visual generation on image-text pairs for 150K iterations, training projection, fusion, flow, and structural modules while initially freezing the language model.The stage uses 432 × 432 images, context length 1024, αNTP = 0.2, and caption dropping probability 0.1.
  • A.8 Training Schedule: Stage 1.5 refines generation on high-quality image-text pairs for 40K iterations, enabling graph and alignment losses for paired samples.Relation supervision is restricted to samples with relation annotations.
  • A.8 Training Schedule: Stage 2 fine-tunes the full model except the frozen VAE for 35K iterations using multimodal instruction, generation, reasoning, and video-text data.Cycle-consistency and intervention losses are activated, while αNTP increases to 1.0.
  • A.8 Training Schedule: Video training samples 17 frames per clip, while inference decodes text autoregressively and visual outputs through the flow head and frozen VAE decoder.Controllable generation edits graph node slots or edges before UDS projection to condition image or video generation.
  • A.8 Training Schedule: Stage 3 oversamples relation-rich and compositional data to improve structural preservation, compositional generation, and intervention consistency.The LSA, Lcyc, and Lint weights increase by a factor of 1.5 while base Show-o2 losses remain active.

B Supplementary Experimental Data

Supplementary results compare C3-UniMM with generative-only and unified methods, showing strong detailed-prompt performance and high alignment and reasoning scores.

  • B Supplementary Experimental Data: 89.84 Overall leads DALL-E 3 (83.5) and SD3-Medium (84.08) on the detailed-prompt benchmark.C3-UniMM also scores 94.12 in Relation and 92.71 in Attribute.
  • B Supplementary Experimental Data: 0.865 Alignment and 0.335 Reasoning are the highest reported scores in the category, ahead of SD3.5-Large and Flux-1-dev.The comparison is presented in Table 9 across Gen-Only and Unified methods.
  • B Supplementary Experimental Data: The supplementary comparisons position C3-UniMM as a unified architecture that combines image-generation quality with multimodal understanding.The reported result emphasizes prompt adherence and multimodal understanding together.
Loading 2608.28603v1…