PaperlayerView exact revision on arXiv

Source-linked AI summary

UniGame: Turning a Unified Multimodal Model Into Its Own Adversary

Zhaolong Su, Wang Lu, Hao Chen, Sharon Li, Jindong Wang

arXiv:2511.19413v3cs.LGcs.AIcs.CV

TL;DR

UMMs’ understanding and generation pathways favor conflicting representations, leaving their consistency difficult to improve directly. UniGame turns generation into an adversary that produces plausible challenging examples for understanding. The paper reports gains in consistency, understanding, generation, and robustness.

  • Problem

    UMMs’ understanding and generation branches favor different representations, and direct objectives for improving their consistency remain unclear.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    UniGame uses a lightweight shared-token perturber and minimax self-play to generate plausible adversarial examples that challenge the understanding branch.

    Source: §1 · p. 2, §3.3 · p. 4, §3.4 · p. 4

  • Results

    +4.6% consistency, +3.6% understanding, +0.02 on GenEval, +4.8% on NaturalBench, and +6.2% on AdVQA are reported for UniGame.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    The paper concludes that adversarial self-play can improve UMM consistency, performance, and robustness by optimizing within the model itself.

    Source: §5 · p. 8

    The evaluation primarily uses Janus-Pro-7B and a limited dataset set, so broader model coverage and more diverse, challenging benchmarks remain to be tested.

    Source: §5 · p. 8

Abstract

from arXiv · show

Unified Multimodal Models (UMMs) have shown impressive performance in both understanding and generation with a single architecture. However, UMMs still exhibit a fundamental inconsistency: understanding favors compact embeddings, whereas generation favors reconstruction-rich representations. This structural trade-off produces misaligned decision boundaries, degraded cross-modal coherence, and heightened vulnerability under distributional and adversarial shifts. In this paper, we present UniGame, a self-adversarial post-training framework that directly targets the inconsistencies. By applying a lightweight perturber at the shared token interface, UniGame enables the generation branch to actively seek and challenge fragile understanding, turning the model itself into its own adversary. Experiments demonstrate that UniGame significantly improves the consistency (+4.6%). Moreover, it also achieves substantial improvements in understanding (+3.6%), generation (+0.02)on GenEval, out-of-distribution and adversarial robustness (+4.8% and +6.2% on NaturalBench and AdVQA). The framework is architecture-agnostic, introduces less than 1% additional parameters, and is complementary to existing post-training methods. These results position adversarial self-play as a general and effective principle for enhancing the coherence, stability, and unified competence of future multimodal foundation models. The official code is available at: https://github.com/AIFrontierLab/TorchUMM

1. Introduction

UMMs can perform understanding and image generation within one architecture, yet their branches favor conflicting representations and remain inconsistent. UniGame uses self-adversarial post-training to challenge understanding with generated examples, with reported gains across consistency, task performance, and robustness.

Motivation

  • UMMs’ understanding and generation pathways favor different representations, contributing to semantic, capability, and feature-compactness mismatches.These inconsistencies can arise with unexpected inputs, unseen compositions, counterfactual queries, distribution shifts, and adversarial attacks.Source: §1 · p. 1
  • Prior reconstruction- and reward-based approaches use handcrafted objectives on a fixed training distribution without explicitly constraining the coupled branches.The paper argues these approaches reproduce behavior within a fixed manifold, leaving inconsistency largely unresolved.Source: §1 · p. 2

UniGame

  • UniGame uses generation as an active adversary, perturbing shared visual tokens to produce plausible counterexamples that understanding learns to solve.A semantic-consistency check and hard-example buffer support this minimax self-play process.Source: §1 · p. 2

Empirical contributions

  • UniGame yields improvements in consistency (4.6%), understanding (+3.6%), generation (+0.02 on GenEval), OOD (+4.8%), and adversarial robustness (+6.2%).Source: §1 · p. 2

2. Related Work

Prior unified multimodal models and post-training methods address different balances between understanding and generation, but inconsistency remains. UniGame introduces adversarial self-play within the UMM to target that gap.

Unified Multimodal Models

  • UMMs combine multimodal understanding and generation, while their shared embeddings still face objective and representation-granularity tensions.Prior approaches include hybrid autoregressive and diffusion training, interleaved image-text token prediction, and dual-granularity codebooks.Source: §2 · p. 2

UniGame

  • Figure 3 presents UniGame as adversarial self-play with a lightweight perturber and a hard-example filtering buffer.The framework targets understanding robustness and understanding-generation consistency.Source: Figure 3 · p. 3

Post-training of UMMs

  • Reconstruction and reward-based post-training can improve fidelity or downstream metrics, but generally do not use generation to adversarially strengthen understanding.The paper identifies off-manifold perturbations, lack of semantic plausibility guarantees, and computational cost as concerns for existing approaches.Source: §2 · p. 3

3. UniGame

UniGame treats understanding and generation as coupled pathways whose representation mismatch leaves fragile regions in a shared manifold. Its self-play framework uses constrained perturbations, decoded hard examples, and replay to train the branches against each other.

3.1. Preliminary

  • Understanding minimizes discriminative prediction loss while generation reconstructs the input image from shared visual representations.The unified visual tokenizer maps images to tokens aligned with the language-model vocabulary space; the branches can be jointly optimized with a trade-off weight.Source: §3.1 · p. 3

3.2. Motivation

  • Existing methods optimize individual goals on fixed data distributions, while consistency improvement requires expanding shared-manifold coverage around fragile decision boundaries.The paper motivates using constrained adversarial signals to expose failures with semantically valid, decodable cases.Source: §3.2 · pp. 3–4

3.3. Overview of UniGame

  • UniGame adds a compact perturber that produces bounded perturbations of fused visual states and a buffer that retains hard, semantically plausible decoded candidates.The framework is described as a minimax game; under bounded perturbation and decoder constraints, the perturber optimizes a lower bound on worst-case understanding loss.Source: §3.3 · p. 4

3.4. The Self-Play Training Process

  • The clean path trains understanding on original tokens, while generation decodes perturbed embeddings into challenging candidates and the understanding branch learns from clean and buffered examples.Semantic plausibility and regularization are enforced on the generation side through the perturber’s norm and CLIP-based filtering.Source: §3.4 · p. 4

3.5. Discussion

  • UniGame differs from GANs by reusing a UMM’s own branches, and from prior adversarial training by constraining perturbations to decoder-constrained, on-manifold images.The paper describes the framework as architecture-agnostic and complementary to existing post-training methods.Source: §3.5 · p. 5

4. Experiments

UniGame is evaluated on understanding, generation, consistency, and robustness benchmarks, with additional ablations, case studies, architecture tests, and training-dynamics analysis. The reported results show gains across multiple tasks and settings, with decoder-constrained perturbations and semantic constraints examined as design choices.

4.1. Experimental Setup

  • The experiments evaluate UniGame on understanding, generation, consistency, out-of-distribution robustness, and adversarial robustness benchmarks, using Janus-Pro-7B for the main experiments.The perturber is a 3-layer MLP with 2.1M parameters, operating on shared visual tokens; baselines include other UMMs and post-training methods.Source: §4.1 · p. 5

4.2. Consistency Evaluation

  • Consistency evaluation combines UnifiedBench reconstruction similarity and WISE prompt-image alignment, with the score weighted 0.6 and 0.4, respectively.The reconstruction protocol samples LAION-5B images, captions each image, reconstructs it, and compares original and reconstructed images using four vision-language backbones.Source: §4.2 · pp. 5–6
  • UniGame substantially improves the unification score across all reported metrics compared with Janus-Pro and SFT-only training.Consistency evaluation uses UnifiedBench and WISE; the table caption marks the base model with †.Source: Table 1 · p. 5, §4.2 · p. 6

4.3. Benchmark Results

  • Understanding improves by 3.6% over the baseline, while NaturalBench and AdVQA gains are 4.8% and 6.2%, respectively; UniGame also outperforms most competitors in generation.Figure 4 reports robustness evaluation, and its hard-sample loss remains above Clean/Adversarial loss over 5K training steps.Source: §4.3 · p. 6, Figure 4 · p. 6, Table 2 · p. 6

4.4. Ablation Study

  • Decoder-only perturbations reach 81.5% VQAv2 accuracy, and CLIP text-image matching raises accuracy to 82.7%; a 3-layer perturber and buffer size 50 each yield 83.4%.The reported comparisons also find that decoding alone improves over embedding perturbation, and that the 3-layer perturber outperforms 2- and 4-layer variants.Source: §4.4 · p. 7

4.5. Case Study

  • Case studies report stronger understanding across four visual-reasoning categories and more faithful, accurate, and stylistic image generation, including improved adherence to layout constraints.The examples include counting, object interaction, spatial relations, and crowd detection.Source: §4.5 · pp. 7–8

4.6. Extensibility and Efficiency

  • UniGame adds gains when integrated with RecA, transfers across tested architectures, and uses fewer trainable parameters than compared post-training approaches.The RecA integration reports +0.5 on MMMU and +1.27 on UnifiedBench while GenEval remains the same; the table caption states that 5K steps take about 10 GPU-hours.Source: §4.6 · p. 8, Table 5 · p. 8, Table 6 · p. 8

4.7. Convergence and Hyperparameter Analysis

  • Training-dynamics analyses report a stable tug-of-war between the two opponents, smooth alternation without divergence in the best configuration, and robustness to different hyperparameter choices.The analysis varies the hard-buffer threshold, trade-off, and perturbation budget.Source: §4.7 · p. 8

5. Conclusion and Limitation

UniGame frames self-adversarial post-training as a way to improve consistency by letting the model discover its own failures. The evaluation is limited in backbone and dataset coverage.

5. Conclusion and Limitation

  • UniGame formulates understanding and generation as a minimax game that lets the model autonomously discover its own failures.The paper reports increased consistency, performance, and robustness.Source: §5 · p. 8
  • The evaluation primarily uses Janus-Pro-7B, so broader model coverage may reveal additional insights.Source: §5 · p. 8
  • The study uses a limited set of datasets; future work should test UniGame on more diverse and challenging benchmarks.Source: §5 · p. 8

Supplementary Material

The supplementary material includes detailed experimental and theoretical appendices, and presents UniGame’s training algorithm as alternating challenge and understanding updates.

Appendix Contents

  • The appendix contents span algorithm and training details, detailed experiments, robustness, case studies, hyperparameter analysis, and theoretical insights.Source: Appendix Contents · p. 11

A. Algorithm Details

  • The algorithm samples images and questions, encodes images into unified tokens, and creates bounded perturbations that are decoded into candidate images.Source: A. Algorithm Details · p. 11
  • UniGame alternates perturber challenge updates with understanding updates on clean samples and hard buffer examples.Candidates passing a CLIP threshold are added to the hard-example buffer.Source: A. Algorithm Details · p. 11

B.1. Training and Testing Data

The supplementary experiments specify the data, implementation, and perturber design, and report positive consistency gains on two additional UMM backbones. They also describe hard-sample selection and training dynamics.

B.1. Training and Testing Data

  • The training data include VQAv2 train-split and a filtered 100k-pair CC3M subset for text-image supervision; evaluation spans understanding, generation, and robustness benchmarks.The listed benchmarks include VQAv2, MMMU, POPE, MMBench, GenEval, UnifiedBench, WISE, NaturalBench, and AdVQA.Source: §B.1 · p. 11

B.2. Hyperparameter Details

  • A generation-to-understanding learning-rate ratio of approximately 250 is reported as optimal, using gen_lr=5×10^-3 and und_lr=2×10^-5.Source: §B.2 · p. 11
  • The implementation computes perturbation updates, regularizers, and losses in float32 while running other forward and backward passes in bfloat16.The authors say this preserves the perturbation signal while retaining mixed-precision speed benefits.Source: §B.2 · pp. 11–12

B.3. Perturber

  • The perturber is a three-layer MLP over fused visual tokens, with a shared learnable scalar gate that controls perturbation strength within a budget.Source: §B.3 · p. 12

B.4. Hard Samples

  • A hard-sampler buffer selects challenging adversarial samples for training; the illustrated cases are failures that successfully challenged the model.Source: §B.4 · p. 12, Figure 6 · p. 12

C.1. Additional Backbone Validation

  • VQA accuracy across adversarial ratios is plotted in the training-dynamics analysis, with the best configuration reaching 83.4%.Source: Figure 7 · p. 12
  • BLIP-3o gains +2.5 in consistency, while Chameleon gains +2.2, supporting generalization across diffusion-based and autoregressive UMMs.Reported scores include BLIP-3o MMMU 51.2 and GenEval 0.54, and Chameleon VQAv2 40.2 (+1.7 gain), MMMU 24.0, and GenEval 0.40.Source: §C.1 · p. 12

C.2. Learning Rate Ratio Ablation

The appendix describes a sweep of generation-to-understanding learning-rate ratios and motivation experiments on token perturbations and CLIP constraints. A small Gaussian-noise level slightly improves VQAv2 soft accuracy before larger noise degrades it.

C.2. Learning Rate Ratio Ablation

  • The learning-rate sweep tests generation and understanding configurations, with Table 8 listing the tested pairs and their Gen/Und ratios.The table caption identifies R250 as gen lr=5×10^-3 and und lr=2×10^-5; Figure 9 is labeled Perturbation Sweetspot.Source: §C.2 · pp. 12–13, Table 8 · p. 13, Figure 9 · p. 13

C.3. Motivation Experiments

  • The CLIP-constraint heatmap compares VQAv2 accuracy across CLIP-weight and CLIP-similarity-threshold configurations.Source: Figure 8 · p. 13
  • VQAv2 soft accuracy rises from 74.50 to 75.58 near σ≈0.01, then degrades at larger Gaussian-noise levels.The experiment injects i.i.d. Gaussian noise into projected visual tokens.Source: §C.3 · p. 13, Figure 9 · p. 13

D. Robustness Results

UniGame significantly improves robustness on the out-of-distribution and adversarial evaluations reported in Table 9.

D. Robustness Results

  • UniGame significantly improves robustness on the out-of-distribution and adversarial evaluations.Table 9 reports the OOD and adversarial robustness results; the supplied passages do not provide its individual values.Source: D. Robustness Results · p. 13, Table 9 · p. 13
  • Table 9 reports results for both out-of-distribution and adversarial robustness.Source: Table 9 · p. 13

E.1. Case Study on Understanding Tasks

The case studies describe gains in visual grounding, reasoning over difficult scenes, and prompt-faithful image generation. The convergence analysis recommends faster perturber updates and short alternations to balance the two branches.

E.1. Case Study on Understanding Tasks

  • UniGame improves understanding in cluttered counting, object relations, spatial reasoning, and crowd detection, while open-ended captions capture finer visual details.The examples contrast baseline failures with UniGame outputs that better attend to decision-critical evidence; the authors say these qualitative improvements align with quantitative gains.Source: §E.1 · pp. 13–14

E.2. Case Study on Generation Tasks

  • UniGame generations better respect specified layouts and attributes and preserve depth, lighting, and visual texture across the described examples.The examples include object counts and spatial relations, multiple attributes, Grand Canyon depth and lighting, and a Siamese cat with green velvet texture.Source: §E.2 · p. 14

F.1. Convergence

  • Short interleaved updates and a higher perturber learning rate are used to balance the branches and avoid instability from either branch dominating.The passage warns that prolonged generation phases can saturate attack success before understanding catches up, while generation overpowering understanding can produce off-manifold candidates and hurt clean accuracy.Source: §F.1 · p. 14

F.2. Hyperparameter Sensitivity Analysis

The hyperparameter analysis specifies perturbation, mining, and replay-buffer settings for the main experiments. The theoretical section presents conditional convergence results for an idealized minimax procedure, alongside a broader summary of the framework’s theoretical interpretations.

F.2. Hyperparameter Sensitivity Analysis

  • The main experiments use δ = εmax = 0.02, a 60th-percentile hardness threshold, minimum CLIP similarity of 0.6, and β = 0.5.The replay buffer stores up to 50 decoded images; the authors report numerical stability for β in [0.3, 1.0].Source: §F.2 · p. 15

G. Theoretical Insights

  • The paper summarizes its theoretical account as conditional self-play convergence, implicit gradient-norm regularization, and decoder-constrained manifold expansion.It presents these properties as theoretical support for the empirical improvements in understanding, consistency, OOD robustness, and adversarial robustness.Source: G. Theoretical Insights · p. 15

G.1. Convergence of the Minimax Self-Play Dynamics

  • Under the stated assumptions, sufficiently small learning rates yield a bounded gradient descent–ascent sequence that converges to a neighborhood of a first-order stationary point.The analysis is an idealized account rather than a full convergence proof for the deep-network implementation; the assumptions include smoothness, compactness, and local nonconvex–concave structure.Source: §G.1 · pp. 15–16

G.2. Robustness Improvement via Worst-Case Regularization

The theoretical analysis links adversarial self-play to reduced sensitivity of understanding decisions and to expanded coverage of challenging, decodable regions. Its summary presents these mechanisms as support for the reported robustness and other empirical gains.

G.2. Robustness Improvement via Worst-Case Regularization

  • To first order, adversarial self-play is equivalent to adding a Jacobian-norm penalty, encouraging the understanding branch to reduce sensitivity to token perturbations.The analysis says this leads to locally flatter decision boundaries and offers it as an explanation for robustness gains across in-distribution, OOD, and adversarial settings.Source: §G.2 · p. 16

G.3. Manifold-Expanding Effect of Decoder-Constrained Perturbations

  • Decoder-constrained perturbations produce semantically valid, plausible samples near fragile decision boundaries and expand the decodable manifold toward regions of high understanding loss.The paper says the hard-sample buffer collects samples near the understanding decision boundary, where training reduces empirical risk; the analysis assumes a locally bi-Lipschitz decoder.Source: §G.3 · p. 16

G.4. Summary of Theoretical Insights

  • The theoretical summary presents convergence, boundary flattening, and manifold expansion as properties that support the paper’s reported empirical gains.Source: §G.4 · p. 17

Related papers

Loading 2511.19413v3…