Gestalt: Large Multimodal Interplay Model

1 Introduction

• Existing models increasingly share architectures or mixed-modal data, but the paper argues that merely accommodating modalities overlooks their distinct characteristics and relations.The introduction describes this direction as reaching a bottleneck.Source: §1 · p. 1
• The interplay pyramid progresses from preserving modality-unique information, through aligning shared information, to integrating cues for synergy unavailable from either modality alone.The dog example combines textual size-collar and visible-collar cues to identify the smaller dog.Source: §1 · p. 2
• Gestalt uses a unified discrete-diffusion model with two successive zones: modality-specific processing and mediated exchange precede full multimodal attention and shared processing.Learnable interplay tokens mediate cross-modal information flow throughout the zones.Source: §1 · p. 2
• The pyramid also guides training: aligned image-caption pretraining precedes instruction fine-tuning across redundant, text-unique, visual-unique, and synergistic categories.The fine-tuning curriculum shifts emphasis toward deeper multimodal synergy across three phases.Source: §1 · p. 2
• Gestalt scores 73.30 on UniGenBench and 74.57/78.86 on TIIF-Bench short/long settings, while leading evaluated diffusion-based unified models overall.The TIIF-Bench scores are for short and long instructions, respectively.Source: §1 · pp. 2–3
• Gestalt scores 78.9 and 86.1 on the two CV-Bench tracks, outperforming the representative autoregressive-based model BAGEL on both.The paper also reports strong multimodal understanding and leading overall text-only performance among evaluated diffusion-based unified models.Source: §1 · p. 3

2 Methodology

• Gestalt combines a unified discrete-diffusion framework with interplay-centric architecture, data organization, and progressive training guided by the interplay pyramid.Its pretrained backbone establishes the modality-specific foundation before multimodal training.Source: §2 · p. 3, §2.3 · p. 4

2.1 Unified Interplay-Centric Architecture

• Vision and text share a discrete token space, while learnable interplay tokens mediate exchange in Zone I before full multimodal integration in Zone II.Zone I retains modality-specific feed-forward networks and restricts direct vision-text attention; Zone II uses full multimodal self-attention and a shared feed-forward network.Source: §2.1 · pp. 3–4

2.2 Training Data Organization

• The training corpus uses 78M aligned image-caption pairs for pretraining and continual pretraining, then organizes 13.7M SFT examples into four interplay categories.The categories are redundant, text-unique, visual-unique, and synergistic.Source: §2.2 · p. 4

2.3 Progressive Training Strategy

• Training moves from joint multimodal masked prediction to conditional masked prediction, with bidirectional cross-modal conditioning and no fixed generation order.The conditional stage leaves one modality visible while predicting the other, strengthening vision-to-text and text-to-vision dependencies.Source: §2.3 · pp. 4–5
• Three SFT phases retain all four interplay categories while shifting emphasis from alignment and language capability to visual-specific capability and finally deeper integration.The sampling ratios are progressively adjusted toward visual-unique and synergistic information.Source: §2.3 · p. 5

3.1 Training and Evaluation Overview

• Training comprises joint pretraining, conditional continual pretraining, and interplay-organized supervised fine-tuning; evaluation then examines benchmarks, representations, language capability, and interplay.The configuration uses 70M image-caption pairs, 8M image-text pairs, and 13.7M instruction examples across the three stages.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2.1 Fine-Grained Text-to-Image Generation

• Gestalt scores 73.30 on UniGenBench and 74.57/78.86 on short/long TIIF-Bench settings, leading generation-only and unified baselines in the reported overall comparisons.It ranks first across six UniGenBench dimensions and all three TIIF-Bench categories in both prompt-length settings.Source: §3.2.1 · pp. 5–6, Table 1 · p. 7, Table 2 · p. 7

3.2.2 Multimodal Understanding

• Gestalt leads diffusion-based models on five of eight multimodal-understanding benchmarks and ranks in the top two on seven; it also surpasses BAGEL on both CV-Bench tracks.On CV-Bench, Gestalt scores 78.9 and 86.1, versus BAGEL’s 77.7 and 84.2.Source: §3.2.2 · p. 7, Table 3 · p. 8

3.3 Cross-modal Representation Analysis

• Gestalt shows more interleaved visual and textual representations than the compared diffusion and autoregressive models, while retaining high cosine similarity and CKA to pretrained LLaDA.Its cosine similarity and CKA are 0.9968 and 0.9949, respectively; text-only evaluation also finds it leads diffusion-based unified models on five of six benchmarks.Source: §3.3 · pp. 8–9, Table 4 · p. 9, Table 5 · p. 9

3.4 Understanding the Role of Cross-Modal Interplay

• Gestalt performs strongly on visual-oriented and synergy-oriented benchmarks, and its interplay-token representations form distinct patterns for visual, textual, and synergistic information demands.Visual- and text-dominated samples occupy largely separated regions, while synergy samples form a distinct distribution that partially bridges them.Source: §3.4 · pp. 9–10, Table 6 · p. 9

4 Related Work

• Early systems connect modality-specific vision encoders to pretrained AR-based language models through learned projectors [2], whereas native unified multimodal models jointly pretrain visual and textual data in a shared model [41].Source: §4 · p. 10
• Unified multimodal models use autoregressive next-token prediction, discrete diffusion, hybrid AR-diffusion, or flow-based formulations.The passage associates these approaches with [20] [42] [43], [31] [44], and, respectively.Source: §4 · p. 10
• Existing work largely brings visual and textual data into a shared model without explicitly organizing their cross-modal information relationships and emergent properties.Source: §4 · p. 10

5 Discussion

• Gestalt’s multistage pyramid retains modality-specific processing and controlled exchange before full integration, as a way to balance specialization and cross-modal synergy.Source: §5 · p. 10
• The discussion suggests multimodal scaling may depend not only on model and data size, but also on how information exchange is organized across stages.This connects to mixed-modal scaling-law studies that treat multimodal synergy as a key component.Source: §5 · p. 10
• The interplay pyramid is presented as a structured setting for examining this possibility and as a potential basis for future scaling-law research.Source: §5 · p. 10

A.1 Data curation.

• For multimodal pretraining, image-text pairs are checked for consistency and samples with weak or mismatched correspondence are removed.The selected pairs come from LLaVA-OneVision-1.5-Midtraining data [10], with Qwen3-VL-32B-Instruct used for verification.Source: §A.1 · p. 14
• For supervised fine-tuning, filtering and model-based verification check that required visual evidence remains available after images are compressed into discrete tokens.The curation process accounts for fine-grained details potentially lost during VQ-VAE tokenization.Source: §A.1 · p. 14
• The 13.7M instruction-tuning examples are organized by primary interplay type: redundant, visual-unique, text-unique, or synergistic information.Source: Table 7 · p. 14

A.2 Interplay-aware data organization.

• Sub-task data are characterized by sampling 100 examples and assigning each to its dominant redundant, visual-unique, text-unique, or synergistic information type.Task-specific heuristics and model-assisted verification assess whether evidence is shared, modality-specific, or jointly required.Source: §A.2 · p. 14
• The 13.7M-example instruction corpus contains 2.8M redundant, 4.0M visual-unique, 4.2M text-unique, and 2.7M synergistic examples, supporting a three-stage curriculum.The curriculum shifts emphasis from preserving pretrained capabilities to visual perception and then cross-modal synergy.Source: §A.2 · p. 14
• The FGRC evaluation did not complete successfully.This limits the reported evaluation for that benchmark.Source: §A.2 · p. 15

A.3 Image generation data.

• The 2.46M image-generation examples are distributed approximately uniformly across the three stages because they maintain and improve generation rather than determine the interplay curriculum.Source: §A.3 · p. 15

• The controlled comparison starts from the same CPT checkpoint and uses the same 10% SFT-data subset for uniform mixing and curriculum training.Source: §B.1 · p. 15
• 113.8 points higher MME-Perception than uniform mixing in the early stage, while MMStar eventually surpasses uniform mixing as training progresses.Broad visual perception is initially stronger, while fine-grained visual perception remains weaker; MMStar improves across stages.Source: §B.1 · p. 15
• Earlier redundancy- and text-oriented stages favor broad perception, whereas later modality-unique and synergy-oriented stages increasingly strengthen task-sensitive visual reasoning and fine-grained perception.The curriculum reshapes capability emphasis rather than yielding uniform gains across benchmarks.Source: §B.1 · p. 15

• The evaluation covers complementary tasks: FGRS assesses fine-grained recognition and understanding, while FGRC emphasizes reasoning about relations among objects and regions.The controlled comparison also adapts the official Lumina-DiMOO checkpoint using the same 50K samples.Source: §B.2 · pp. 15–16
• 79.34 on FGRS, approaching SkySenseGPT’s 79.76 despite using 50K remote-sensing samples versus 1.415M.Gestalt also outperforms the general-purpose BAGEL and GeoChat baselines on FGRS.Source: §B.2 · p. 16
• On relation-oriented FGRC, Gestalt improves over BAGEL and GeoChat, but remains behind the heavily specialized SkySenseGPT.Source: §B.2 · p. 16

• Across diverse styles, multi-object compositions, attributes, and spatial relations, Gestalt produces coherent scenes while following elaborate prompts.The figure presents qualitative UniGenBench examples; the accompanying analysis reports instruction-following and visual quality across these challenging prompts.Source: §B.3 · p. 16, Figure 7 · p. 17
• Generated scenes preserve object identities, bind attributes to the correct entities, and arrange elements according to specified spatial relationships.Examples include complex multi-object layouts and fine-grained properties such as different bottle contents and flower-vase pairings.Source: §B.3 · p. 16
• Style constraints affect the whole composition, with lighting, palette, brushwork, perspective, and atmosphere remaining mutually consistent.Examples span watercolor, oil painting, Impressionism, product photography, cinematic landscapes, and anime.Source: §B.3 · p. 16
• The examples jointly illustrate modeling what appears, where elements are placed, and how the scene is rendered—key requirements for preserving compositional and attribute constraints.The passage relates these qualitative examples to quantitative UniGenBench results.Source: §B.3 · p. 16