PaperlayerView exact revision on arXiv

Source-linked AI summary

JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces

Jiahan Li, Huangyuan Su, Shengbang Tong, Shusheng Yang, Ruihan Guo, Suozhi Huang, Hexu Zhao, Pratyusha Sharma, Sherry Yang, Jiatao Gu, Rob Fergus, Kyunghyun Cho

arXiv:2610.28801v1cs.CV

TL;DR

Most unified multimodal systems generate images and text through different processes and in a fixed order, so the first output cannot adapt to the second. JUICE jointly models both modalities in frozen pretrained representation spaces with one flow-matching Transformer. Coupled sampling yields 93.7% maze joint success, compared with 54.6% for independent sampling, without fixing a generation order.

  • Problem

    Most unified multimodal systems use different processes and a fixed generation order, limiting how image and text outputs can adapt to one another.

    Source: §1 · p. 3

  • Method

    JUICE trains one flow-matching backbone from scratch on frozen image and text representations, using separate noise levels for conditional and joint generation.

    Source: §1 · p. 3

  • Results

    Coupled sampling reaches 93.7% maze joint success versus 54.6% with independent sampling, without choosing which modality goes first.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Joint sampling can make image and text outputs agree on one of several valid solutions without a fixed generation order.

    Source: §1 · p. 4, §5.4 · p. 18

    Models run at 224×224, limiting fine visual detail; each pretraining configuration was run once, and joint reasoning uses synthetic planning problems.

    Source: §6 · p. 23

Abstract

from arXiv · show

Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second. We ask whether both modalities can instead be generated by one process that is trained on them jointly from scratch. Pretrained encoder spaces hold both modalities in one continuous form and carry knowledge learned at scale, and recent work finds them effective as generative targets for vision and language. We therefore model the joint distribution directly in these spaces. We introduce JUICE (Joint Unified multimodal modeling In Continuous Encoder spaces), a single flow matching Transformer over frozen image and text representations, in which each modality has its own noise level. Which modalities are clean and which are generated then decides the task: image generation, captioning, visual question answering, or joint generation. Controlled pretraining runs test each design choice. At equal guidance, generating text as continuous states leaves image generation intact (48.41 GenEval, against 47.86 for a model trained only for text-to-image generation), while captions lose some exact wording. Training on joint and conditional denoising together beats either alone, and semantic encoder spaces beat VAE latents and raw pixels as targets. At 2.1B parameters, with no language model initialization and at 224x224, JUICE reaches up to 0.88 on GenEval and 73.1 on VQAv2. By sampling both modalities from the joint distribution (coupled sampling), it can also draw a route on a map while writing out the matching moves. On mazes with several valid solutions, joint success rises from 54.6% with independent sampling to 93.7%, and no modality has to go first. On visual spatial planning, JUICE solves 99.8% of maps at its training sizes.

1 Introduction

JUICE replaces fixed-order, asymmetric multimodal generation with one model that jointly learns image and text in frozen pretrained representation spaces. Its controlled studies and larger-scale evaluations test this design across image generation, understanding, and joint reasoning.

1 Introduction

  • Existing unified systems commonly use different generative processes or quantized representations and generate image-text outputs in a fixed order, preventing the first output from adapting to the second.The proposed alternative trains both modalities jointly from scratch.Source: §1 · pp. 3–4
  • One flow-matching Transformer operates on frozen image and text representations, with separate noise levels that let clean context and generated targets specify different tasks.The same model supports text-to-image generation, captioning, visual question answering, and coupled joint generation.Source: §1 · p. 3, Figure 1 · p. 4
  • At equal guidance, continuous-text JUICE scores 48.41 GenEval versus 47.86 for text-to-image-only training, while continuous captions lose some exact wording.Joint and conditional denoising together outperform either alone; semantic spaces work better for generation and captioning than reconstruction-oriented alternatives.Source: §1 · p. 3
  • From 179M to 2.1B parameters, SigLIP2 leads captioning at every size, while DINO’s early image-generation lead disappears.At 179M, DINO leads image generation; at 2.1B, SigLIP2 leads.Source: §1 · p. 3
  • At 224×224, starting from no language model, the two models reach 0.87 and 0.88 GenEval, while the SigLIP2 model scores 73.1 on VQAv2.Source: §1 · p. 4
  • 93.7% maze joint success with coupled sampling, versus 54.6% independently; JUICE also solves 99.8% of visual spatial-planning maps at training sizes.Sequential sampling reaches 95.2% with image first and 85.3% with text first, but requires choosing an order.Source: §1 · p. 4

2 Related Work

Prior unified multimodal models vary in representation spaces and generation processes, and often generate modalities separately or in a fixed order. JUICE instead generates both modalities as continuous pretrained encoder states with one flow-matching process.

Unified multimodal models

  • Most surveyed unified models use different processes for images and text and generate outputs containing both modalities in a fixed order; JUICE uses one flow-matching process in frozen representation spaces.The comparison covers discrete token and image-code autoregression, mixed autoregressive and diffusion systems, and discrete masked diffusion.Source: §2 · p. 5

Generation in pretrained representation spaces

  • Earlier continuous-space work generates only one modality: BLM predicts text as tokens, while ELF generates text without images.JUICE generates both modalities in pretrained spaces, where the same representations serve as targets and cross-modal context.Source: §2 · p. 5

Joint and conditional multimodal modeling

  • Existing joint and conditional methods use specialized components or representations, whereas JUICE keeps both modalities as continuous encoder states and trains one backbone from scratch.In this setting, joint denoising improves conditional generation beyond what its additional targets explain, and coupled sampling aligns multiple valid solutions.Source: §2 · p. 6

Continuous diffusion for language

  • JUICE follows the line that diffuses contextual pretrained language-model states, but generates those states jointly with images rather than using a text-only process.Earlier approaches also diffuse learned word embeddings, categorical distributions, or compressed and contextual pretrained language representations.Source: §2 · p. 6

3 Method

JUICE uses one flow-matching model over frozen image and text representations, with each modality’s noise level determining whether it is a clean condition or a generation target. It supports conditional and joint tasks by training and sampling along paths through a two-dimensional time domain.

3.1 Preliminaries

  • Flow matching interpolates noise and clean data, and MM-RiT predicts clean states whose estimates define the velocity field used for denoising.The linear-path loss weights clean-state denoising by 1/d(t)^2, with the denominator floored near t=1 for numerical stability.Source: §3.1 · p. 6

3 Method

  • A single MM-RiT predicts clean image and text states from noisy representations, using modality-specific times to distinguish conditions from generation targets.Frozen pretrained encoders provide the representations; fixed decoders map generated states back to pixels and tokens.Source: §3.2 · p. 7, §3.3 · pp. 7–8

3.2 Formulation

  • Each modality has its own time, so clean inputs and generated targets define paths through a unit square; training covers conditional edges and the joint interior.Text-to-image keeps text clean, captioning keeps the image clean, and question answering keeps the image and question clean while generating the answer.Source: §3.2 · p. 7

3.3 Architecture

  • Separate modality projections and bidirectional joint attention connect the streams, while per-token time signals identify clean conditions and generation targets.Fixed decoders map generated image representations to pixels and text representations to tokens; frozen encoder features serve as both targets and conditioning context.Source: §3.3 · pp. 7–8

3.4 Training

  • Image and text use separate noise scales and time schedules, and training mixes joint denoising with conditional text-to-image and image-to-text denoising.The image schedule shifts probability toward noisier states, whereas text is not shifted; joint denoising trains the joint distribution used for coupled sampling.Source: §3.4 · p. 8

3.5 Inference

  • At inference, tasks differ by their paths through the time domain, with conditions held clean and generation targets integrated from noise to clean data.Coupled sampling computes both velocity fields from the current joint state before either stream is updated, so each stream conditions step by step on the other’s evolving state. Conditional tasks use classifier-free guidance with an uninformative reference input.Source: §3.5 · pp. 8–9

4 Pretraining Study

The pretraining studies test how generation frameworks, denoising-task coverage, encoder spaces, and scale affect JUICE. Across these choices, continuous text preserves image generation at shared guidance, joint-plus-conditional training performs best bidirectionally, semantic spaces outperform reconstruction targets, and scaling changes image-space rankings.

4.1 Setup

  • The shared reference recipe uses a 668M MM-RiT backbone, frozen SigLIP2 image features and Flan-T5 text states, trained for 50k updates on image-caption pairs and Cambrian-7M.The studies vary one design choice at a time; framework and representation comparisons change complete pipelines rather than isolated components.Source: §4.1 · pp. 9–10, §4 · p. 9

4.2 Generative Frameworks

  • 48.41 GenEval for JUICE versus 47.86 for a text-to-image-only model at shared guidance; continuous text matches image generation but yields lower CIDEr than autoregressive text.JUICE scores 88.80 CIDEr versus 160.52 for autoregressive text, while CLIPScore is 80.20 versus 80.37.Source: §4.2 · pp. 10–11, Table 2 · p. 10, Figure 3 · p. 11

4.3 Training Task Mixtures

  • Mixing joint and conditional denoising leads bidirectional recipes on every metric; at best guidance it reaches 48.41 GenEval versus 44.29 for Multi-task and 31.33 for Joint only.Matching target count or blocking cross-modal attention lowers GenEval relative to Mixing, while longer Multi-task training closes the CIDEr gap but not the GenEval gap.Source: §4.3 · pp. 11–13, Table 3 · p. 12, Figure 4 · p. 12

4.4 Representation Spaces

  • 48.41 GenEval with SigLIP2 image features versus 35.42 with FLUX VAE latents; Flan-T5 states lead the tested text spaces on all four metrics.SigLIP2 leads DINO by 20.0 CIDEr; the VAE has better reconstruction quality, with rFID 0.288 versus 0.435 for SigLIP2.Source: §4.4 · pp. 13–14, Figure 5 · p. 13

4.5 Scaling

  • From 179M to 2.1B parameters, all four metrics improve, but image-generation rankings flip from DINO’s 32.90 versus SigLIP2’s 18.99 GenEval to 56.81 versus 52.78.SigLIP2 leads captioning at every size, reaching 86.64 CLIPScore at 2.1B versus 72.32 for DINO.Source: §4.5 · p. 14, Figure 6 · p. 14

5 Experiments

At 224×224, the experiments evaluate JUICE’s image generation, visual understanding, and joint reasoning after pretraining and finetuning. Across these tasks, the results show strong image-generation and planning performance, alongside weaker exact wording and domain-knowledge performance.

5.1 Setup

  • The study trains two 2.1B models, one per image space, pretraining each for 60k rather than 50k updates and then finetuning for 15k updates at global batch size 1,024.The models are finetuned on 20.6M text-to-image and visual-understanding examples.Source: §5.1 · p. 15

5.2 Image Generation

  • 0.87 and 0.88 GenEval are the scores for SigLIP2 and DINO, respectively; DINO ties TUNA (0.88) among 1B–3B models.Both JUICE variants have the highest position score (0.85), while TUNA leads on counting and attribute binding.Source: §5.2 · p. 15, Table 4 · p. 15
  • 0.45 WiScore makes SigLIP2 the highest-scoring reported unified model of 1B–3B parameters on WISE, below FLUX.1-dev (0.50) and BAGEL (0.52).WISE separates the image spaces: SigLIP2 scores 0.45 and DINO 0.37.Source: §5.2 · p. 16, Table 5 · p. 17

5.3 Visual Understanding

  • 73.1 on VQAv2 places SigLIP2 above Show-o (69.4), Muddit (68.2), and OneFlow (67.7), but below JanusFlow (79.8).JUICE is competitive on several perception benchmarks but weaker on domain-knowledge tasks such as AI2D and MMMU.Source: §5.3 · pp. 17–18, Table 6 · p. 18

5.4 Joint Reasoning

  • 93.7% maze joint success with coupled sampling contrasts with 54.6% for independent SigLIP2 sampling, without requiring a fixed generation order.Coupled sampling also reaches 81.7% for DINO; sequential performance varies with which modality is generated first.Source: §5.4 · pp. 19–20, Table 7 · p. 20
  • 99.8% of VSP maps at levels 3–6 are solved, on par with fine LatentUM (99%) and above coarse LatentUM (85%).These levels match the training map sizes; larger unseen levels are reported separately.Source: §5.4 · p. 21, Table 8 · p. 21

(a) Solved maps of each size

On VSP maps larger than those used in adaptation, JUICE still solves many maps, though move-sequence accuracy falls with map size.

(a) Solved maps of each size

  • 94% of level-7 maps and 69% of level-8 maps are solved, although these sizes were not seen in adaptation.On those levels, drawn paths are shortest routes on 99% and 96% of maps, respectively; moves reach the goal on 94% and 69%.Source: (a) Solved maps of each size · p. 22
  • On unseen sizes, move errors rise with repeated-run length: 7 of 9 runs of six moves lose a move, while runs of one or two lose under 2%.Of 37 failures, 29 lose one move from the reference route while the drawn path is exactly that route.Source: (a) Solved maps of each size · p. 22
  • Coupled sampling restores failed moves on nine maps, seven with the drawn path fixed throughout; no initially solved map ends unsolved.Both outputs are fixed from the first sampling step on 574 of 600 maps.Source: (a) Solved maps of each size · p. 22

6 Conclusion

JUICE uses frozen pretrained image and text representation spaces with a single flow-matching backbone to generate, understand, and coordinate both modalities. The conclusion reports strong performance in several tasks while identifying resolution, wording, domain knowledge, and evaluation scope as limitations.

6 Conclusion

  • One model handles image generation, visual understanding, and joint generation by treating tasks as paths through modality-specific noise levels.Pretraining should include both conditional denoising at the square’s edges and joint denoising in its interior.Source: §6 · pp. 22–23
  • Semantic encoder spaces are better generation targets than reconstruction-oriented spaces, while exact wording remains a cost of continuous text generation.The conclusion also reports that image-space preferences carry over across scales for captioning, but not for image generation.Source: §6 · pp. 22–23
  • The models run at 224×224, limiting fine visual detail; each pretraining configuration was run once, and joint reasoning was tested on synthetic planning tasks.The conclusion also notes that exact wording and benchmarks requiring domain knowledge remain weaker areas.Source: §6 · p. 23

A.1 Representation encoders

  • The image stream uses frozen encoder features arranged as a 16×16 token grid, with learned linear projections into MM-RiT.SigLIP2 and WebSSL-DINO use their last patch states; the VAE uses packed 2×2 latent neighborhoods, while pixels use a trainable patch stem.Source: §A.1 · p. 31, Table 9 · p. 31

A.2 Representation normalization

  • Text states are normalized before entering the shared backbone; Flan-T5 uses z = h/0.2, while other text encoders are standardized per channel.For the other encoders, the fixed per-channel means and standard deviations are computed over valid training text positions.Source: §A.2 · p. 31, §A.1 · p. 31

A.3 MM-RiT architecture

MM-RiT processes projected image and text representations through joint attention while retaining modality-specific streams. Fixed decoders read out generated representations, with parallel decoding preserving caption fidelity under noisy states.

A.3 MM-RiT architecture

  • Image and text representations are projected to a shared width for joint attention, then use modality-specific residual and feedforward paths and separate output projections.The default 668M model has two text-only input layers followed by 17 joint attention blocks, with four time tokens per modality.Source: §A.3 · p. 32

A.4.1 Image decoders

  • Image representations use their corresponding reconstruction decoder: pretrained Scale-RAE decoders for SigLIP2-SO400M and WebSSL-DINO, a pretrained VAE decoder, or pixel unpatchification.The Scale-RAE construction follows Zheng et al., using decoders [7].Source: §A.4.1 · p. 33

A.4.2 Text decoders

  • The final parallel text decoder maps 128 generated states to 128 tokens in one pass, with every position attending to every other position and no answer-length input.It is trained on text alone, without MM-RiT, and decodes up to the first predicted EOS.Source: §A.4.2 · p. 33
  • For Qwen3-E, Qwen3-VL-E, and SigLIP2 text, parallel decoders add an eight-layer bidirectional trunk to a linear branch initialized from each representation’s readout head.On held-out captions, each decoder recovers more tokens than its starting linear head.Source: §A.4.2 · pp. 33–34
  • 98.4% of held-out captions are recovered exactly from clean states by the parallel decoder, versus 93.7% when the clean fraction is 0.7; the autoregressive decoder reaches 58.5% at that fraction.The linear head recovers 98.7% of tokens but only 50.8% of captions exactly.Source: §A.4.2 · p. 34, Table 14 · p. 35

A.5 Frozen components and total parameters

  • The parameter accounting distinguishes trainable backbone parameters, modules executed for each direction, and the union of modules loaded to support both directions.Source: §A.5 · p. 36

A.6.1 Training details

Training samples joint and conditional denoising tasks, corrupts normalized target representations, and updates MM-RiT with a stabilized velocity loss. Per-position corruption extends the tasks to partial inputs.

A.6.1 Training details

  • The stabilized velocity loss divides predicted and target velocities by the same d(t) = max(1 − t, 0.05), matching ordinary linear-path flow-matching velocity loss below the floor region.Above 0.95, the denominator floor turns the loss into a stabilized surrogate near the clean endpoint rather than exact velocity supervision.Source: §A.6.1 · p. 37
  • Task losses average squared velocity error over valid generated positions and feature channels, with active counts excluding padding and clean context.A task or modality with no active targets contributes zero.Source: §A.6.1 · p. 38
  • Target streams receive span corruption with 15% probability and independently sampled per-position times with 10% probability; otherwise positions share a stream time.The operations apply to normalized target latents and do not construct the clean VQA question condition.Source: §A.6.1 · p. 38
  • Frozen encoders produce representations for each microbatch, and the loss updates MM-RiT parameters, including its input and output projections, while encoders and decoders remain fixed.Two exponential moving averages of MM-RiT weights are also maintained.Source: §A.6.1 · p. 38

A.6.2 Inference details

Inference follows separate conditional or joint sampling paths, advancing both generated streams from the same current pair. Guidance uses a generic caption reference by default; reference choice changes measured image-generation quality.

A.6.2 Inference details

  • Each joint sampling step computes both stream velocities from the same current image-text pair before either stream advances.A stream with no target positions stays fixed.Source: §A.6.2 · p. 39
  • Text-to-image sampling uses 25 Euler steps and guidance w = 5, while captioning and question answering use 16 steps and w = 3.Joint reasoning uses 16 backbone calls, and unconditional joint generation uses 64 steps; neither uses guidance.Source: §A.6.2 · p. 39
  • 48.41 GenEval with the generic-caption reference exceeds 44.64 with time swap for the default mixture at w = 5; the corresponding DPG-Bench scores are 78.85 and 78.45.The generic caption scores higher for every tested mixture and guidance scale from 2 to 5.Source: §A.6.2 · p. 39, Table 20 · p. 40
  • For the default mixture, GenEval improves through w = 5, while T2I-only and Multi-task peak at w = 3; with each model’s best tested scale, T2I-only reaches 51.23 and leads the default mixture by 2.8 points.At those per-model scales, the default mixture leads Multi-task (44.29) and Joint only (31.33).Source: §A.6.2 · p. 39, Table 20 · p. 40
  • On WISE, the finetuned SigLIP2 model scores 0.447 with the generic-caption reference at w = 5 and 0.392 with time swap at w = 5.Time swap at w = 3 gives the highest WISE scores measured, but the study keeps one reference and scale for all text-to-image benchmarks.Source: §A.6.2 · p. 39, Table 21 · p. 40
  • For question answering, the image and separately encoded question remain fixed at inference while a fixed-length answer canvas starts from noise and is updated.The question condition is encoded separately from the question-and-answer target sequence; the answer content and length do not enter that conditioning representation.Source: §A.6.2 · pp. 39–40

A.7 Data, optimization, and training stages

The final models proceed through pretraining, supervised finetuning, and task-specific planning adaptation. Pretraining combines caption pairs and visual-understanding supervision, while SFT separates image generation from image-to-text training.

A.7 Data, optimization, and training stages

  • The final 2.1B models are pretrained for 60k updates, then undergo supervised finetuning and separate planning-task adaptation.Both models are adapted for mazes; the SigLIP2 model is also adapted for visual spatial planning.Source: §A.7 · p. 40, Table 23 · p. 41
  • Pretraining uses 38M image-caption pairs with recaptioned text, including CC12M, SA-1B, and JourneyDB images, plus Cambrian-7M for visual-understanding supervision.The listed dataset sources are cited as, [60], [61], and.Source: §A.7 · p. 41
  • SFT mixes text-to-image rows with image-to-text question answering, caption-prompt, and long-answer rows, but includes no joint denoising.Generated answer-window length depends on prompt type rather than answer length.Source: §A.7 · p. 41, Table 24 · p. 41

B.1 Generative frameworks

  • The framework comparison uses the 668M d19 backbone family and a 50k-update schedule; its configurations distinguish autoregressive, flow-matching, and masked-diffusion generation.The controlled comparisons share pretraining settings, except budget controls that vary only update count.Source: Table 22 · p. 40, Table 26 · p. 42, §B.1 · p. 42

B.1.1 Framework implementations

The appendix details the framework controls and design studies, including how text conditioning is represented, how joint and conditional supervision is compared, and how representation choice relates to reconstruction and downstream generation.

B.1.1 Framework implementations

  • The four framework controls vary representations, generation objectives, attention masks, and output interfaces while holding the MM-RiT trunk and training schedule fixed.JUICE uses frozen T5 states and continuous text generation; the other configurations generate discrete text tokens, and the fully discrete control also uses VQ image codes.Source: §B.1.1 · pp. 42–44, Figure 13 · p. 43

B.1.2 Matching the text-to-image conditioning encoder

  • Replacing token embeddings with T5 conditioning raises Transfusion-style GenEval from 44.23 to 46.07 and DPG from 76.84 to 78.24, but explains only part of its gap to JUICE.The corresponding GenEval gains are 3.08 for D-DiT-style and 1.74 for MMaDA-style; the comparison does not isolate the remaining differences.Source: §B.1.2 · p. 44

B.2.1 Supervision and cross-modal information

  • The task-mixture controls distinguish target assignment from cross-modal information by comparing dual-conditional, blocked-joint, and single-target-joint variants with Mixing.Dual conditional replaces a joint pass with two conditional passes; blocked joint masks cross-modal paths, while single-target joint supervises only one modality per joint example.Source: §B.2.1 · pp. 44–45
  • Mixing finishes 2.51 GenEval and 1.08 CIDEr ahead of matched DC, and 2.92 GenEval and 2.22 CIDEr ahead of blocked joint, after 50k updates.The authors caution that DC changes both context and computation; against Multi-task at matched work, it gains 6.9 GenEval but loses 7.9 CIDEr and 2.5 DPG.Source: §B.2.1 · p. 45, Table 27 · p. 45

B.3.1 Diagnostics of encoders and decoders

  • The text-readout diagnostic decodes held-out caption representations without MM-RiT generation, separating representation recovery from the generation of continuous states.Table 29 reports token accuracy for linear readout heads and the parallel decoders used in the pretraining study.Source: §B.3.1 · p. 46, Table 29 · p. 46
  • Across representation configurations, the most faithful reconstructions do not yield the strongest downstream generation: FLUX VAE has the lowest rFID, while DINO and SigLIP2 achieve stronger GenEval.The passage concludes that reconstruction fidelity alone does not determine the downstream generation ranking.Source: §B.3.1 · p. 46, Table 30 · p. 46, Table 28 · p. 46

C Extended benchmark comparisons

The extended comparisons provide additional views of image generation and visual understanding, including sampling-step results and benchmark coverage.

C Extended benchmark comparisons

  • The extended understanding results list ten benchmarks for both JUICE variants, including two VQAv2 splits, and compare additional reported benchmarks across models.Only values reported in each model’s paper are included in the remaining-benchmark comparison.Source: C Extended benchmark comparisons · p. 47
  • The paper includes an image-generation comparison with larger models and reports GenEval results for JUICE across different sampling-step counts; the main text uses 25 steps.Source: Table 31 · p. 47, Table 32 · p. 47

D.1 Mazes: dataset, adaptation, and evaluation

The appendix specifies planning-task datasets and sampling evaluations, then gives qualitative examples across generation, captioning, and visual spatial planning. Maze sampling compares agreement between paired outputs, while visual spatial planning separates training-size results from larger-map extrapolation.

D.1 Mazes: dataset, adaptation, and evaluation

  • Maze joint success requires the drawn route and move sequence to show the same valid shortest route, and evaluation accepts any valid shortest route rather than only the training target.The dataset contains puzzles with two to sixteen shortest routes and tests 500 layouts absent from training.Source: §D.1 · pp. 47, 49

D.2 Maze sampling budgets

  • On the same 500 mazes, image-first sampling solves more paired cases than coupled sampling for SigLIP2, while the DINO difference is not statistically significant.At 16 calls, coupled sampling alone solves 10 mazes versus 25 for image-first with SigLIP2 (p=0.017), and 19 versus 29 with DINO (p=0.19); halving the budget changes coupled sampling by at most 0.3 points.Source: §D.2 · p. 49, Table 37 · p. 50

D.3 Maze trajectory case studies

  • In maze case studies, coupled sampling can revise either modality toward agreement: the route changes to match fixed moves in one example, while moves change to match a fixed route in another.Independent counterparts end with two valid but different routes.Source: §D.3 · p. 50

D.4 Visual spatial planning

  • Visual spatial planning evaluates move sequences on FrozenLake maps, with levels 7 and 8 testing grid sizes absent from JUICE’s adaptation training.Any move sequence reaching the goal without entering a hole counts as success; the official benchmark score checks moves, while held-out maps also assess joint consistency.Source: §D.4 · pp. 50–51
  • On visual spatial planning, moves reach the goal on 100 maps at each of levels 3–5, 99 at level 6, 94 at level 7, and 69 at level 8.The drawn path is a shortest route on 98, 100, 100, 99, 99, and 96 maps across levels 3–8, respectively; levels 7–8 are larger than training maps.Source: §D.4 · p. 51

E.1 Image generation conditioned on text

  • The image-generation examples use shared prompts to show imagined scenes, varied visual styles, and natural or architectural compositions.Examples include a lighthouse wave, a watercolor sailboat, a fox in a snowy forest, and a cathedral interior.Source: §E.1 · p. 52, Figure 14 · p. 52, Figure 15 · p. 53

E.2 Image captioning

  • The qualitative caption examples show that SigLIP2+T5 and DINO+T5 can mention different entities, attributes, and scene details for the same images.The long-answer examples use the prompt “Describe this image in detail,” with the two model outputs shown in separate columns.Source: §E.2 · p. 55, Figure 17 · p. 55, Figure 18 · p. 56

E.3 Joint generation

  • The joint-generation examples pair images with text produced from noise without a text prompt, and the sampling visualization shows both streams’ estimates evolving over time.The text is read out with an autoregressive caption decoder; in the illustrated samples, later steps revise both streams.Source: §E.3 · p. 57, Figure 19 · p. 57, Figure 20 · p. 58

E.5 Visual question answering

The selected visual-question-answering examples span relational, material, knowledge, existence, action, and object queries. Both model configurations match the listed reference on several examples, while other examples show their responses side by side.

E.5 Visual question answering

Related papers

Loading 2610.28801v1…