Oddin: Latent Visual Reasoning in Unified Multimodal Models via Self-Distillation from Image Generation

1. Introduction

• Generating intermediate images incurs computational overhead because the model reconstructs pixel-level details, including redundant background regions, despite reasoning steps passing more compact information.Source: §1 · p. 1
• Most visual latent reasoning methods use VLMs that cannot generate images, making them inapplicable to tasks such as VBVR-Pro that require visual outputs.ULMMs have a generation branch that can produce images.Source: §1 · p. 1
• Oddin replaces intermediate images with latent token blocks processed by the understanding branch, reserving the generation branch for the final image.Offline self-distillation uses final-image generation, text-distillation, and special-token cross-entropy losses.Source: §1 · p. 1
• Oddin surpasses interleaved-generation and other baselines.The paper also reports that latent tokens attend to input regions relevant to each reasoning step.Source: §1 · pp. 1–2

2. Related work

• Interleaved-generation ULMMs produce intermediate images during reasoning; recent image-output benchmarks include VBVR-Pro, RIG-Bench, and GenPuzzle.Interleaved generation can serve as a teacher on suitable tasks, which Oddin exploits through self-distillation.Source: §2 · p. 2
• Visual latent reasoning replaces intermediate images with compact tokens, but existing VLM methods answer in text and cannot handle image-output tasks.LatentUM uses full sequences of generated visual tokens and requires new pretraining; Oddin adapts a pretrained ULMM through offline self-distillation.Source: §2 · p. 2

3.1. Preliminaries

• SenseNova-U1.5 uses shared self-attention with separate understanding and generation branches; predicting <img> triggers image generation.The understanding branch controls image generation, so intermediate images can be replaced without disrupting final-image generation.Source: §3.1 · p. 2

3.2. Interleaved Latent Reasoning of the Unified Model

• Oddin replaces each intermediate image with a latent block of K learnable embeddings enclosed by <lnt> and </lnt>, which the understanding branch processes during inference.The generation branch produces only the final image, and the model determines the number and timing of latent blocks.Source: §3.2 · p. 2

3.3. Offline Self-Distillation

• The teacher generates interleaved traces, and the student sequence replaces intermediate images with latent blocks while retaining the reasoning text and final image target.Teacher traces are filtered by keeping those whose final image passes the task-specific scorer.Source: §3.3 · p. 2, Figure 1 · p. 3

3.4. Training Objectives

• Training combines final-image generation, text-distribution distillation, and special-token cross-entropy losses.The image-generation objective penalizes errors in edited regions; text distillation aligns image-conditioned teacher and latent-conditioned student distributions, while special-token supervision targets <lnt> at intermediate events and <img> at the final event.Source: §3.4 · pp. 2–3

3.5. Trainable Parameters and Gradient Flow

• Training updates latent tokens, special-token embeddings, and latent-block-specific LoRA adapters while freezing other parameters.Generation and text-distillation losses backpropagate through frozen operations; special-token cross-entropy detaches hidden states to update only the two output embeddings.Source: §3.5 · p. 3

4.1. Experimental Setup

• Evaluation covers 700 held-out problems across seven VBVR-Pro-RL tasks, comparing Oddin with closed VLMs, ULMMs, and SenseNova-U1.5 in two modes.Training uses 17,051 successful teacher traces from seven selected tasks.Source: §4.1 · p. 4

4.2. Result

• 0.65 mean accuracy makes Oddin the top-performing baseline overall, while token generation falls 84% versus Interleaved Reasoning.It replaces an average of 7.9 images with one final image and four latent blocks.Source: §4.2 · p. 4
• In Dot To Dot, latent-block attention concentrates around the newly connected dot at each reasoning step, following the task’s drawing order.Figure 2 reports Gaussian-smoothed attention relative to uniform attention over the image, where 1 indicates an even spread.Source: §4.2 · p. 4, Figure 2 · p. 4

• Oddin uses offline self-distillation to train ULMMs for visual reasoning with compact latent tokens, without explicit intermediate-image supervision.The latent tokens are processed by the understanding branch.Source: §5 · p. 4
• Latent tokens attend to regions relevant to each reasoning step, supporting visual reasoning within the understanding branch.The paper concludes that the unified model need not generate intermediate images for this reasoning.Source: §5 · p. 5
• Oddin performs below the strongest baselines on some VBVR-Pro tasks; the authors identify limited precise spatial information and target detailed visual manipulation for future work.Source: §5 · p. 5

A. Training Objectives Details

• The appendix provides formulations and implementation specifics for Oddin’s three training-loss components.Source: A. Training Objectives Details · p. 6

A.1. Image Generation Loss

• The image-generation objective reconstructs the final image from a noisy state and explicitly penalizes errors in edited regions.The masked term averages errors across selected pixels and RGB channels, and evaluates to zero for empty masks.Source: §A.1 · p. 6

A.2. Text Distillation Loss

• Text distillation aligns teacher and student output distributions at aligned text positions, using the teacher’s top 50 tokens plus an aggregated tail category.The teacher-to-student KL divergence is summed across text positions per trace and scaled by a T^2 multiplier.Source: §A.2 · pp. 6–7

A.3. Special-token Cross-Entropy Loss

• Special-token cross-entropy targets <lnt> at intermediate image events and <img> at final image events, averaging the two losses equally when both occur.Traces without intermediate images optimize only the final <img> loss; detached preceding hidden states restrict gradients to the special-token output rows.Source: §A.3 · p. 7

A.4. Trainable Parameters

• Latent-token embeddings are independently trainable and reused in order across blocks, while LoRA adapters operate only within latent blocks and their boundary tokens.All other parameters remain frozen; image-generation and text-distillation losses update latent embeddings and adapters, while special-token cross-entropy updates only output rows.Source: §A.4 · p. 7

B.1. Hyperparameters

• The main configuration uses four latent tokens per block and LoRA adapters with rank 16 and α = 32.Source: §B.1 · p. 7
• Training runs for two epochs with a global batch size of 16 and an initial learning rate of 2 × 10^-4.The learning rate follows a cosine decay schedule.Source: §B.1 · p. 7
• Loss coefficients are λgen = 10, λKL = 0.005, and λCE = 0.1, with temperature T = 1.5.Source: §B.1 · p. 7

B.2. Training Data

• Seven VBVR-Pro tasks were selected because the teacher exceeded 20% success under interleaved reasoning and final images could be automatically verified.The tasks are Traffic Light, Separate Object Spinning, Dot to Dot, Rotation Puzzle, Track Object Movement, Domino Chain, and Sliding Puzzle.Source: §B.2 · p. 7
• The training pool combines 6,300 candidate problems from VBVR-Pro-RL with 21,000 randomly sampled VBVR-Pro-SFT problems, after holding out 700 RL problems for evaluation.SFT input images are rendered at 1024 × 1024 resolution.Source: §B.2 · p. 7
• 17,051 training traces were retained after teacher-generated final images passed task verification, and those images serve as reconstruction targets.The grader is adapted from the official VBVR-Pro evaluation code.Source: §B.2 · p. 7

B.3. Evaluation Details

• Closed-source gpt-4o and gpt-6.1-sol use GPT-Image-2 for final images, while open-weight baselines include BAGEL-7B, ThinkMorph, and SenseNova-U1.5.SenseNova-U1.5 is evaluated with Text CoT and Interleaved reasoning; Oddin replaces intermediate images with latent blocks while retaining final-image generation.Source: §B.3 · p. 7
• Open-weight models use greedy decoding with at most 4,096 new text tokens, while closed-source models receive one API call per problem and may make multiple image-tool calls.Open-weight image settings differ by model, and closed-source denoising steps and CFG are not controllable.Source: §B.3 · pp. 7–8
• Oddin autonomously chooses whether to emit another latent block or the final image, with a maximum of 12 blocks; Interleaved reasoning can generate up to 13 images.For Interleaved reasoning, scoring uses the last image before </think> when available; Text CoT and Oddin are scored on their final generated image.Source: §B.3 · p. 8

B.4. Prompts Setting

• Oddin’s prompt directs the model to use <latent> blocks for intermediate visual reasoning and generate a visible image only for the final requested state.Open-source models receive the original task prompt; closed-source models receive a modified prompt asking for an image of the final state.Source: §B.4 · p. 8

C.1. Additional Qualitative Analysis

• On Domino Chain, latent-block attention moves from the starting dominoes toward the two branches and often concentrates on dominoes about to fall next.The figure caption describes attention at each step as concentrating on dominoes falling at the current and next step.Source: §C.1 · p. 8, Figure 4 · p. 8

C.2. Masking Analysis of Latent Blocks

• Masking every latent block lowers accuracy in most tasks and brings it close to zero in Separate Object Spinning, Dot to Dot, and Sliding Puzzle.Figure 3 compares accuracy when blocks are masked from the first or last block; intervals are 95% bootstrap confidence intervals over 30 problems per task.Source: §C.2 · p. 9, Figure 3 · p. 9
• In Separate Object Spinning, masking from the last block reduces accuracy more, while in Dot to Dot, masking from the first block reduces it more.The results indicate that later blocks matter more for the former task, while earlier blocks carry information that later blocks cannot replace in the latter.Source: §C.2 · p. 9

C.3. Inference Cost

• 84% fewer total generated tokens are used by Oddin than by Interleaved reasoning, which generates 7.9 images on average versus Oddin’s single final image and about 4 latent blocks.Oddin also generates 49% fewer text tokens (725 →371), and its total is close to Text CoT’s 1,209 tokens.Source: §C.3 · p. 9, Table 2 · p. 9