PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

Zican Hu, Xuyang Hu, Yiming Liu, Zuwei Long, Wei Liu, Yunzhuo Hao, Jiawei Gu, Linjie Li, Yu Cheng, Zhenhong Sun, Weibo Gu, Xing Sun, Zhi Wang

arXiv:2607.03748v1cs.AI

TL;DR

Existing RL formulations optimize text while leaving intermediate image generation outside the training loop, limiting joint learning for interleaved reasoning. BRAID casts text-image-text generation as a unified MDP with shared advantages and vision-thinking rewards, improving BAGEL by +5.73 avg. across seven benchmarks and surpassing GPT-4o with 7B parameters.

  • Problem

    Existing RL formulations optimize text tokens while leaving intermediate image generation outside the training loop, leaving interleaved reasoning only partially optimized.

  • Method

    BRAID casts interleaved text-image-text reasoning as a unified two-level MDP, propagating shared advantages through modality-native text and image policy gradients.

  • Results

    +5.73 avg. across seven benchmarks, BRAID improves BAGEL and surpasses GPT-4o with only 7B parameters.

  • Takeaways & Limitations

    Across seven spatial-reasoning and visual-perception benchmarks, BRAID consistently outperforms various baselines.

  • Takeaways & Limitations

    BRAID depends on BAGEL’s hybrid AR-diffusion backbone, an external VLM judge, and a fixed interleaving pattern.

Abstract

from arXiv · show

Unified multi-modal models (UMMs) have shown promising interleaved text-image reasoning capabilities, yet effectively optimizing such multi-turn generation via reinforcement learning (RL) remains an open challenge. Existing approaches apply RL exclusively to text steps, relegating image generation to supervised surrogates, preventing policy gradients from propagating through the full interleaved trajectory across heterogeneous modalities. This leaves the potential of RL for UMMs largely untapped. In the paper, we introduce \textbf{BRAID} (\textbf{B}ridging inte\textbf{R}le\textbf{A}ved mult\textbf{I}-modal reasoning as a unified \textbf{D}ecision process), a simple framework that casts multi-turn text-image-text reasoning as a unified Markov decision process (MDP), enabling joint optimization of textual and visual generation via a single, principled RL objective. BRAID computes a shared trajectory-level advantage and propagates it coherently into both text tokens and image denoising paths, each optimized through its modality-native policy gradient mechanism. To further address long-horizon credit assignment, BRAID employs a vision-language model (VLM) judge that scores each intermediate image on its reasoning utility, supplying dense turn-level feedback to sharpen learning at critical visual branches. Experiments on spatial reasoning and visual perception benchmarks show that BRAID consistently outperforms various baselines, confirming that a unified MDP formulation with vision-thinking guidance is essential for effective multi-modal reasoning.

1 Introduction

BRAID frames interleaved text-image-text reasoning as a unified MDP, jointly optimizing textual and visual generation under one RL objective. It addresses incomplete prior optimization and improves BAGEL across seven benchmarks, including SAT and V*Bench.

  • Introduction: Interleaved Text-Image-Text reasoning alternates verbal thinking with intermediate image generation to externalize perceptual steps beyond language.UMMs bridge understanding and generation across modalities within a single framework.
  • Introduction: Supervised fine-tuning on curated TIT-CoT traces is constrained by the cost and scarcity of high-quality interleaved data.RL is presented as an alternative that learns from task rewards rather than curated data.
  • Introduction: Prior methods leave TIT-CoT half-optimized because policy gradients reach text tokens but not intermediate image generation.COOPER uses a separate supervised stage for visual generation, while DeepEyes reaches visual tool calling rather than image generation.
  • Introduction: BRAID casts multi-turn text-image-text reasoning as a unified MDP, jointly optimizing textual and visual generation under one RL objective.Each turn in the text→image→text→···→answer trajectory is formulated as a macro-action, without separate training stages or modality-specific reward designs.
  • Introduction: +5.73 avg. across seven benchmarks is BRAID’s improvement over BAGEL, surpassing GPT-4o with only 7B parameters.Gains are +14.00 on SAT and +10.76 on V*Bench; Maj@n scales efficiently with sampling budget while baselines plateau.

2 Related Work

Prior work extends chain-of-thought reasoning with interleaved visual states and applies reinforcement learning separately to textual reasoning or image-generation processes. However, existing unified-model approaches commonly exclude visual generation from the RL loop or optimize it in a separate supervised stage.

  • Interleaved Multi-Modal Reasoning: Pure text limits expressiveness for visually grounded tasks, while unified architectures enable interleaved text-image reasoning with intermediate visual states.These architectures integrate generation and understanding within a single backbone.
  • RL Fine-tuning: Verifiable-reward RL has advanced reasoning in LLMs and VLMs, while diffusion denoising and flow matching have been formulated as RL-compatible decision processes.VLM applications remain limited to textual action spaces, whereas image-generation methods model denoising as an MDP.
  • RL for Interleaved Multi-Modal Reasoning: Unified-model RL studies often exclude visual generation from policy optimization, using separate supervised training or restricting policy gradients to visual tool calling.For autoregressive backbones, Emu3.5 and other studies apply verifiable-reward or hybrid-reward RL; for hybrid architectures, COOPER separates image-generation training and DeepEyes backpropagates to visual tool calling.

3 Preliminaries

The preliminaries describe GRPO’s group-normalized policy optimization and flow matching as the foundation for reinforcement learning in text and image generation. DiffusionNFT injects reward into flow-matching training without likelihood or score estimation.

  • Group Relative Policy Optimization: GRPO samples G rollouts per prompt, scores each with a scalar reward, and assigns a group-normalized advantage.The advantage normalizes each rollout reward by the group mean and standard deviation, with κ ensuring numerical stability.
  • Group Relative Policy Optimization: GRPO updates the policy with a clipped surrogate objective using per-token importance ratios and KL regularization against a frozen reference model.ε sets the clipping threshold, while η controls KL regularization strength.
  • RL for Image generation: Flow matching models image generation with a conditional velocity field that transports Gaussian noise to a clean image through a probability-flow ODE.Training regresses the velocity field toward the ground-truth velocity on interpolated noisy samples xt = (1 − t)x0 + tϵ.
  • RL for Image generation: DiffusionNFT converts flow-matching training into reward-aware regression by splitting samples into positive and negative branches weighted by an optimality probability.The current velocity field is linearly mixed with a frozen behavior velocity; high-reward samples move toward the target, while low-reward samples move away, without likelihood or score estimation.

4 Method

BRAID formulates interleaved text-image reasoning as a two-level MDP, enabling shared trajectory advantages to optimize text tokens and image denoising through modality-native policy gradients. A VLM-based vision-thinking process reward provides dense, decoupled credit assignment for intermediate image turns.

  • Two-level MDP: BRAID models each text chunk or intermediate image as a macro-action in an outer MDP, then expands it into token-generation or denoising micro-actions.The outer process spans interleaved reasoning turns, while the inner process uses autoregressive tokens for text and flow-matching denoising for images.
  • Unified policy optimization: Text and image branches share one interleaved trajectory and turn-level advantage while using GRPO-style clipping for text and DiffusionNFT for images.The unified RL loop combines distinct generative mechanisms in one joint backward pass, with λ balancing the modalities.
  • Unified policy optimization: BRAID injects image-turn advantages into flow matching by converting them into soft reward labels, steering the velocity field toward high-advantage images and away from low-advantage images.The soft label is rk = σ(Â(sk, Ik)/υ) ∈[0, 1].
  • Vision-thinking process reward: A VLM judge scores each intermediate image for Visual Correctness, Visual Fidelity, Reasoning Utility, and Trustworthiness, supplying dense feedback for image turns.The process reward addresses the weakness of broadcasting a single sparse terminal scalar across long trajectories.
  • Vision-thinking process reward: Decoupled advantage estimation normalizes terminal and visual rewards against separate baselines before combining them, reducing scale mismatch and temporal sensitivity.Text turns use terminal advantage alone, whereas image turns combine terminal advantage with λvis · visual advantage.

5 Experiments

Experiments evaluate BRAID across multimodal reasoning benchmarks against VLM and UMM baselines, showing strong gains from SFT initialization and subsequent RL. Ablations and qualitative analyses indicate that jointly optimizing image and text branches with vision-thinking rewards improves reasoning capacity and visual faithfulness.

  • Comparison among Unified Models: BAGEL reaches 59.46 average, outperforming Janus-Pro (46.05) and Chameleon (27.49) among comparable-scale unified models.These results motivate building BRAID on BAGEL’s hybrid autoregressive-diffusion architecture.
  • Comparison among Unified Models: Qwen2.5-VL-7B scores 62.11, exceeding BAGEL by 2.65 points, while Qwen2.5-VL-72B reaches 70.53.The comparison shows that unified generative versatility does not automatically yield strong visual reasoning without dedicated post-training.
  • Spatial Reasoning and Visual Perception Performance: +2.53 points from SFT and +2.73 points from RL improve over BAGEL on average, lifting BRAID to 65.19.The SFT stage provides a reliable initialization for on-policy exploration, after which RL adds further average gains.
  • Reasoning Capacity Boundary: BRAID’s VStar Maj@n rises from 66.5 to 70.7 as samples increase from 3 to 17, while SAT reaches 61.3 at Maj@17.On SAT, BRAID exceeds SFT (56.7) and BAGEL (52.3) by +4.6 and +9.0, respectively; Avg@n remains consistently higher as well.
  • Ablation Study: Removing rvis lowers average performance from 65.19 to 62.68, while removing the image branch causes steeper declines on VStar (−6.00) and CV-Bench 2D (−4.32).The rvis ablation especially hurts VStar (−4.48) and SAT (−2.67), whereas the shared trajectory advantage still propagates terminal reward across modalities.
  • Interleaved Reasoning: Only BRAID produces faithful intermediate visuals and standalone images satisfying both spatial and numerical constraints; its ablations induce distortions, artifacts, or numerical errors.Qualitative visual-search results show hallucinated reasoning without LNFT and fabricated artifacts without rvis, while text-to-image tests expose instruction-following failures in both variants.

6 Conclusions, Limitations, and Future Work · Appendix

BRAID unifies interleaved text-image-text reasoning as an MDP, jointly optimizing textual and visual generation with shared trajectory-level advantages and VLM-based turn-level feedback. The paper identifies extending beyond BAGEL’s hybrid backbone and replacing the external VLM judge as key future directions, while the appendix details data-allocation regimes and SFT/RL overlap.

  • 6 Conclusions, Limitations, and Future Work: BRAID casts interleaved text-image-text reasoning as a unified MDP for joint textual and visual generation.Its shared trajectory-level advantage propagates reward signals across modality boundaries through modality-native policy gradients.
  • 6 Conclusions, Limitations, and Future Work: A VLM judge supplies vision-thinking process rewards that provide turn-level feedback and sharpen credit assignment.The feedback complements BRAID’s trajectory-level optimization.
  • 6 Conclusions, Limitations, and Future Work: Extending BRAID to fully autoregressive UMMs with discrete image-token prediction would broaden its applicability beyond BAGEL’s hybrid AR-diffusion backbone.The proposed direction targets architectures where images are generated through next-token prediction.
  • 6 Conclusions, Limitations, and Future Work: Replacing the external VLM judge with a compact, jointly trained PRM could reduce inference cost and dependency while improving reproducibility.The current vision-thinking reward relies on an external VLM judge.
  • Appendix: The appendix training corpus spans eight task families, each contributing both SFT and RL splits, and defines overlap ratio ω = |D_SFT ∩ D_RL|/|D_RL| ∈ [0, 1].The task families are Matterport3D, RealSee3D, Perspective, DirectionalQuery, 360+x, PositionedDirection, Jigsaw, and VisualSearch.
  • Appendix: The appendix compares three data-allocation regimes, including a disjoint D2:1 regime with ω = 0 and an overlapping D∩ regime.In D∩, the RL pool adds 4,800 SFT-pool samples, yielding ω ≈ 0.32.

A.2 Evaluation Benchmarks

The evaluation covers seven benchmarks across spatial reasoning and visual perception, using VLMEvalKit with accuracy as the primary metric. The benchmarks include multi-image spatial intelligence, dynamic and static spatial reasoning, and fine-grained visual perception tasks.

  • Evaluations cover seven benchmarks across spatial reasoning and visual perception under the VLMEvalKit framework, with accuracy as the primary metric.
  • Spatial Reasoning: Spatial reasoning includes MMSI-Bench, with 1,000 multiple-choice questions across 10 spatial tasks curated from over 120,000 images, and SAT for dynamic and static reasoning.
  • Visual Perception: Visual perception includes BLINK, with 3,807 questions reformulated from 14 classic computer vision tasks, and V∗Bench for fine-grained perception in high-resolution crowded images.

A.3 Training Recipe · A.4 Vision-Thinking Process Reward Prompt · B Additional Results

The training recipe uses a two-stage SFT→RL pipeline built on BAGEL-7B, with jointly optimized trainable modules during RL and controlled interleaved image rollouts. The vision-thinking reward process uses GPT-5.2 to score each intermediate image across four criteria with integer ratings from 1 to 10.

  • A.3 Training Recipe: Training proceeds from SFT to RL on BAGEL-7B, using FSDP2 with hybrid sharding and gradient checkpointing while keeping the VAE frozen.During RL, the LLM, ViT, understanding head, and generation head are unfrozen and jointly optimized.
  • A.3 Training Recipe: The SFT stage follows ThinkMorph on curated TIT-CoT trajectories under the D2:1 data regime, while RL defaults to the partially overlapping D∩ replay regime with ω≈0.32.This replay setup bridges imitation learning and on-policy exploration.
  • A.3 Training Recipe: RL rollouts use temperature 1.0, up to 3 interleaved image rounds, 2048 thinking tokens per round, and 32768 total thinking tokens.These settings define the rollout horizon and token budget for interleaved reasoning.
  • A.3 Training Recipe: Image generation uses 20 denoising steps, timestep shift 3.0, classifier-free guidance scales of 4.0 for text and 2.0 for images, and guidance interval [0.4, 1.0].Intermediate images use noise level 0.7, with the same 20 denoising steps for training and evaluation.
  • A.4 Vision-Thinking Process Reward Prompt: GPT-5.2 evaluates each generated intermediate image after considering the question, problem image(s), and image Ik through question understanding, image analysis, and scoring.The judge emits four integer scores corresponding to r(vc), r(vf), r(ru), and r(tr).
  • A.4 Vision-Thinking Process Reward Prompt: The evaluation prompt requires precise comparison of the generated image with the problem image(s), including preserved, changed, new, and potentially fabricated content.It instructs the evaluator to identify the needed visual information, ambiguities, and the visual operation best suited to the question.
  • A.4 Vision-Thinking Process Reward Prompt: Each intermediate image receives integer scores from 1 to 10 for action correctness, visual faithfulness, reasoning contribution, and misleading risk.The evaluator must return only valid JSON containing these four fields.

B.1 Ablation Study · B.2 Analysis of Reasoning Capacity

The ablation study evaluates BRAID components across seven benchmarks, while reasoning-capacity analysis shows BRAID leading across sampling budgets on VStar and SAT. Results highlight text-branch RL gains and benefits from both per-sample quality and diversity.

  • B.1 Ablation Study: Table 4 evaluates BAGEL, SFT, BRAID, and two ablated variants across seven spatial reasoning and perception benchmarks.The study reports complete numerical ablation results for all seven benchmarks.
  • B.1 Ablation Study: On MMSI, all RL variants outperform SFT, showing gains from text-branch RL even without rvis or LNFT.This pattern specifically concerns multi-image spatial tasks.
  • B.1 Ablation Study: On CV-Bench 3D, BRAID and w/o rvis recover the SFT regression, scoring 82.17 versus 81.67, while w/o LNFT reaches 81.50.The reported comparison indicates differing effects of the ablated components on this benchmark.
  • B.2 Analysis of Reasoning Capacity: Table 5 compares methods on VStar and SAT using majority voting and average scores at sampling budgets n ∈{3, 5, 9, 17}.The results correspond to the curves in Figure 3.
  • B.2 Analysis of Reasoning Capacity: BRAID achieves the highest VStar and SAT scores at every sampling budget, with performance gaps widening as n increases.This conclusion covers both Maj@n and Avg@n results.
  • B.2 Analysis of Reasoning Capacity: On VStar at n=17, BRAID’s gap over SFT is +3.15 for Maj@17 and +2.34 for Avg@17, indicating stronger quality and diversity.The passage attributes these differences to higher per-sample quality and greater diversity.

B.3 Analysis of Training Recipe

Training performance depends on both data allocation and RL update design. D2:1 provides a stable SFT foundation, replay improves downstream RL broadly, and a 2:1 rollout-to-update ratio yields the best average performance.

  • Data allocation: D1:1 SFT underperforms BAGEL on CV-Bench 2D (65.24 vs. 73.53), a −8.29 drop attributed to insufficient visual perception data.The reduced SFT budget fails to cover sufficient visual perception data.
  • Data allocation: D2:1 provides a stable SFT initialization for effective downstream RL, while replay D∩ improves D2:1 RL on every benchmark except tied BLINK (57.76).Replay produces the largest gains on SAT (+9.34) and VStar (+3.43).
  • Rollout-to-update ratio: The 2:1 rollout-to-update ratio achieves the best average (65.19 vs. 61.93), with advantages on MMSI (+7.20), SAT (+8.67), and VStar (+8.15).The 1:1 ratio leads on CV-Bench 3D (84.83) and BLINK (59.07), exceeding the 2:1 setting on both.

C Interleave Reasoning Case

The visual search case shows that BRAID generates faithful, diverse visual intermediates that support correct textual reasoning, whereas ablations can reach the same answer through hallucinated or corrupted evidence. Thus, answer correctness alone can conceal fragile reasoning and lower robustness.

  • Qualitative comparison: Without rvis, the generated image includes a hallucinated second license plate, creating unnecessary ambiguity despite the model selecting the correct answer.The fabricated plate reads “E-EI SV” alongside the actual plate.
  • Qualitative comparison: Without LNFT, a distorted blurry crop garbles the plate, leading the text model to hallucinate “GI01 EPD” and rely on faulty reasoning.The model incorrectly maps the corrupted reading to option A, although the final answer is correct.
  • Qualitative comparison: All variants answer correctly, but only BRAID produces visual evidence that genuinely supports the conclusion.The ablated variants achieve correctness despite their visual intermediates, not because of them.
  • Qualitative comparison: The ablated variants’ fragile correctness manifests as lower robustness under Maj@n evaluation.This distinction separates genuinely supported reasoning from answers produced despite unreliable visual intermediates.

D Text-to-Image Generation Quality

BRAID’s RL training improves standalone text-to-image quality and instruction adherence across spatial, numerical, and creative concept-binding prompts. The full model outperforms the compared variants by correctly satisfying compositional relations, exact counts, and semantic bindings.

  • Overall generation quality: BRAID improves image quality and instruction adherence across spatial compositionality, numerical accuracy, and creative concept-binding prompts.The comparison includes BAGEL, SFT, BRAID w/o rvis, BRAID w/o LNFT, and full BRAID.
  • Spatial compositionality: Only BRAID renders the cat inside the fishbowl while preserving higher visual fidelity with a fish coexisting in the scene.BAGEL, SFT, and w/o LNFT place the cat next to the bowl; w/o rvis gets the relation right but lacks BRAID’s additional fidelity.
  • Numerical accuracy: Only BRAID generates exactly five balloons, whereas BAGEL produces approximately eight, SFT four, w/o LNFT many, and w/o rvis seven to eight.The result demonstrates improved precise counting under the joint RL objective.
  • Creative concept binding: BRAID clearly depicts a galaxy serving as the flowing sand in an hourglass, improving on w/o LNFT’s galaxy swirl and the other variants’ regular sand.BAGEL, SFT, and w/o rvis merely place the hourglass against a galaxy background.
  • Reward transfer: The vision-thinking process reward rvis provides optimization direction that transfers to general text-to-image generation by teaching the image branch to follow compositional instructions.The reward evaluates visual correctness and faithfulness during TIT-CoT training; w/o LNFT shows the weakest instruction following among RL-trained variants.
Loading 2607.03748v1…