Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

Minh-Quan Le; Armand Comas; Alexandros Lattas; Stylianos Moschoglou; Pedro Vélez; Amit Raj; Aaron Germuth; Thabo Beeler; Dimitris Samaras; Di Qiu

Source-linked AI summary

Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

Minh-Quan Le, Armand Comas, Alexandros Lattas, Stylianos Moschoglou, Pedro Vélez, Amit Raj, Aaron Germuth, Thabo Beeler, Dimitris Samaras, Di Qiu

arXiv:2607.13188v1Jul 14, 2026cs.LG

TL;DR

Existing multimodal samplers do not let text and images revise one another within denoising steps. The paper introduces SC-CMJP and CO2Jump, which couple confidence-driven transitions with remasking, achieving best joint performance across understanding, editing, and visual reasoning while scaling with denoising steps.

Problem
Existing multimodal samplers generate text and images without tightly coupling their within-step decisions, limiting concurrent multimodal generation.
Method
SC-CMJP couples modalities’ transition rates through cross-modal confidence and attention, while CO2Jump adds contradiction-triggered remasking in a training-free single-pass sampler.
Results
CO2Jump achieves best performance across joint image understanding, editing, and visual reasoning, with performance scaling monotonically with denoising steps.
Takeaways & Limitations
The results support coupled, self-correcting multimodal denoising as a unified approach to concurrent text-and-image generation.
Takeaways & Limitations
Standard masked diffusion cannot remask unmasked tokens, while modality-agnostic remasking scores fail to detect cross-modal contradictions.

Abstract

from arXiv · show

Human cognition does not separate understanding and generation. A teacher at a whiteboard speaks and draws $\textit{together}$, each modality reshapes the other. In this paper, we bring this coupled loop to artificial systems. Masked Diffusion Models (MDMs) are ideally suited to this task, yet existing samplers either decode text and image interleavedly or independently update them in parallel branches that share only previous-step history, but not the other modality's latest decisions $\textit{within}$ the same step; combined with MDMs' inability to remask, cross-modal contradictions are neither detected nor repaired. We introduce $\textbf{Self-Correcting Coupled Markov Jump Processes (SC-CMJP)}$, a framework in which one modality's transition rates are functionals of the other modality's confidence score, as weighted by cross-modal attention. Furthermore, a remasking jump retracts commitments the moment cross-modal evidence turns against them. In conjunction with SC-CMJP, we introduce $\texttt{CO}_\texttt{2}\texttt{Jump}$ (Self-$\underline{\text{CO}}$rrecting $\underline{\text{CO}}$upled $\underline{\text{Jump}}$), a novel training-free single-pass sampler for joint multimodal geneneration. For training and evaluation purposes, we have created and will release three large-scale joint multimodal generation corpora: $\text{JEdit-1M}$, $\text{JMaze-200K}$, $\text{JNono-200K}$, with matching in- and out-of-distribution benchmarks. $\texttt{CO}_\texttt{2}\texttt{Jump}$ achieves best joint performance for image understanding and editing as well as visual reasoning (maze and nonogram solving). The performance of the sampler scales monotonically with the number of denoising steps, evidence that the benefits of cross-modal coupling $\textit{compound}$ across the trajectory. Project page: https://coupled-jump.github.io

1. Introduction

The paper frames multimodal understanding and generation as a concurrent, tightly coupled loop and introduces SC-CMJP to operationalize cross-modal influence and self-correction during denoising. Its training-free CO2Jump sampler, supported by three released corpora and benchmarks, improves joint image-text performance and scales monotonically with denoising steps.

Motivation: Existing parallel MDM samplers factorize each denoising step, so text and image updates cannot use the other modality’s latest within-step decisions.This limits concurrency to interleaving updates over a shared previous-step history and leaves cross-modal contradictions unrepaired.
Core framework: SC-CMJP models text and image generation as a unified stochastic process in which each modality continuously informs and revises the other.Transition rates depend on the other modality’s confidence, weighted by cross-modal attention from the same backbone pass.
Sampler: CO2Jump is a training-free, single-pass sampler that couples per-modality transition rates through cross-modal attention and retracts commitments with remasking jumps.It runs on a frozen MDM without architectural changes or an auxiliary evaluator, using one forward pass per step.
Data and evaluation: The authors curate three large-scale joint-generation corpora—JEdit-1M, JMaze-200K, and JNono-200K—with matching benchmarks for image editing and visual reasoning.The reasoning tasks are maze and nonogram problems whose text and image outputs are logically interlocked and verifiable against algorithmic ground truth.
Results: CO2Jump consistently improves both modalities, beats existing sampling methods on concurrent joint image understanding and generation, and scales monotonically with denoising steps.The reported gains span image editing, maze solving, and nonogram solving, with especially strong joint-performance metrics.

2. Background

Masked discrete diffusion models can be represented as continuous-time Markov jump processes, but standard reverse sampling cannot remask erroneous commitments. ReMDM permits remasking, yet modality-agnostic schedules cannot detect cross-modal contradictions, motivating coupled jump processes whose rates use complementary-modality evidence.

Masked diffusion background: Masked discrete diffusion models corrupt tokens by replacing them with an absorbing mask state and learn to invert this corruption.The paper adopts the continuous-time formulation because it supports cross-modal coupling and remasking extensions.
Continuous-time formulation: The continuous-time formulation is equivalent to a CTMC whose stochastic jumps between clean tokens and the mask state recover the same marginals as discrete-time MDLM.The two formulations parameterize the same marginals, posteriors, and likelihood bounds and are used interchangeably.
Remasking limitation: Standard MDLM reverse sampling cannot remask unmasked tokens, so an error committed at one timestep persists through the rest of the trajectory.This is identified as a limitation of the standard reverse posterior.
Remasking limitation: ReMDM allows committed tokens to revert to the mask state, but its modality-agnostic schedules score remasking only from intra-modal likelihoods and miss cross-modal contradictions.The modified reverse posterior preserves the marginal under a stated bound on σ_t and recovers MDLM when σ_t = 0.
Coupled jump processes: Coupled Markov jump processes make one modality’s birth and death rates depend on the complementary modality’s representations and instantaneous confidence through cross-modal attention.This enables localized cross-modal self-correction during sampling.

3. Related Work

Prior work advances discrete diffusion through improved objectives, LLM scaling, decoding strategies, and unified text-image tokenizations, while concurrent multimodal samplers still lack within-step cross-modal feedback. Remasking has been pursued through predictor-corrector samplers and training-based modifications, whereas CO2Jump couples confidence signals with Death and Birth jumps.

Discrete Diffusion: Discrete diffusion evolved from absorbing-state D3PM into score-entropy, simplified-ELBO, and any-order autoregressive formulations, with subsequent scaling to LLMs.The passage attributes scaling to LLaDA, Dream, and SDAR.
Discrete Diffusion: Decoding improvements include block-wise generation and Top-K confidence selection, while image-token methods use VQ-VAE tokenization and unified text-image vocabularies.Examples of unified-vocabulary systems include Lumina-DiMOO and MMaDA.
Concurrent Multimodal Samplers: CO2Jump combines per-token self-confidence, cross-modal attention, and entropy-gated Coupled Confidence, then uses Death jumps to remask low-confidence tokens and Birth jumps to reveal high-confidence masked tokens.The sampler obtains confidence for both modalities from one forward pass and applies Birth jumps under the noise schedule.
Self-Correction via Remasking: Remasking research includes predictor-corrector samplers that reduce τ-leaping error without explicit remask jumps and training-based methods that modify the model.The training-based examples include GIDD, a separate hollow-transformer evaluator, and other cited approaches.
Concurrent Multimodal Samplers: UD-VLA factorizes modalities independently, while MMaDA-Parallel interleaves updates across steps but samples independently within each step, providing no instantaneous cross-modal feedback.Both updates condition only on the previous joint state, so coupling occurs through shared history rather than feedback within the same step.

4. Self-Correcting Coupled Markov Jump Processes

SC-CMJP reformulates joint multimodal generation as a coupled birth-death jump process in which cross-modal confidence guides decisions within each denoising step. CO2Jump combines asymmetric confidence scoring, entropy-based trust, and remasking to retract contradictions while preserving single-pass inference.

Coupled jump-process formulation: SC-CMJP replaces independent, irreversible modality updates with a shared birth-death jump process whose transition intensities are coupled across text and image.This targets modality drift caused by independent unmasking schedules.
Coupled jump-process formulation: The joint model implicitly learns cross-modal denoising because each position conditions on the entire joint state, including the complementary modality.It can reconstruct masked image patches from textual clues and masked text tokens from partially decoded images.
Confidence-based negotiation: CO2Jump uses self-confidence for text and an entropy-gated mixture of image self-confidence with cross-modal attention-weighted text confidence.The image side uses a single gate per denoising step, deferring to text when image uncertainty is high and relying on itself when text is uncertain.
Self-correction through remasking: Unlike standard masked diffusion, CO2Jump permits tokens in either modality to die by remasking when their scores fall below the schedule-driven death rate.Remasking retracts intra-modal errors and cross-modal contradictions.
Self-correction through remasking: The sampler performs death and birth jumps using asymmetric scores, remasking the lowest-scoring unmasked tokens and unmasking the highest-scoring remaining masked tokens.The image uses CoupledConf, while text uses SelfConf for both remasking and unmasking.

5. Datasets and Benchmarks for Joint Multimodal Generation

The paper introduces three joint-generation corpora—JEdit-1M, JMaze-200K, and JNono-200K—covering image editing and logically interlocked visual reasoning, each paired with held-out evaluation. Their shared multimodal records and task-specific benchmarks are designed to assess understanding and generation jointly.

Dataset overview: Three corpora—JEdit-1M, JMaze-200K, and JNono-200K—cover image editing, maze solving, and nonogram solving, each paired with a held-out benchmark.All share a record schema containing a prompt, source image, target image, structured understanding, and thinking trace.
JEdit-1M: JEdit-1M combines 724k ImgEdit pairs and 368k OmniEdit pairs into 1M tuples augmented with pixel-aligned scene graphs and logic-based thinking traces.The added supervision is synthesized by Qwen3-VL-235B and conditions the trace on both images, the prompt, and the scene graph.
Visual reasoning corpora: JMaze-200K contains 200k DFS-perfect mazes with grid sizes uniformly sampled from {6, . . . , 20}, while JNono-200K contains 200k nonogram puzzles spanning grid sizes {5, . . . , 25}.Maze outputs pair a blue solution-path image with an (r, c) text sequence; nonogram generation emphasizes multi-run row and column clues.
Visual reasoning corpora: Both reasoning datasets use parallel-form supervision, representing maze solutions with global coordinates and nonograms with bidirectional row/column constraint propagation.These representations avoid dependencies on autoregressive ordering and support parallel decoding.
Benchmarks and metrics: Each reasoning task has a 500-sample test set extending beyond the training grid-size range: JMaze-Test500 spans {3, . . . , 22} and JNono-Test500 spans {3, . . . , 27}.The held-out sets include in-distribution and out-of-distribution sizes, with joint accuracy requiring both text and image correctness.

6. Experiments

Across image editing and visual reasoning, CO2Jump delivers the strongest joint performance, with gains that grow across denoising steps and transfer to out-of-distribution grid sizes. Ablations and trajectories attribute these results to coupled confidence, entropy-based gating, and self-correction.

Image editing: CO2Jump achieves the strongest understanding score (0.369 overall) and best ImgEditBench score (1.93), while MMaDA-Parallel scores 0.335 and 1.44.The sampler with stronger understanding also achieves better image-editing performance.
Scaling sampling steps: CO2Jump improves monotonically from 8 to 512 NFE, with ImgEditBench rising from 1.72 to 1.93 and overall mAP from 0.074 to 0.369.At 512 NFE, its lead is +0.015/+0.016/+0.034 over MDM / ReMDM / MMaDA-Parallel.
Visual reasoning: CO2Jump outperforms other samplers in all six visual-reasoning columns, with gains ranging from +0.008 on Maze In-Dist to +0.062 on Nonogram OOD.No baseline is the runner-up everywhere, supporting sampler consistency across both task structures.
Out-of-distribution generalization: On OOD grids, CO2Jump leads Maze at 0.320 versus 0.312 and Nonogram at 0.175 versus 0.113, while remaining roughly flat across splits.MDM drops 55% and MMaDA-Parallel 21% from In-Dist to OOD on Nonogram.
Ablation: Removing Shared Percentile Rank causes the largest image-edit fidelity drop, from 1.93 to 1.87; removing Entropy-Based Gating cuts target-mAP by 0.030 and overall mAP by 0.015.The ablation links shared scaling to balanced confidence signals and gating to late-stage target-image commitment.
Mechanistic analysis: CO2Jump’s image-side remasks mostly occur when Self-Confidence and Cross Signal are both low, while high self-confidence with low cross-signal triggers coupling-driven revocations.Entropy-based gating shifts λ_image from approximately 0.05 to approximately 0.85 as text commits.

7. Conclusion

The paper presents SC-CMJP, where two modalities in a unified MDM negotiate commitments within every denoising step. CO2Jump instantiates this framework as a training-free single-pass sampler that couples confidence-driven transitions with contradiction-triggered remasking.

SC-CMJP lets two modalities in a unified MDM negotiate their commitments within every denoising step.
CO2Jump is a training-free single-pass sampler instantiated on a frozen backbone.
Cross-modal attention couples one modality’s transition rates to the other’s emerging confidence.
A remasking jump retracts committed tokens when cross-modal contradictions arise, closing the concurrent-generation loop left open by prior parallel samplers.

A. Qualitative Example from CO2Jump: Joint Image Editing and Understanding

A CO2Jump example jointly produces the prompt, edited image, scene-graph analyses, bounding boxes, and thinking trace within one denoising loop. The text branch plans a localized edit, and the image branch executes it consistently.

Joint image editing and understanding: CO2Jump concurrently generates the prompt, target image, scene-graph understanding, bounding boxes, and thinking trace in a single denoising loop.The sample comes from ImgEditBench and includes both source and generated images.
Joint image editing and understanding: The model’s text branch emits the overlaid bounding boxes itself, without an external grounder, using a normalized 1001×1001 canvas.The boxes are model predictions rather than ground truth.
Joint image editing and understanding: The text branch identifies only the shell as dynamic, while the image branch realizes the planned localized texture edit.All other entities retain identical bounding boxes in the source and target analyses.

B. Qualitative Example from CO2Jump: Maze Solving

CO2Jump jointly solves a JMaze maze by generating a textual thinking trace, coordinate path, and rendered solution concurrently in one denoising loop. Its reasoning identifies dead-end regions and main corridors before tracing the committed coordinate sequence onto the grid.

Joint maze solving: CO2Jump concurrently generates the maze-solving thinking trace, coordinate sequence, and rendered path in a single denoising loop.The text and image branches produce aligned outputs during the same trajectory.
Global reasoning: The thinking trace globally identifies the upper maze as a dead-end region and the lower corridors as the main artery.For the 12x12 maze, the start is at (8, 1) and the target at (6, 10).
Path commitment: After reasoning about corridor structure, the text branch commits an explicit (r, c) coordinate sequence that the image branch traces precisely on the grid.The solved maze overlays the resulting path in blue.

C. Qualitative Example from CO2Jump: Nonogram Solving

CO2Jump solves a 7×7 JNono nonogram through bidirectional constraint propagation, prioritizing globally constraining rows and columns before producing per-row filled-cell answers and the completed grid. Forced fills in one dimension tighten the other before commitments are made.

Constraint-propagation strategy: CO2Jump solves the JNono puzzle by propagating constraints bidirectionally between rows and columns rather than solving row by row.The trace uses forced fills from one dimension to constrain the other.
Example outputs: The example uses an empty 7×7 grid with row and column clues, a generated thinking trace, per-row filled-cell answers, and a solved black-cell grid.Figure 10 presents the input, reasoning, answer, and final visualization together.
Constraint-propagation strategy: The reasoning starts with the most globally constraining lines, including rows whose largest clues force blocks and columns whose clue sums match the grid width.These lines provide the earliest forced placements before the model commits the per-row ranges.

D. System Prompts for Dataset Curation … D.4. Thinking Prompt: Nonogram Parallel Reasoning

The dataset-curation prompts define structured visual understanding and logic-based reasoning traces for image editing, maze solving, and nonogram solving. They emphasize pixel-aligned evidence, explicit transition checks, and parallel or holistic constraint reasoning rather than purely sequential processes.

D.1. Understanding Prompt: Pixel-Aligned Scene Graph Extraction: The image-editing understanding prompt maps source-to-target visual changes while preserving spatial anchors and detailing edited-region deltas.It uses high-fidelity grounding focused on how visual tokens change in response to the editing prompt.
D.1. Understanding Prompt: Pixel-Aligned Scene Graph Extraction: The prompt requires explicit comparison of static, transformed, and newly added or deleted entities before answering.This comparison separates persistent context, attribute or state changes, and structural changes across the two images.
D.1. Understanding Prompt: Pixel-Aligned Scene Graph Extraction: Annotations must cover the main subject, edited objects, and key background anchors to provide global scene context.The required coverage includes contextual elements such as floors, skies, and walls.
D.2. JEdit-1M Thinking Prompt: Reasoning Trace Synthesis: The JEdit-1M reasoning prompt synthesizes a logic-based trace that evaluates whether the target image fulfills or deviates from the editing instruction.It uses source and target images, the editing prompt, and bounding-box object labels as inputs.
D.3. Thinking Prompt: Maze Solving Parallel Reasoning: The maze prompt combines maze imagery, textual adjacency structure, and a ground-truth path to elicit global-to-local parallel refinement.The trace should identify boundaries, bottlenecks, dead ends, and anchor corridors while forming the path concurrently.
D.4. Thinking Prompt: Nonogram Parallel Reasoning: The nonogram prompt supplies an empty and solved grid, textual clues, and filled cell ranges as inputs for parallel constraint propagation.The objective explicitly rejects row-by-row sequential solving and prioritizes globally constraining lines and definite-fill overlaps.