PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Multi-Image Re-Representation in Multi-Image Understanding

Gengyuan Zhang, Xiao Han, Xinyu Xie, Tong Liu, Volker Tresp

arXiv:2609.39363v1cs.CVcs.AI

TL;DR

Multi-image understanding requires organizing evidence distributed across images, and the paper asks when textual or visual re-representation is more useful. It introduces Mosaic and compares re-representation approaches, finding task-dependent benefits and showing that reinforcement learning can teach an agent to compose visual operations over multiple steps.

  • Problem

    Multi-image tasks can depend on subtle, differently oriented, or distributed evidence, and existing benchmarks mix task demands that make it difficult to identify when visual re-representation helps.

    Source: §1 · pp. 1–2

  • Method

    The paper compares textual and visual re-representation, introduces Mosaic for constructing image intermediates, and trains MosaicAgent-8B with reinforcement learning using accuracy and format rewards.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Visual re-representation benefits are task-dependent, and accuracy-and-format-reward training enables multi-step visual operation composition with diverse problem-solving patterns.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The relative value of textual and visual re-representation depends on task demands, with visual re-representation particularly effective when precise visual evidence is needed.

    Source: §6 · p. 10

    The closed-evidence conditional-independence property follows from stated assumptions rather than empirical findings and concerns the complete original input.

    Source: §3.1 · pp. 3–4

Abstract

from arXiv · show

Multi-image understanding requires MLLMs not only to recognise the content of individual images, but also to organise visual evidence distributed across them. We study this problem through multi-image re-representation, viewing prompted Chain-of-Thought reasoning and agentic visual tool use as different ways of re-organising visual evidence during reasoning. We introduce Mosaic, a general-purpose multi-image visual harness that enables an MLLM to actively construct visual intermediates with ten composable image operations. We compare five re-representation settings on existing multi-image benchmarks and on MosaicBench, a new grounding-focused benchmark for fine-grained multi-image understanding. Our experiments show that the relative benefits of textual and visual re-representation are strongly task-dependent. Visual re-representation is particularly effective for tasks requiring precise visual evidence, including hypothesis testing, precision comparison, and orientation-sensitive reasoning, while tasks dominated by higher-level semantic content show smaller or less consistent gains. Building on this finding, we train MosaicAgent-8B to use Mosaic with reinforcement learning using only accuracy and format rewards. Without demonstration trajectories or rewards for specific tool-use, the agent learns to compose visual operations over multiple steps and exhibits diverse problem-solving patterns unpromptedly. Code and data will be released at https://github.com/gengyuanmax/Mosaic.

1 INTRODUCTION

The paper frames multi-image re-representation as a way to reorganize visual evidence, asking when visual representations help more than textual reasoning. It compares re-representation approaches, introduces Mosaic and MosaicBench, and studies whether reinforcement learning can teach an agent visual tool use.

1 INTRODUCTION

  • Visual re-representation helps most on tasks requiring precise visual evidence or fine-grained cross-image relations, while semantic tasks show smaller or less consistent gains.The paper introduces MosaicBench to examine fine-grained multi-image understanding.Source: §1 · p. 2
  • The paper compares direct answering, free-form and guided textual reasoning, online visual re-representation, and prefabricated visual intermediates.Textual re-representation includes standard CoT prompting (Wei et al., 2022); visual approaches construct or use image views.Source: §1 · pp. 1–2
  • Mosaic lets an MLLM construct and reuse intermediate image views through operations such as cropping, geometric transformation, and image composition.MosaicBench provides a grounding-focused evaluation for fine-grained multi-image understanding.Source: §1 · p. 2
  • Reinforcement learning with accuracy and format rewards, without demonstrations or rewards for specific tool sequences, teaches MosaicAgent-8B multi-step visual operation composition.The resulting trajectories exhibit diverse problem-solving patterns not explicitly prescribed by the training objective.Source: §1 · p. 2
  • The paper formulates multi-image re-representation and defines five settings for comparing ways to organize evidence.Its contributions also include Mosaic, MosaicBench, and training MosaicAgent-8B.Source: §1 · p. 2

2 RELATED WORK

Related work covers multi-image benchmarks and models, textual approaches to organizing visual evidence, and visual agents that manipulate inputs or construct intermediates.

2 RELATED WORK

  • Multi-image research spans video-frame temporal reasoning, spatial reasoning across views, and image-collection comparison; MANTIS develops multi-image abilities through interleaved instruction tuning (Jiang et al., 2024).BLINK, MMIU, and M4Bench evaluate perception-intensive tasks and semantic, temporal, spatial, or alignment relationships (Fu et al., 2024; Meng et al., 2024).Source: §2 · p. 2
  • Textual methods organize visual evidence through intermediate rationales, question-guided descriptions, or multi-image caption chains (Zhang et al., 2023; Kao et al., 2025).These approaches operate in an encode-then-reason setting with interleaved images and text (Alayrac et al., 2022; Jiang et al., 2024).Source: §2 · p. 2
  • Visual-agent work includes local image inspection, sketching, executable-code intermediates, and perception tools, alongside reinforcement-learning training and task-specific multi-image agents.Examples include DeepEyes, Visual Sketchpad, PyVision, and VipAct (Zheng et al., 2025; Hu et al., 2024; Zhao et al., 2025).Source: §2 · p. 3

3 MULTI-IMAGE RE-REPRESENTATION FOR MULTI-IMAGE UNDERSTANDING

The paper defines re-representation as an intermediate record that organizes visual evidence for answering a question, then distinguishes textual and visual forms. Mosaic provides a persistent workspace for composing image operations and reusing their outputs.

3.1 MULTI-IMAGE RE-REPRESENTATION

  • Multi-image re-representation constructs a variable-length intermediate record from the question and source images to organize visual evidence for answering the question.The solver uses this record alongside the original input.Source: §3.1 · p. 3
  • Textual re-representation uses a text sequence as its intermediate record, while online visual re-representation creates image intermediates through operations and records the model-tool interaction trace.The trace contains model-generated text and tool requests or stop actions, together with returned observations.Source: §3.1 · p. 3
  • Under the closed-evidence assumptions, the intermediate record is conditionally independent of the answer given the complete original input; the stated property is not an empirical finding.The discussion distinguishes the complete input from compressed visual tokens, which may not expose details that cropping and re-encoding make accessible to the solver.Source: §3.1 · pp. 3–4
  • The five settings range from no explicit intermediate reasoning through free-form or guided textual reasoning to online visual re-representation and prefabricated visual intermediates.Prefabricated intermediates separate their use from their construction by omitting the interaction traces and tool access.Source: §3.1 · p. 4

3.2 MOSAIC: A MULTI-IMAGE VISUAL HARNESS

  • Mosaic stores source images and derived views as persistent assets, and its ten deterministic operations can transform individual views, combine assets, expose pixel differences, or add spatial references.Operation outputs become new reusable assets; Figure 1 depicts asset selection, structured calls, returned views, and subsequent reuse.Source: §3.2 · pp. 4–5, Figure 1 · p. 4
  • Figure 2 presents accuracy by re-representation setting and model, with task-level cells and overall benchmark accuracy bars.The caption identifies BLINK, M4Bench, and MosaicBench as the benchmarks represented.Source: Figure 2 · p. 5

4 COMPARING MULTI-IMAGE RE-REPRESENTATION METHODS

This section compares five forms of multi-image re-representation on BLINK and M4Bench to examine which approaches benefit which tasks. It then introduces MosaicBench for evaluating tasks that require precise visual evidence and relations.

4.1 EVALUATING ON EXISTING BENCHMARKS

  • On Detailed Difference, Qwen3-VL-8B scores 7.3% with No-Re2, 46.1% with T-Re2, and 59.9% with V-Re2; V-Re2 also beats T-Re2 by 6.0 points for Qwen3-VL-32B and GPT-5.4.Prompt-guided textual re-description reaches 45.7% for Qwen3-VL-8B; on State Comparison, all three models perform worse with V-Re2 than T-Re2.Source: §4.1 · p. 5
  • On BLINK Relative Depth, Qwen3-VL-8B reaches 82.3% with prefabricated visual intermediates, versus 78.2% with T-Re2 and 74.2% with V-Re2.Providing visual intermediates can help where online re-representation does not.Source: §4.1 · p. 6

4.2 MOSAICBENCH: A GROUNDING-FOCUSED VISUAL UNDERSTANDING BENCHMARK

  • MosaicBench task coverage spans six visual challenges: resolution, orientation, precision comparison, hypothesis testing, context interference, and spatial reference.Its task examples contain one to five images and task-specific questions.Source: §4.2 · p. 6
  • The benchmark uses annotation augmentation from 12 public datasets, with controlled transformations creating related views through geometric changes, local edits, and rearrangements.Ground-truth answers follow source annotations or known construction parameters.Source: §4.2 · p. 6
  • MosaicBench evaluation contains 560 examples across 28 task types, with 20 examples per type.The benchmark excludes overlap with its training suite at the sample, source, and image levels.Source: §4.2 · p. 6

5 LEARNING MULTI-IMAGE VISUAL RE-REPRESENTATION

The paper trains MosaicAgent-8B to use Mosaic for multi-image visual re-representation and analyzes its benchmark performance and evolving tool-use behavior. Training improves benchmark results, with gains tied to constructed trajectories and a more varied mix of problem-solving modes.

5.1 TRAINING AN AGENT WITH MOSAIC

  • MosaicAgent-8B is initialized from Qwen3-VL-8B-Instruct and trained with GRPO using accuracy and format rewards, without supervised warm-up or demonstration trajectories.The training suite retains 3,509 examples across 32 task types, selected from examples where tool-enabled runs sometimes answer correctly when the no-tool run does not.Source: §5.1 · p. 7

5.2 EXPERIMENTAL ANALYSIS

  • MosaicAgent-8B achieves 63.3% on MosaicBench and 61.8% on M4Bench, exceeding the strongest evaluated open-weight baselines by 2.4 and 4.3 percentage points, respectively.On Mantis, it reaches 81.6%, close to the highest open-weight result of 81.9%.Source: §5.2 · p. 7
  • With the harness, MosaicAgent-8B scores 63.3% on MosaicBench and 61.8% on M4Bench, beating same-suite CoT RL by 10.3 and 10.2 points.CoT RL alone raises MosaicBench accuracy from 45.2% to 53.0%, while M4Bench changes from 51.3% to 51.6%.Source: Table 2 · p. 8, §5.2 · p. 8
  • Over 218 RL steps, mean accuracy reward rises from 0.48 to 0.63, while MosaicBench tool calls increase from 1,830 to 3,037.Cropping grows from 36.7% to 54.8% of calls and collage from 0.5% to 6.2%, while pixel differencing falls from 16.1% to 3.3%.Source: §5.2 · p. 8, Figure 4 · p. 9

5.3 DOES TRAINING IMPROVE RE-REPRESENTATION CONSTRUCTION OR UTILISATION?

  • Post-training trajectories raise final-prefix answerability by 11.0 points with the backbone solver and 10.5 points with the trained solver.The advantage is absent over the first two tool steps; post-training trajectories average 5.41 steps versus 3.16 before training.Source: §5.3 · p. 8, Figure 5 · p. 9

5.4 HOW ANSWERABILITY EVOLVES DURING RE-REPRESENTATION

  • Progression's share falls from 41.9% to 35.5%, while self-correction and post-stabilisation continuation become more frequent, producing a mix of progression, revision, and continued processing.Self-correction can follow an intermediate representation that moves the model away from a correct answer before later operations recover it; continuation may verify an answer or be unnecessary.Source: §5.4 · pp. 9–10, Figure 6 · p. 9

6 CONCLUSION

The study finds that textual and visual re-representation have task-dependent benefits, with visual tools particularly effective when tasks require precise visual evidence. It also finds that accuracy- and format-reward training can produce multi-step visual tool use without demonstrations or prescribed tool sequences.

6 CONCLUSION

  • Visual re-representation is particularly effective for hypothesis testing, precision comparison, and orientation-sensitive tasks, while its relative benefit depends on the task.Reinforcement learning with accuracy and format rewards also enables multi-step tool composition and diverse problem-solving patterns without demonstrations or rewards prescribing specific behaviors.Source: §6 · p. 10

AI USE STATEMENT

  • The authors report using generative AI for translation, methodological feedback, qualitative analysis, literature work, and figure creation or modification, and say they reviewed AI-assisted work.They state that at least three authors reviewed all code and that AI-assisted figure generation did not alter underlying data.Source: AI USE STATEMENT · p. 10

APPENDIX

  • The appendix covers Mosaic tool definitions and training, MosaicBench construction, and extended tool-ablation, model-comparison, and failure-pattern results.The appendix overview also lists MosaicBench task coverage, data provenance, validation, and reproducibility details.Source: APPENDIX · p. 14

A MOSAIC: A VISUAL HARNESS FOR MULTI-IMAGE UNDERSTANDING

A.1 TOOL DEFINITIONS

  • The operations include collaging, cropping, rotation, flipping, affine transformation, and homography transformation of images.They also include overlaying, resizing, drawing a normalized coordinate grid, and producing per-pixel difference heatmaps.Source: §A.1 · p. 14

A.2 PROMPT ENGINEERING

  • Tool-mode and no-tool comparisons combine tool access with differences in image metadata and reasoning instructions, so they do not isolate tool availability.The paper says an isolating control would match the metadata and general reasoning instructions across settings.Source: §A.2 · p. 16

A.3 TRAINING SETUP

The training setup specifies GRPO optimization and rollout limits, while the following appendix section documents MosaicBench and its associated training data.

A.3 TRAINING SETUP

  • GRPO uses eight rollouts per prompt with group-relative advantage normalization and no learned critic, alongside AdamW, a 1 × 10^-6 learning rate, batch size 32, and one epoch.The KL and entropy coefficients are zero, and training uses four NVIDIA H100 GPUs.Source: §A.3 · p. 16
  • Each multi-turn episode is limited to ten assistant turns, with maximum prompt and response lengths of 16,384 and 20,480 tokens, respectively.Source: §A.3 · p. 16

B MOSAICBENCH: A DATASET FOR MULTI-IMAGE UNDERSTANDING

B.1 DATASET OVERVIEW AND TASK COVERAGE

The dataset overview describes a 32-task construction for the training suite and MosaicBench, with tasks organized by visual challenge.

B.1 DATASET OVERVIEW AND TASK COVERAGE

  • The construction contains 32 task types for the training suite and MosaicBench.Source: §B.1 · p. 17
  • The task coverage draws on source material listed in Table 3, with source documentation and training/evaluation selection described in later sections.Source: §B.1 · p. 17, Table 3 · p. 17
  • Table 3 groups the tasks by their main visual challenge and reports the number of input images per example.Source: Table 3 · p. 17

B.2 DATA SOURCES AND PROVENANCE

The dataset is assembled from public and procedural sources, with provenance records, task-specific construction and validation, and exact-match scoring.

B.2 DATA SOURCES AND PROVENANCE

  • The task collection uses 12 public datasets and two procedural generators, with Table 4 listing source subsets and associated task IDs.Source: §B.2 · p. 18, Table 4 · p. 18
  • Each sample records its source, split, fingerprint, and licence; the authors exclude an HPatches stitching-matrix task because its corpus overlaps an external benchmark task.The source-level split and source-specific exceptions are also recorded.Source: §B.2 · p. 18

B.3 SAMPLE FORMAT AND SHARED CONSTRUCTION

  • Each task defines input selection or generation, a question template, and a ground-truth relation; 400 examples per task yield 12,800 examples before train/evaluation selection.Each example is stored as a JSON record, and labels come from source annotations or known construction parameters.Source: §B.3 · p. 19
  • Spatial coordinates are normalized in prompts, with pixel equivalents retained in metadata for auditing; stored dimensions and task-specific preprocessing are specified separately.Source: §B.3 · p. 19

B.4 VALIDATION AND SHORTCUT CHECKS

  • Parsed answers are scored by exact match, with missing or unparseable answers incorrect; Table 7 describes rollout outcomes and partitions of constructed examples.Its caption identifies a row of examples answered incorrectly in a single no-tool run and three final rows partitioning the examples.Source: §B.3 · p. 19, Table 7 · p. 20
  • Validation checks structural consistency and task-specific label criteria, including geometric reprojection, angular error, assembly seams, and mask-based conditions.Point-localization checks include tolerance around the target and minimum separation from competing locations.Source: §B.4 · p. 19
  • Shortcut probes test heuristics using supplied matrices, angles, or coordinates without inspecting image content.Table 6 describes these probes as based on transformation and spatial geometry.Source: §B.4 · p. 19, Table 6 · p. 20

B.5 TRAINING AND EVALUATION SET CONSTRUCTION

Training and evaluation examples are selected through repeated tool-enabled and no-tool rollouts, with benchmark examples filtered to reduce overlap with training. The task-specific construction covers small targets and dense objects as well as geometric transformations and document orientation.

B.5 TRAINING AND EVALUATION SET CONSTRUCTION

  • The training suite retains examples missed without tools but solved in two to seven of eight tool-enabled runs, yielding 3,509 examples across 32 task types.A separate hard subset contains 4,032 examples that were missed without tools and solved in at most one tool-enabled run; it is not used for training.Source: §B.5 · p. 20
  • MosaicBench examples are filtered for overlap by sample identity, source provenance, and image similarity, including perceptual-hash checks for synthetic and CARPK images.The resulting benchmark contains 560 examples across 28 task types, with 20 examples per type.Source: §B.5 · pp. 20–21

B.6 TASK-SPECIFIC CONSTRUCTION

  • Task construction specifies each input, question, ground-truth relation, and validation checks, with resolution tasks targeting small visual objects or dense object collections.The examples include tracking small targets across frames, identifying lesions or defects in candidate windows, and counting annotated cars.Source: §B.6 · p. 21, §B.6.1 · p. 21, §1.1 · p. 21

1.1 Small-target cross-frame tracking.

  • Small-target tracking follows an annotated target across four frames, while dense-object counting asks for the total cars in a fully annotated image.The tracking example uses a target averaging 1.9% of frame width, or about 15 pixels at the reference input scale.Source: §1.1 · p. 21, §1.7 · pp. 21–22

B.6.2 ORIENTATION

  • Orientation tasks test recovery of rotations, reflections, and perspective geometry, with labels derived from known transformations and checked using angular or corner-reprojection errors.Assembly orientation alignment asks for the rotation aligning two views; homography tasks ask for the transformation reversing an applied warp.Source: §B.6.2 · p. 22, §2.3 · p. 22

2.9 Oblique document rectification.

  • Document rectification tasks ask for a homography that restores an oblique page or an additional counterclockwise rotation that makes text upright.Ground truth is based on annotated document corners or the construction-derived rotation, with corner-error or angular checks.Source: §2.9 · p. 22

B.6.3 PRECISION COMPARISON

The task collection probes fine-grained visual comparisons, transformations, context interference, and spatial reference. Construction checks use annotations or known image operations to define targets and distinguish distractors.

B.6.3 PRECISION COMPARISON

  • Precision-comparison tasks include detecting building changes across dates and finding a defect absent from a reference despite independent photometric jitter.The comparison labels are grounded in change masks or composited defect masks, rather than unrelated appearance differences.Source: §B.6.3 · p. 23, §3.10 · p. 23

3.11 Symmetric self-comparison.

  • Symmetry-breaking and spot-the-difference tasks locate real defect patches or local edits amid reflected counterparts, compression noise, and other image variation.Examples include symmetric self-comparison, JPEG-encoded image pairs, and altered interface elements.Source: §3.11 · p. 23, §3.12 · p. 23

3.13 GUI regression verification.

  • GUI regression verification asks which candidate region contains an edited interface element, using annotated element boxes to define the target.Edits may recolour, shift, or remove an element.Source: §3.13 · p. 23

B.6.4 HYPOTHESIS TESTING

  • Hypothesis-testing tasks ask models to identify known transformations, including an affine map between views and a rotation from template to observed image.Transformation labels follow the construction, with distractors checked against reprojection error or angle separation.Source: §B.6.4 · pp. 23–24, §4.3 · p. 24

4.3 Template rotation for grasping.

  • Template-matching tasks distinguish a rotated figure from reflected alternatives and test whether shuffled image tiles can be assigned to restore the original arrangement.Multi-tile reassembly uses seam discontinuity thresholds to separate the original layout from distractors.Source: §4.3 · p. 24

B.6.5 CONTEXT INTERFERENCE

  • Context-interference tasks separate target properties from surrounding structure or appearance, including mirror reflections and subtle grey-value differences.Mirror windows are checked against a reflection mask, while patch labels depend on printed intensities rather than surrounding backgrounds.Source: §B.6.5 · p. 24, §5.2 · p. 24, §5.5 · p. 24

B.6.6 SPATIAL REFERENCE

  • Spatial-reference tasks ask for coordinates tied to a grid, drivable region, or annotated change, using track positions or change masks as ground truth.Examples include a four-frame grid-cell sequence, a point on the drivable corridor, and a point at the centre of a building change.Source: §B.6.6 · p. 25

6.6 Lesion-centre localisation.

  • Lesion-centre localisation uses the lesion annotation to define a target point and applies a 0.07 tolerance with distractors at least 0.2 away; reproducibility details record construction seeds and source-access procedures.Source attribution and licence information are recorded per sample.Source: §6.6 · p. 25, §B.7 · p. 26

C EXTENDED RESULTS

The extended results compare the full visual toolset with crop-only use and examine failure profiles after training. Additional operations provide the largest reported accuracy gain on MosaicBench, while training changes the types of failures more than their total number.

C.1 BEYOND CROPPING

  • 9.4 percentage points on MosaicBench and 3.0 points on M4Bench, but only 1.0 on Mantis and 0.4 on BLINK, while MMIU falls by 1.1 points with the full toolset versus crop alone.The benefit of operations beyond cropping is most pronounced on MosaicBench.Source: §C.1 · p. 27, Table 8 · p. 27

C.2 FAILURE PATTERNS

  • Training leaves failures nearly unchanged at 231 →221 but reshapes them: fewer trajectories lose an initially correct answer, while more lose a correct intermediate answer or settle on a wrong answer early.The figure profiles answerability before and after RL on failed tasks.Source: Figure 7 · p. 27, §C.2 · p. 27
Loading 2609.39363v1…