Source-linked AI summary
Visual Jigsaw Post-Training Improves MLLMs
Penghao Wu, Yushan Zhang, Haiwen Diao, Bo Li, Lewei Lu, Ziwei Liu
TL;DR
Current MLLM post-training is largely text-centric, leaving intrinsic fine-grained visual understanding comparatively under-addressed and often requiring extra visual-generation designs. Visual Jigsaw instead trains natural-language permutation recovery with verifiable rewards, and its application across images, videos, and 3D data consistently improves the corresponding visual capabilities.
Problem
MLLM post-training predominantly uses dense visual inputs as sparse evidence for text reasoning, while existing visual reconstruction approaches add generative components and objectives.
Method
Visual Jigsaw partitions and shuffles visual inputs, then trains MLLMs to output the correct permutation in natural language using RLVR-compatible, self-supervised rewards.
Results
Visual Jigsaw consistently improves fine-grained perception, temporal reasoning, and 3D spatial understanding across images, videos, and 3D data.
Takeaways & Limitations
Vision-centric self-supervised post-training is a complementary path for strengthening MLLM visual understanding without altering text-only architectures.
Takeaways & Limitations
The approach has not been extensively scaled to larger training datasets and models, and some 3D jigsaw variants did not yield expected improvements.
Abstract
from arXiv · showhide
Reinforcement learning based post-training has recently emerged as a powerful paradigm for enhancing the alignment and reasoning capabilities of multimodal large language models (MLLMs). While vision-centric post-training is crucial for enhancing MLLMs' intrinsic understanding of visual signals, current post-training paradigms are predominantly text-centric, where dense visual inputs are only leveraged to extract sparse cues for text-based reasoning. There exist a few approaches in this direction, however, they often still rely on text as an intermediate mediator or introduce additional visual generative designs. In this work, we introduce Visual Jigsaw, a generic self-supervised post-training framework designed to strengthen visual understanding in MLLMs. Visual Jigsaw is formulated as a general ordering task: visual inputs are partitioned, shuffled, and the model must reconstruct the visual information by producing the correct permutation in natural language. This naturally aligns with reinforcement learning from verifiable rewards (RLVR), requires no additional visual generative components, and derives its supervisory signal automatically without any annotations. We instantiate Visual Jigsaw across three visual modalities, including images, videos, and 3D data. Extensive experiments demonstrate substantial improvements in fine-grained perception, temporal reasoning, and 3D spatial understanding. Our findings highlight the potential of self-supervised vision-centric tasks in post-training MLLMs and aim to inspire further research on vision-centric pretext designs. Project Page: https://penghao-wu.github.io/visual_jigsaw/
1 INTRODUCTION
Visual Jigsaw addresses the underuse of dense visual information in predominantly text-centric MLLM post-training by introducing a self-supervised ordering task. Applied across images, videos, and 3D data, it improves fine-grained perception, temporal understanding, and 3D spatial reasoning without additional generative components.
- 1 INTRODUCTION: The framework targets a gap in text-centric multimodal post-training, where dense visual signals are often reduced to sparse evidence for text reasoning.Prior visual reconstruction approaches add generative components and objectives, while the optimality of pixel-level reconstruction remains uncertain.
- 1 INTRODUCTION: Visual Jigsaw partitions and shuffles visual inputs, then trains MLLMs to generate their correct permutation in natural language during RL post-training.The task uses deterministic ground truth and requires no additional annotations or visual generative designs.
- 1 INTRODUCTION: The method is positioned for post-training because solving the ordering task requires an existing foundation of visual understanding.The authors also motivate RL post-training as a way to transfer vision-centric skills to downstream applications.
- 1 INTRODUCTION: Across images, videos, and 3D data, Visual Jigsaw improves fine-grained perception, temporal understanding, and 3D spatial reasoning.The reported improvements follow modality-specific ordering tasks for patches, temporal clips, and 3D structures.
2 RELATED WORKS
Related work has largely emphasized reconstruction, text-based reasoning, or task-specific multimodal objectives, while jigsaw tasks provide a lightweight ordering signal suited to text-output MLLMs. Visual Jigsaw extends this direction as a verifiable, architecture-preserving post-training framework across multiple visual modalities.
- RELATED SELF-SUPERVISED LEARNING: Jigsaw-style objectives formulate visual learning as recovery of spatial or temporal structure, but their traditional representation-learning performance has been weaker than dominant alternatives.Prior work includes image patch ordering and video frame or spatiotemporal ordering.
- MLLM VISUAL UNDERSTANDING: Visual Jigsaw adapts ordering tasks to understanding-based MLLMs by using a lightweight, verifiable objective compatible with textual outputs and requiring no additional generative modules.Its inputs can be partitioned into modality-specific elements, shuffled, and reordered by the model.
- MLLM VISUAL UNDERSTANDING: Visual reconstruction methods improve visual understanding but require extra generative modules and objectives, with limited validation in stronger MLLMs.The cited work also leaves open whether dense reconstruction is the optimal strategy.
- MLLM RL POST-TRAINING: Figure 2 depicts image patch, video clip, and 3D depth-ordering variants, each evaluated by comparing the predicted ordering with ground truth.Partial correctness receives a partial accuracy reward in the illustrated framework.
- MLLM RL POST-TRAINING: Most multimodal RL post-training methods target text-based reasoning or task-specific objectives rather than intrinsic visual perception.Some perception-oriented methods still derive training signals from caption errors, textual responses, or text–image alignment.
3 METHOD
Visual Jigsaw formulates visual understanding as a verifiable ordering task: modality-specific elements are partitioned, shuffled, and reconstructed through natural-language permutation predictions. The framework applies this design to images, videos, and practical RGB-D-based 3D inputs.
- 3 METHOD: Visual Jigsaw partitions, shuffles, and reconstructs modality-specific visual elements by having the MLLM predict their original permutation.The same ordering formulation covers images, videos, and 3D data, with modality-specific rules for creating jigsaw elements.
- 3 METHOD: A graded reward gives exact permutations reward 1, valid partial permutations discounted credit, and invalid permutations reward 0.The partial-credit term is scaled by γ, while validity checks prevent repeated-index outputs from exploiting the reward.
- 3.1.2 IMAGE JIGSAW: For images, the method splits inputs into m × n raster-ordered patches, randomly permutes them, and trains recovery of the original patch arrangement.The reported training setup uses 118K COCO images and a 3 × 3 grid, producing 9 patches per image.
- 3.1.3 VIDEO JIGSAW: For videos, the method uniformly segments frames into K non-overlapping clips, shuffles them, and predicts the permutation restoring chronological order.Training uses 100K LLaVA-Video videos divided into 6 clips, with boundary trimming and minimum-duration filtering.
4 EXPERIMENTS
Visual Jigsaw post-training consistently improves vision-centric MLLM capabilities across image, video, and 3D benchmarks, with gains also examined under alternative training and task settings.
- 4.2.1 IMAGE JIGSAW: Image Jigsaw consistently improves fine-grained perception, monocular spatial understanding, and compositional visual understanding benchmarks.The reported explanation emphasizes attention to local patch details, global layouts, and inter-patch relations.
- 4.2.3 3D JIGSAW: 3D Jigsaw significantly improves all evaluated 3D benchmarks, including depth comparison, single-view, multi-view, and egocentric video inputs.The largest gain occurs on DA-2K, which is directly related to the depth-ordering task.
- 4.3 ABLATION STUDIES AND DISCUSSIONS: Reinforcement learning yields larger gains than supervised fine-tuning, while SFT can degrade LISA-Grounding and OVD-Eval performance.The SFT results suggest weaker transfer from the jigsaw task on some benchmarks.
- 4.3 ABLATION STUDIES AND DISCUSSIONS: Easier image and video jigsaw tasks still improve performance, indicating that task difficulty affects but does not eliminate the post-training benefit.The ablation reduces image grids from 3 × 3 to 2 × 2 and video clips from six to four.
- 4.3 ABLATION STUDIES AND DISCUSSIONS: Visual Jigsaw is also evaluated on the reasoning-oriented ThinkLite-VL model for improved visual perception while preserving reasoning ability.The setup enables the KL constraint during image-jigsaw training.
5 CONCLUSION
Visual Jigsaw formulates visual understanding as a verifiable ordering problem optimized with RLVR, avoiding dense reconstruction and additional generative modules. Across images, videos, and 3D data, it consistently improves vision-centric capabilities and supports perception-focused self-supervised post-training.
- 5 CONCLUSION: Visual Jigsaw is a verifiable self-supervised post-training framework that formulates visual understanding as an ordering problem optimized with RLVR.The framework avoids dense visual reconstruction and integrates with text-only MLLMs.
- 5 CONCLUSION: The approach yields consistent improvements across images, videos, and 3D data in fine-grained perception, temporal reasoning, and 3D spatial understanding.The conclusion presents these results as evidence of the framework’s generality.
- 5 CONCLUSION: The work highlights perception-focused self- and weakly-supervised tasks as a complementary path toward more capable and robust multimodal models.This conclusion motivates further exploration of vision-centric post-training objectives.
A.1 ADDITIONAL EXPERIMENTS ON 3D JIGSAW
The study compares depth ordering with view–motion matching and BEV–pose matching as alternative 3D jigsaw designs. Preliminary results find the two matching variants do not significantly improve downstream benchmarks and underperform depth ordering.
- View–Motion Matching: View–motion matching selects an anchor view and shuffled candidates, requiring the model to match candidates using ego-motion descriptions.
- BEV–Pose Matching: BEV–pose matching uses an annotated bird’s-eye-view image and shuffled candidate views, requiring each camera pose to be matched to its view.
- The evaluation compares depth ordering, view–motion matching, and BEV–pose matching on 3D benchmarks.
- Analysis and Results: The two matching variants do not significantly improve downstream benchmarks and underperform the depth-ordering formulation.The authors hypothesize that limited base-model 3D perception and reasoning constrain transfer to downstream tasks.
A.2 VISUAL JIGSAW EXAMPLES
The paper provides examples of the visual jigsaw tasks for images, videos, and 3D data. These examples are presented in Figures 4, 5, and 6.
- Examples of the visual jigsaw tasks for all three modalities are provided in Figures 4, 5, and 6.
- The examples cover image, video, and 3D visual inputs.
- The figures illustrate the task examples rather than reporting experimental results.
A.3 QUALITATIVE EXAMPLES
Qualitative examples are provided for models trained with image, video, and 3D jigsaw tasks. The examples appear in Figures 7, 8, and 9.
- Qualitative examples are shown for models trained with image, video, and 3D jigsaw tasks.
- Image-task qualitative examples appear in Figure 7.
- Video- and 3D-task qualitative examples appear in Figures 8 and 9.
A.4 LIMITATIONS AND FUTURE WORKS
The paper identifies limitations in the simplicity and scalability of its image and video jigsaw formulations, as well as in some 3D variants. It proposes richer jigsaw designs, larger-scale studies, stronger 3D foundations, and broader vision-centric self-supervision as future directions.
- Limitations and Future Work: Image and video jigsaw use relatively simple formulations, motivating richer or hybrid designs such as joint spatial-temporal partitioning and heterogeneous piece sizes.
- Limitations and Future Work: The study does not extensively scale training data or model size because of computational constraints.
- Limitations and Future Work: Some 3D jigsaw variants do not yield the expected improvements, motivating evaluation with stronger 3D reasoning capabilities and richer 3D priors.
- Limitations and Future Work: The paper suggests exploring self- and weakly supervised vision-centric tasks beyond jigsaw.
A.5 TASK PROMPTS
The task prompts instantiate visual ordering across images and videos, requiring MLLMs to reconstruct spatial or chronological structure and output the resulting index sequence in natural language.
- Image jigsaw: Image prompts require arranging shuffled tiles into the correct 3×3 layout and returning their indices in raster-scan order.The tiles are presented with shuffled indices, and the answer is a comma-separated sequence from left to right and top to bottom.
- Response format: The prompts specify answer formatting with reasoning enclosed in <think> tags and the final sequence enclosed in <answer> tags.This format appears in both the tile-ordering and point-ordering instructions.
- Video jigsaw: Video prompts require mentally reassembling shuffled clips chronologically, from the earliest segment to the latest, and outputting their clip indices.The clips are shown with shuffled indices and must be returned as a comma-separated chronological sequence.
- Monocular spatial ordering: A separate image prompt asks the model to order marked points from closest to farthest relative to the camera.Depth is judged using the centers of red circular markers, with the ordered point numbers returned in sequence.