Source-linked AI summary
PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs
Ahmadreza Jeddi, Hakki Can Karaimer, Hue Nguyen, Zhongling Wang, Ke Zhao, Javad Rajabi, Ran Zhang, Raghav Goyal, Konstantinos G. Derpanis, Babak Taati, Radek Grzeszczuk
TL;DR
Puzzle-based RLVR for VLMs is limited by costly verification, flat or sparse rewards, collapsed exploration, and reasoning–answer inconsistency. PuzzleCraft addresses these issues with automatically verifiable puzzles, an exploration-aware curriculum, and RAC, producing consistent gains across Qwen2.5-VL and Qwen3-VL benchmarks.
Problem
Puzzle-based RLVR commonly uses sparse or flat rewards and curricula based mainly on reward statistics, while visual RLVR also faces costly verification and reasoning–answer inconsistency.
Method
PuzzleCraft uses three supervision-free, automatically verifiable puzzles with a curriculum combining difficulty and solution-space exploration, plus RAC for tracking reasoning–answer consistency.
Results
Across Qwen2.5-VL and Qwen3-VL, PuzzleCraft improves robustness, reasoning consistency, and downstream performance across broad image benchmarks, with positive transfer to video reasoning.
Takeaways & Limitations
Scalable puzzle-based RLVR benefits from curricula that account for both difficulty and rollout collapse together with explicit consistency-enhancing schemes.
Takeaways & Limitations
The paper identifies richer verifiable puzzle families, better structured-output exploration signals, and stronger consistency diagnostics as directions for future work.
Abstract
from arXiv · showhide
RL post-training with verifiable rewards (RLVR) has become a practical route to eliciting chain-of-thought reasoning in vision--language models (VLMs), but scaling it in the visual domain remains challenging due to costly or noisy supervision and reliance on external verifiers. Puzzle-based RLVR is a promising alternative, yet existing approaches often treat puzzle rewards as flat or sparse, which weakens group-relative learning signal. Existing curriculum strategies are overly restrictive: they rely mainly on reward statistics and do not account for exploration in the solution space, which can lead to collapsed rollout dynamics. Further, RL post-training can induce reasoning--answer inconsistency as training progresses. To address these shortcomings, we present PuzzleCraft, a supervision-free framework that scales vision-centric RLVR using a set of lightweight puzzle environments with built-in verification. PuzzleCraft instantiates three puzzles inspired by classic visual pretext tasks: PatchFit, Rotation, and Jigsaw. We introduce a curriculum that combines difficulty with an exploration signal derived from solution-space dispersion, and use it to downweight collapsed prompt groups. In addition, we introduce a new post-training metric, Reasoning-Answer Consistency (RAC), to measure the degree that the chain-of-though supports the answer, and show our exploration-aware curriculum improves RAC and downstream performance. Across a broad suite of vision-centric benchmarks, PuzzleCraft improves robustness and reasoning consistency, yielding consistent downstream gains on both Qwen2.5-VL and Qwen3-VL backbones. Overall, our results suggest that scalable puzzle-based RLVR benefits from curricula that account for both difficulty and solution-space collapse, together with explicit consistency-enhancing schemes.
1 Introduction
PuzzleCraft addresses weaknesses in vision-centric puzzle RLVR by combining verifiable puzzle environments with difficulty- and exploration-aware curriculum weighting and reasoning–answer consistency monitoring. Across Qwen2.5-VL and Qwen3-VL, it improves consistency, robustness, and downstream performance.
- Contribution: PuzzleCraft provides supervision-free RLVR through lightweight puzzle environments with built-in automatic verification, using PatchFit, Rotation, and Jigsaw.The environments require neither external verifiers nor teacher models.
- Contribution: Its curriculum upweights medium-hard prompts using reward variance and downweights groups whose rollouts collapse to the same solution.Exploration is measured with class entropy for binary puzzles and permutation diversity for Jigsaw.
- Contribution: RAC measures whether a reasoning trace supports its final answer, enabling analysis of consistency drift and consistency-aware reward schemes during post-training.The metric is used alongside curriculum and optional consistency-aware optimization.
- Results: PuzzleCraft’s exploration-aware curriculum consistently outperforms difficulty-only curricula and improves reasoning–answer consistency and downstream accuracy.Ablations attribute gains to both curriculum and consistency-aware reward schemes.
- Results: On Qwen2.5-VL-7B, Jigsaw improves average performance across nine image benchmarks by 2.2 points over the strongest puzzle-based baseline.On Qwen2.5-VL-3B, the best variant improves over baselines by up to 1.39 points.
- Results: Jigsaw improves Qwen3-VL average performance over corresponding Instruct checkpoints by +2.6, +2.99, and +1.87 for 2B, 4B, and 8B models.The method also transfers positively to video reasoning benchmarks without video data during post-training.
2 Related Work
Related work positions PuzzleCraft within RLVR, supervision-free post-training, GRPO curricula, and reasoning-consistency research. It responds to known weaknesses involving costly verification, sparse or difficulty-agnostic rewards, collapsed rollouts, and consistency drift.
- Vision-language RLVR: Vision-language RLVR has extended verifiable-reward post-training beyond language models, but visual scaling remains difficult because verification is task-specific and often depends on curated annotations or external verifiers.
- Supervision-free post-training: Supervision-free post-training reduces dependence on costly or noisy labeled answers through heuristics, self-consistency, imperfect-reward training, and verifier-based pipelines.
- Curricula for GRPO post-training: Vanilla GRPO is largely difficulty-agnostic, and homogeneous rollouts shrink group-relative advantages; sparse rewards can push groups toward all-success or all-failure regimes.
- Consistency drift in reasoning: Reasoning–answer inconsistency can worsen later in GRPO training, motivating monitoring and objectives that better couple final answers to reasoning chains.The related work describes the same trend in visual puzzle post-training and reports that PuzzleCraft’s exploration-aware curriculum mitigates the decline.
- Consistency drift in reasoning: PuzzleCraft combines exploration-aware curriculum weighting with GRPO-CARE, further amplifying gains in consistency and downstream performance.
3 PuzzleCraft
PuzzleCraft reframes puzzle-based RLVR as a curriculum and reasoning-consistency problem, using automatically verifiable visual puzzles and rollout diversity to improve training signals. Its curriculum combines medium-difficulty weighting with solution-space exploration, while RAC monitors alignment between reasoning traces and answers.
- 3 PuzzleCraft: PuzzleCraft combines lightweight, automatically verifiable puzzle environments with a difficulty- and exploration-aware curriculum for vision-centric RL post-training.The framework uses multiple puzzles rather than treating puzzle RLVR as a purely outcome-driven objective.
- 3 PuzzleCraft: Jigsaw supplies graded supervision by rewarding the fraction of correctly placed tiles, allowing partial progress to receive credit while penalizing localized errors.Unlike binary puzzle rewards, this reward does not collapse every imperfect rollout to failure.
- 3 PuzzleCraft: The framework samples rollout groups and assigns weights so training focuses on prompts that yield informative group-relative updates.The curriculum is motivated by the limitation that difficulty alone cannot distinguish exploration from repeated identical guesses.
- 3 PuzzleCraft: For Rotation and PatchFit, exploration is measured through selected-option entropy; for Jigsaw, it is measured through diversity of induced tile permutations.Normalized dispersion approaches zero when rollouts collapse and increases as solutions become more diverse.
- 3 PuzzleCraft: Difficulty weighting emphasizes medium-difficulty prompts, while exploration weighting downweights groups whose rollouts collapse to the same solution.The combined weight is w(x) = λ d(x)(1 − d(x))(d_π(x))^γ, with λ = 4 and γ = 0.5 in the experiments.
- 3 PuzzleCraft: PuzzleCraft also monitors Reasoning–Answer Consistency as a post-training diagnostic because puzzle reward can improve while reasoning faithfulness degrades.RAC is introduced to track alignment between the reasoning trace and final answer during training.
4 Experiments
Experiments evaluate PuzzleCraft’s design choices, consistency dynamics, transfer behavior, and broad benchmark performance across models, puzzles, and prompting modes. Results show that exploration-aware curriculum and consistency-aware optimization improve reasoning consistency and downstream performance, with Jigsaw and Mix achieving the strongest overall transfer.
- 4.2 Ablating design choices: The combined curriculum and GRPO-CARE setup maintains the highest RAC through most training, while both components improve downstream accuracy.Curriculum+GRPO-CARE outperforms GRPO-CARE alone despite the latter achieving higher puzzle reward, showing puzzle reward alone is not a reliable downstream indicator.
- 4.2 Ablating design choices: The exploration-aware curriculum outperforms the difficulty-aware Observe-R1 baseline on Jigsaw under the same training setup.This comparison supports tracking rollout exploration in addition to reward statistics.
- 4.3 Main Results: 65.18% Avg. is achieved by Qwen2.5-VL-7B-Jigsaw across nine image benchmarks, improving by more than 2.2% over Visual Jigsaw.Similar gains are reported for Qwen2.5-VL-3B and across Qwen3-VL sizes.
- 4.3 Main Results: PuzzleCraft’s Jigsaw also outperforms VisualSphinx by about 1.5% on video benchmarks and approaches Video-R1 without video post-training.The video evaluation covers seven reasoning benchmarks.
- 4.3 Main Results: Jigsaw transfers best overall, Rotation transfers complementarily, PatchFit transfers weakly, and Mix remains robust across tasks.The authors attribute part of Jigsaw’s advantage to its graded partial-credit reward.
- 4.5 Performance on Puzzles: Direct-mode results are generally close to the base model, although Mix achieves the best overall average, indicating that most gains appear with explicit reasoning traces.Direct-mode evaluation uses default prompts without an explicit chain-of-thought request.
- 4.5 Performance on Puzzles: Puzzle-specific training improves its matching puzzle but does not reliably transfer to other setups, whereas mixed-puzzle training alleviates this problem.The transfer evaluation uses Jigsaw, PatchFit, and Rotation environments.
5 Conclusion
PuzzleCraft combines automatically verifiable puzzles, an exploration-aware curriculum, and Reasoning–Answer Consistency diagnostics to improve vision-language model post-training. Across Qwen2.5-VL and Qwen3-VL, it improves robustness, consistency, and transfer to video reasoning without video post-training data.
- PuzzleCraft combines lightweight, automatically verifiable puzzles with an exploration-aware curriculum and a Reasoning–Answer Consistency diagnostic, with optional consistency-aware optimization.
- Across Qwen2.5-VL and Qwen3-VL backbones, PuzzleCraft improves robustness and consistency on a broad suite of image benchmarks and transfers to video reasoning without video data during post-training.
- The results suggest that flat weighting, collapsed rollouts, and reward-only optimization can limit puzzle-based RLVR by wasting compute or poorly reflecting faithful reasoning.
A Additional Discussion and Details on our Puzzles
PuzzleCraft uses configurable visual puzzles whose difficulty is controlled by task-specific parameters. Rotation varies angle choices, PatchFit varies distractor hardness, and Jigsaw varies grid size with graded tile-placement rewards.
- Puzzle difficulty is controlled by angle-set cardinality for Rotation, distractor hardness for PatchFit, and grid size for Jigsaw.Experiments use four Rotation angles, D ∈{3, 5, 7} PatchFit decoys, and Jigsaw grids with 2 ≤ MN ≤ 9.
- Under random guessing, Rotation succeeds at 25%, PatchFit averages approximately 18%, and Jigsaw’s graded reward averages approximately 26% in the sampled configurations.
- The Rotation puzzle uses an image from Microsoft COCO as an example training instance.
B.1 Difficulty-Only Curricula
PuzzleCraft’s exploration-aware curriculum outperforms difficulty-only curricula, showing that rollout exploration adds useful learning signal beyond reward statistics.
- PuzzleCraft improves average performance by more than 2.5 points over the difficulty-only curricula of Observe-R1 and VCRL.The comparison is reported in Table S1.
B.2 Frozen Vision Encoder
PuzzleCraft remains effective when the vision encoder is frozen, while allowing the encoder to adapt provides additional benefits. The results indicate that language-model reasoning policy improvements account for a substantial portion of the gains.
- With the vision encoder frozen, PuzzleCraft still clearly outperforms the base model, indicating substantial gains from improving the language-model reasoning policy.
- The full model performs better than the frozen-vision-encoder variant, suggesting that adapting the visual encoder provides additional benefits.
C RAC Measurement
RAC evaluates whether generated rationales explicitly support final answers, using sampled post-training rollouts judged with binary scores. The metric illustrates consistency patterns and serves as a monitoring diagnostic during training.
- RAC evaluation: RAC assigns binary scores by asking a fixed open-source judge whether each rollout’s rationale explicitly supports its final answer.Rollouts are sampled uniformly across the post-training trajectory and evaluated at regular intervals.
- RAC evaluation: The RAC evaluation uses each rollout’s rationale and final <answer>, with the evaluation prompt shown in Figure S5.Representative Jigsaw examples in Figure S4 illustrate faithful and inconsistent reasoning–answer patterns.
- Observed pattern: Vanilla GRPO shows an early faithfulness increase followed by a later decline, while curriculum mitigates the decline and CARE further improves RAC throughout training.Figure 2b compares vanilla GRPO, GRPO+curriculum, GRPO+CARE, and GRPO+curriculum+CARE on Jigsaw.
- Observed pattern: RAC is used to monitor consistency dynamics rather than as a strict criterion for selecting models.This defines RAC’s role as a post-training diagnostic.