PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaC-GRPO: Towards Mastering Long-Horizon Visual Sudoku Puzzles via Adaptive Curriculum Learning

Sida Li, Yuanxin Liu, Linli Yao, Shicheng Li, Xu Sun, Haoli Bai, Xiaohui Li

arXiv:2610.13184v1cs.AI

TL;DR

The paper asks whether current multimodal models can solve visual Sudoku without external symbolic solvers and how to improve their performance on long-horizon reasoning tasks. The paper introduces FG-Sudoku, a fine-grained benchmark, and AdaC-GRPO, which uses online rollout accuracy to sample tasks within the model’s Zone of Proximal Development. AdaC-GRPO raises Qwen2.5-VL-7B’s C-group accuracy from near zero to 0.219, approaches Gemini-2.5-Flash on the hardest 9×9 tier, and closes 62.2% of the overall gap to Gemini-2.5-Pro.

  • Problem

    The paper asks whether current multimodal models can solve visual Sudoku without external symbolic solvers and how to improve their performance on long-horizon reasoning tasks.

    Source: §1 · p. 1

  • Method

    The paper introduces FG-Sudoku, a fine-grained benchmark, and AdaC-GRPO, which uses online rollout accuracy to sample tasks within the model’s Zone of Proximal Development.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    AdaC-GRPO raises Qwen2.5-VL-7B’s C-group accuracy from near zero to 0.219, approaches Gemini-2.5-Flash on the hardest 9×9 tier, and closes 62.2% of the overall gap to Gemini-2.5-Pro.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    AdaC-GRPO, a competence-driven curriculum, stabilizes gradient signals and raises Qwen2.5-VL-7B’s C-group accuracy from a near-zero baseline to 0.219 on visual Sudoku.

    Source: §6 · p. 9

    AdaC-GRPO assumes access to a verifiable reward and meaningful task partitions, which may be difficult to construct in open-ended domains without reliable metadata or stable empirical difficulty tiers.

    Source: Limitations · p. 9

Abstract

from arXiv · show

While reinforcement learning with verifiable reward (RLVR) enhances reasoning in large language models, its efficacy in long-horizon multimodal tasks remains underexplored. Visual Sudoku puzzles pose a representative challenge, requiring multi-step deductive reasoning over both local constraints and global dependency chains. To this end, we introduce FG-Sudoku, a fine-grained benchmark categorized by grid size and required deductive steps. Evaluations reveal that frontier MLLMs fail on complex long-range dependencies (e.g., 9*9 puzzles). During post-training, this near-zero success rate causes severe reward sparsity, halting standard RLVR. We propose Adaptive Curriculum Group Relative Policy Optimization (AdaC-GRPO) to overcome this. Unlike rigid or prior-based schedules, AdaC-GRPO dynamically constructs training batches by tracking the model's "Zone of Proximal Development" via online rollout accuracy. Experiments demonstrate that AdaC-GRPO enables Qwen2.5-VL-7B to approach the performance of Gemini-2.5-Flash while bridging 62.2% of the overall accuracy gap to Gemini-2.5-Pro, with its generalizability further validated across InternVL-3.5 and Qwen3-VL.

1 Introduction

The paper studies whether multimodal models can solve visual Sudoku without symbolic solvers and how to improve their long-horizon reasoning. It introduces FG-Sudoku and AdaC-GRPO, which uses online rollout accuracy to sample tasks within the model’s Zone of Proximal Development.

1 Introduction

  • The paper asks whether current multimodal models can solve visual Sudoku without external symbolic solvers and how their performance can be improved on long-horizon reasoning tasks.Sudoku requires multi-step deduction across local constraints and long-range global dependencies.Source: §1 · p. 1
  • AdaC-GRPO uses online rollout accuracy to identify the model’s Zone of Proximal Development and sample tasks that are challenging but still yield positive reinforcement.This competence-driven progression contrasts with static schedules and error-driven sampling of the hardest instances.Source: §1 · p. 2
  • AdaC-GRPO raises Qwen2.5-VL-7B’s C-group accuracy from near zero to 0.219, approaches Gemini-2.5-Flash on the hardest 9×9 tier, and closes 62.2% of the overall gap to Gemini-2.5-Pro.It outperforms AdaRFT and CurrReFT by over 5× on the hardest tier and generalizes to InternVL-3.5 and Qwen3-VL.Source: §1 · p. 2
  • FG-Sudoku is a fine-grained diagnostic benchmark for visual Sudoku, with difficulty measured by grid size and the deductive steps required by standard solving algorithms.Its solver-derived difficulty and verifiable correctness support evaluation without an LLM-as-judge.Source: §1 · pp. 1–2

2 Related Work

The related work situates FG-Sudoku among symbolic reasoning, multimodal benchmarks, and reinforcement fine-tuning methods. The paper positions its benchmark as a controlled way to study long-range constraint propagation and its curriculum as competence-driven sampling for sparse-reward tasks.

2.1 Symbolic and constraint reasoning

  • Prior symbolic and hybrid approaches use explicit rule supervision or modular solvers; this work instead treats Sudoku as an end-to-end environment where models infer constraints through reward feedback.The approach is presented as complementary to integrating symbolic solvers into networks.Source: §2.1 · pp. 2–3

2.2 Multimodal reasoning and benchmarks

  • Existing multimodal and puzzle benchmarks leave long-range constraint propagation open; FG-Sudoku offers a controlled, solver-derived difficulty spectrum for analyzing global consistency.It is framed as a diagnostic counterpart to large-scale open-ended reasoning suites.Source: §2.2 · p. 3

2.3 Reinforcement fine-tuning and curriculum learning

  • Unlike static schedules or prior-based scoring, AdaC-GRPO tracks the model’s Zone of Proximal Development through online rollout accuracy to maintain dense positive feedback.It routes batches before rollouts, whereas DAPO filters zero- or all-one-reward groups after sampling, potentially spending rollouts without reward contrast.Source: §2.3 · p. 3

3 FG-Sudoku: Dataset Construction

FG-Sudoku provides visual Sudoku instances across grid sizes and difficulty tiers, with solver-derived traces and costs supporting controlled analysis of deductive complexity. Its construction pipeline ensures unique solutions and uses quantile partitioning to balance difficulty groups.

3 FG-Sudoku: Dataset Construction

  • FG-Sudoku contains 3,072 training and 768 held-out test puzzles across 4×4, 6×6, and 9×9 grids, stratified into four difficulty tiers.Gemini 2.5 Pro also synthesizes chain-of-thought traces for initial supervised fine-tuning.Source: §3 · p. 4

3.1 Generation and Validation Pipeline.

  • A generation pipeline combines valid-board generation, dynamically sampled clue removal, and deterministic solving with backtracking and an MRV heuristic to ensure unique solutions.It records structural operations to estimate each puzzle’s deductive cost; roughly 3K instances can be synthesized in about 60 seconds on a standard CPU.Source: §3.1 · p. 4

3.2 Difficulty Stratification.

  • Because deductive cost is heavily right-tailed, FG-Sudoku uses four count-balanced quantile tiers rather than equal-width bins that would leave the hard tail under-sampled.This yields balanced data buffers across the deductive-complexity spectrum.Source: §3.2 · p. 4

3.3 Advantages and Solver Traces.

  • Recorded solver traces expose dependency-chain structure and the steps over which global exclusivity constraints propagate, enabling controlled comparisons of model behavior.The 4×4 trace visualization uses quadrants for operation classes, links for operations on the same cell, and blue-to-red color for temporal order.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

4 AdaC-GRPO

AdaC-GRPO routes training toward difficulty groups whose online rollout accuracy places them within the model’s Zone of Proximal Development. It pairs this competence-driven curriculum with binary outcome rewards and an auxiliary format reward for strict Sudoku constraints.

4.1 Competence-Driven Curriculum via ZPD Scheduling

  • AdaC-GRPO tracks group-level exact-solution rollout accuracy and routes batches to difficulty groups inside a ZPD feasibility band, avoiding uniform sampling’s sparse signals.The ZPD excludes both trivially mastered and overwhelmingly complex tasks.Source: §4.1 · p. 5
  • When no group lies within the ZPD, training stops if all groups are mastered; otherwise, it returns to the previously trained group for consolidation.This rollback lets the model reenter the ZPD before progressing again.Source: §4.1 · p. 5

4.2 Group Relative Policy Optimization for Strict Constraints

  • For strict constraints, the reward combines exact-grid success with an auxiliary output-format term, but format compliance does not guide ZPD selection.The authors report that dense partial-assignment rewards can give false-positive advantages for locally valid assignments that violate global constraints.Source: §4.2 · p. 5
  • Groups with zero-success rollouts provide no task-success learning signal, while finding an informative group by post-sampling can require rapidly more rollouts as success probability falls.AdaC-GRPO uses online competence estimates to defer groups below the lower ZPD bound until their accuracy rises into the feasible range.Source: §4.2 · p. 6
  • When just one rollout succeeds, it supplies the sole positive credit, which may reinforce a success the model cannot reliably reproduce.Restricting training to the ZPD reduces dependence on such rare events.Source: §4.2 · p. 6

5 Experiments

The experiments test FG-Sudoku’s ability to distinguish reasoning difficulty and compare AdaC-GRPO with other training approaches. AdaC-GRPO improves Qwen2.5-VL-7B on hard Sudoku, while ablations examine its curriculum, reward, modality, and diagnostic behavior.

5.1 Experimental Setup

  • Evaluation uses Pass@1 across proprietary and open-weight MLLMs; training cold-starts reasoning with 300 Gemini-2.5-Pro-annotated traces and routes sampling to groups within [0.3, 0.8] validation accuracy.Each prompt receives eight candidate completions; selection-rule and threshold ablations are also reported.Source: §5.1 · p. 6

5.2 Main Results (RQ1 & RQ2)

  • Frontier models perform competently on 4×4 Group A but fall to near-zero on 9×9 Group C as deductive cost grows, pointing to constraint propagation as the bottleneck.The authors identify multi-step constraint propagation, rather than perception, as the limiting factor.Source: §5.2 · p. 6, Table 1 · p. 7
  • 0.586 overall Pass@1 for AdaC-GRPO versus 0.427 for GRPO; on Group C, AdaC-GRPO reaches 0.219 versus DAPO’s 0.086.AdaC-GRPO also outperforms CurrReFT and AdaRFT at every difficulty level, with wider gaps on harder instances.Source: §5.2 · pp. 6–7, Table 1 · p. 7
  • 62.2% of the overall accuracy gap to Gemini-2.5-Pro is closed, and AdaC-GRPO’s 0.219 on the hardest 9×9 tier approaches Gemini-2.5-Flash’s 0.293.The comparisons are reported for Block C and the hardest tier, respectively.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 Curriculum Trajectory

  • By activating subsets according to [τlow, τhigh], AdaC-GRPO keeps the policy in its ZPD as rollout accuracy progresses across sequentially activated subsets.Figure 4 plots rollout accuracy against the activated training subsets and marks the global ZPD bounds.Source: §5.3 · p. 8, Figure 4 · p. 8

5.4 Ablation and Analysis

  • Coarse grid-size-only stratification lowers C-group accuracy from 0.219 to 0.012; partial-grid rewards yield 0.000 on Group C, supporting fine-grained tiers and binary outcome rewards.The authors attribute the coarse curriculum’s result to a difficulty gap too large to bridge between 6×6 and 9×9, and argue local correctness may not lead to a globally valid solution.Source: §5.4 · p. 8, Table 2 · p. 8
  • Matched visual training performs better overall and on Groups B and C, while text is stronger on Group A; visual-to-text transfer is also stronger than the reverse.The authors suggest the two-dimensional layout adds benefit as constraint reasoning becomes more difficult.Source: §5.4 · p. 8, Table 2 · p. 8
  • AdaC-GRPO reduces solver-detected constraint conflicts by 67.4% versus SFT on the evaluated traces, and improves medium and hard tiers on ZebraLogicBench and CMM-Math.Remaining Sudoku errors more often involve higher-order patterns such as Hidden Pairs; ZebraLogicBench tiers have only 64–72 examples, so those differences warrant caution.Source: §5.4 · pp. 8–9, Table 3 · p. 9

6 Conclusion

The conclusion presents FG-Sudoku as a benchmark for multi-step constraint reasoning and AdaC-GRPO as a competence-driven curriculum for reward starvation. Its reported results show gains on hard Sudoku and comparisons to proprietary models.

6 Conclusion

  • AdaC-GRPO raises Qwen2.5-VL-7B’s C-group accuracy from a near-zero baseline to 0.219, substantially outperforming standard GRPO and existing curriculum-learning methods.The conclusion also reports that the method approaches Gemini-2.5-Flash on the hardest tier and closes 62.2% of the overall gap to Gemini-2.5-Pro.Source: §6 · p. 9
  • FG-Sudoku is introduced as a diagnostic benchmark for isolating multi-step constraint bottlenecks.The benchmark focuses on visual Sudoku puzzles.Source: §6 · p. 9
  • AdaC-GRPO confines active sampling to the model’s Zone of Proximal Development to address optimization collapse on complex tasks.The conclusion describes the curriculum as competence-driven.Source: §6 · p. 9

Limitations

The paper identifies limits in statistical coverage, task assumptions, and evaluation scope.

Limitations

  • Only GRPO and AdaC-GRPO are repeated across three seeds; other training baselines use single runs, and ZebraLogicBench’s hard tiers have few test examples.The authors note that these small test tiers increase sampling uncertainty in reported differences.Source: Limitations · p. 9
  • AdaC-GRPO assumes verifiable rewards and meaningful task partitions, which may be difficult to construct in open-ended domains without reliable metadata or stable difficulty tiers.Source: Limitations · p. 9
  • FG-Sudoku covers standard 4×4, 6×6, and 9×9 puzzles, not irregular variants or larger grids; hard-case and cross-task evaluation remain limited in scope.The paper also reports low performance on the hardest C3 and C4 tiers and calls for broader validation on open-ended visual reasoning.Source: Limitations · p. 9

A.3 Dataset Statistics

The appendix reports dataset statistics, training setup, and ablations of ZPD selection and thresholds. Its results favor consolidation-first selection and a moderately aggressive lower threshold, with limited sensitivity to the upper bound.

A.3 Dataset Statistics

  • Table 4 summarizes exact train/test counts across four difficulty tiers for each grid size; discrete deductive step counts produce numerical differences in final bin capacities.The dataset preserves this discreteness to maintain the mathematical threshold boundaries.Source: §A.3 · p. 12, Table 4 · p. 12

B.1 Experimental Setup

  • Training uses Qwen2.5-VL-7B on four NVIDIA A800-SXM4 GPUs with ZeRO-3 and FlashAttention-2, taking approximately 30 hours.Source: §B.1 · p. 13

B.2 Hyperparameters

  • Detailed training configurations and hyperparameters are listed in Table 5; the other parameters use trl.GRPOConfig and trl.ModelConfig defaults.Source: §B.2 · p. 13, Table 5 · p. 13

B.3 Selection Strategy within the ZPD

  • Argmax selection records C1 accuracy of 0.304 and C2 accuracy of 0.153, outperforming median-accuracy and random selection in the hardest C-tiers.The authors describe argmax as consolidation-first progression; Figure 5 visualizes selection-strategy accuracy.Source: §B.3 · p. 13, Table 6 · p. 14, Figure 5 · p. 14

B.4 Sensitivity to ZPD Threshold [τlow, τhigh]

  • The two L = 0.3 threshold configurations tie for C-group average accuracy at 0.138, while the default [0.3, 0.8] is best or tied-best on 9 of 12 tiers.The results indicate robustness to U but stronger sensitivity to L; Figure 6 visualizes the threshold sweep.Source: §B.4 · p. 13, Table 7 · p. 14, Figure 6 · p. 14

B.5 Seed-Level Variance Analysis

Across three training seeds, AdaC-GRPO’s reported result is stable, with per-tier standard deviations below 0.04. The section also covers transfer evaluation, prompt formats, and the unit-testing module’s time and accuracy trade-offs.

B.5 Seed-Level Variance Analysis

  • Per-tier standard deviations stay below 0.04; Seed-1 leads in total accuracy (0.5859 vs. 0.5794 and 0.5807) and C-group accuracy (0.219 vs. 0.207 and 0.211).These margins are smaller than the reported 0.176 absolute C-group gain over AdaRFT, supporting stability under seed perturbation.Source: §B.5 · p. 15, Table 8 · p. 16

C Transfer from the Sudoku-Trained Model

D.1 System Prompt

  • The system prompt instructs the model to solve one empty cell at a time, check its constraints, place only a uniquely supported number, and rescan affected areas.Source: §D.1 · p. 15

D.2 User Prompt

  • The user prompt specifies the puzzle’s size and contents and requires the complete solution as a nested-list matrix, with no other output.Source: §D.1 · p. 15, §D.2 · p. 15

E.1 Time Overhead Analysis

  • Unit testing adds approximately 3.8%, 5.3%, and 8.3% overhead at sampling ratios 1/16, 1/8, and 1/4, respectively.Even the 1/4 configuration remains below 10% overhead; the 1/8 setting stays under 6%.Source: §E.1 · p. 15

E.2 Accuracy Under Different Sampling Ratios

  • At a 1/8 sampling ratio, unit-testing accuracy is 0.7214 versus 0.7391 at full sampling, with average category deviation around 5%.The section identifies 1/8 as a balance between prediction fidelity and testing cost; some categories degrade more at lower ratios.Source: §E.2 · pp. 15–16

E.3 Rollout Efficiency on Hard Instances

On Group C, most DAPO rollout groups lack accuracy-reward variation, while AdaC-GRPO yields more informative groups. Curriculum progression shifts training toward a difficulty region where such groups are more likely.

E.3 Rollout Efficiency on Hard Instances

  • The unit-testing module adds only small time overhead across sampling ratios, and its accuracy remains close to the full-data result even at ratio 1/8.Source: Figure 7 · p. 17, Figure 8 · p. 17
  • DAPO discard rates rise from 99.4% on C1 to 100.0% on C3 and C4, while AdaC-GRPO’s rates range from 22.7% on C1 to 86.8% on C4.On C3 and C4, no sampled DAPO group contains both successful and unsuccessful trajectories.Source: §E.3 · p. 17
  • Increasing DAPO’s rollout-group size from G = 4 to 16 lowers its discard rate only from 99.9% to 99.4%, compared with a 21.3-point reduction for AdaC-GRPO.When per-trajectory success is near zero, sampling more trajectories adds little reward variation.Source: §E.3 · p. 17
  • 99.7% of DAPO rollout groups have zero accuracy-reward variance, versus 54.7% for AdaC-GRPO; every discarded DAPO group is all-wrong.The comparison uses Group C with G = 8; the discard rates reflect near-zero task success rather than saturation.Source: §E.3 · pp. 16–17, Table 11 · p. 17
  • Curriculum progression moves the policy toward a more learnable difficulty region, making informative rollout groups more likely; post-generation filtering alone cannot create reward variation when nearly all trajectories fail.Source: §E.3 · p. 17

F.1 Case-wise Entropy Delta Analysis

Entropy analyses show different post-GRPO patterns across task difficulty: easier cases tend toward predictive sharpening, while harder cases show entropy expansion and more substantial policy updates. The supplied error analysis also describes Claude Sonnet 4’s Sudoku reasoning failures.

F.1 Case-wise Entropy Delta Analysis

  • In easier Size-4 cases, mean entropy delta is predominantly negative; harder cases show entropy expansion, with peaks exceeding 1.0 nat.The authors suggest the expansion may reflect SFT overconfidence on complex tasks and GRPO flattening overly peaked priors to encourage exploration.Source: §F.1 · p. 18

F.2 Temporal Evolution of JS Divergence

  • Low-complexity cases show sparse, intermittent JS divergence, whereas high-complexity cases show sustained, high-magnitude divergence across the generation window.The authors interpret the larger updates as reparameterizing reasoning trajectories away from the reference distribution.Source: §F.2 · pp. 18–19

F.3 Stratified Entropy Distributions

  • Size-9 post-GRPO entropy distributions have greater variance, elevated medians, and heavier tails; Size-4 and Size-6 distributions remain more tightly clustered with minimal shift.The authors describe the scale-dependent pattern as consistent with changes in confidence on longer, more ambiguous reasoning paths.Source: §F.3 · p. 19, Figure 11 · p. 19

G.1 Claude Sonnet 4

  • Claude Sonnet 4’s traces include Sudoku uniqueness violations, unreliable candidate bookkeeping, and coordinate or box-indexing mistakes that produce incorrect deductions.Inconsistent intermediate and final grids further indicate missing global consistency checks, allowing contradictions to propagate.Source: §G.1 · p. 19

G.2 Gemini-2.5-pro

Gemini-2.5-Pro’s Sudoku traces can begin with plausible local deductions and attempt backtracking, yet assigned values and candidate sets become inconsistent across reasoning steps. The analysis also describes a false-positive final validation.

G.2 Gemini-2.5-pro

  • Gemini-2.5-Pro’s corrections are not consistently propagated through one solver state, and incompatible assignments and stale candidate sets accumulate into global inconsistency.Source: §G.2 · p. 19
  • Figure 12’s trace shows natural-language backtracking without restoration of a defined prior grid state, mixing revised and unrevised assignments.The caption identifies state inconsistencies and explicit backtracking among the highlighted excerpts.Source: §G.2 · p. 20, Figure 12 · p. 20
  • Gemini reports that all rows, columns, and boxes are unique, yet the returned grid violates Sudoku constraints, including two 2s in column 6.The authors note that using the same unreliable internal state for deduction and validation permits a false-positive check.Source: §G.2 · p. 20
Loading 2610.13184v1…