PaperlayerView exact revision on arXiv

Source-linked AI summary

Getting Space Right: Agentic Rewards for Image Generation

Xiaofeng Tan, Jiaxin Lin, Haobo Yuan, Tiancheng Shen, Zihan Su, Rui Chen, Hongsong Wang, Ming-Hsuan Yang

arXiv:2610.14852v1cs.CVcs.AI

TL;DR

Spatial T2I reinforcement learning can improve in-domain performance while degrading out of domain, with reward reliability and rollout progression remaining open concerns. The paper combines an agentic spatial reward model with capability-graph-guided curriculum learning; it reports 95.1% SpaRW-Eval accuracy and higher average generation scores on SD3.5-Medium and BAGEL.

  • Problem

    Spatial T2I reinforcement learning faces questions about open-world reward accuracy and whether rollouts can progress from foundational spatial skills to complex compositions.

    Source: §1 · p. 1

  • Method

    The framework combines SPATIALCRITIC, an agentic reward model aligned to spatial tools through preference data and RLVR, with capability-graph-guided curriculum learning.

    Source: §1 · p. 2, §3.3 · p. 5, §4.1 · p. 6

  • Results

    SpatialCritic achieves 95.1% verification accuracy on SpaRW-Eval, while curriculum RL raises average scores to 83.9 on SD3.5-Medium and 86.5 on BAGEL.

    Source: §1 · p. 2, §5.2 · pp. 7, 9

  • Takeaways & Limitations

    The results show accurate spatial preference judgments and improved RL alignment with curriculum learning across SD3.5-Medium and BAGEL.

    Source: §1 · p. 2, §5.2 · pp. 7, 9

    The evaluation uses a fixed tool configuration; transfer to updated or alternative tools and robustness to temporary tool failures or missing outputs remain untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) is promising for spatial text-to-image generation, but existing work largely focuses on policy optimization, assuming accurate rewards and effective online rollouts. We observe in-domain gains with out-of-domain degradation, exposing two limitations: unreliable spatial rewards in open-world settings and rollouts lacking basic-to-complex progression. We propose a curriculum-guided RL framework. First, we introduce SpatialCritic, an agentic reward model equipped with a spatial perception harness that combines reusable verification guidance with specialized vision tools. We then construct spatial preference datasets through targeted counterfactual perturbations and perform agent-harness alignment to improve tool selection and evidence integration efficiently. Second, curriculum learning organizes shared foundational capabilities and their compositional dependencies in a capability graph for mastery-guided rollouts, periodic review, and targeted remediation. On SpaRW-Eval, SpatialCritic achieves 95.1% verification accuracy, outperforming 320B GLM-5.3-Flash with a 40x smaller 8B backbone and raising average baseline scores from 78.4 to 83.9 on SD3.5-Medium and 83.2 to 86.5 on BAGEL. Project page: https://xiaofeng-tan.github.io/projects/spatialcritic/

1 INTRODUCTION

Spatial T2I reinforcement learning improves in-domain performance but can degrade out of domain, raising questions about reward reliability and whether rollouts progress from basic to complex capabilities. The paper combines SPATIALCRITIC with a capability-graph curriculum and reports stronger preference judgments and generation scores.

1 INTRODUCTION

  • The paper frames spatial T2I post-training around three factors—rollout organization, reward accuracy, and policy optimization—and asks whether rewards and rollouts support open-world verification and progressive learning.Existing work mainly advances policy optimization while often assuming effective rollouts and reliable rewards.Source: §1 · p. 1
  • RL-aligned models can improve on in-domain benchmarks yet fall below the pretrained model out of domain when reused prompts and rule-based rewards favor complex compositions without progressive learning.The stated flaws are benchmark-specific rewards that generalize poorly, especially under occlusion, and rollouts that neglect foundational capabilities.Source: §1 · p. 1, Figure 1 · p. 1
  • Prior reward approaches use regression-based preference scores or teacher-guided generative evaluation, but the paper identifies spatial-verification limits in both paradigms.Regression underuses native reasoning and geometric depth priors, while generative evaluation relies on teacher generation and struggles with geometric quantities such as occlusion and orientation.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • The framework pairs SPATIALCRITIC, an agentic reward model using specialized vision tools, with a curriculum progressing from foundational spatial capabilities to complex compositions.Spatial preference data and RLVR alignment improve tool selection and evidence integration; the Compositional Capability Graph guides mastery-based progression, review, and remediation.Source: §1 · p. 2
  • 95.1% verification accuracy on SpaRW-Eval accompanies curriculum-RL averages of 83.9 on SD3.5-Medium and 86.5 on BAGEL.The results support accurate preference judgments and improved RL alignment.Source: §1 · p. 2

2 RELATED WORK

Related work traces visual reward modeling from similarity heuristics and learned preferences to generative evaluations. The paper distinguishes its approach through per-claim evidence acquisition and a capability-dependency curriculum for spatial T2I post-training.

2 RELATED WORK

  • Visual reward modeling progressed from similarity heuristics to learned preferences, including dual encoders trained to predict scalar preference scores and VLMs producing disentangled rewards.More recent generative models emit rubrics, rationales, preference tokens, or calibrated score distributions.Source: §2 · pp. 2–3
  • Unlike SpatialReward’s predefined verification routine, SPATIALCRITIC learns per-claim evidence acquisition from an open tool catalog under outcome supervision.SpatialReward grounds decomposed constraints with expert detectors but determines its verification procedure before observation.Source: §2 · p. 3
  • Prior staged spatial-learning work primarily targets understanding and embodied planning, whereas this work applies capability dependencies and mastery probes to online prompt scheduling for spatial T2I post-training.The scheduling also includes periodic review and targeted remediation.Source: §2 · p. 3

3 SPATIALCRITIC: TOOL-MEDIATED AGENTIC REWARD MODEL

SPATIALCRITIC combines a spatial perception harness with specialized vision tools, and learns to select tools and integrate evidence for spatial preference judgments. Its dataset uses targeted counterfactuals and verification, while RLVR alignment trains the agent under outcome rewards and token-cost penalties.

3.1 AGENT WITH A SPATIAL PERCEPTION HARNESS

  • The harness supports atomic claim verification through specialized tools, using standardized routines for frequent relations and agent-planned tool sequences for unfamiliar compositions.SPATIALCRITIC decomposes prompts, plans and executes targeted tool calls, then integrates observations into a preference verdict.Source: Figure 3 · p. 4, §3.1 · p. 4

3.2 VERIFIABLE SPATIAL PREFERENCE DATASET

  • The preference dataset perturbs targeted spatial constraints while preserving others, then validates prompt pairs, rendered-image compliance, intended violations, and human-reviewed image preferences.Construction covers 36 scene settings and 13 task categories, with 3D pose held out from agent training.Source: §3.2 · pp. 4–5, Figure 4 · p. 5

3.3 AGENT–HARNESS ALIGNMENT VIA RLVR

  • RLVR aligns tool selection and evidence integration with the fixed harness, using hard examples where the tool-free agent fails and a reward combining judgment outcomes with a token-budget penalty.The penalty applies conditionally on correct judgments; GRPO optimizes the policy using group-relative reward advantages.Source: §3.3 · pp. 5–6

4 CURRICULUM LEARNING FOR SPATIAL T2I POST-TRAINING

The curriculum organizes spatial T2I training from foundational capabilities to complex compositions, using a capability graph to represent shared prerequisites. Mastery-guided progression is paired with periodic review and targeted remediation.

4 CURRICULUM LEARNING FOR SPATIAL T2I POST-TRAINING

  • Curriculum learning schedules rollouts from foundational spatial capabilities to complex compositions, guided by policy competence, review, and remediation.This replaces an unstructured mixture of complex tasks with progressive training.Source: §4 · p. 6, §4.2 · p. 6

4.1 COMPOSITIONAL CAPABILITY GRAPH

  • The 13-node directed acyclic CCG represents shared capability dependencies, including Topology as a prerequisite shared by Layout and Occlusion.Layout also depends on Count and Position, while Occlusion depends on Depth.Source: §4.1 · p. 6, Figure 5 · p. 6

4.2 PROGRESSIVE LEARNING WITH REVIEW AND REMEDIATION

  • Capabilities meet mastery through a smoothed-score threshold and Nmin assessments; after Nmax assessments, fallback permits progression, and both sets unlock downstream capabilities.A successor becomes eligible for training when all prerequisites are mastered or have reached the assessment limit.Source: §4.2 · p. 6
  • Periodic review reassesses learned capabilities individually and compositionally, then applies targeted training to capabilities that fail the mastery criterion.For example, counting-specific training addresses Count when it fails a joint assessment with Position and Occlusion.Source: §4.2 · p. 7

5 EXPERIMENTS

Experiments assess spatial preference judgments and reward signals, then test reward-guided RL post-training with and without curriculum learning. Results cover SpaRW-Eval and generation benchmarks using SD3.5-Medium and BAGEL.

5.1 EXPERIMENTAL SETUPS

  • The setups fine-tune Qwen3-VL-8B with GRPO and 4,389 spatial preference pairs, then use Flow-GRPO for SD3.5-Medium and BAGEL post-training.Generation evaluation covers TIIF-Bench, UniGenBench++, and SpatialGenEval; preference evaluation uses SpaRW-Eval, with reward-model, zero-shot VLM, and RL baselines.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • 95.1% verification accuracy on SpaRW-Eval surpasses 320B GLM-5.3-Flash (94.5%); agent–harness alignment raises accuracy from 92.9% to 95.1%.The 8B model also exceeds Qwen3-VL-8B (84.9%) and SpatialScore (87.5%); Table 1 reports spatial preference accuracy.Source: §5.2 · p. 7, Table 1 · p. 8
  • 74.4% Best-of-N selection accuracy exceeds GLM-5.3-Flash (71.8%) and the selection-free baseline (70.8%); it leads nine of ten categories, with position accuracy 73.3% versus 67.6%.Table 2 reports reward-guided Best-of-N generation.Source: §5.2 · pp. 7–8, Table 2 · p. 9
  • 83.9 on SD3.5-Medium and 86.5 on BAGEL are the averages with curriculum learning, versus 78.4 and 83.2 baselines, respectively.Spatial accuracy also reaches 69.5% and 77.8%, respectively; without curriculum learning, the averages are 81.9 and 83.7.Source: Table 3 · p. 9, §5.2 · p. 9

6 CONCLUSION

The conclusion presents SpatialCritic and capability-graph curriculum learning as complementary responses to unreliable spatial rewards and non-progressive rollouts. It reports limitations in tool transfer and failure robustness, alongside data and verification caveats.

6 CONCLUSION

  • The framework pairs grounded spatial preference judgments and agent–harness alignment with mastery-guided progression, review, and remediation across capability dependencies.Experiments report accurate SpaRW-Eval judgments, effective Best-of-N reward signals, and improved RL alignment on SD3.5-Medium and BAGEL.Source: §6 · p. 9
  • The evaluation fixes the tool configuration, leaving transfer to updated or alternative tools and robustness to temporary failures or missing outputs untested.The authors identify cross-tool transfer and failure-aware fallback strategies as future work.Source: §6 · p. 9
  • The authors used generative models for synthetic prompts, images, and candidate-label review, and AI assistants for literature discovery and writing support.AI tools also supported experimental-analysis feedback and inspection of data-processing code and metadata.Source: AI USE STATEMENT · p. 10
  • Synthetic data may inherit generator biases, and model-based verification may introduce systematic annotation errors; human inspection does not guarantee unbiased or error-free labels.The authors caution that spatial correctness is not evidence of authenticity or safety.Source: ETHICS STATEMENT · p. 10
  • The paper locates toolkit, dataset, evaluation, training, and curriculum details in appendices and states that code will be open-sourced upon acceptance.The reproducibility statement points to App. C, B.1–B.3, and D for these materials.Source: REPRODUCIBILITY STATEMENT · p. 10

Related papers

Loading 2610.14852v1…