Source-linked AI summary
Enhancing Spatial Understanding in Image Generation via Reward Modeling
Zhenyu Tang, Chaoran Feng, Yufan Deng, Jie Wu, Xiaojie Li, Rui Wang, Yunpeng Chen, Daquan Zhou
TL;DR
Complex spatial prompts remain difficult for text-to-image models, and reliable reward signals for improving their spatial understanding are lacking. The paper constructs a human-verified adversarial preference dataset, trains SpatialScore, and uses it for online RL with top-k filtering. Across multiple benchmarks, the approach produces substantial and consistent spatial-reasoning improvements over the base model, while extension to video generation remains unexplored.
Problem
Text-to-image models struggle with complex spatial relationships, while reliable reward models for guiding online RL in this setting remain limited.
Method
The paper builds an 80K-pair human-verified adversarial preference dataset, trains SpatialScore, and uses it with GRPO-style online RL and top-k filtering.
Results
SpatialScore surpasses several leading proprietary models in spatial evaluation, and online RL yields substantial and consistent spatial-reasoning improvements across multiple benchmarks over the base model.
Takeaways & Limitations
A specialized spatial reward model can provide reliable signals for online RL that improves spatial understanding in image generation.
Takeaways & Limitations
The integration of spatial understanding with temporal dynamics, particularly for video generation, remains insufficiently explored.
Abstract
from arXiv · showhide
Recent progress in text-to-image generation has greatly advanced visual fidelity and creativity, but it has also imposed higher demands on prompt complexity-particularly in encoding intricate spatial relationships. In such cases, achieving satisfactory results often requires multiple sampling attempts. To address this challenge, we introduce a novel method that strengthens the spatial understanding of current image generation models. We first construct the SpatialReward-Dataset with over 80k preference pairs. Building on this dataset, we build SpatialScore, a reward model designed to evaluate the accuracy of spatial relationships in text-to-image generation, achieving performance that even surpasses leading proprietary models on spatial evaluation. We further demonstrate that this reward model effectively enables online reinforcement learning for the complex spatial generation. Extensive experiments across multiple benchmarks show that our specialized reward model yields significant and consistent gains in spatial understanding for image generation.
1. Introduction
Complex spatial prompts remain difficult for text-to-image models, while existing reward models and GenEval-based approaches provide unreliable feedback. The paper addresses this gap with a human-verified spatial preference dataset, SpatialScore, and online RL.
- Text-to-image models struggle to depict complex spatial relationships among multiple objects as prompt complexity increases.
- Existing reward models often fail to evaluate complex spatial relationships accurately, sometimes assigning higher rewards to spatially incorrect images.
- Proprietary VLM APIs are impractical for online RL because their high cost requires frequent reward queries, while open-source VLMs can hallucinate.
- GenEval-based reinforcement learning fails to generalize from simple templates to long prompts with multiple spatial relationships and is sensitive to occlusion.
- SpatialReward-Dataset contains 80K adversarial preference pairs reviewed and filtered by human experts to support reliable reward-model training.
- SpatialScore evaluates spatial relationship accuracy and, when used with top-k filtering for online RL, produces substantial improvements over the base model across multiple benchmarks.
2. Related Works
Prior text-to-image reward models emphasize aesthetic quality or semantic alignment, while reinforcement learning methods improve generation through adapted optimization procedures. The paper positions spatially focused reward modeling as the missing component for complex spatial understanding.
- Existing text-to-image reward models use CLIP-based or VLM-based approaches to assess human preference, aesthetic quality, and text-image alignment.
- The proposed direction is a reward model focused specifically on spatial understanding rather than primarily aesthetics or semantic alignment.
- Reinforcement learning methods including PPO, DPO, FlowGRPO, and Dance-GRPO have been adapted to diffusion-based generation for alignment and controllability.
3. Dataset
The SPATIALREWARD-DATASET uses adversarial preference pairs to represent complex spatial relationships across diverse real-world scenarios. Human validation and controlled generation are used to improve pair quality and reduce confounding visual differences.
- SPATIALREWARD-DATASET comprises 80K adversarial preference pairs created as the foundation for reward training.
- Each pair is generated with one image-generation model while prompts are varied to reduce the influence of aesthetic differences between models.
- GPT-5 creates initial prompts containing complex spatial relationships among multiple objects and perturbs them by modifying spatial relations.
- Qwen-Image, HunyuanImage-2.1, and seedream4.0 are used for data construction, and human annotators manually review pairs for alignment with spatial constraints.
- The dataset covers diverse real-world scenarios with greater spatial complexity and compositional diversity than GenEval’s simple template-based prompts.
4. Method: SpatialScore
SpatialScore uses a vision-language backbone and preference-pair training to assign reward scores that rank images according to spatial accuracy. The trained model then supplies rewards for online reinforcement learning.
- Architecture: SpatialScore uses Qwen2.5-VL-7B as a backbone to extract image–text features and replaces its language-modeling head with a linear reward head.The model processes preferred and less-preferred images conditioned on the same prompt.
- Preference training: Each preference example contains a preferred winner image and a less-preferred image, enabling pairwise reward learning.The training data is represented as triplets (c, yw, yl).
- Reward representation: A Gaussian distribution s ∼ N(µ, σ^2) models the reward score, with µ and σ predicted from a special token attending to image and text representations.The reward head maps the special-token embedding to the distribution parameters.
- Preference training: The reward model is optimized with Bradley–Terry preference learning by minimizing the negative log-likelihood of the ground-truth preference.Two independent forward passes produce scores for the preferred and less-preferred images.
- Online reinforcement learning: After training, SpatialScore assigns higher scores to preferred images and serves as the reward model for online reinforcement learning.The sigmoid converts preference probabilities to values in [0, 1].
5. SpatialScore in Image Generation
The method uses SpatialScore to evaluate groups of generated images and applies GRPO updates that emphasize the most and least accurate samples. Top-k filtering addresses advantage bias while retaining a large sampling group for diversity and stability.
- Reward-guided sampling: The policy generates image groups through SDE sampling, and SpatialScore assigns each image a spatial-accuracy reward.The group consists of G samples generated by the policy conditioned on prompt c.
- Advantage bias: Group-wise advantage normalization can bias training because easy prompts produce many high-reward samples while difficult prompts produce generally low rewards.For easy prompts, a high group mean can give some high-quality samples negative advantages.
- Top-k filtering: GRPO ranks sampled images by reward, selects the top-k and bottom-k candidates, and uses them to compute balanced group statistics for policy updates.Only the selected subset S is used during training.
- Policy optimization: The GRPO objective updates the policy with clipped policy-gradient ratios and a KL-divergence penalty against a reference policy.The regularization limits excessive deviation from the reference policy.
- Efficiency and stability: Top-k filtering reduces the number of function evaluations while preserving the consistent group size needed for sample diversity and stable training.The method avoids directly shrinking the sampling group because prior findings associated that change with training collapse.
6. Experiments
Experiments evaluate SpatialScore against reward-model and VLM baselines, then test its use for online RL and key design choices. SpatialScore reaches strong preference accuracy, while reward-guided training improves spatial understanding across benchmarks.
- Reward Model Performance: 0.89–0.95 accuracy is attained by leading proprietary models, while Qwen2.5-VL-72B reaches only 0.76 pairwise accuracy.Existing image-reward models and open-source VLMs show weaker preference prediction than proprietary models.
- Reward Model Performance: 95.77% pairwise accuracy is achieved by 7B SpatialScore, surpassing GPT-5 and Gemini-2.5 Pro on multi-object spatial understanding.The model achieves state-of-the-art performance on the reward benchmark.
- Applying SpatialScore for Online RL: 2.18 to 7.81 is the improvement on in-domain SPATIALSCORE evaluation after online RL with Flux.1-dev as the base model.The approach also produces consistent gains on spatial-aware dimensions of short- and long-prompt alignment benchmarks.
- Applying SpatialScore for Online RL: Flow-GRPO trained on GenEval improves short prompts but degrades markedly on long prompts with complex multi-object spatial relationships.Qualitative comparisons report that the SpatialScore-trained model more faithfully reflects these relationships, whereas the GenEval variant can omit objects and produce implausible artifacts.
- Ablation Study: 89.1% to 95.8% pairwise preference accuracy is obtained by scaling SpatialScore from Qwen2.5-VL-3B to Qwen2.5-VL-7B.This ablation evaluates the effect of reward-model backbone size.
- Ablation Study: k = 6 provides a better tradeoff between sampling balance and diversity than k = 4 in top-k filtering.The authors adopt k = 6 as the default and report comparable or superior performance with fewer function evaluations in the policy-update stage.
7. Conclusion
The paper addresses spatial understanding in image generation through online RL by combining a human-verified preference dataset, a specialized reward model, and top-k-filtered GRPO. Across multiple benchmarks, this approach substantially and consistently improves spatial reasoning over the base model.
- 7. Conclusion: The method combines an 80K-pair human-verified dataset, SpatialScore, and GRPO with top-k filtering for online RL.SpatialScore provides the reward signal used to train the image-generation model.
- 7. Conclusion: Substantial and consistent improvements in spatial reasoning are obtained across multiple benchmarks over the base model.The conclusion attributes these gains to leveraging the high-fidelity reward model with top-k-filtered GRPO.
8. Experimental Details
The experimental setup defines how SpatialScore is trained and evaluated, and how it supplies rewards for online RL. Evaluation combines an in-domain spatial benchmark with spatial-aware dimensions from several out-of-domain alignment benchmarks.
- Reward Model Training: SpatialScore is fine-tuned from Qwen2.5-VL-7B with LoRA on 80K preference pairs.A special reward token attends to visual and textual features during reward-model training.
- Reward Model Evaluation: The reward evaluation benchmark contains 365 human-reviewed preference pairs and uses overall preference accuracy as its metric.Each pair contains an image from a perfect prompt and an image from the corresponding perturbed prompt.
- Online RL Training: Flux.1-dev is fine-tuned with LoRA-based GRPO using SpatialScore rewards and perfect prompts from the curated dataset.This setup is used to validate improved spatial understanding through online RL.
- Evaluation Benchmarks: Evaluation uses SpatialScore in-domain and spatial-aware sub-dimensions from out-of-domain text–image alignment benchmarks.The out-of-domain benchmarks include DPG-Bench, TIIF-Bench, and UniGenBench++.
- Reward Criterion: Spatial Position Consistency rates how well image spatial relationships match the prompt on a 0-to-10 scale.The rubric focuses on relative positioning among multiple objects and their relation to the background.
9. Dataset Construction
SpatialReward-Dataset is constructed from adversarial perfect-versus-perturbed image pairs that isolate violations of complex spatial relationships. Multiple image generators and human verification are used to increase diversity and ensure pair quality.
- Pair Construction: 80K adversarial preference pairs are constructed by varying prompts while keeping each pair’s image-generation model fixed.The prompts describe complex spatial relationships among multiple objects.
- Dataset Analysis: Figures visualize preference pairs and report distribution statistics for the SpatialReward-Dataset.The visualizations distinguish perfect images and generated images from perturbed prompts.
- Dataset Examples: The dataset examples encode relations such as left/right, front/behind, alignment, centering, and relative object ordering.Examples span dining rooms, rooftops, stadiums, offices, kitchens, markets, and outdoor scenes.
- Human Verification: Human verification checks whether perfect images satisfy all specified spatial relations and discards pairs when clear violations occur.A further verification round is used to ensure dataset reliability and overall quality.
10. Additional Experiment Results
Additional experiments show that SpatialScore-guided reinforcement learning improves spatial understanding across model families and evaluation settings, while GenEval has limited reliability and generalization for complex prompts.
- Applying SpatialScore to Qwen-Image: 6.74 to 8.25: SpatialScore-guided RL improves Qwen-Image on the in-domain SpatialScore evaluation.The method also yields consistent improvements across short- and long-prompt settings on spatial-aware text–image alignment benchmarks.
- Evaluations on GenEval: GenEval rewards are unreliable under occlusion and show limited generalization to long texts with complex inter-object spatial relationships after RL training.These limitations motivate treating GenEval as a complementary evaluation rather than a fully reliable reward signal for complex spatial generation.
- Evaluations on GenEval: Significant zero-shot improvements occur across all evaluated metrics on the GenEval benchmark after SpatialScore-guided RL training.GenEval uses simple, fixed-template compositions, so this result assesses generalization to that benchmark rather than complex long-prompt reasoning.
- SpatialScore Backbone Ablation: 0.891 to 0.958 to 0.973: pairwise preference-prediction accuracy increases as the SpatialScore backbone scales from 3B to 7B and 32B.The 7B SpatialScore backbone is reported to exceed Gemini 2.5 Pro in the referenced comparisons.
- Qualitative Comparisons: SpatialScore-guided RL produces images that more accurately reflect complex spatial relationships, whereas GenEval-guided models omit objects and generate implausible artifacts.Examples include missing candles, soap, and toothbrushes, plus a napkin floating in mid-air.
11. Limitations and Future Works
The paper validates reward-modeling improvements for image-level spatial understanding but identifies temporal spatial reasoning in video generation as an open direction.
- Limitations: Image-level reward modeling has been validated, but its integration with temporal dynamics remains unexplored.The stated scope boundary is particularly relevant to video generation.
- Future Work: Video generation must account for both static spatial relationships and their changes over time.The paper gives sequential object movements and position swaps as examples of this requirement.
- Future Work: Extending reward modeling to spatially accurate and temporally consistent video is identified as an essential challenge for future work.The paper connects this direction to sim-to-real embodied simulation scenarios.