PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Prompts

Shuyue Stella Li, Xiaochuang Han, Yulia Tsvetkov, Luke Zettlemoyer

arXiv:2609.35641v1cs.AIcs.CV

TL;DR

Image-generation models often fail to follow precise visual instructions, and learned reward evaluators can make errors. The paper introduces program-verifiable visual rewards and uses them for reinforcement learning; training improves instruction following across harder tasks and natural prompts, with further gains when combined with existing objectives.

  • Problem

    Image generators often miss requested counts, colors, or positions, while learned evaluators make errors on instruction-following judgments and can be exploited by trained policies.

    Source: §1 · p. 1

  • Method

    VVR derives prompts and deterministic program verifiers from satisfiable visual constraints, and RLVVR uses verifier scores as reinforcement-learning rewards.

    Source: §1 · pp. 1–2

  • Results

    RLVVR improves instruction following on tasks harder than those seen in training, transfers from synthetic shapes to natural prompts, and improves benchmark performance and human preference when mixed with existing objectives.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors’ results motivate including VVR in standard image-generation post-training recipes.

    Source: §1 · p. 2

    RLVVR is evaluated on SD3.5-M with Flow-GRPO; applying it to other generators or RL algorithms is left to future work.

    Source: LIMITATIONS AND FUTURE DIRECTIONS · p. 10

Abstract

from arXiv · show

Precise instruction following in image generation, such as satisfying object counts and spatial relations, remains an open challenge at least in part because it is learned using unreliable reward models such as object detectors and vision-language models. We introduce Verifiable Visual Rewards (VVR), the first framework for programmatically verifiable image rewards, and show that training on it generalizes to natural prompts. Each VVR task is a scene of geometric objects and relations among them, from which we derive both the prompt and a deterministic verifier, so tasks can be generated in any number and at any chosen complexity. We release VVRBench, with 10,000 tasks over 32 constraint types, and VVRBench-Challenge, with 720 more complex tasks; the strongest model we evaluate---GPT-Image-2.5---solves 21.4% of VVRBench-Challenge. Using VVR scores as rewards for reinforcement learning (RLVVR) raises the accuracy of Stable Diffusion 3.5 Medium on VVRBench from 2.8% to 28.3% and demonstrates consistent easy-to-hard generalization. These gains extend to out-of-domain benchmarks, and mixing VVR into existing objectives further improves overall performance and human preference, motivating the adoption of VVR into standard image generation post-training recipes.

1 INTRODUCTION

Image generators often miss visual requirements, while learned evaluators can misjudge them and be exploited. The paper introduces programmatically verifiable visual rewards and reports that reinforcement learning with these rewards improves instruction following and transfers to natural prompts.

1 INTRODUCTION

  • Image generators often miss requested counts, colors, or positions, and learned preference models, VLMs, and detectors make errors that trained policies can exploit.Source: §1 · p. 1
  • VVR pairs composable, jointly satisfiable visual constraints with natural-language prompts and deterministic program verifiers, enabling fine-grained diagnosis of generator capabilities.The framework supports generating tasks in any number and at chosen complexity.Source: §1 · pp. 1–2
  • The strongest open-weight model, FLUX.2-dev, achieves 19.2% accuracy on VVRBENCH; the strongest model overall, GPT-Image-2.5-Sunburst, solves 21.4% of VVRBENCH-Challenge tasks.The benchmark can be updated with higher complexity as frontier models evolve.Source: §1 · p. 2
  • RLVVR improves instruction following beyond training complexity, transfers from colored shapes to natural prompts, and, mixed with existing objectives, improves general benchmark performance and human preference.The authors motivate adding VVR to standard image-generation post-training recipes.Source: §1 · p. 2

2 VERIFIABLE VISUAL REWARDS

VVR represents visual-generation tasks as constraints on object groups and generates prompts from satisfiable scenes. Deterministic pixel-based verification then checks those constraints and supplies both pass/fail decisions and partial-credit rewards.

2.1 VVR TASK REPRESENTATION

  • Each task packages object groups, background constraints, active constraints, forbidden content, and a prompt; valid tasks are well-formed, jointly satisfiable, and faithfully expressed.The constraint library contains 46 types grouped into five families.Source: §2.1 · p. 3, Table 1 · p. 3

2.2 TASK GENERATION

  • The generator samples scenes, selects constraints already true in them, chooses active subsets, and templates prompts, ensuring well-formed, satisfiable tasks with faithful wording.Each requirement maps to a fixed phrase, and every prompt phrase corresponds to a task requirement.Source: §2.2 · p. 3
  • Structural complexity sums type-specific constraint costs that grow with the number of evaluated objects; datasets can target selected constraint families and difficulty ranges.For the paper’s dataset instantiation, the fixed background-color and forbidden-content constraints are excluded from complexity.Source: §2.2 · p. 3

2.3 DETERMINISTIC, REFERENCE-FREE CONSTRAINT VERIFICATION

  • The reference-free verifier extracts candidate objects from pixels, then applies a program verifier for each constraint to return a pass/fail decision and partial-credit score.Extraction uses fixed color thresholds, connected components, and contour measurements to classify supported shapes.Source: §2.3 · p. 4
  • Exact success requires every background, active, and forbidden-content constraint to pass; training’s dense reward combines partial-credit scores with a penalty against focusing on easy constraints.The penalty is multiplicative, and reward weights are fixed by constraint type.Source: §2.3 · p. 4

3 VVRBENCH

VVRBench evaluates image generators on tasks with verifiable visual constraints, and results show substantial difficulty on complex tasks. Failures are especially common when requirements apply across sets of objects.

3.1 PRECISE INSTRUCTION FOLLOWING IS FAR FROM SOLVED

  • GPT-Image-2 accuracy drops from 97.65% in C1 to 65.72% in C5, while no open-weight model exceeds 20% overall.FLUX.2-dev, the strongest open-weight model, solves 19.15% overall and 2.24% in C5.Source: Table 3 · p. 5, §3.1 · p. 5
  • 21.39% of VVRBench-Challenge tasks are solved by GPT-Image-2.5-Sunburst, versus 10.28% by GPT-Image-2; every other model solves at most 8%.Sunburst accuracy falls to 7.92% at complexity 69–80.Source: Table 4 · p. 5, §3.1 · p. 5
  • The uniform accuracy drop across complexity bins supports structural complexity as a model-independent heuristic for generating tasks with controlled difficulty.The VVRBench table divides tasks into five complexity ranges.Source: §3.1 · p. 5, Table 3 · p. 5

3.2 FAILURES CONCENTRATE IN COUNTING AND OBJECT MATCHING

  • On VVRBench-Challenge, API models satisfy 95% of Grounding constraints but only 56% of Topology constraints; same count passes in 31% of checks.Times as many passes in 33% of checks, and each contains passes in 34%.Source: §3.2 · p. 6, Figure 3 · p. 6

3 VVRBENCH

  • Generators handle individual objects and pairs better than requirements constraining whole sets of objects.Table 2 inventories training and evaluation datasets produced by the VVR task generator.Source: Table 2 · p. 4, §3.2 · p. 6

4 RLVVR: VVR FOR DIFFUSION POST-TRAINING

The section evaluates reinforcement learning with Verifiable Visual Rewards (RLVVR), testing whether training complexity shapes instruction following, whether synthetic-scene skills transfer to natural prompts, and whether VVR complements existing rewards. Results show gains on VVR tasks and natural-prompt benchmarks, with further improvements from reward mixtures.

4.1 EXPERIMENTAL SETUP

  • Each 100K-task corpus trains Stable Diffusion 3.5 Medium with Flow-GRPO; VVR-Easy caps tasks at one constraint family and complexity 20, while VVR-Matched follows benchmark complexity 3–48.Evaluation covers VVRBENCH and non-VVR metrics including GenEval, OCR, human-preference, and reward-model scores.Source: §4.1 · pp. 6–7

4.2 RQ1: RLVVR TEACHES PRECISE INSTRUCTION FOLLOWING

  • 2.81% to 28.27%: VVR-Easy raises VVRBENCH accuracy and still gains 17.16, 8.31, and 1.35 points on C3–C5; VVR-Matched reaches 46.60% overall.Every C3–C5 task exceeds the complexity of all VVR-Easy training tasks.Source: Figure 4 · p. 7, §4.2 · p. 7
  • On tasks outside its training range, VVR-Easy closes 79% and 64% of the pretrained–VVR-Matched gap for individual count and relation scores, but 55% and 46% for satisfying all counts or relations.The authors conclude that easy tasks make individual constraints reliable, while large scenes teach compositionality.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 RQ2: SKILLS LEARNED FROM SYNTHETIC SCENES TRANSFER TO NATURAL PROMPTS

  • Trained only on colored shapes, VVR-Easy improves eight of ten non-VVR metrics and is preferred on 71.6% of 160 natural prompts; it also scores higher than GenEval2 training on six of nine natural-prompt benchmarks.Against GenEval2 training, gains include GenEval position (+0.150), counting (+0.103), and color attribution (+0.025).Source: §4.3 · p. 7, Table 6 · p. 8, Table 5 · p. 8

4.4 RQ3: SUPERVISION FROM SYNTHETIC SCENES COMPLEMENTS EXISTING REWARDS

  • Mixed with GenEval2, VVR-Easy raises all ten non-VVR metrics, including GenEval2 itself (+0.025), and annotators prefer the mixture to GenEval2 alone on 58.6% of 160 natural prompts.VVR-Easy mixtures with OCR and the five-reward objective each raise eight of ten metrics; those mixtures also show small native OCR declines, and the five-reward mixture slightly lowers PickScore.Source: §4.4 · p. 8, Table 5 · p. 8, Table 6 · p. 8

5 RELATED WORK

Prior work uses executable checks for language outputs and learned evaluators for image generation. VVR instead derives prompts and constraints from generated scenes and verifies generated pixels directly, supporting both evaluation and post-training.

5 RELATED WORK

  • Verifiable-reward work checks language-model outputs with executable rules, while procedural environments generate tasks at controlled difficulty; VVR derives image prompts and constraints from generated scenes.Earlier visual-scene work derives questions and answers, whereas VVR derives image-generation prompts and their constraints.Source: §5 · p. 8
  • Text-to-image post-training commonly uses learned preference models or rules over detector and vision-language-model outputs; RLVVR computes prompt-specific rewards directly from generated pixels and can be mixed with those objectives.The cited work also includes a compressibility reward that avoids learned models but is not prompt-dependent.Source: §5 · pp. 8–9
  • Text-to-image evaluation metrics and compositional benchmarks generally score images with learned models; VVRBENCH verifies each constraint exactly using the same program verifiers that provide RLVVR rewards.Some prior synthetic-visual-concept prompts are still evaluated with detectors or vision-language models.Source: §5 · p. 9

6 CONCLUSION

The paper presents VVR as a way to score open-ended image generation with deterministic program verifiers, providing evaluation feedback and post-training signals. It reports transfer to natural prompts and further gains when VVR is mixed with existing post-training objectives.

6 CONCLUSION

  • VVR uses deterministic program verifiers to score open-ended image generation for evaluation feedback and post-training signals.Tasks can be procedurally generated for target distributions over constraint types and complexity levels.Source: §6 · p. 9
  • The released VVRBENCH contains 10K tasks, and GPT-Image-2.5-Sunburst solves 21.4% of VVRBENCH-Challenge tasks.The tasks ask models to draw geometric objects with specified colors, shapes, counts, and spatial relations.Source: §6 · p. 9
  • RLVVR improves instruction following on VVR tasks and natural prompts unseen during training; mixing VVR with existing objectives yields further evaluation-suite and human-preference gains.The authors suggest incorporating VVR into standard image-generation post-training recipes.Source: §6 · p. 9

LIMITATIONS AND FUTURE DIRECTIONS

The paper identifies limits in VVR’s current visual scope, in the models and RL algorithm tested, and in its analysis of incorrect refusals. It also describes possible extensions to other visual tasks and reward designs.

LIMITATIONS AND FUTURE DIRECTIONS

  • Current VVRBENCH covers 2D geometric objects, eight colors, three shapes, and plain backgrounds; extending it to richer objects or 3D renderings remains future work.The authors also identify image editing as a possible application beyond current text-to-image generation.Source: LIMITATIONS AND FUTURE DIRECTIONS · p. 10
  • The reported post-training study uses Stable Diffusion 3.5 Medium with Flow-GRPO; applying RLVVR to other image generators and RL algorithms is left to future work.The authors suggest that supervised fine-tuning or DPO before RL could further improve instruction following.Source: LIMITATIONS AND FUTURE DIRECTIONS · p. 10
  • Verifier feedback on failed constraints permits alternative reward designs, and the complexity-controlled generator permits adaptive curricula for RLVVR.Source: LIMITATIONS AND FUTURE DIRECTIONS · p. 10
  • Several API models incorrectly decline satisfiable VVR tasks as contradictory, but the analysis remains case-study-level because abstentions are few.Source: LIMITATIONS AND FUTURE DIRECTIONS · p. 10

A TASK REPRESENTATION

VVR represents tasks through object groups, their constraints, and a natural-language prompt; its constraint library assigns supported values and complexity costs, and a worked example shows the representation in use.

A.1 CONSTRAINT LIBRARY

  • Table 7 lists the exact 46 constraint types, their supported values, and their contributions to structural complexity; † marks types that appear only in VVRBENCH-Challenge.The background and forbidden-content constraints apply to every task and do not contribute to C(s).Source: Table 7 · p. 16

A.2 EXAMPLE TASK

  • The example task uses one purple circle below one yellow square, a pale-pink background, and no unrequested objects; its prompt binds attributes unambiguously.The task representation includes two object groups and seven constraints.Source: §A.2 · pp. 16–17

B TASK GENERATION

This section describes how VVR tasks and their verifier are constructed, validated, and distributed across datasets, including human audits of verifier behavior and perceptual thresholds. It also examines how constraint complexity predicts task failure.

B.1 PROCEDURE

  • Generation samples scenes and constraints, selects active constraints within a target complexity range, renders prompts and references, then filters duplicates and verifier failures.A changed-constraint counterfactual must also fail verification; relation-specific placement templates and seeded sampling produce scenes and constraints.Source: §B.1 · pp. 17–18

B.2 VALIDATION

  • All 10,000 VVRBench and 720 Challenge tasks pass verification on their reference images and fail on their counterfactuals.For the worked example, four nested tasks share a scene; each reference is accepted and each counterfactual is rejected.Source: §B.2 · p. 18, §B.3 · p. 19

B.3 WORKED EXAMPLE

  • The worked example yields four nested tasks from one scene by adding constraints in a fixed order while staying within the target complexity range.The largest task has 12 constraints; its prompt omits the yellow squares’ exact count because a same-count relation supplies that information.Source: §B.3 · p. 19

B.4 STRUCTURAL COMPLEXITY

  • For 19 models, C(s) has the highest failure-prediction AUC and R2 for 18, with median AUC 0.857 and R2 0.283.C(s) counts color, shape, relation, and layout constraints; the reported analysis also compares object-instance, group, and relation counts.Source: §B.4 · pp. 19–20

B.5 DATASETS

  • VVRBench selects from 11,250 candidates across five scene-size settings and uses 32 constraint types, while Challenge cycles through all 46 types and balances complexity.VVR-Easy adds one constraint type to object attributes, and VVR-Matched is accepted only when its complexity distribution matches VVRBench by a Kolmogorov–Smirnov test.Source: §B.5 · p. 20

C.1 PIXEL-TO-OBJECT EXTRACTION

  • The verifier estimates the background, segments saturated foreground pixels by color, cleans masks morphologically, and groups connected pixels into candidate objects.It scores components geometrically for circle, square, and triangle shape, and uses fixed safeguards for ambiguous colors, imperfect contours, and stray pixels.Source: §C.1 · p. 21

C.2 PROGRAM VERIFIERS

  • Program verifiers compare detected objects with requested attributes and relations using explicit geometric rules, including centroid margins, component boundaries, and relative visual extent.Count and color checks use matched components; extra visible components are unmatched, and relative size is assessed between objects because absolute size lacks a stable human decision boundary.Source: §C.2 · pp. 22, 24

C.3 VERIFIER VALIDATION

  • Verifier validation includes 290 historical edge cases, 360 direct checks, 14,788 metamorphic checks, 320 single-constraint cases, 200 threshold cases, and geometry tests.The listed coverage includes blur, compression, low contrast, irregular contours, touching and merged components, and missing objects.Source: §C.3 · p. 24
  • During development, 3,947 human decisions set predicate boundaries for color hues, object-touch margins, shape-contour tolerances, and the size ratio defining “larger.”Source: §C.3 · p. 25
  • After calibration, the released verifier agrees with 481 of 508 decisive labels (94.7%, Cohen’s κ = 0.89) in the larger audit.This released version scores every result in the paper.Source: §C.3 · p. 25
  • Before calibration, the frozen verifier agrees with 454 of 508 decisive labels (89.4%, Cohen’s κ = 0.78) in the larger audit; an earlier audit reports 421 of 452 (93.1%, κ = 0.86).The larger audit covers 512 images of 128 prompts generated by pretrained SD3.5-M, FLUX.1-dev, and two SD3.5-M models trained with earlier VVR rewards.Source: §C.3 · p. 25

C.4 SCORES AND TRAINING REWARD

The section specifies how dense training rewards combine partial-credit constraint scores and reports evaluation details and API-model outcomes, including difficulty with complex prompts and some no-image responses.

C.4 SCORES AND TRAINING REWARD

  • Dense reward combines group-level partial-credit scores for count, shape, layout, and size with weighted object, relation, forbidden-content, extra-component, and background scores.The relation score is included only when a task has a relation.Source: §C.4 · p. 25

D.1 EVALUATION DETAILS

  • Open-weight models use the same prompt-specific hashed seed; VVRBENCH images are scored at 512 × 512 and Challenge images at 1024 × 1024.Completed API responses are not resampled, and transient errors are retried.Source: §D.1 · p. 25

D.2 COMPLETE VVRBENCH-FAST RESULTS

  • GPT-Image-2 models exceed 80% overall on VVRBENCH-Fast but fall to 51%–57% in C5, while Gemini models reach 37%–47%.VVRBENCH-Fast is an 820-task subset with 20 tasks at each attainable integer complexity from 3 to 44.Source: Table 10 · p. 26

D.3 RESPONSES WITHOUT AN IMAGE

  • 62 API responses contain no image—48 include text and 14 are empty—and each scores zero, although every task is satisfiable.Examples include models incorrectly calling compatible arrangement constraints contradictory or treating unassigned objects as grid-bound.Source: §D.3 · pp. 26–27

D.4 PER-CONSTRAINT PASS RATES OF API MODELS

  • Table 11 reports per-constraint-type pass rates across 5,040 outputs from seven API models, pooling checks by type and counting responses without images as failures.The table also gives the number of checks per type and the unweighted mean across the seven models.Source: §D.4 · p. 27

D.5 COMPLEXITY-MATCHED FAMILY ANALYSIS

Complexity-matched comparisons identify model-specific weaknesses associated with particular constraint families, and a sensitivity analysis preserves the largest negative profiles.

D.5 COMPLEXITY-MATCHED FAMILY ANALYSIS

  • The largest negative profiles persist in a linear probability model with all five family indicators and integer-complexity fixed effects, while GPT-Image-2 positive associations are less stable.The negative differences shift to −14.6 for GPT-Image-1-mini Spatial, −5.9 for FLUX.2-dev Cardinality, and −12.6 for HunyuanImage-2.1 Size.Source: §D.5 · p. 28
  • GPT-Image-1-mini loses 9.7 points on Spatial tasks, HunyuanImage-2.1 loses 10.7 on Size tasks, and FLUX.2-dev loses 4.0 on Cardinality tasks versus tasks without those families at matched complexity.The comparison matches task complexity; Table 12 reports VVRBENCH accuracy and the difference from prompts without each family.Source: §D.5 · p. 27, Table 12 · p. 29

D.6 FAILURE EXAMPLES

  • Figure 10 shows API-model failures that add content excluded by the prompt, including extra objects around requested scenes.Examples include added vases, bowls, apples, lemons, mugs, squares, and other shapes.Source: Figure 10 · p. 29, §D.6 · p. 29

E TRAINING SETUP

The section defines evaluation settings and reports that VVR training improves benchmark accuracy, with matched training strongest at high complexity; partial-credit gains exceed all-constraint satisfaction gains increasingly at C3–C5.

E TRAINING SETUP

  • Table 13 documents the final SD3.5 Medium post-training configuration; released resolved configurations and data manifests retain the remaining implementation metadata.Reward proportions are fractions of prompt groups in each update, and each rollout is scored only by the reward attached to its prompt.Source: E TRAINING SETUP · p. 29, Table 13 · p. 30

E.1 POST-TRAINING EVALUATION

  • Evaluation uses one fixed-seed image per prompt except four on GenEval, with separate held-out training-objective, preference, and cross-domain metric panels.The cross-domain panel averages each metric within a prompt set and then across four prompt sets.Source: §E.1 · pp. 30–31

F.1 VVRBENCH RESULTS BY COMPLEXITY

  • 2.81% VVRBENCH accuracy (%) for SD3.5 M rose to 28.27% with VVR-Easy and 46.60% with VVR-Matched; matched training gained most at high complexity.Table 14 reports accuracy by complexity range and gives subscripts as 95% confidence margins.Source: Table 14 · p. 31, §F.1 · p. 31

F.2 PARTIAL AND JOINT CONSTRAINT SATISFACTION

  • Across C3–C5, VVR-Easy closes more of the pretrained-to-Matched gap in partial scores than in tasks satisfying every constraint of a kind, increasingly with complexity.The partial scores are graded count and relation scores; “all” measures the fraction of tasks satisfying every count or every relation.Source: §F.2 · p. 31, Table 15 · p. 32

F.3 EXTERNAL TASK, QUALITY, AND ALIGNMENT METRICS

Adding VVR-Matched to a GenEval2 mixture raises VVRBench and GenEval2 scores and improves seven non-VVR metrics; annotators also prefer VVR-trained images across both studied comparisons.

F.3 EXTERNAL TASK, QUALITY, AND ALIGNMENT METRICS

  • Table 16 evaluates the pretrained model and nine post-training conditions, retaining native units for VVR accuracy, GenEval, GenEval2, and OCR and macro-averaging cross-domain metrics.HPSv3 values are means over four cross-domain prompt sets.Source: Table 16 · p. 32, §F.3 · p. 32
  • 21.82% to 33.50% VVRBENCH accuracy and 0.478 to 0.491 GenEval2 when VVR-Matched replaces VVR-Easy in the GenEval2 mixture; seven of ten non-VVR metrics rose.Intervals excluded zero for HPSv2.1 (+0.004) and ImageReward (+0.038).Source: Table 17 · p. 33, §F.3 · p. 33

G HUMAN PREFERENCE STUDY

  • Each annotator favored the VVR-trained model in every suite of both comparisons; pairwise agreement when both chose was 83.8%, and Fleiss’ κ was 0.49.Three annotators evaluated 400 image pairs across VVR-Easy versus pretrained and GenEval2 mixed with VVR-Easy versus GenEval2.Source: G HUMAN PREFERENCE STUDY · p. 33, Table 18 · p. 33
Loading 2609.35641v1…