PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrate What You SHIP: Post-Selection Risk Control for Verifier-Guided Text-to-Image Generation

Xuanhua Yin, Shunqi Mao, Wei Guo, Chuanzhi Xu, Weidong Cai

arXiv:2608.21748v1cs.CV

TL;DR

Verifier-guided T2I search changes the distribution of released outputs, making candidate-level risk calibration insufficient for the deployed policy. The paper introduces SHIP, which calibrates complete policy rollouts on held-out prompts using an independent target judge and confidence bounds. Across search procedures, SHIP identifies lower-risk operating points and policy-dependent risk, coverage, and compute tradeoffs, while validity depends on replay and calibration assumptions.

  • Problem

    Candidate-level risk control need not control released-output risk because search changes both which prompts receive outputs and which images are released.

  • Method

    SHIP runs or replays the complete inference policy on held-out prompts, evaluates its released image with an independent target judge, and selects the most permissive threshold satisfying a risk budget.

  • Results

    SHIP recovers feasible released-output operating points across fixed, sequential, and adaptive procedures while revealing policy-dependent tradeoffs among risk, coverage, and inference cost.

  • Takeaways & Limitations

    Reliable inference-time scaling requires calibrating the output distribution induced by the complete deployed policy rather than independently generated candidates.

  • Takeaways & Limitations

    Exact calibration for threshold-dependent history-dependent search may require a complete rollout for every prompt and threshold, so adaptive-policy operating points are descriptive.

Abstract

from arXiv · show

Verifier-guided text-to-image systems increasingly use test-time search to select, refine, or stop among multiple candidates, yet release thresholds are often calibrated on individual images. This creates a candidate-to-policy calibration mismatch: search changes both which prompts receive an output and which candidate is released, so candidate-level risk control need not imply control of released-output risk. We formalize this estimand shift through prompt reweighting and within-prompt selection, and introduce SHIP, Selection-aware Held-out calibration of Inference Policies. SHIP runs or replays the complete deployed policy on held-out prompts, evaluates the image it actually releases using an independent target judge, and selects the most permissive threshold whose risk upper bound satisfies a prescribed budget. For replayable policies with a prespecified threshold grid, simultaneous confidence control provides finite-sample validity. Experiments across fixed, sequential, and adaptive T2I inference procedures show that policy-level calibration recovers lower-risk operating points while exposing policy-dependent tradeoffs among risk, coverage, and compute. On GenEval2 with FLUX at N=16, a pooled-candidate threshold yields released risk 0.310, whereas SHIP reduces it to 0.162. Across 200 cached-stream splits, the fixed-grid certificate has no target crossing. Reliable inference-time scaling therefore requires calibrating the output distribution induced by the complete deployed policy.

1 Introduction

Verifier-guided T2I search changes both which prompts receive outputs and which images are released, so candidate-level calibration may not control deployed policy risk. SHIP calibrates the complete release policy on held-out prompts and evaluates its actual outputs under a target risk budget.

  • Test-time T2I systems search across generation paths, verification outcomes, prompt revisions, image edits, stopping decisions, and abstentions.
  • Candidate-level threshold calibration controls individual passing-candidate loss but does not generally control risk for images released by the complete search, selection, and stopping policy.
  • Search changes both the prompt distribution represented among released outputs and the conditional image distribution within each prompt.
  • SHIP runs or replays complete policies on held-out prompts, scores released images with an independent target judge, and selects the most permissive threshold meeting an upper-confidence-bound risk budget.
  • Experiments across fixed, sequential, and adaptive procedures recover feasible released-output operating points while exposing policy-dependent tradeoffs among risk, coverage, and inference cost.

2 Related Work

Related work spans T2I verification, verifier-guided inference-time search, and selective risk control. The paper distinguishes its contribution as a T2I-specific estimand analysis and evidence across multiple release-policy types.

  • T2I systems use fine-grained vision-language evaluators and learned reward models to assess prompt following and image quality.
  • Verifier-guided inference-time scaling searches seeds or trajectories, revises prompts, and edits images using verifier feedback.
  • Existing selective-risk methods calibrate decisions under exchangeability, trade coverage for conditional accuracy, or control risks for generated outputs and multimodal response sets.
  • The paper does not claim policy-level calibration or confidence bounds as new, instead contributing a T2I-specific estimand analysis and evidence across first-hit, best-of-N, and history-dependent policies.

3 Method

SHIP calibrates the complete text-to-image release policy on held-out prompts, aligning the calibration estimand with the image actually released. It uses independent target losses and simultaneous fixed-grid bounds to select the most permissive threshold meeting the risk budget.

  • Policy-level estimand: SHIP models each held-out prompt with a complete policy rollout, recording whether an image is released and the independent target loss of that released image.The policy observes selector risk s(Y,p), while the held-out target loss ℓ(Y,p) is used only for calibration.
  • Atomic prompt verification: SHIP uses atomic prompt requirements and geometric-mean Soft-TIFA risk to aggregate selector and target judge scores across existence, count, attributes, relations, and actions.Continuous next-token probabilities from separate selector and target VLMs avoid hard yes/no decoding brittleness.
  • Candidate-to-policy mismatch: Candidate-level calibration does not generally control released-output risk because selection changes both prompt weights and within-prompt conditional losses.These changes cover best-of-N selection, first-hit stopping, tournaments, and history-dependent refinement.
  • Calibration procedure: For each prespecified threshold, SHIP executes or replays the deployed policy on calibration prompts and evaluates the loss of the output it would release.Threshold-independent streams permit counterfactual replay, while history-dependent procedures require complete trajectories preserving native state transitions.
  • Finite-sample validity: SHIP returns the largest feasible threshold whose Hoeffding upper bound is at most α, assigning δ′ = δ/|T| for simultaneous validity across the fixed grid.If no calibration prompt is accepted at a threshold, its bound is set to +∞; if no threshold is feasible, SHIP returns abstention.
  • Scope and limitations: Formal fixed-grid certification applies to replayable streams, whereas threshold-dependent history search can require naive offline cost proportional to M|T| and remains an open calibration-cost problem.Adaptive-policy operating points are therefore treated as descriptive in the reported certification framework.

4 Experiments and Results

Experiments show that calibrating complete inference policies, rather than pooled candidates, controls released-output risk while exposing tradeoffs among coverage, speed, and verifier cost across search procedures and model settings.

  • Candidate-to-Policy Mismatch: SHIP reduces every descriptive policy-level UCB to at most 0.299, with test risk at most 0.208 across four FLUX policies.Pooled thresholds produce pointwise UCBs from 0.355 to 0.472; these pooled rows are not valid prompt-level certificates.
  • Risk, Coverage, and Compute Tradeoffs: 0.253-to-0.557 coverage increases as the requested risk budget rises from 0.20 to 0.50, while risk tracks the requested budget.Strict-budget abstention explains lower coverage, whereas Best-of-4 rejection covers more prompts but always uses four generations.
  • Risk, Coverage, and Compute Tradeoffs: At risk 0.166, Flash-BoN is fastest, RAISE has the highest coverage, and rewriting uses the fewest VLM calls; no policy dominates every cost measure.Sequential SHIP is intermediate in the displayed tradeoff profile.
  • Robustness and Generalization: ProbeSelect with SHIP reaches risk 0.196 and UCB 0.298 on SD3.5-Large using 1.80 rather than 5.00 full-denoise equivalents, while ADE-CoT reaches edit loss 0.171 and UCB 0.298.These results support calibration transfer, not lower native editing cost.
  • Robustness and Generalization: Across 200 cached-stream splits, the certificate shows no target crossing, with mean risk 0.148 and coverage 0.391.The splits reuse generations and omit rerun variability; SDXL coverage falls to 0.113 and SD3 transfer requires recalibration at risk 0.303.

5 Conclusion

SHIP calibrates the complete deployed inference policy by bounding released-image loss, while recognizing validity assumptions and boundaries. The paper concludes that reliability should be calibrated on the decision process that releases outputs.

  • SHIP calibrates a policy-selected output by replaying the complete policy on held-out prompts and bounding released-image loss.
  • Validity requires a prespecified threshold grid, exchangeability, bounded loss, and faithful policy replay.
  • Threshold-dependent adaptive search remains descriptive because exact calibration may require one rollout per prompt and threshold.
  • SHIP controls average rather than subgroup risk, with evaluator drift, hard-prompt abstention, and distribution shift remaining boundaries.
  • Reliability should be calibrated on the complete deployed decision process that releases an output, not upstream candidates.

Overview

The paper frames SHIP as a policy-level calibration approach for verifier-guided T2I systems and establishes finite-sample validity for replayable policies on prespecified threshold grids. Its scope distinguishes this application from generic concentration tools and related decision-level calibration methods.

  • The document specifies generation, verification, calibration, and uncertainty procedures for multiple diffusion models, evaluators, and GenEval2 atom questions.
  • One prompt and its complete policy trajectory form one exchangeable SHIP calibration unit, unlike pooled candidates that share prompts.
  • Test intervals use prompt bootstrap resampling of complete release or abstention outcomes, while bootstrap replicates do not count as independent generation runs.
  • For fixed prespecified thresholds, Hoeffding bounds and a union bound provide simultaneous finite-sample control for the selected threshold.
  • SHIP contributes candidate-to-policy estimand analysis and a T2I instantiation spanning fixed and adaptive release policies, rather than a new concentration inequality.

C Diagnosing the Candidate-to-Policy Mismatch

The mismatch diagnosis separates prompt reweighting from within-prompt selection and shows that calibration-unit choice changes the released-risk estimate. Replication and image-level examples support the distinction while preserving its empirical scope.

  • Mechanism decomposition: +0.108 risk arises from replacing candidate pass weights with policy acceptance weights in sequential release at N=16.The interval is 0.083 to 0.133.
  • Mechanism decomposition: −0.009 arises from replacing mean passing-candidate loss with first released-candidate loss, while best-of-N has a more negative within-prompt component.The sequential component interval is −0.035 to 0.017.
  • Calibration-unit sensitivity: 0.310 released risk is reproduced by the pooled sequential heuristic, compared with 0.196 for one fixed candidate per prompt and 0.162 for direct policy replay.
  • Calibration-unit sensitivity: The pooled heuristic reports 6400 correlated candidate observations from 16 candidates per prompt, so its empirical operating points do not inherit simultaneous certification.
  • Replication: Two nonoverlapping seed streams preserve the same ordering, but their smaller prompt subset yields wider intervals and does not replace the full experiment.
  • Release decisions: Real release examples show policies rejecting early relation failures and releasing later candidates that satisfy the relation, without adding quantitative evidence.

E Robustness across Models, Data, Judges, and Splits

Robustness analyses examine transfer across prompt distributions, target judges, cached-stream splits, and inference policies. They show preserved qualitative orderings and operating tradeoffs, while delimiting certification and distribution-shift claims.

  • Prompt-distribution transfer: Transfer at threshold τ=0.322 remains below the nominal budget in four cases, but coverage and risk shifts preclude interpreting this as a distribution-shift guarantee.
  • Alternative target judge: Replacing Qwen3 with InternVL3 changes absolute risks while preserving policy ordering across acceptance, best-of-4, and sequential policies.
  • Fixed-grid certification: Across 200 cached-stream splits at N=16, the fixed-grid row has no empirical target crossing and is the only simultaneously valid comparison row.
  • Fixed-grid certification: The pointwise row achieves higher mean coverage but is reported only as an empirical operating rule, not the fixed-grid certificate.
  • Image-level diagnostics: Figure 8 displays failures and faithful generations across counting, contextual, and structural requirements, while quantitative evidence uses the complete held-out prompt set.
  • Policy tradeoffs: Matched-risk FLUX policies occupy different points on a common frontier, while SHIP changes released outcomes without claiming native search contributions or internal compute savings.

G Fixed Search, Evaluators, and Failure Diagnostics

The evaluation compares policy operating points across backbones, budgets, evaluators, and failure types. Sequential stopping reduces cost, while released-output quality and coverage vary by generator, prompt set, and atomic failure category.

  • Operating points: Coverage rises with N, while compute grows more slowly than the maximum budget because sequential stopping ends after the first pass.The budget sweep uses the complete first-pass policy at risk budget α=0.30.
  • Operating points: Held-out risk is below budget for all three generators, while achievable coverage varies substantially across backbones.The cross-backbone operating points evaluate the sequential policy at α=0.30.
  • Failure diagnostics: Counting, spatial, and verb atoms remain bottlenecks, whereas object existence is nearly solved.Accepted-output quality is broken down by atom type under calibrated rejection.
  • Human evaluation: Human auditing reused shared outputs and adjudicated disagreements after two annotators labeled each atom as yes, no, or uncertain.The audit evaluated both N=16 sequential policies on the same 400 test prompts, whose released sets contained 267 distinct images and shared 137 outputs.
  • Evaluator robustness: Each alternative target was calibrated separately, with selector-target agreement measured by Spearman correlation over 12,800 candidates.The robustness summary also covers held-out coverage and risk across three backbones, fixed-threshold transfer across four FLUX prompt sets, and target-crossing rates over 200 cached-stream splits.
  • Evaluator robustness: Qualitative streams show the held-out target agreeing with every displayed Qwen2.5 selector decision, while sequential stopping releases the first passing candidate.Examples include candle counting, an ocean-versus-pool shark mismatch, and cup-hole prompts.

I Evaluator Ablations and Modular Cascades

The section tests evaluator choices, adaptive-baseline fidelity, and cheaper selector cascades. Fine-grained selectors are necessary for feasible release sets, while sending two candidates to the expensive gate improves coverage at reduced QA cost.

  • Evaluator ablations: Only fine-grained atom-wise selectors yield a nonempty release set under the empirical feasibility criterion.Coverage and risk are measured for best-of-4 policy-level rejection at α=0.30.
  • Modular cascades: Sending two candidates to Soft-TIFA-GM improves coverage while using half of the QA calls required by scoring all four candidates.The cascade first ranks four candidates with a cheaper holistic score and sends only the top one or two to the fine-grained gate.
  • Adaptive baselines: Official adaptive-baseline metrics are evaluated on each method’s native benchmark before applying the common SHIP protocol.Small residual gaps arise from stated implementation or hardware differences.
  • Operating frontier: The FLUX N=4 frontier reports pointwise risk, coverage, and cost operating points, with the requested risk budget changing by row.The rows therefore do not share a single horizontal risk constraint.

J Controlled Study and Deployment Boundaries

Controlled mixtures demonstrate that candidate calibration can fail after best-of-N selection, motivating policy-level calibration. Deployment evidence further shows subgroup and transfer boundaries, while recalibration and sensitivity analyses identify practical operating conditions.

  • Controlled study: The two-component controlled mixture calibrates candidate risk at 0.30 before deploying best-of-N or sequential selection.Hard prompts occur with probability 0.5 and candidate failure probability 0.9; easy prompts have failure probability 0.05.
  • Controlled study: Released risk reaches 0.64 at ρ=0.7 with N=256 while candidate risk remains 0.30.The limiting construction assigns loss 1 to hard-prompt candidates and loss 0 to easy-prompt candidates.
  • Controlled study: As N →∞, best-of-N released risk converges to the hard-prompt share ρ among accepted prompts while candidate-level risk remains α.The resulting gap can approach 1−α, showing that candidate calibration cannot generally be converted into a policy guarantee.
  • Deployment boundaries: A global threshold controls average released risk, not subgroup risk; the 9 to 10 atom group has risk 0.398 at 0.05 coverage from five releases.Group-SHIP calibrates groups separately and abstains where no threshold is feasible, remaining a diagnostic rather than a service guarantee.
  • Deployment boundaries: On SD3, transferred risk reaches 0.303 > α, while a small deployment-side calibration set returns average held-out risk below α as ncal grows.The result is reported for GenAI-Bench transfer and recalibration over 200 splits.
  • Sensitivity and certification: Coverage is 0.36 at |T|=25 and 0.39 at full resolution, while calibration sets of 200 to 400 prompts yield coverage from 0.36 to 0.39.Cached-stream crossing rates remain below δ under the tested fixed-grid certification, while sharper concentration remains future work.
Loading 2608.21748v1…