Calibrate What You SHIP: Post-Selection Risk Control for Verifier-Guided Text-to-Image Generation

• Test-time T2I systems search across generation paths, verification outcomes, prompt revisions, image edits, stopping decisions, and abstentions.Source: §1 · p. 1
• Candidate-level threshold calibration controls individual passing-candidate loss but does not generally control risk for images released by the complete search, selection, and stopping policy.Source: §1 · pp. 1–2
• Search changes both the prompt distribution represented among released outputs and the conditional image distribution within each prompt.Source: §1 · p. 2
• SHIP runs or replays complete policies on held-out prompts, scores released images with an independent target judge, and selects the most permissive threshold meeting an upper-confidence-bound risk budget.Source: §1 · p. 2
• Experiments across fixed, sequential, and adaptive procedures recover feasible released-output operating points while exposing policy-dependent tradeoffs among risk, coverage, and inference cost.Source: §1 · p. 2

• T2I systems use fine-grained vision-language evaluators and learned reward models to assess prompt following and image quality.Source: §2 · p. 2
• Verifier-guided inference-time scaling searches seeds or trajectories, revises prompts, and edits images using verifier feedback.Source: §2 · p. 2
• Existing selective-risk methods calibrate decisions under exchangeability, trade coverage for conditional accuracy, or control risks for generated outputs and multimodal response sets.Source: §2 · p. 2
• The paper does not claim policy-level calibration or confidence bounds as new, instead contributing a T2I-specific estimand analysis and evidence across first-hit, best-of-N, and history-dependent policies.Source: §2 · p. 2

Policy-level estimand

• SHIP models each held-out prompt with a complete policy rollout, recording whether an image is released and the independent target loss of that released image.The policy observes selector risk s(Y,p), while the held-out target loss ℓ(Y,p) is used only for calibration.Source: §3.1 · p. 3, §3.4 · p. 4

Atomic prompt verification

• SHIP uses atomic prompt requirements and geometric-mean Soft-TIFA risk to aggregate selector and target judge scores across existence, count, attributes, relations, and actions.Continuous next-token probabilities from separate selector and target VLMs avoid hard yes/no decoding brittleness.Source: §3.2 · p. 3

Candidate-to-policy mismatch

• Candidate-level calibration does not generally control released-output risk because selection changes both prompt weights and within-prompt conditional losses.These changes cover best-of-N selection, first-hit stopping, tournaments, and history-dependent refinement.Source: §3.3 · p. 3

Calibration procedure

• For each prespecified threshold, SHIP executes or replays the deployed policy on calibration prompts and evaluates the loss of the output it would release.Threshold-independent streams permit counterfactual replay, while history-dependent procedures require complete trajectories preserving native state transitions.Source: §3.1 · p. 3, §3.4 · pp. 4–5

Finite-sample validity

• SHIP returns the largest feasible threshold whose Hoeffding upper bound is at most α, assigning δ′ = δ/|T| for simultaneous validity across the fixed grid.If no calibration prompt is accepted at a threshold, its bound is set to +∞; if no threshold is feasible, SHIP returns abstention.Source: §3.4 · p. 4

Scope and limitations

• Formal fixed-grid certification applies to replayable streams, whereas threshold-dependent history search can require naive offline cost proportional to M|T| and remains an open calibration-cost problem.Adaptive-policy operating points are therefore treated as descriptive in the reported certification framework.Source: §3.4 · p. 4

Candidate-to-Policy Mismatch

• SHIP reduces every descriptive policy-level UCB to at most 0.299, with test risk at most 0.208 across four FLUX policies.Pooled thresholds produce pointwise UCBs from 0.355 to 0.472; these pooled rows are not valid prompt-level certificates.Source: §4.2 · p. 6

Risk, Coverage, and Compute Tradeoffs

• 0.253-to-0.557 coverage increases as the requested risk budget rises from 0.20 to 0.50, while risk tracks the requested budget.Strict-budget abstention explains lower coverage, whereas Best-of-4 rejection covers more prompts but always uses four generations.Source: §4.3 · p. 7
• At risk 0.166, Flash-BoN is fastest, RAISE has the highest coverage, and rewriting uses the fewest VLM calls; no policy dominates every cost measure.Sequential SHIP is intermediate in the displayed tradeoff profile.Source: Figure 5 · p. 6, §4.3 · p. 7

Robustness and Generalization

• ProbeSelect with SHIP reaches risk 0.196 and UCB 0.298 on SD3.5-Large using 1.80 rather than 5.00 full-denoise equivalents, while ADE-CoT reaches edit loss 0.171 and UCB 0.298.These results support calibration transfer, not lower native editing cost.Source: §4.4 · p. 7
• Across 200 cached-stream splits, the certificate shows no target crossing, with mean risk 0.148 and coverage 0.391.The splits reuse generations and omit rerun variability; SDXL coverage falls to 0.113 and SD3 transfer requires recalibration at risk 0.303.Source: §4.4 · p. 7

• SHIP calibrates a policy-selected output by replaying the complete policy on held-out prompts and bounding released-image loss.Source: §5 · p. 7
• Validity requires a prespecified threshold grid, exchangeability, bounded loss, and faithful policy replay.Source: §5 · p. 7
• Threshold-dependent adaptive search remains descriptive because exact calibration may require one rollout per prompt and threshold.Source: §5 · p. 7
• SHIP controls average rather than subgroup risk, with evaluator drift, hard-prompt abstention, and distribution shift remaining boundaries.Source: §5 · p. 7
• Reliability should be calibrated on the complete deployed decision process that releases an output, not upstream candidates.Source: §5 · p. 7

• The document specifies generation, verification, calibration, and uncertainty procedures for multiple diffusion models, evaluators, and GenEval2 atom questions.Source: A Evaluation Protocol and… · p. 10, Overview · p. 10
• One prompt and its complete policy trajectory form one exchangeable SHIP calibration unit, unlike pooled candidates that share prompts.Source: A Evaluation Protocol and… · p. 10
• Test intervals use prompt bootstrap resampling of complete release or abstention outcomes, while bootstrap replicates do not count as independent generation runs.Source: A Evaluation Protocol and… · p. 10
• For fixed prespecified thresholds, Hoeffding bounds and a union bound provide simultaneous finite-sample control for the selected threshold.Source: B Formal Scope and Finite-Sample… · p. 10
• SHIP contributes candidate-to-policy estimand analysis and a T2I instantiation spanning fixed and adaptive release policies, rather than a new concentration inequality.Source: B Formal Scope and Finite-Sample… · p. 10, Table 3 · p. 11

Mechanism decomposition

• +0.108 risk arises from replacing candidate pass weights with policy acceptance weights in sequential release at N=16.The interval is 0.083 to 0.133.Source: C Diagnosing the Candidate-to-Policy… · p. 11
• −0.009 arises from replacing mean passing-candidate loss with first released-candidate loss, while best-of-N has a more negative within-prompt component.The sequential component interval is −0.035 to 0.017.Source: C Diagnosing the Candidate-to-Policy… · p. 11

Calibration-unit sensitivity

• 0.310 released risk is reproduced by the pooled sequential heuristic, compared with 0.196 for one fixed candidate per prompt and 0.162 for direct policy replay.Source: C Diagnosing the Candidate-to-Policy… · p. 11
• The pooled heuristic reports 6400 correlated candidate observations from 16 candidates per prompt, so its empirical operating points do not inherit simultaneous certification.Source: C Diagnosing the Candidate-to-Policy… · p. 11, Table 5 · p. 12

Replication

• Two nonoverlapping seed streams preserve the same ordering, but their smaller prompt subset yields wider intervals and does not replace the full experiment.Source: C Diagnosing the Candidate-to-Policy… · p. 11, Table 6 · p. 12

Release decisions

• Real release examples show policies rejecting early relation failures and releasing later candidates that satisfy the relation, without adding quantitative evidence.Source: C Diagnosing the Candidate-to-Policy… · p. 11, Figure 6 · p. 11

Prompt-distribution transfer

• Transfer at threshold τ=0.322 remains below the nominal budget in four cases, but coverage and risk shifts preclude interpreting this as a distribution-shift guarantee.Source: E Robustness across Models, Data,… · p. 12

Alternative target judge

• Replacing Qwen3 with InternVL3 changes absolute risks while preserving policy ordering across acceptance, best-of-4, and sequential policies.Source: E Robustness across Models, Data,… · p. 12

Fixed-grid certification

• Across 200 cached-stream splits at N=16, the fixed-grid row has no empirical target crossing and is the only simultaneously valid comparison row.Source: E Robustness across Models, Data,… · p. 12
• The pointwise row achieves higher mean coverage but is reported only as an empirical operating rule, not the fixed-grid certificate.Source: E Robustness across Models, Data,… · p. 12

Image-level diagnostics

• Figure 8 displays failures and faithful generations across counting, contextual, and structural requirements, while quantitative evidence uses the complete held-out prompt set.Source: F Qualitative Evidence from Real… · p. 12

Policy tradeoffs

• Matched-risk FLUX policies occupy different points on a common frontier, while SHIP changes released outcomes without claiming native search contributions or internal compute savings.Source: Table 7 · p. 13

Operating points

• Coverage rises with N, while compute grows more slowly than the maximum budget because sequential stopping ends after the first pass.The budget sweep uses the complete first-pass policy at risk budget α=0.30.Source: G Fixed Search, Evaluators, and… · p. 13, Table 15 · p. 13
• Held-out risk is below budget for all three generators, while achievable coverage varies substantially across backbones.The cross-backbone operating points evaluate the sequential policy at α=0.30.Source: G Fixed Search, Evaluators, and… · p. 13, Table 16 · p. 13

Failure diagnostics

• Counting, spatial, and verb atoms remain bottlenecks, whereas object existence is nearly solved.Accepted-output quality is broken down by atom type under calibrated rejection.Source: G Fixed Search, Evaluators, and… · p. 13, Table 17 · p. 13

Human evaluation

• Human auditing reused shared outputs and adjudicated disagreements after two annotators labeled each atom as yes, no, or uncertain.The audit evaluated both N=16 sequential policies on the same 400 test prompts, whose released sets contained 267 distinct images and shared 137 outputs.Source: H Human Released-Output Audit · p. 14, Table 18 · p. 14, Table 19 · p. 15

Evaluator robustness

• Each alternative target was calibrated separately, with selector-target agreement measured by Spearman correlation over 12,800 candidates.The robustness summary also covers held-out coverage and risk across three backbones, fixed-threshold transfer across four FLUX prompt sets, and target-crossing rates over 200 cached-stream splits.Source: Table 11 · p. 14, Figure 7 · p. 14, Table 10 · p. 14
• Qualitative streams show the held-out target agreeing with every displayed Qwen2.5 selector decision, while sequential stopping releases the first passing candidate.Examples include candle counting, an ocean-versus-pool shark mismatch, and cup-hole prompts.Source: Figure 8 · p. 15

Evaluator ablations

• Only fine-grained atom-wise selectors yield a nonempty release set under the empirical feasibility criterion.Coverage and risk are measured for best-of-4 policy-level rejection at α=0.30.Source: Table 20 · p. 15

Modular cascades

• Sending two candidates to Soft-TIFA-GM improves coverage while using half of the QA calls required by scoring all four candidates.The cascade first ranks four candidates with a cheaper holistic score and sends only the top one or two to the fine-grained gate.Source: I Evaluator Ablations and Modular… · pp. 15–16, Table 21 · p. 16

Adaptive baselines

• Official adaptive-baseline metrics are evaluated on each method’s native benchmark before applying the common SHIP protocol.Small residual gaps arise from stated implementation or hardware differences.Source: Table 13 · p. 16

Operating frontier

• The FLUX N=4 frontier reports pointwise risk, coverage, and cost operating points, with the requested risk budget changing by row.The rows therefore do not share a single horizontal risk constraint.Source: Table 14 · p. 16

Controlled study

• The two-component controlled mixture calibrates candidate risk at 0.30 before deploying best-of-N or sequential selection.Hard prompts occur with probability 0.5 and candidate failure probability 0.9; easy prompts have failure probability 0.05.Source: J Controlled Study and Deployment… · p. 16
• Released risk reaches 0.64 at ρ=0.7 with N=256 while candidate risk remains 0.30.The limiting construction assigns loss 1 to hard-prompt candidates and loss 0 to easy-prompt candidates.Source: J Controlled Study and Deployment… · p. 16, Figure 9 · p. 17
• As N →∞, best-of-N released risk converges to the hard-prompt share ρ among accepted prompts while candidate-level risk remains α.The resulting gap can approach 1−α, showing that candidate calibration cannot generally be converted into a policy guarantee.Source: J Controlled Study and Deployment… · pp. 16–17

Deployment boundaries

• A global threshold controls average released risk, not subgroup risk; the 9 to 10 atom group has risk 0.398 at 0.05 coverage from five releases.Group-SHIP calibrates groups separately and abstains where no threshold is feasible, remaining a diagnostic rather than a service guarantee.Source: J Controlled Study and Deployment… · p. 17, Table 22 · p. 17
• On SD3, transferred risk reaches 0.303 > α, while a small deployment-side calibration set returns average held-out risk below α as ncal grows.The result is reported for GenAI-Bench transfer and recalibration over 200 splits.Source: J Controlled Study and Deployment… · p. 17, Table 23 · p. 17

Sensitivity and certification

• Coverage is 0.36 at |T|=25 and 0.39 at full resolution, while calibration sets of 200 to 400 prompts yield coverage from 0.36 to 0.39.Cached-stream crossing rates remain below δ under the tested fixed-grid certification, while sharper concentration remains future work.Source: J Controlled Study and Deployment… · p. 17