PaperlayerView exact revision on arXiv

Source-linked AI summary

Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback

Huaisong Zhang, Hao Yu, Yuxuan Zhang, Jiahe Wang, Xinrui Chen, Haoxiang Cao, Feng Lu, Wendong Zhang, Changqian Yu, Chun Yuan

arXiv:2606.06113v2cs.CV

TL;DR

T2I models need instance-level feedback that localizes defects and explains their type, reason, and importance beyond scalar scores or heatmaps. This paper introduces SDG and its diagnosis-to-alignment pipeline, showing improved defect grounding, T2I alignment, and actionable image refinement.

  • Problem

    Scalar scores and heatmap-based feedback do not provide instance-level localization while binding each defect’s type, reason, and importance.

  • Method

    SDG predicts variable-length sets of box-grounded (location, type, reason, importance) tuples, supported by SDG-30K, SDG-Eval, and BoxFlow-GRPO.

  • Results

    SDG outperforms leading proprietary VLMs on structured defect grounding, while SDG-guided rewards improve T2I alignment and support actionable defect-guided refinement.

  • Takeaways & Limitations

    SDG provides a practical instance-level interface for evaluating and improving modern generative models through structured defect feedback.

  • Takeaways & Limitations

    SDG covers only artifacts and misalignments, while importance scores may differ from human preferences and BoxFlow-GRPO’s spatial projection may be unreliable.

Abstract

from arXiv · show

Despite generating increasingly photorealistic images, text-to-image (T2I) models still exhibit localized, subtle, and structurally complex failures. Diagnosing these failures requires instance-level feedback that answers where a defect occurs, what type it is, why it is defective, and its importance to overall image quality. While recent dense-feedback methods move beyond scalar supervision, their heatmap-centric representations still formulate diagnosis as pixel-field regression, making it difficult to localize variable-cardinality defects and bind semantic reasons to individual failures. To address this representation bottleneck, we propose Structured Defect Grounding (SDG), which casts T2I diagnosis as structured set prediction by modeling each defect as a (location, type, reason, importance) tuple. To make this formulation trainable and measurable, we introduce SDG-30K, a 30K-image dataset with box-grounded annotations across four modern T2I generators, together with a dedicated evaluation protocol, SDG-Eval. Building on this structured representation, we further present a diagnosis-to-alignment framework in which a Vision-Language Model (VLM) serves as the SDG detector, and BoxFlow-GRPO converts predicted defect sets into box-derived, importance-weighted spatial rewards for diffusion model alignment. Extensive experiments show that our SDG detector outperforms leading proprietary VLMs on structured defect grounding, while SDG-guided rewards consistently improve T2I alignment and support localized image refinement. These results establish SDG as a unified, instance-level interface for diagnosing, evaluating, and enhancing modern generative models.

1 Introduction

The paper introduces Structured Defect Grounding (SDG) to replace heatmap-based pixel-field regression with variable-cardinality, instance-level defect-set prediction, enabling structured diagnosis and spatially targeted T2I alignment.

  • Motivation: T2I failures are localized, subtle, and structurally heterogeneous, while scalar preference scores collapse them into one global value.Examples include malformed text, implausible geometry, and semantic mismatches.
  • Motivation: Heatmap-based diagnosis remains limited because point-derived maps depend on chosen radii and cannot bind defect types, reasons, or importance to individual failures.This reflects a representation bottleneck in spatial, semantic, and architectural aspects of diagnosis.
  • Structured Defect Grounding: SDG formulates diagnosis as variable-cardinality set prediction over structured (location, type, reason, importance) tuples.It unifies artifacts and prompt-conditioned misalignments within one instance space.
  • Diagnosis-to-Alignment: BoxFlow-GRPO converts VLM-predicted defect sets into box-derived, importance-weighted spatial rewards for diffusion-model alignment.Boxes define spatial support, defect types and reasons provide semantic diagnoses, and importance scores calibrate reward strength.
  • Resources: 30,096 images comprise SDG-30K, a box-grounded defect dataset spanning four modern T2I generators, alongside SDG-Eval for image- and defect-level evaluation.The dataset and protocol make structured defect grounding trainable and measurable.
  • Results: The SDG detector outperforms leading proprietary VLMs on structured defect grounding, while SDG-guided rewards improve T2I alignment and support faithful, actionable image refinement.These results position SDG as an interface for diagnosing, evaluating, and enhancing generative models.

2 Related Work

Prior work advances dense T2I feedback, structured spatial reasoning in VLMs, and reinforcement-learning-based diffusion alignment. These efforts motivate structured, spatially grounded diagnosis and refinement.

  • From scalar evaluation to dense T2I feedback: T2I evaluation has progressed from scalar scores to heatmap-based feedback, hierarchical explanations, and fine-grained artifact taxonomies.RichHF introduces heatmap-based dense feedback; ImageDoctor predicts heatmaps with a VLM-plus-decoder; HEIE and MagicMirror add hierarchical explanations and artifact taxonomies.
  • Structured spatial reasoning in VLMs: Modern VLMs increasingly generate explicit spatial structures, including boxes, points, and point-sequence segmentation in language space.Qwen2.5-VL supports box- and point-based grounding, Qwen3-VL strengthens image-grounded reasoning, and SimpleSeg reformulates segmentation as point-sequence generation.
  • RL for diffusion alignment and image refinement: Reinforcement learning aligns diffusion models with human preferences through policy gradients, direct preference optimization, and group relative policy optimization.DDPO frames denoising as a multi-step MDP, Diffusion-DPO adapts direct preference optimization, and Flow-GRPO applies group relative policy optimization to flow-matching models.

3 SDG-30K: Dataset and Evaluation

SDG-30K provides prompt-disjoint, box-grounded annotations for structured defect instances across four T2I generators, while SDG-Eval scores defect detection at image and instance levels. Its statistics show that localized failures are common, with artifact defects more frequent than misalignments and 25.1% of images defect-free.

  • Structured annotations: Each defect is represented as a variable-cardinality tuple containing a quantized box, type, free-form reason, and 1–100 importance score.Types are artifact and misalignment; importance reflects perceptual impact on overall image quality.
  • Dataset construction: SDG-30K is the first related dataset to jointly annotate artifact and misalignment defects with natural-language reasons in one instance-level space.The dataset comparison identifies unified coverage of both defect types and natural-language reasons as its distinguishing scope.
  • Evaluation protocol: SDG-Eval measures defect-type presence and clean-image accuracy at the image level, then matches same-type predictions to ground-truth instances at the defect level.Image-level metrics include DetTypeF1 and ClnAcc; defect-level matching uses class-aware Hungarian matching.
  • Dataset construction: 30,096 1024×1024 images were generated from Pick-a-Pic prompts using FLUX.2-dev, Z-Image-Turbo, LongCat-Image, and SANA-1.5-1.6B.Each generator contributes approximately 7.8K images.
  • Dataset statistics: 25.1% of images are defect-free, 46.3% are artifact-only, 5.4% are misalignment-only, and 23.2% contain both types.Artifact instances are more frequent than misalignment instances across all four generators, with SANA-1.5 reaching 3.22 artifact instances per image.

4 Structured Defect Grounding Framework

The framework formulates diagnosis as structured vision-language generation, emitting reasoning traces and defect tuples, then uses BoxFlow-GRPO to convert detected defects into spatially varying diffusion rewards. The detector is trained with supervised formatting and GRPO refinement, while BoxFlow-GRPO applies importance-weighted, box-derived penalties and spatially normalized advantages.

  • SDG Detector: SDG generates a reasoning trace followed by a structured defect set whose instances specify location, type, reason, and importance.This representation treats defect diagnosis as a structured vision-language generation task conditioned on an image and its prompt.
  • SDG Detector: The detector uses SFT for structured-format adherence, followed by GRPO that improves localization, description consistency, and importance estimation under format-validity gating.The validity check requires well-formed reasoning and answer delimiters, parseable JSON, and geometrically valid bounding boxes.
  • BoxFlow-GRPO: Unlike scalar-advantage heatmap feedback, BoxFlow-GRPO computes per-location advantages and likelihood ratios, preserving spatial variation throughout diffusion alignment.The design is motivated by the limitation that DenseFlow-GRPO assigns an image-level scalar reward and scalar advantage, with its heatmap entering only as a multiplier.
  • BoxFlow-GRPO: BoxFlow-GRPO constructs latent-space reward maps by subtracting type-specific, importance-weighted penalties from a base scalar reward at locations covered by predicted defect boxes.The method sets cart=0.5 and cmis=0.05, so high-importance defects receive stronger spatial penalties while minor defects receive lighter corrections.

5 Experiments

Experiments evaluate SDG through grounding quality, cross-dataset generalization, diffusion alignment, and defect-guided refinement. Across these settings, structured defect feedback improves localization, preserves realism during alignment, and enables targeted editing.

  • Defect grounding: GRPO achieves the strongest artifact/misalignment BoxF1@0.5 (0.263/0.387) and importance accuracy (0.887/0.893), while SFT attains the highest clean-image accuracy (0.697/0.799).Both SDG variants substantially reduce localization error compared with zero-shot VLM baselines.
  • Cross-dataset generalization: SDG achieves misalignment F1 0.655 versus 0.250/0.007 on RichHF-18K without fine-tuning, outperforming heatmap-based generalization despite ImageDoctor’s artifact F1 0.952 at a loose threshold.ImageDoctor’s misalignment recall is only 0.143/0.004, whereas SDG captures prompt-conditioned misalignments more effectively.
  • Ablations: GRPO improves localization over SFT, raising artifact/misalignment BoxF1@0.5 from 0.255/0.376 to 0.263/0.387.Removing the reasoning trace lowers misalignment BoxF1@0.5 from 0.387 to 0.352, while unfreezing the vision encoder substantially degrades localization.
  • Diffusion alignment: BoxFlow-GRPO achieves the best average relative change (+2.4%) and highest P(real) (0.228, above Base), while maintaining competitive preference and quality metrics.Baseline RL variants improve reward-model scores by drifting toward illustration- or anime-like outputs, whereas BoxFlow-GRPO reduces artifacts and prompt misalignment while preserving photographic realism.
  • Defect-guided refinement: SDG refinement yields higher Good than Bad rates against ImageDoctor (11.00% vs. 3.90%) and Fixed (10.31% vs. 2.75%) across 873 valid samples.The structured box-and-text feedback supports targeted semantic correction, including distinguishing a modern Ford Fiesta from the prompted Mark 2.

6 Conclusion … B Prompt Templates

The paper defines SDG as instance-level, variable-cardinality defect-set prediction and provides SDG-30K, SDG-Eval, and a BoxFlow-GRPO alignment pipeline. Its appendices specify annotation rules, defect taxonomy, importance scoring, and standardized prompts for training, inference, and distillation.

  • 6 Conclusion: SDG models each T2I defect as a (location, type, reason, importance) tuple and casts diagnosis as variable-cardinality structured set prediction.
  • 6 Conclusion: SDG-30K contains 30K box-grounded images from four modern T2I generators, while SDG-Eval measures structured defect-set performance.
  • A.1 Interface and Workflow: The annotation workflow uses synchronized image and metadata panels, followed by independent annotation and global-scan passes without machine-generated candidate boxes.
  • A.2 Defect-Type Taxonomy: The taxonomy separates prompt-content misalignment from prompt-independent visual artifacts, excluding intentional artistic stylization from defect labels.
  • A.3 Boxing and Description Principles: Annotators tightly box each concrete defect, split independent issues into separate boxes, and provide concise reason statements grounded in specific visual evidence.
  • A.4 Clean Images; A.6 Placeholder Box Handling: Clean images receive placeholder boxes and explicit no-problem tags during annotation, then become empty defect sets for consistent training and evaluation.
  • A.5 Prompt Interpretation Protocol: Annotators may consult search engines to clarify unfamiliar prompt concepts, reducing interpretation noise without obtaining defect labels externally.
  • A.7 Importance Scoring Rubric: Each defect receives an integer importance score from 1 to 100 based on visual prominence, semantic impact, area coverage, and location, used for supervision and rewards.The five tiers are Critical (90–100), Major (70–89), Moderate (40–69), Minor (15–39), and Negligible (1–14).

C Evaluation Metric Details

SDG-Eval represents ground-truth and predicted defects as typed tuples and evaluates them at both image and defect levels. It combines type-specific image-level F1, class-aware localization matching at multiple IoU thresholds, and matched-pair semantic and importance measures.

  • Metric definitions: Ground-truth and predicted defect sets are represented as tuples containing location, type, reason, and importance for each image.Defect types are artifact and misalignment.
  • Image-level metrics: DetTypeF1 measures type-specific image-level F1 using binary indicators for whether each defect type appears in an image.The indicators compare ground-truth and predicted presence for each type.
  • Defect-level metrics: Defect-level localization metrics use class-aware Hungarian matching constrained to equal defect types.Matched pairs are grouped by type and retained when their IoU reaches τ ∈ {0.1, 0.5}.
  • Defect-level metrics: Localization precision, recall, and F1 are computed over valid matched defect pairs at each IoU threshold.The matching-based evaluation is performed separately for each defect type.
  • Defect-level metrics: DescCos evaluates reason similarity with Qwen3-Embedding-0.6B, while ImpAcc measures normalized absolute-error accuracy for importance.Both metrics are computed over valid matched pairs.

D Experimental Details … D.4 GRPO Hyperparameters

The appendix details SDG-guided GRPO training, including format-gated composite rewards, structured matching and scoring, the clipped objective, and SFT/GRPO hyperparameters.

  • D Experimental Details: The appendix provides complete reward computation and training details for the GRPO stage described in Section 4.1.
  • D.1 Reward Formulation: The composite reward is 0.6 Rdiou + 0.25 Rdesc + 0.15 Rimp when format is valid, and −1 otherwise.Validity requires <think> tags, parseable JSON in <answer>, and properly ordered xyxy coordinates.
  • D.1 Reward Formulation: Predicted and ground-truth boxes receive Hungarian one-to-one matching using DIoU cost, with normalized, clipped spatial scores and explicit edge-case penalties.Correct rejection gives Rdiou = 0.3, misses −0.8, false alarms −0.3, and unmatched boxes −0.5; scores are normalized by max(|G|, |P|, 1) and clipped to [−1, 1].
  • D.1 Reward Formulation: Description rewards use Qwen3-Embedding-0.6B cosine similarity, while importance rewards decrease linearly with absolute error and reach zero beyond 50 points.Description similarities are transformed to [0, 1], summed over matches, and divided by max(|G|, |P|, 1); unmatched boxes contribute zero.
  • D.2 GRPO Objective: The GRPO objective combines clipped importance-ratio policy optimization with KL regularization using β=0.01.
  • D.3 SFT Hyperparameters: SFT uses 16 GPUs, ZeRO-2, bfloat16, learning rate 3 × 10−5, cosine scheduling, 5% warmup, one epoch, and effective batch size 16.Images are capped at 1024 pixels on the longer side; 3× pre-baked jitter exposes each example to three passes within the single epoch.
  • D.3 SFT Hyperparameters: Coordinate jitter independently samples offsets from [−10, +10] in normalized [0, 1000] space, clamps boxes, and resamples offsets during SFT loading.
  • D.4 GRPO Hyperparameters: GRPO uses 16 GPUs, learning rate 5 × 10−6, β=0.01, S=8 sampled responses, temperature 1.0, top-p 0.85, and two epochs with per-device batch size 4.Rollouts use colocated vLLM, with maximum completion length 4,096 tokens.

D.5 Compute Resources … E Extended Experimental Results

The paper reports GPU requirements for detector training and diffusion alignment, while defect-guided refinement relies on external API throughput. It also documents the assets, models, and language-model usage underlying dataset construction, experiments, evaluation, and editing.

  • D.5 Compute Resources: A single SDG detector SFT run takes approximately 2 hours on 16 GPUs.Detector GRPO takes approximately 36 hours on 16 GPUs.
  • D.5 Compute Resources: A single detector GRPO run takes approximately 36 hours on 16 GPUs.This is substantially longer than the reported SFT run.
  • D.5 Compute Resources: One BoxFlow-GRPO diffusion-alignment run takes approximately 24 hours on 16 GPUs.The reported runtime is for a single run on 16 GPUs.
  • D.5 Compute Resources: Defect-guided image refinement uses GPT-Image-1.5 through an external API, with wall-clock time driven primarily by API throughput and request concurrency.The refinement experiments do not primarily depend on local GPU compute.
  • D.7 Existing Assets and Licenses: The study uses existing assets only for research dataset construction, training, evaluation, or API-based editing, citing their original sources.Pick-a-Pic prompts are used under the MIT License.
  • D.7 Existing Assets and Licenses: Qwen3-VL-4B-Instruct and Qwen3-Embedding-0.6B are released under Apache-2.0.The T2I generators used for SDG-30K follow their respective public licenses or terms.
  • D.8 LLM Usage Declaration: Gemini 3 Pro supports description expansion, reasoning-trace distillation, and importance scoring during data preparation.Qwen3-VL-4B-Instruct is fine-tuned as the SDG detector, while Qwen3-Embedding-0.6B supports description-similarity evaluation and reward computation.
  • D.8 LLM Usage Declaration: General-purpose LLMs assist manuscript writing through language polishing and wording refinement, with technical claims, results, and final text reviewed and edited by the authors.GPT-Image-1.5 is used for defect-guided image refinement.

E.1 SDG Detector Ablation … E.4 Extended BoxFlow-GRPO Qualitative Comparison

The appendix reports SDG ablations, extended qualitative comparisons, a complete structured-output example, and additional evidence that BoxFlow-GRPO improves prompt fidelity while preserving realism.

  • E.1 SDG Detector Ablation: Table 7 reports the complete SDG detector ablation results on the SDG-30K test set.The ablation groups cover training stage, chain-of-thought steps, output component, and architecture or augmentation.
  • E.2 Extended Comparison with ImageDoctor on SDG-30K: Figure 7 adds six SDG-30K cases spanning artifact defects, misalignments, and absent target defect types.The comparison separates ImageDoctor’s artifact and misalignment heatmaps and overlays SDG’s predicted boxes with per-instance labels.
  • E.2 Extended Comparison with ImageDoctor on SDG-30K: SDG grounds prompt-conditioned misalignments that ImageDoctor’s misalignment representation misses, including incorrect depictions of “Nucleosome” and “Dumbo.”The cited examples depict “Nucleosome” as a double helix and “Dumbo” without clown makeup.
  • E.3 Full SDG Output Example: The full Figure 1 example shows SDG receiving an image and prompt, then producing a reasoning trace followed by a JSON defect set.The output format is explicitly organized as <think> followed by <answer>.
  • E.3 Full SDG Output Example: In the example, SDG identifies the requested upside-down star jump as a prominent pose misalignment because the ballerina is standing upright on the skateboard.The misalignment receives importance 95 and is localized with box_2d [223, 48, 657, 840].
  • E.4 Extended BoxFlow-GRPO Qualitative Comparison: Across additional DrawBench prompts, BoxFlow-GRPO with UR2+SDG better follows fine-grained attributes while preserving photographic realism than FlowGRPO and DenseFlow-GRPO.Examples include correct color binding for “a red book and a yellow vase” and coherent composition for “Darth Vader playing with raccoon,” while avoiding illustration or anime drift.

E.5 Extended Defect-Guided Refinement Results · F Limitations and Broader Impact

Extended GPT-Image-1.5 cases show that SDG feedback enables localized, instance-level image corrections across varied defects. The comparison covers caption-only editing, ImageDoctor heatmap/text feedback, and SDG box-structured feedback.

  • E.5 Extended Defect-Guided Refinement Results: SDG feedback provides localized, instance-level guidance for targeted image corrections.The extended cases use GPT-Image-1.5 editing.
  • E.5 Extended Defect-Guided Refinement Results: SDG guidance removes an extra lion cub when the prompt requests a single cub.This demonstrates correction of an object-count defect.
  • E.5 Extended Defect-Guided Refinement Results: SDG identifies an incorrect webpage title and replaces it with the correct title.This demonstrates targeted correction of textual image content.
  • E.5 Extended Defect-Guided Refinement Results: SDG eliminates a large poster-like obstruction from a city-skyline image.This demonstrates localized removal of a prominent visual obstruction.
  • E.5 Extended Defect-Guided Refinement Results: The extended comparison evaluates whether structured feedback supports targeted corrections across multiple defect types.The illustrated defects include object count, webpage text, and skyline obstruction errors.
  • E.5 Extended Defect-Guided Refinement Results: Figure 9 compares original images with Fixed caption-only editing, ImageDoctor heatmap/text-feedback editing, and SDG box-structured-feedback editing.The comparison is presented as an extended qualitative evaluation via GPT-Image-1.5.

F.1 Limitations · F.2 Broader Impact

SDG’s current scope, supervision, and grounding assumptions leave important limitations in coverage, reliability, and generalization. Its structured feedback improves interpretability and targeted refinement, but raises misuse and release-governance risks.

  • F.1 Limitations: SDG labels only artifact and misalignment defects, excluding aesthetics, style, composition, safety, and cultural appropriateness.Expanding the label space would require additional annotation guidelines and evaluation metrics.
  • F.1 Limitations: The dataset uses four contemporary T2I generators and Pick-a-Pic prompts, so performance may change across generators, domains, resolutions, or prompt distributions.The current construction provides broad coverage but does not establish transferability beyond these settings.
  • F.1 Limitations: Importance scores distilled from Gemini 3 Pro under a fixed rubric may differ from human preferences and inherit teacher-model biases.The scores provide a scalable severity signal, but their alignment with human judgment is not guaranteed.
  • F.1 Limitations: The SDG detector may miss subtle defects, hallucinate defects in clean locations, or generate boxes too coarse for very small or diffuse failures.These failure modes limit the reliability and spatial precision of defect-level feedback.
  • F.2 Broader Impact: Structured defect feedback exposes localized failure modes, supports dataset auditing, and enables targeted image refinement beyond scalar preference scores.These properties can help users diagnose generation errors more directly.
  • F.2 Broader Impact: Improved diagnosis and refinement may increase synthetic-image realism, creating potential for deceptive or harmful-content misuse.The release includes code, model weights, and sampled data, while the complete dataset remains under internal review.
  • F.2 Broader Impact: Dataset and model releases should preserve annotation provenance, document intended use, and include safeguards for controlled release where appropriate.The passage frames release practices as necessary safeguards accompanying SDG’s capabilities.
Loading 2606.06113v2…