PaperlayerView exact revision on arXiv

Source-linked AI summary

From Structural Feedback to Prompt Policies: Learning Faithful Text-to-Image Prompt Editors

Wentao Ye, Yali Ye, Zhiqing Xiao, Ru Peng, Xinpeng Ti, Liyao Li, Zhanming Shen, Peng Lu, Jie Zhang, Haobo Wang

arXiv:2610.36146v1cs.AI

TL;DR

Prompt optimization must improve image quality while preserving the user’s intended scene, despite uncertainty in structural grounding and stochastic image generation. StruPO uses a calibrated scene graph to condition local prompt edits and supervise them from generated images, improving the faithfulness–aesthetics trade-off across prompt distributions and frozen backbones, especially on relations.

  • Problem

    Prompt optimization must improve perceptual quality while preserving the scene specified by the user, even though grounding can be uncertain and image generation is stochastic.

    Source: §1 · pp. 1–2

  • Method

    StruPO shares a calibrated scene graph between policy conditioning and image-grounded supervision, uses local deletion, reordering, and insertion edits, and retains grounding confidence in its structural reward.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5

  • Results

    Across prompt distributions and frozen T2I backbones, StruPO improves the faithfulness–aesthetics trade-off, with its most consistent gains in relations.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Structural feedback can shape how a prompt policy learns, rather than serving only to score its outputs.

    Source: §5 · p. 9

    Object–relation graphs are less expressive for counting, occlusion, negation, emotion, metaphor, and other abstract intent.

    Source: §5 · p. 9

Abstract

from arXiv · show

Prompt optimization should improve the image a user receives without changing the scene the user requested. Yet stylistic rewriting can weaken object attributes and relations even when global text--image similarity remains high. We propose StruPO, a structure-aware prompt editor that uses a calibrated scene graph as a shared representation for policy conditioning and image-grounded supervision. The graph exposes what should be preserved, while deletion, reordering, and insertion provide a token-aligned way to modify the prompt. This turns structural feedback from an external critique into a training signal for a reusable editing policy. Learning from that signal requires handling both uncertain grounding and stochastic image generation. StruPO retains grounding confidence through smooth reward shaping and averages multiple images for each edit before computing group-relative advantages. Experiments span three disjoint 2K-prompt test sets and four frozen generation backbones, with comparisons to learned prompt optimizers, iterative structural feedback, diffusion-model fine-tuning, and inference-time attention control. StruPO improves the faithfulness--aesthetics trade-off, with consistent gains in relational fidelity. Independent automatic evaluators and a 15-annotator study provide complementary evidence, and human preference margins increase with relation complexity. Controlled ablations connect these gains to graph conditioning, local editing, and image-grounded learning. The resulting policy produces reusable optimized text in one pass after graph calibration, without an iterative critique--rewrite loop.

1 Introduction

Prompt optimization must improve image quality while preserving the requested scene, but vivid or globally similar rewrites can still alter objects, attributes, or relations. StruPO uses calibrated structural intent both to guide local prompt edits and to evaluate their rendered consequences.

1 Introduction

  • Prompt optimization must improve perceptual quality while preserving the user’s specified scene, even though generators also respond to composition, lighting, and style.A prompt’s formulation affects the resulting image.Source: §1 · p. 1
  • High global text–image similarity can conceal local failures, such as changed object attributes or relations, that style-oriented and embedding-constrained optimizers may leave unresolved.A prompt may remain topically similar despite a wrong coat color or a missing requested crown.Source: §1 · pp. 1–2
  • StruPO shares calibrated structural intent between policy conditioning and image-grounded supervision, aligning edit decisions with the object–attribute–relation constraints checked in generated images.Deletion, reordering, and insertion bias edits toward modifying the input rather than replacing it wholesale.Source: §1 · p. 2
  • Learning from structural feedback must address uncertain grounding and stochastic image scores, which can otherwise make a policy react to evaluator thresholds or render noise.StruPO uses confidence-preserving rewards and averages multiple renders per edit before comparing rollouts.Source: §1 · p. 2
  • The paper’s contributions include controlled ablations, cross-backbone transfer, independent evaluators, and human judgments to assess relation-sensitive benefits and quality–cost trade-offs.The stated comparisons include learned and iterative optimizers.Source: §1 · p. 2

2 Related Work

Prior work optimizes prompt quality through learned reformulation, structural feedback, scene-graph evaluation, and interventions in text or generation. StruPO’s distinction is a reusable editor that shares calibrated graph structure between its decisions and image-grounded reward.

Prompt optimization for T2I generation

  • Learned prompt optimizers span gradient-based token or embedding updates and supervised, multimodal, reinforcement-learning, and LLM-based reformulation; PAE [18] and RePrompt broaden learned objectives.StruPO instead shares a calibrated scene graph between editor input and atomic grounding reward, then learns token-aligned edits.Source: §2 · pp. 2–3

Structure-aware feedback for prompt optimization

  • OPT2I iteratively maximizes structural consistency, while VisualPrompter uses atomic visual feedback for targeted refinement; StruPO learns graph-conditioned editing from that feedback.The distinction is learned editing versus test-time feedback-driven revision.Source: §2 · p. 3

Scene graphs and fine-grained T2I evaluation

  • Scene-graph evaluation such as DSG exposes semantic failures global similarity can miss; StruPO uses object and relation grounding for learning and evaluates with held-out OR/RR, Gemini-VQA, human judgments, and independent evaluators.Attribute-bearing phrases remain in graph features and grounding queries.Source: §2 · p. 3

Text editing and broader faithfulness interventions

  • Text editors decompose edits into deletion, reordering, and insertion [28] [29], whereas DPOK updates diffusion parameters and Attend-and-Excite guides latent trajectories.These approaches have different control and access requirements from StruPO’s reusable-text interface.Source: §2 · p. 3

3 Proposed Method: StruPO

StruPO formulates structure-aware prompt editing as a single-step policy problem: calibrated scene graphs condition edits and define image-grounded constraints. Its policy uses token-aligned actions, while training addresses uncertain grounding and stochastic rendering.

3 Proposed Method: StruPO

  • A calibrated graph of attribute-bearing object phrases and relation triplets conditions a policy that edits a prompt and receives a scalar image-grounded reward from the frozen generator.The same graph both supplies policy context and specifies the constraints used to score the edited prompt.Source: §3 · p. 3

3.1 Scene-Graph-Aware Prompt Editing Policy

  • DSE calibrates parsed structure with visual evidence for ambiguous auxiliary elements while retaining text-core objects, including requested objects missing from the calibration render.MHE encodes prompt tokens and graph elements into a shared representation for the editor.Source: §3.1 · p. 4
  • EAG applies deletion, reordering, and insertion to preserve explicit correspondence with retained input tokens while allowing new phrasing and style.This is an inductive bias toward conservative edits, not a guarantee that constraints are preserved; image reward determines which edits are reinforced.Source: §3.1 · pp. 4–5, Figure 3 · p. 4

3.2 Policy Training

  • Supervised fine-tuning initializes edit behavior from token-aligned input–target pairs, after which on-policy reinforcement learning optimizes rendered outcomes.Unmatched input tokens define deletions, retained tokens define successor pointers, and unmatched output tokens define insertions.Source: §3.2 · p. 5
  • The structural reward retains grounding confidence rather than thresholding constraints, avoiding abrupt score changes near a confidence threshold.Object and relation confidence are scored atomically, independently of CLIP.Source: §3.2 · p. 5
  • Environment-aware GRPO averages multiple image rewards for each edit before group normalization, reducing render variation when comparing edit rollouts.Reward smoothing addresses sensitivity within a render, while multi-image averaging addresses variation across renders of the same edit.Source: §3.2 · pp. 5–6

4 Experiments

The experiments compare StruPO with prompt optimizers and structural-feedback methods across three test distributions, then examine transfer, human judgments, and design choices. Results emphasize relational fidelity, while ablations and examples probe how structural conditioning and local editing relate to those gains.

4.1 Experimental Settings

  • Evaluation uses three disjoint 2K-prompt test sets; held-out images are scored for aesthetic change, object recall, relation recall, and SGR, averaging three matched-seed generations per prompt unless noted.The sets are Lexica, DiffusionDB, and out-of-domain COCO captions.Source: §4.1 · p. 6

4.2 Main Results

  • 7.96-point RR gain over the best listed baseline on Lexica accompanies SGR 80.74; StruPO also leads AES∆ on all datasets and all four metrics on DiffusionDB.Table 1 describes its in-domain faithfulness–aesthetics trade-off as the strongest; Gemini-VQA ranks it highest on Lexica.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • On COCO, VisualPrompter has the highest SGR but AES∆ −0.36, while StruPO has the largest aesthetic improvement and consistency comparable to GPT-4o baselines.The passage reports that independent metrics show the same distinction on COCO.Source: §4.2 · p. 7
  • Without retraining, StruPO achieves the highest RR and SGR on all three unseen generators; on FLUX.1-dev, RR is 62.43 versus Promptist’s 53.41.VisualPrompter has higher AES∆ on SD v2.1, while StruPO has higher structural recall.Source: Table 2 · p. 7, §4.2 · p. 7
  • 15 annotators gave StruPO positive win–loss margins on both criteria against all five comparators; consistency margins were +9, +10, and +7 points versus three named baselines.Margins versus Promptist, VisualPrompter, and few-shot GPT-4o rose from +6 to +16, +6 to +17, and +3 to +14 points, respectively, between the lowest and highest relation-count buckets.Source: §4.2 · p. 7, Table 3 · p. 8

4.3 Ablation Study

  • Removing DSE, MHE, EAG, RL, or rcon lowers SGR by 7.05, 5.41, 7.06, 6.70, or 5.60 points, respectively.The w/o-EAG comparison retains graph conditioning, reward, and GRPO; removing rcon slightly increases aesthetics while lowering structural fidelity.Source: Table 4 · p. 8, §4.3 · p. 8
  • As constraints grow, StruPO has the smallest object-recall degradation and stays near 70% RR across relation-count buckets, where generic rewriters fluctuate.Human preference margins in the high-relation bucket provide a perceptual counterpart to this robustness.Source: Figure 4 · p. 8, §4.3 · pp. 8–9

4.4 Qualitative Analysis

  • Representative SD v1.5 examples show StruPO retaining attribute-bearing phrases and requested relations while adding visual detail when baselines alter attributes, omit objects, or miss relations.The examples illustrate local semantic errors that aggregate object recall alone can obscure.Source: Figure 5 · p. 9, §4.4 · p. 9

5 Conclusions and Limitations

StruPO uses calibrated structural intent for both policy conditioning and image-grounded supervision, and improves the faithfulness–aesthetics trade-off with consistent relational gains. Its scope is bounded by graph expressiveness, conservative editing, model quality, training cost, and English-only prompt distributions.

5 Conclusions and Limitations

  • StruPO improves the faithfulness–aesthetics trade-off, with its most consistent gains in relational fidelity.Independent evaluators and human judgments provide complementary evidence; controlled ablations connect gains to graph conditioning, local editing, and image feedback.Source: §5 · p. 9
  • After graph calibration, StruPO produces reusable optimized text in one policy pass rather than an iterative critique–rewrite loop.The editor shares calibrated structural intent between policy conditioning and image-grounded supervision.Source: §5 · p. 9
  • The approach depends on parser, grounding, and generator quality; object–relation graphs are less expressive for abstract intent, and conservative editing may restrict needed global reformulation.Other stated boundaries include multi-image averaging cost, an initial calibration render at inference, and no established multilingual coverage from the studied English prompt distributions.Source: §5 · p. 9

A Pipeline and Data Construction

  • The pipeline section specifies how prompt pairs become supervised edit targets and how image feedback subsequently trains the policy.Appendices B and C provide reward analysis and implementation settings.Source: A Pipeline and Data Construction · p. 16

A.1 Data Collection

The appendix describes how optimized prompts are turned into supervised inputs and token-aligned edit actions, then points to the environment-aware reinforcement-learning algorithm. The action trace preserves links between retained input tokens and their optimized-prompt positions.

A.1 Data Collection

  • SFT inputs pair each optimized prompt with two variants: its semantic skeleton and a GPT-4o rewrite of that skeleton.The optimized-prompt source is a public Lexica-derived collection; Table 5 illustrates an input–target pair.Source: §A.1 · p. 16, Table 5 · p. 16

A.2 Supervised Fine-Tuning

  • For each mapping from raw to optimized prompt, greedy token alignment favors a long shared subsequence before deriving deletion, reordering, and insertion targets.The three action types are identified by comparing input and target tokens and their aligned positions.Source: §A.2 · p. 16

A.3 Edit-Action Illustration

  • EAG removes unretained tokens, predicts successor links among those retained, then inserts new tokens into the resulting slots.The trace links retained input tokens to optimized-prompt positions and records inserted material; Figure 6 illustrates the action space.Source: §A.3 · p. 16, Figure 6 · p. 16

A.4 Reinforcement Learning

  • Algorithm 1 provides the detailed pseudocode for environment-aware GRPO.Source: §A.4 · p. 16

B Theoretical Analysis

The theoretical analysis contrasts smooth grounding-confidence rewards with thresholded rewards and describes environment-aware GRPO’s sampling and update procedure. Its stated stability results concern reward variation and gradient bounds, not a universal ordering of complete gradient variance or a convergence guarantee.

B Theoretical Analysis

  • For each prompt, environment-aware GRPO samples policy actions, averages rewards across multiple generated images per action, then updates with a clipped objective and KL and entropy regularization.The algorithm also extracts and calibrates the prompt’s scene graph and compares group-relative rewards.Source: B Theoretical Analysis · p. 17
  • The smooth reward averages grounding confidences for the graph’s atomic semantic items, whereas the discrete reward thresholds each confidence at a fixed value.The analysis defines the confidence vector over objects or relations and contrasts the two reward forms.Source: B Theoretical Analysis · p. 17
  • Smooth confidence rewards admit a Lipschitz-type gradient-fluctuation bound, while the discrete thresholded reward generally does not.The proof’s counterexample shows the thresholded reward’s discontinuity can transfer to the gradient contribution.Source: B Theoretical Analysis · pp. 17–18
  • Near the threshold, a discrete reward can retain variance 1/4 as confidence perturbations shrink, while the smooth reward’s variance tends to zero.The authors frame continuous confidence as less sensitive to small grounding perturbations, not as a universal ordering of complete gradient variance or a convergence theorem.Source: B Theoretical Analysis · p. 19

C Implementation Details

The implementation freezes the generator and reward evaluators while training the graph encoder and prompt editor, with optimization and compute settings selected or reported for reproducibility. The appendix also records how prompt-optimization and alternative-intervention baselines were configured.

C.1 Models and Frozen Evaluators

  • The default setup uses Stable Diffusion v1.5, GLIP-T for object grounding, BLIP-base for relation verification, and BART-base for the editor; only the GCN and editor are updated.The T2I generator, grounding and verification models, CLIP, and reward models remain frozen; the aesthetic predictor is also frozen during training and evaluation.Source: §C.1 · pp. 19–20

C.2 Optimization Hyperparameters

  • Hyperparameters are selected on a reserved Lexica validation split and fixed across test sets and generation backbones; SFT uses 144K pairs and RL uses a 24K-prompt pool.Training uses 56K SFT updates and 6K RL updates, with four policy rollouts and three independently sampled images per rollout for environment-aware averaging.Source: §C.2 · p. 20

C.3 Computational Budget

  • Reported StruPO training uses about 80 GPU-hours for SFT and 100 for RL; the DPOK run matches the RL-stage budget, while StruPO also incurs SFT cost.The runs use NVIDIA GeForce RTX 3090 GPUs with 24GB memory.Source: §C.3 · p. 20, Table 8 · p. 20

D Reproduction Details

  • The reproduction appendix records baseline configurations, the human-study protocol, and statistical procedures.It also describes VisualPrompter’s official-code setup, PAE’s dynamic-conditioning evaluation, and the GPT-4o structural-critique baseline’s three rounds.Source: D Reproduction Details · p. 21, §D.1 · p. 22

D.1 Baseline Setup

  • Prompt baselines include templated and LLM-rephrased prompts, few-shot prompting, continuation, learned reprompting, and human-refined targets where available.The human-crafted baseline is not reported for COCO because it lacks corresponding optimized targets.Source: §D.1 · pp. 21–22
  • Alternative interventions include DPOK updates to diffusion parameters and Attend-and-Excite changes to cross-attention or latent trajectories for each image.DPOK uses the same 24K RL prompts and composite reward, capped at 100 RL GPU-hours; Attend-and-Excite follows official inference settings.Source: §D.1 · p. 22

D.2 Human Evaluation Setup

The human study compared anonymized, matched-seed images on aesthetics and semantic consistency, with ties retained. StruPO was preferred in most dataset–comparator–criterion comparisons, and its consistency margin grew with relation complexity.

Participants and task

  • Fifteen independent annotators judged aesthetics and semantic consistency separately on anonymized, matched-seed image pairs, with ties allowed and retained.Left–right order was randomized, and prompt-level labels were aggregated by majority vote.Source: §D.2 · pp. 22–23, Figure 7 · p. 22

Human preference results

  • On Lexica, Fleiss’ κ was 0.40 for aesthetics and 0.43 for consistency, with prompt-bootstrap 95% half-widths of approximately ±2.1 and ±2.4 percentage points.Table 9 summarizes aggregate results across Lexica, DiffusionDB, and COCO.Source: §D.2 · p. 23, Table 9 · p. 23
  • StruPO had a positive W–L margin in 17 of 18 dataset–comparator–criterion comparisons.On COCO, VisualPrompter was preferred for consistency (31/33/36 from StruPO’s perspective), while StruPO was preferred for aesthetics (71/14/15).Source: §D.2 · p. 23, Table 9 · p. 23

Forced-choice sensitivity analysis

  • Under strict forced choice, StruPO remained preferred, consistent with the direction of the tie-allowed results.A win required a majority of annotators to prefer StruPO.Source: §D.2 · p. 23, Table 11 · p. 23

Relation-complexity analysis

  • The W–L margin rose by 10–11 points from the lowest to highest relation-count bucket for all three strong baselines.The trend remained significant after Holm correction.Source: §D.2 · p. 24, Table 12 · p. 24

Study documentation

  • The study reports consent, privacy protections, and hourly compensation, but not the exact rate or IRB/equivalent approval or exemption status.Table 10 reports exact Lexica consistency counts and defines its intervals and sign tests.Source: §D.2 · pp. 23–24, Table 10 · p. 23

D.3 Error Bars and Statistical Stability

The supplementary analyses test statistical stability, prompt sensitivity, attention patterns, template choices, interactive editing, and additional examples. They report a long-input exception to the otherwise strong prompt-setting results and show that inspectable edit traces can still contain fluency errors.

D.3 Error Bars and Statistical Stability

  • Across three datasets, average 95% bootstrap half-widths were approximately ±0.02 for AES∆, ±0.6 for OR, ±1.1 for RR, and ±0.8 for SGR.The pairwise ranking remained unchanged in 91.2% of trials for SGR and 90.6% for AES∆; evaluator uncertainty was not included.Source: §D.3 · p. 24

E.1 Sensitivity to Prompt Settings

  • Full StruPO led all four metrics for prompts up to 40 tokens, but the SFT-only variant exceeded it on AES∆, relation recall, and SGR for prompts of at least 40 tokens.For the longest bucket, StruPO retained the highest object recall at 76.65.Source: §E.1 · p. 24, Table 13 · p. 24
  • Within the Lexica evaluation, StruPO had the highest AES∆, OR, RR, and SGR for both rewritten and semantic-skeleton inputs.The analysis also examines input length and prompt format.Source: §E.1 · pp. 24–25, E Additional Analyses · p. 24, Table 14 · p. 25

E.2 Token-Level Attention Visualization

  • In randomly selected Lexica examples, DAAM maps showed more localized attention to scene-graph objects after prompt optimization.The maps illustrate token-level attention patterns alongside quantitative grounding results.Source: §E.2 · p. 25, Figure 8 · p. 25

E.3 Comparisons with Various Templates

  • Some shorter style-suffix variants improved SGR over the full suffix, but all reduced AES∆; the strongest, Tag4, reached SGR 76.36, 4.38 points below StruPO.Table 15 reports the template comparison, and Table 16 lists the tested suffixes.Source: §E.3 · p. 25, Table 15 · p. 26, Table 16 · p. 26

F.1 Interactive Prompt Optimization

  • Deletion, reordering, and insertion edits expose controls for locking selected object phrases, while the prototype interface was not evaluated in a separate usability study.Figure 9 shows a front end for inspecting edits and specifying preservation constraints.Source: §F.1 · p. 25, Figure 9 · p. 27

F.2 Case Study of Editing

  • Intermediate traces reveal where tokens are retained, moved, or added, but also expose awkward word order and repeated modifiers.Additional image examples are shown for Stable Diffusion v1.5, v1.4, and v2.1.Source: §F.2 · p. 26, §F.3 · p. 26, Figure 10 · p. 27

G. Additional Evaluation and Robustness Analyses

  • The supplementary evaluation asks whether gains transfer across evaluators, how the editor compares with alternatives, and which design choices matter; Gemini scores prompt satisfaction from 0 to 100.Gemini receives the original prompt and one generated image and is held out from training, model selection, and reward construction.Source: G Additional Evaluation and Robustness… · p. 30, §G.1 · p. 31

G.2 Independent Automatic Evaluators

Independent evaluators support StruPO’s quality–faithfulness pattern, while comparison with a learned optimizer shows distinct strengths across metrics. Results are reported on three disjoint test sets and a separate Lexica evaluation.

G.2 Independent Automatic Evaluators

  • StruPO led all three independent metrics on Lexica and DiffusionDB; on COCO, VisualPrompter led TIFA and VQAScore but had substantially lower ImageReward.The evaluators were absent from default training, and each prompt score averaged three generated images.Source: §G.2 · p. 31, Table 17 · p. 31
  • On Lexica, HPSv2.1 ranked StruPO essentially level with few-shot GPT-4o, while ImageReward ranked StruPO first.StruPO also gained 0.21 PickScore points and 0.8 CLIPScore points over Promptist; relation-specific conclusions rely on grounding, separate VQA evaluation, and human judgments.Source: §G.2 · p. 31, Table 18 · p. 32

G.3 Closest Learned Prompt Optimizers

  • Against RePrompt, StruPO improved RR by 2.30 points and SGR by 1.83 points, while RePrompt scored higher on TIFA (.795 versus .789).StruPO had higher AES∆ and ImageReward in the matched SD v1.5/Lexica comparison.Source: §G.3 · p. 32, Table 19 · p. 32

G.4 Attribute-Bearing Phrase Ablation

The supplementary comparisons test attribute-bearing phrases, CLIP-derived components, alternative generation interventions, reward weighting, and graph calibration. They also identify dense and abstract prompts as boundaries where structural representation remains less complete.

G.4 Attribute-Bearing Phrase Ablation

  • Full attribute-bearing phrases gained 0.05 over the policy-side ablation and 0.03 over the reward-side ablation on average, with improvements in all three binding categories.Both ablations reduced binding accuracy, linking attribute preservation to complete phrases in policy inputs and grounding queries.Source: §G.4 · p. 32, Table 20 · p. 32

G.5 CLIP Sensitivity

  • Replacing CLIP with T5 changed SGR by −0.14, within run-to-run variation; removing all CLIP-derived training components jointly lowered SGR by 0.94.The resulting score of 79.8 remained 4.40 points above Promptist.Source: §G.5 · p. 33, Table 21 · p. 33

G.6 Cross-Paradigm Comparison

  • DPOK achieved the largest aesthetic gain, while StruPO had the highest OR, RR, and SGR and improved RR by 3.34 points and SGR by 1.64 points.StruPO and Attend-and-Excite had comparable TIFA scores (.789 and .787); StruPO produced reusable text without per-image latent optimization.Source: §G.6 · p. 33, Table 22 · p. 33

G.7 Full GPT-4o Structural-Feedback Baseline

  • On the full Lexica test set, StruPO exceeded GPT-4o-SG-Iter by 0.74 OR, 2.96 RR, 1.85 SGR, and 0.7 Gemini-VQA points.StruPO used fewer test-time feedback rounds; the comparison used three independent matched seed lists.Source: §G.7 · pp. 33–34, Table 23 · p. 34

G.8 Reward Sensitivity

  • Across tested structural reward weights, SGR ranged from 79.6 to 80.7 while AES∆ decreased as structural weight increased, with SGR peaking at an intermediate setting.The default α1 = 0.6 was selected on validation data as a balanced operating point.Source: §G.8 · p. 34, Table 24 · p. 34

G.9 DSE Stress Test

  • Under synthetic medium parser noise, the full model remained above the text-only policy, while using the raw parsed graph without DSE performed worse.The perturbations inserted distractor objects, swapped predicates, and sometimes dropped relation endpoints.Source: §G.9 · p. 34, Table 25 · p. 34

G.10 Complexity-Bucket Protocol

  • Automatic complexity analyses bucketed the same 2,000 Lexica prompts by retained object phrases and relation triplets, with matched methods and image seeds within each bucket.The human reaggregation used the same relation-count partition.Source: §G.10 · pp. 34–35

G.12 Dense and Abstract Failure Cases

  • On dense scenes with at least four objects, SGR fell from 80.7 to 74.8 but remained above Promptist and few-shot GPT-4o; abstract prompts expose under-representation of non-visual intent.The authors identify denser-scene reasoning and intent beyond visually grounded object–relation constraints as extension priorities; reported runtime depends on hardware and implementation.Source: §G.12 · p. 35, §G.11 · p. 35

H Structural Feedback in Learned Prompt Editing

This section presents StruPO’s scene-graph-conditioned, locally edited prompt policy and its image-outcome supervision, alongside discussion of deployment risks and reproducibility limits. It also includes NeurIPS checklist disclosures on human-subjects research and LLM use, distinguished from the structural reward.

H Structural Feedback in Learned Prompt Editing

  • StruPO couples policy conditioning and image-grounded supervision through one calibrated graph and a local edit representation.DSE calibrates structural context, MHE exposes it to the editor, EAG links edits to input tokens, and confidence-preserving, environment-aware optimization handles noisy image feedback.Source: H Structural Feedback in Learned… · p. 35
  • Replacing local edits with free-form generation lowers SGR by 7.06 points, while removing structural reward lowers SGR by 5.60 points.Both interventions retain the other components, supporting complementary roles for the editing parameterization and its supervision.Source: H Structural Feedback in Learned… · p. 35
  • After graph calibration, one policy pass produces reusable optimized text rather than requiring an iterative critique–rewrite loop.The paper evaluates the online trade-off in Table 23 and reuse across frozen backbones in Table 2.Source: H Structural Feedback in Learned… · p. 35
  • Faithful prompt editing may reduce manual effort and unintended scene changes, but controllable generation can also facilitate misleading synthetic content or reproduce model biases.The paper recommends retaining generator safety filters and usage restrictions, communicating synthetic provenance, and respecting privacy and copyright constraints; editing does not establish that a requested use is appropriate.Source: H Structural Feedback in Learned… · pp. 35–36

I Declaration of LLM and Multimodal-Model Usage

  • GPT-4o and Gemini-VQA serve as external baseline and evaluation components, while GPT-6 Pro assisted with polishing, grammar, and code rather than research design or scientific conclusions.The declarations state that these models did not design StruPO or define its objective, and that authors developed and verified the research and analyses.Source: I Declaration of LLM and… · p. 36

NeurIPS Paper Checklist

  • The supplied revision does not establish full reproducibility: clipped-log constants and a verified executable code/checkpoint package are absent, and total compute for preliminary or failed runs is unavailable.The materials also lack a complete implementation package with run commands, environment specification, and licensing documentation.Source: NeurIPS Paper Checklist · pp. 38–40
  • The paper reports prompt-bootstrap uncertainty, human-study confidence intervals and sign tests, and mean and standard deviation over three seed lists for the full iterative baseline.The checklist identifies these analyses as support for reporting uncertainty and statistical significance.Source: NeurIPS Paper Checklist · p. 40
  • The supplied materials do not establish human-study IRB or equivalent approval, the exact compensation rate, or exhaustive asset licensing and documentation.The appendix describes consent, privacy, and study procedures, but the checklist notes missing administrative and release details.Source: NeurIPS Paper Checklist · pp. 42–43
  • The checklist marks crowdsourcing and human-subjects research as not applicable.It also states that IRB approval may be required depending on the country and that authors should follow institutional guidelines and preserve anonymity in initial submissions.Source: NeurIPS Paper Checklist · p. 43
  • The paper reports GPT-4o, GPT-3.5, and Gemini use while distinguishing these roles from the GLIP/BLIP structural reward.GPT-4o was used in SFT-input construction and comparison methods, GPT-3.5 as a baseline, Gemini as a held-out evaluator, and LLMs also provided editorial assistance.Source: NeurIPS Paper Checklist · p. 43

Related papers

Loading 2610.36146v1…