PaperlayerView exact revision on arXiv

Source-linked AI summary

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty

Meera Hahn, Wenjun Zeng, Nithish Kannen, Rich Galt, Kartikeya Badola, Been Kim, Zi Wang

arXiv:2412.06771v3cs.AIcs.CVcs.LG

TL;DR

Underspecified prompts can leave T2I models misaligned with users’ intended images. The paper proposes proactive agents that represent uncertainty in belief graphs and ask users for clarifications; across three datasets, they achieve at least 2 times higher VQAScore than traditional single-turn generation within five turns.

  • Problem

    Underspecified prompts can leave T2I models’ assumptions misaligned with users’ intentions, including culturally preferred content and scene details.

    Source: §1 · p. 2

  • Method

    Proactive T2I agents represent uncertainty about image entities, attributes, and relations in belief graphs and use multi-turn questions to gather information toward a target image.

    Source: §1 · pp. 2–3

  • Results

    At least 2 times higher VQAScore than traditional single-turn T2I generation was achieved within 5 turns; over 90% expected proactive clarifications to help.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Belief graphs provide tools for agent interpretability and controllability, and presenting them may offer a generalizable method for AI transparency.

    Source: Impact Statement · p. 15

    The automatic NLL approximation does not account for semantic similarity between entity or attribute names, which can cause approximation errors.

    Source: §7.2 · p. 11

Abstract

from arXiv · show

User prompts for generative AI models are often underspecified, leading to a misalignment between the user intent and models' understanding. As a result, users commonly have to painstakingly refine their prompts. We study this alignment problem in text-to-image (T2I) generation and propose a prototype for proactive T2I agents equipped with an interface to (1) actively ask clarification questions when uncertain, and (2) present their uncertainty about user intent as an understandable and editable belief graph. We build simple prototypes for such agents and propose a new scalable and automated evaluation approach using two agents, one with a ground truth intent (an image) while the other tries to ask as few questions as possible to align with the ground truth. We experiment over three image-text datasets: ImageInWords (Garg et al., 2024), COCO (Lin et al., 2014) and DesignBench, a benchmark we curated with strong artistic and design elements. Experiments over the three datasets demonstrate the proposed T2I agents' ability to ask informative questions and elicit crucial information to achieve successful alignment with at least 2 times higher VQAScore (Lin et al., 2024) than the standard T2I generation. Moreover, we conducted human studies and observed that at least 90% of human subjects found these agents and their belief graphs helpful for their T2I workflow, highlighting the effectiveness of our approach. Code and DesignBench can be found at https://github.com/google-deepmind/proactive_t2i_agents.

1. Introduction

Underspecified prompts can leave T2I models misaligned with users’ cultural preferences and intended scene details, prompting repeated refinement. The paper proposes agents that expose uncertainty in editable belief graphs, ask prioritized clarification questions, and evaluate alignment through simulated users and curated data.

1. Introduction

  • Prompt underspecification can produce culturally mismatched images or miss intended scene details and spatial relationships, leading users to repeatedly refine prompts.The authors also note potential frustration, dissatisfaction, bias propagation, and offense from model assumptions.Source: §1 · p. 2
  • The work frames proactive, understandable interaction as a way to give users direct control over agent beliefs and make image creation more accessible.The prototypes are modular, allowing components such as question strategy, belief-graph construction, and underlying models to be changed.Source: §1 · pp. 2–3
  • The agents represent uncertainty about image entities, attributes, and relations in an editable belief graph, including likely elements not stated in the prompt.The graph can surface alternatives such as cuisine types and possible objects, and lets users edit uncertain items.Source: §1 · p. 2
  • Clarification questions are selected using both uncertainty and importance, with likely answer options offered while users may also enter their own responses.After user answers or graph edits, an LLM updates the prompt and belief graph before an off-the-shelf T2I model generates images.Source: §1 · p. 3
  • The evaluation pipeline pairs a target image and underspecified prompt with a simulated user, then assesses generated images against the target using text and image similarity metrics.DesignBench adds curated aesthetic scenes, multiple entities and interactions, short and long captions, and varied realism and styles for artist- and designer-oriented evaluation.Source: §1 · p. 3
  • At least 2 times higher VQAScore than traditional single-turn T2I generation was achieved within 5 turns across three datasets, and human participants reported helpfulness and image preferences.Over 90% expected proactive clarifications to help, about 85% found belief graphs helpful, and participants preferred agent-generated images in more than 80% of 550 prompt-image pairs.Source: §1 · p. 3

2. Related Work

The related work situates proactive T2I agents among research on knowledge-seeking agents, human-AI interaction, interpretability, image evaluation, and prompt expansion. The paper distinguishes its use of clarification questions for underspecification from prior work addressing ambiguity.

2. Related Work

  • Prior agent and data-acquisition research motivates agents that actively seek important information from users to reduce uncertainty and pursue goals.The paper connects this goal to work on active learning, Bayesian optimization, reinforcement learning, and experimental design.Source: §2 · p. 4
  • Human-AI interface research addresses transparency, uncertainty about AI capability, and complex outputs, while interpretability work raises questions about whether language-model explanations are truthful or grounded.The paper’s belief graph represents the agent’s distribution over image-prompts it may construct, not the T2I model’s output distribution given the underspecified prompt.Source: §2 · p. 4
  • Unlike clarification work aimed at resolving ambiguity, this paper uses questions to address underspecification: prompts that lack information to fully describe the intended image.The cited T2I literature describes prompts as potentially ambiguous, subjective, or difficult to represent visually.Source: §2 · p. 4
  • T2I alignment research uses embedding-, VQA-, and captioning-based metrics, alongside human-preference models and image-diversity measures.The paper evaluates agent-user conversations and generated images using several such metrics and human ratings.Source: §2 · pp. 4–5
  • Prompt expansion can improve image generation, but evaluating the proposed approach as a prompt-expansion tool is outside this paper’s scope.The paper characterizes its approach as adaptively expanding prompts based on user feedback.Source: §2 · p. 5

3. Background

The background frames image generation as constructing a picture of an imagined world represented through entities, attributes, and relations. Belief graphs extend this representation with uncertainty and task-specific importance, while using language models to derive predicates rather than relying on fixed symbols.

3. Background

  • Classical symbolic world representations encode entities, their attributes, and relations, such as blocks and their positions in a blocks-world state.Such representations support an agent’s ability to know and act in the world, with predicates often predefined and hardcoded.Source: §3 · p. 5
  • Scene-graph approaches represent image descriptions as atomic propositions, while this paper treats image generation as taking a picture of an imagined world.That imagined world should include all entities in the picture and their attributes and relations.Source: §3 · p. 5
  • A belief state represents a distribution over possible states; for T2I, probabilities can express whether entities appear and which attribute values they take.For example, an agent may assign equal probability to a block being on or not on a table.Source: §3 · p. 5
  • The paper’s belief graphs add importance scores and use LLMs to derive predicates, making them adaptable rather than dependent on rigid, hardcoded rules.Samples from the agent belief can be used to construct expanded prompts.Source: §3 · p. 5

4. Belief Graphs

Belief graphs are generated adaptively from image descriptions and encode entities, their attributes, and relations with probabilities and importance scores. They distinguish explicit, implicit, and background entities, including uncertain elements that may not appear in the prompt.

4. Belief Graphs

  • LLMs adaptively generate belief graphs from image descriptions, avoiding the hardcoded symbols used in classical belief representations.The authors describe this simple approach as flexible across image types, while identifying more sophisticated methods as future work.Source: §4 · p. 6
  • Graph nodes represent entities and edges represent relations; entities carry names, descriptions, types, probabilities, importance scores, and attributes.Entities are explicit when mentioned in the prompt, implicit when likely but unmentioned, or background elements such as style, time, and location.Source: §4 · p. 6
  • Entity probabilities indicate likelihood of appearance, with explicit and background entities typically near 0 or 1 and implicit entities usually between them.Importance scores range from 0 to 1 and represent relevance and impact on the image; the paper assumes the agent does not model variation in people’s importance judgments.Source: §4 · p. 6
  • Attributes carry value-probability pairs and importance scores, while relations are attributes connecting entities and can encode spatial values.The graph can include potential implicit relations, such as the position of a fork relative to a table.Source: §4 · p. 6

5. Proactive T2I Agent Design

The agent seeks desired images through multi-turn interaction, balancing interaction cost against image quality by gathering information and updating its belief graph and prompt. Its questions target relevant uncertainty while avoiding redundancy and minimizing the effort required of users.

5. Proactive T2I Agent Design

  • The agent constructs an initial belief graph, asks questions or presents the graph, incorporates feedback, and updates the graph and prompt to approximate the intended image with minimal back-and-forth.Its decision strategy trades off interaction cost against generated-image quality.Source: §5 · p. 6

5.1. Belief Graph Transitions

  • User feedback from dialogue or direct graph manipulation is combined with the prompt, conversation history, and previous belief to regenerate an updated belief graph.The prototype summarizes gathered information in a merged prompt before regenerating the belief.Source: §5.1 · p. 7

5.2. Asking Questions to Gather Information

  • Questions should avoid redundancy, reduce uncertainty about image contents, relate to the prompt, and be concise and direct.The agent can ask about uncertain image characteristics, update its uncertainty after an answer, and offer likely answer choices to reduce user effort.Source: §5.2 · pp. 7–8

6. Implemented Agent Prototypes

The three prototypes share belief-graph construction but differ in how they ask questions. Their approaches trade off question-answerability, redundancy, and information gained.

6. Implemented Agent Prototypes

  • All three prototypes use the same belief-graph construction approach and differ in their question-selection decision strategies.Ag1 uses a heuristic score, while Ag2 and Ag3 prompt an LLM using different information.Source: §6 · p. 8, §6.2 · p. 8, §6.3 · p. 9

6.1. Ag1: Heuristic Score Agent

  • Ag1 selects questions about important, uncertain entities, attributes, or relations using a heuristic that multiplies importance by entropy.It prompts an LLM to phrase the selected question as an easy-to-answer multiple-choice query.Source: §6.1 · p. 8
  • Ag1’s heuristic and post-processing do not consistently prevent redundant questions, including asking whether an entity already mentioned in the prompt exists.The passage attributes this example to possible LLM errors in parsing beliefs from prompts.Source: §6.1 · p. 8

6.2. Ag2: Belief-prompted Agent

  • Ag2 tends to outperform Ag1 with fewer question turns, but produces fewer easy-to-answer questions.Ag2 prompts an LLM with the belief graph, merged prompt, and conversation history to generate questions, without Ag1’s post-processing for belief transitions.Source: §6.2 · p. 8

6.3. Ag3: Principle-prompted Agent

  • Ag3 asks more open-ended and complex questions, which are harder for users to answer but gain more information about intent.It relies on conversation history and question-asking principles rather than Ag2’s belief graph and merged prompt.Source: §6.3 · p. 9

7. Experiments

The experiments combine simulated multi-turn evaluation and human studies to assess whether proactive agents improve alignment with image intent across datasets and workflows.

7. Experiments

  • The evaluation has two parts: simulated user-agent conversations for automatic metrics and human-subject studies of the framework.Source: §7 · p. 9

7.1. Automatic Evaluation

  • Simulated conversations begin from a target image and an underspecified prompt, run for 15 turns, and are evaluated at each turn against the target.The ground-truth data tuple includes the image, initial prompt, detailed caption, and belief graph; the prompt is intentionally less detailed than the caption.Source: §7.1 · p. 9, §7.1.2 · p. 9

7.1.1. Setups for agents and baseline

  • The comparison uses a no-question T2I baseline alongside Ag1, Ag2, and Ag3, with Gemini 1.5 for agent interaction and Imagen 3 for image generation.Source: §7.1.1 · p. 9

7.1.2. Datasets.

  • The evaluation combines COCO-Captions and ImageInWords with DesignBench, a 30-scene dataset adding artistic, cartoon, and improbable imagery absent from the former datasets.ImageInWords uses dense human captions; COCO-Captions starts from the shortest caption, which is expanded into a detailed ground-truth caption.Source: §7.1.2 · p. 10

7.1.3. Metrics

  • Alignment is measured across text-text, image-image, text-image, and belief-state dimensions using similarity metrics and an approximate belief-state NLL.The NLL calculation assumes independence among entities, attributes, and relations.Source: §7.1.3 · p. 11

7.3. Human Studies on Generated Images and Dialogues

  • Human raters selected agent-generated images over single-turn T2I in 80%+ of cases for both content and aesthetics/style.Across turns, most metrics generally improve, while T2T and T2I scores plateau or decline after about 10 interactions and I2I continues to increase.Source: §7.2 · p. 11, §7.3 · p. 13

7.2. Automatic Metrics and Qualitative Results

  • Across COCO-Captions, ImageInWords, and DesignBench, multi-turn agents outperform the standard T2I baseline on all reported metrics, with similar patterns across datasets.The table caption describes large gains over the standard T2I model; the results text says all agents exceed the baseline on both datasets and all metrics.Source: §7.2 · p. 11, Table 1 · p. 13

7.4. Human Studies on the Agent Interface

  • Among 143 regular T2I users, 91% considered clarifications potentially helpful, 88% entity graphs, and 86% relationship graphs.58% expected clarifications to deliver value immediately or very soon.Source: §7.4 · pp. 13–14

8. Discussion and Conclusion

The paper presents proactive question asking and belief-graph refinement as an interactive approach to image generation, while noting image-generation alignment errors as a constraint. It identifies direct image generation from belief graphs and trajectory fine-tuning as future work.

8. Discussion and Conclusion

  • The agent design combines proactive questions with belief-graph refinement to update its understanding of user intent and support collaborative image generation.The authors describe belief graphs as a potentially generalizable method for AI transparency.Source: §8 · p. 14
  • T2I prompt-image alignment errors can limit agent effectiveness, though QA-pair evaluation can filter misaligned images across N seeds.With the T2I model ablated, agents’ T2T scores in Table 1 are 92%+ on caption-level alignment.Source: §8 · p. 15
  • The authors suggest clarification questions may help models adapt to diverse user needs and mitigate risks tied to differing perceptions of harmful content.Source: §8 · p. 15
  • Future work could generate images directly from belief graphs and fine-tune VLMs on image-text trajectories, potentially requiring gold-standard trajectories or trajectory-quality annotations.Source: §8 · p. 15

Impact Statement

The impact statement frames belief graphs as tools for interpretability and controllability, and connects clarification questions to personalized content creation and potential risk mitigation. The formalism models interactive agents through beliefs, actions, observations, transitions, and action selection.

Impact Statement

  • Belief graphs are presented as tools for interpretability and controllability, and as a potentially generalizable method for AI transparency.Source: Impact Statement · p. 15
  • Clarification questions may support personalized content creation and potentially mitigate risks from content perceived as harmful by particular users.Source: Impact Statement · p. 15

B. Formalism of the Agent and its Objective

  • An interactive agent is formalized as a tuple of belief states, actions, observations, a belief-transition function, and an action-selection strategy.The agent aims to maximize an alignment score between its belief and user intent with as few interaction turns as possible.Source: B. Formalism of the Agent and its… · p. 22

C. Visualization of Multi-Turn Agent-User Dialogs and Generated Images

D. Implementation Details of Agent Prototypes

The prototypes use different configurations for question generation and represent agent beliefs both as merged natural-language prompts and as structured belief graphs. The appendix also shows example dialogs and generated images.

D. Implementation Details of Agent Prototypes

  • The appendix visualizes five-turn dialogs for Ag1–Ag3 and generated outputs compared with goals, including 15-turn agent runs on DesignBench and COCO-Captions.Source: Figure 5 · p. 23, Figure 6 · p. 24, Figure 7 · p. 25
  • The three prototypes use distinct question strategies: Ag1 uses heuristic importance scores, Ag2 conditions an LLM on conversation history and belief structure, and Ag3 relies on conversation history and LLM reasoning.Source: D. Implementation Details of Agent… · p. 28

D.1. Implementation of Agent Beliefs

  • Each agent belief has two forms: a merged prompt summarizing the conversation and a symbolic belief graph derived from that prompt.The graph captures entities, attributes, relations, and associated probabilities for reasoning and decision-making.Source: §D.1 · p. 28
  • Specialized in-context-learning parsers extract entities, their attributes, and relations, with associated importance scores and probability information.Source: §D.1 · p. 28

D.2. Implementation of Action Selection

The appendix describes action-selection heuristics and the implementation components for simulating users, updating beliefs, parsing prompt content, and constructing questions and revised prompts. The automated evaluation uses user-agent self-play.

D.2. Implementation of Action Selection

  • The action-selection discussion frames the ideal action as maximizing information gain, then presents heuristic strategies based on importance scores or LLM-generated clarification questions.Ag1 uses heuristic importance scores; Ag2 uses belief-structured prompts, while Ag3 relies on conversation history.Source: §D.2 · pp. 28, 30

D.3. User Simulation

  • The user simulator builds a belief graph for the intended image and answers agent questions using that graph and conversation history.Source: §D.3 · p. 30

D.4. Implementation of Belief Transition

  • At each turn, prompt merging and belief parsing update the graph; Ag1 additionally removes redundant questions and retains information missing from the refreshed graph.Source: §D.4 · pp. 30–31

D.5. Entity Parser Prompt Instruction

  • Entity parsing identifies explicit, implicit, and background entities, assigning likelihoods and importance scores to ask about.The example instructions prioritize primary or layout-relevant entities and reduce scores for elements already specified.Source: §D.5 · pp. 31–32

D.6. Attribute Parser Prompt Instruction

  • Attribute parsing generates plausible candidate values with probabilities summing to one, while scoring how important each attribute is to clarify.Already specified values receive a single candidate with probability 1.0.Source: §D.6 · p. 32

D.7. Relation Parse Prompt Instruction

  • Relation parsing identifies entity pairs and possible spatial relations, with an importance score reflecting uncertainty and relevance to image layout.Source: §D.7 · p. 33

D.8. Verbalization Prompt Instruction

  • Prompt-construction instructions turn question-answer pairs into declarative statements and combine user feedback with the original prompt without adding unsupported information.Question prompts request concise clarifications about important image details and avoid asking about information already covered in the dialog.Source: §D.8 · p. 33, §D.9 · p. 34

D.12. HSA Question Prompt Instruction

E.1. DSG Evaluations

The DSG evaluation compares ground-truth and agent-generated captions at each turn. Every multi-turn agent exceeds the no-update T2I baseline on all benchmarks, and agent performance increases with more turns.

E.1. DSG Evaluations

  • All multi-turn agents far exceed the no-update T2I baseline on every benchmark for DSG.DSG compares the ground-truth caption with the generated caption.Source: §E.1 · p. 35
  • DSG scores compare the ground-truth caption with the generated caption at each turn.The evaluation adapts DSG to parse prompts into Davidsonian scene graphs.Source: §E.1 · p. 35
  • Performance increases with the number of turns for all agents.Figure 11 reports the comparison at each turn.Source: Figure 11 · p. 35
  • The self-play algorithm takes an initial prompt, user, agent, and maximum turn count as inputs, and outputs a refined prompt.The provided algorithm excerpt initializes the refined prompt from the initial prompt and iterates over turns.Source: §E.1 · p. 36

E.2. Mitigation of T2I Errors

The authors test QA-pair-based filtering as a way to improve image fidelity when T2I outputs miss prompt details, while noting that the pipeline remains constrained by the employed model’s prompt-following ability. The interface proposal lets users clarify prompts and edit entity attributes and relations in a belief graph.

E.2. Mitigation of T2I Errors

  • DINO (I2I) increased from 0.7637 to 0.7838 (δ=+.02) after QA/VQA filtering across 30 DesignBench prompts.For each prompt, the authors generated ten images from different seeds and selected the image with the highest VQA score; pipeline capability remains constrained by the T2I model’s prompt-following ability.Source: §E.2 · p. 36

F. Details on the Agent Interface

  • The proposed interface is an intentionally simple hypothesis that could be iterated for different applications and users.The interface would place prompt input and output images alongside clarifications or a graph view.Source: F. Details on the Agent Interface · pp. 36–37

1. Default state

  • The default view pairs a text-prompt input and output-image area with space for clarifications or a graph that can inform later iterations.Figure 12 illustrates this proposed default state.Source: §1 · p. 37, Figure 12 · p. 37

2. Output images, with Clarifications

  • Clarifications ask specific questions to increase prompt specificity or help users realize their intent, and answers can be selected or entered freely before regeneration.The proposed interface offers likely options and incorporates the user’s answer into the primary prompt.Source: §2 · p. 37, Figure 13 · p. 38

3. Graph Entities & Attributes

  • The entity graph distinguishes explicit from implicit entities and lets users confirm or deny entities and edit their assumed attributes.Attribute alternatives are ordered by model probability, which is shown as a percentage.Source: §3 · pp. 37–38, Figure 14 · p. 39

4. Graph Relationships

  • The relationship view lets users choose two entities, inspect their described relation, and replace it with an alternative before regenerating images.The proposed controls use probability-ordered alternatives.Source: §4 · p. 38, Figure 15 · p. 39

G.1. Hypothesized Frustrations

  • The user study asked participants to rate four hypothesized frustrations: misinterpretation, repeated iterations, inconsistent generations, and incorrect assumptions.Participants were asked how much these issues related to their current T2I usage.Source: §G.1 · p. 40

G.2. Hypothesized Features

  • The proposed features pair clarifying questions with editable graphs of prompt entities and relationships.Users would be able to edit entity attributes and entity-to-entity relations.Source: §G.2 · p. 40

G.3. Human Study Results

The human-study section summarizes participants’ T2I usage, reported frustrations, and expected speed of value from proposed agent features. Its appendix describes North America-based recruitment and three human-rating tasks covering question issues, image-dialog correspondence, and similarity to a goal image.

G.3. Human Study Results

  • Tables 3–5 summarize T2I usage frequency, reported frustrations, and expected speed of value from agent features.Table 3 covers 143 participants; the supplied passages do not state the tables’ cell values.Source: §G.3 · p. 42, Table 3 · p. 42, Table 4 · p. 42

H. Details on Recruitment and Participant Consent

  • Participants were recruited in North America, were English-fluent adults with recent T2I experience, and completed the survey on desktop computers.The study review described the study as low risk, and generated images were pre-approved and served to participants.Source: H. Details on Recruitment and… · p. 43
  • Raters assessed question issues, image correspondence to dialogue and prompt, and similarity to a goal image.The question-rating task covered approximately 8,000 questions per agent; the correspondence task covered approximately 1,500 image-dialog pairs rated by three raters; the similarity task covered approximately 550 image-dialog pairs per agent rated by three raters.Source: Figure 19 · p. 44, H. Details on Recruitment and… · pp. 44–46, Figure 20 · p. 45

Related papers

Loading 2412.06771v3…