STaR-GATE: Teaching Language Models to Ask Clarifying Questions

Motivation

• Users’ unstated dietary, stylistic, or procedural preferences can make a generic response ineffective for the underlying task.The same request may correspond to different tasks depending on the user.Source: §1 · p. 1
• The approach is motivated by limitations of prompting and alignment strategies that can produce ineffective or irrelevant questions.Prior methods include free-form elicitation and constrained optimal-experimental-design approaches.Source: §1 · pp. 1–2

Results

• 72% win rates against the initial model are reached after two STaR-GATE iterations.The method improves question asking while retaining response generation through response regularization.Source: §1 · p. 2, Figure 3 · p. 3

Approach

• STaR-GATE combines active preference elicitation with STaR-inspired self-improvement and filters simulated conversations using oracle-generated gold responses.The dataset contains 25,500 unique persona-task prompts, and response regularization reduces distribution shift.Source: §1 · p. 2, Figure 2 · p. 2

Scope

• The finetuned model generalizes beyond the roleplayer used during training.This contribution is stated alongside the method’s improved elicitation and response quality.Source: §1 · p. 2

Preference Optimization

• Preference-optimization methods such as RLHF, DPO, and KTO train models to produce preferred single-turn responses from the initial prompt.The paper contrasts this response-focused setting with follow-up questioning for user-specific needs.Source: §2 · p. 3

Preference Elicitation with LMs

• GATE and related methods use language-model interaction or preference inference to resolve task ambiguity, while other approaches constrain elicitation to fixed feature comparisons.The cited literature includes free-form questioning, observation-based inference, and structured elicitation.Source: §2 · pp. 3–4

Synthesis

• The paper combines active preference elicitation with self-improvement rather than treating response optimization, elicitation, and reasoning self-training as separate problems.This combination motivates STaR-GATE’s bootstrapped questioning strategy.Source: §2 · pp. 3–4

Self-Improving Reasoning

• STaR showed that iterative training on reasoning traces leading to correct answers can improve performance on increasingly difficult problems.STaR combines rationalization with supervised finetuning on successful rationales.Source: §2 · p. 4

Objective

• STaR-GATE maximizes E_sij log p_QBASE(g_ij|t_i, s_ij), rewarding conversations that make gold personalized responses more probable.The Oracle supplies gold responses from task-persona pairs, while the Questioner and Roleplayer generate the conversation.Source: §3 · p. 5
• The objective’s simulated conversation contains Questioner questions and Roleplayer answers conditioned on the task, persona, and preceding dialogue.This defines how elicited information enters the gold-response likelihood.Source: §3 · p. 5

Optimization

• Each iteration samples multiple conversations, selects the top trajectory by the objective, and finetunes the initial model on the selected conversation.The implementation uses expert iteration with k = 1 selected trajectory.Source: §3 · p. 5

Regularization

• Response regularization adds model-sampled responses conditioned on selected conversations so the Questioner learns to answer as well as ask questions.The loss masks roleplayer answers, training question and response policies without imitating the answers.Source: §3 · p. 5

Algorithm

• Task and persona splits alternate across iterations to avoid generating new data from tasks or personas present during training.The algorithm trains the initial model for N = 3 iterations using intermediate models Q_1, Q_2, ..., Q_η.Source: §3 · p. 5

Tasks

• The task set consists of the first 550 conversations from an open-source instruct human-assistant prompt dataset, using human queries rather than assistant responses.The queries are divided into two training splits of 250 and one test split of 50.Source: §4 · p. 6

Personas

• GPT-4 generates 110 personas from content-filtered examples, split into two 50-person training sets and one 10-person test set.The persona splits support held-out persona evaluation.Source: Persona Generation · p. 6

Dataset Construction

• 25,500 task–persona–gold-response triples are created from 550 human queries and 110 generated personas.The construction uses 250-task training splits, a 50-task test split, and persona splits of 50, 50, and 10.Source: Persona Generation · p. 6, §4 · p. 6

Gold Responses

• An Oracle prompted with each persona and task generates the personalized gold response without dialogue history.These responses define the task–persona–gold-response triples used by the evaluation setup.Source: Persona Generation · p. 6

Gold Log-probabilities

• Figure 4 reports log-probability trajectories for STaR-GATE and STaR-GATE without regularization, with error bars showing ± SEM across held-out persona-task prompts.The regularized and unregularized variants are evaluated across iterations.Source: Figure 4 · p. 7
• Gold-response log probabilities increased over iterations for the Q-Experimental condition.The evaluation used held-out persona-task prompts and compared experimental, random-persona, negative-control, and positive-control conditions.Source: §5 · pp. 6–7
• The Q-Random baseline also improved, but its log probabilities remained much lower than the Q-Experimental condition.The paper attributes the additional experimental improvement to persona-specific information, while noting that random personas can contain partially relevant information.Source: §5 · pp. 6–7

Win Rates

• 72% maximum win rate was reached after two iterations, comparing STaR-GATE responses with those from the initial model.Win rates were computed from GPT-4 pairwise judgments of responses conditioned on sampled conversations.Source: §5 · p. 7

Training Ablations

• STaR-GATE without response regularization loses win rate over iterations because finetuning on questions alone makes the Questioner forget how to respond.This ablation finetunes on questions but not sampled responses.Source: §6 · p. 8
• STaR-GATE finetuned on gold responses has slightly higher log probabilities but its win rate initially falls and then converges to 50%.Gold responses expose information from the Oracle that is absent from elicited preferences, which the authors expect can cause hallucination.Source: §6 · p. 8

Response Model

• Using QBASE to generate responses after Qη elicits the conversation yields a slight 57% win rate after two iterations, but returns to 50% at iteration three.The initial model was not trained to predict responses conditional on conversation history, unlike the STaR-GATE finetuned model.Source: §6 · p. 8

Self-Oracle

• A self-oracle setup using llama3-8b-instruct achieves a 65% win rate after one iteration, close to 66% when GPT-4 supplies the oracle.This exploratory result suggests a sufficiently strong Questioner may reduce reliance on a more powerful external Oracle.Source: §6 · p. 9

Limitations

• STaR-GATE depends on gold responses, so the current approach is not full self-play or self-improvement.A stronger Questioner could potentially serve as a self-oracle and eliminate external gold responses.Source: §7 · p. 9
• Replacing the training Roleplayer with mistral-7b-instruct or gemma-7b-instruct lowers win rates, motivating training with multiple Roleplayers for greater robustness.The observed degradation may partly reflect lower Roleplayer capability.Source: §7 · p. 9

Conclusion

• STaR-GATE significantly enhances effective dialog through targeted questioning.The conclusion frames better question asking as a route to better personalized responses.Source: §8 · p. 9
• Finetuning on self-generated questions and responses outperforms training on questions alone or on questions with gold responses.The authors attribute this to regularization that prevents forgetting how to answer and helps avoid hallucinations.Source: §8 · p. 9

• Gold-response log probabilities increase over iterations for STaR-GATE with positive control.Error bars represent ± SEM across held-out persona-task prompts.Source: Figure 6 · p. 13
• Gold-response log probabilities increase over iterations for STaR-GATE without regularization.The figure reports uncertainty using ± SEM across held-out persona-task prompts.Source: Figure 6 · p. 13
• The STaR-GATE-with-gold-response result is evaluated with ± SEM error bars across held-out persona-task prompts.Source: Figure 7 · p. 13

• The examples cover practical advice about reducing negative thoughts and preparing for a job interview.Source: §A.2 · p. 14
• The set includes questions about contacting a Congressman and choosing whether to lease a car.Source: §A.2 · p. 14
• Other examples ask for Spanish-learning courses, recent artificial-intelligence developments, and a book recommendation.Source: §A.2 · p. 14

Persona Generation

• The examples include personas named June, Reece, and Zara with distinct biographies, professions, interests, and personal circumstances.June owns a Brooklyn bistro, Reece is a retired criminal profiler and novelist, and Zara is an astrophysicist in the Andes.Source: §A.3.1 · p. 15

GPT-4 Prompts

• The persona-generation materials include dedicated system and task prompts, and the examples identify the roleplayer model as mixtral-8x7b-instruct.Source: Figure 8 · p. 16, Figure 9 · p. 16, Figure 10 · p. 17

Example Conversations

• For Zara’s interview task, the regularized response uses her astrophysics background, Andes location, sailing, and communication difficulty to tailor preparation advice.Source: §A.4.1 · p. 17, §A.5.1 · p. 19
• The regularized setup asks about background and interests, then uses the elicited information to produce advice for an astrophysics research position and its collaborative culture.Source: §A.5.1 · pp. 18, 20
• The unregularized setup asks follow-up questions but is described as highlighting an inability to provide a response, whereas the win-rate response supplies personalized interview guidance.Source: §A.5.1 · pp. 20–21