Source-linked AI summary
Whom to Query for What: Adaptive Group Elicitation via Multi-Turn LLM Interactions
Ruomeng Ding, Tianwei Gao, Thomas P. Zollo, Eitan Bachmat, Richard Zemel, Zhun Deng
TL;DR
Surveys and collective assessments must reduce uncertainty about latent group properties while limiting costly questions and responses amid missing data. This paper jointly selects questions and respondents using LLM-based expected information gain and heterogeneous GNN propagation, consistently improving inference across three opinion datasets, including a >12% relative improvement on CES at a 10% respondent budget.
Problem
Surveys and collective assessments infer latent population properties under costly questioning and responding, while missing data and breakoffs complicate estimation.
Method
The framework jointly selects questions and respondent subsets under budgets, combining LLM-based expected information gain with heterogeneous GNN propagation for imputation and respondent selection.
Results
The method consistently improves inference accuracy and calibration across multiple real-world opinion datasets under tight budgets, including a >12% relative improvement on CES at a 10% respondent budget.
Takeaways & Limitations
Effective elicitation depends on question choice, respondent selection, and propagation of partial information across the population.
Takeaways & Limitations
The experiments pretrain and meta-learn on South-region respondents and evaluate adaptive inference on West-region respondents, leaving deployment dependent on this regional generalization setting.
Abstract
from arXiv · showhide
Eliciting information to reduce uncertainty about latent group-level properties from surveys and other collective assessments requires allocating limited questioning effort under real costs and missing data. Although large language models enable adaptive, multi-turn interactions in natural language, most existing elicitation methods optimize what to ask with a fixed respondent pool, and do not adapt respondent selection or leverage population structure when responses are partial or incomplete. To address this gap, we study adaptive group elicitation, a multi-round setting where an agent adaptively selects both questions and respondents under explicit query and participation budgets. We propose a theoretically grounded framework that combines (i) an LLM-based expected information gain objective for scoring candidate questions with (ii) heterogeneous graph neural network propagation that aggregates observed responses and participant attributes to impute missing responses and guide per-round respondent selection. This closed-loop procedure queries a small, informative subset of individuals while inferring population-level responses via structured similarity. Across three real-world opinion datasets, our method consistently improves population-level response prediction under constrained budgets, including a >12% relative gain on CES at a 10% respondent budget.
1 Introduction
Adaptive group elicitation addresses costly, incomplete collective assessments by jointly selecting questions and respondents while using population structure to infer unobserved responses. The framework combines LLM-based information-gain scoring with heterogeneous GNN propagation and reports consistent gains under constrained budgets.
- Motivation: Surveys infer latent population properties from limited questions and respondents, but cost, respondent burden, and breakoffs produce sparse, partially observed data.These costs include interviewer time and participation incentives.
- Research gap: Existing LLM elicitation methods mainly optimize which questions to ask with a fixed respondent pool, leaving respondent selection and population structure underused.The paper identifies completed responses as a dominant deployment bottleneck.
- Framework: Adaptive group elicitation jointly selects questions and respondents across multiple rounds to reduce uncertainty about a latent group quantity under query and participation budgets.The framework uses an LLM to score candidate questions by expected information gain and a heterogeneous GNN to impute responses and refine respondent similarity.
2 Preliminaries
The preliminaries frame elicitation as predictive inference about latent entities and introduce heterogeneous graphs for relational reasoning under sparse observations. They also establish notation and entropy-based uncertainty measures used by the framework.
- De Finetti’s predictive perspective: A generalized de Finetti perspective represents uncertainty about future observable responses through an underlying latent entity governing behavioral patterns.Predictive distributions are updated as new observations arrive.
- De Finetti’s predictive perspective: Under a martingale condition, predictive distributions converge to a limiting measure from which the latent entity can be recovered.The paper notes that this assumption is discussed and justified later.
- Relational modeling: Heterogeneous GNNs model multiple node and relation types, propagating information across them to jointly inform respondent representations and predictions under sparse observations.Related work represents individuals, subgroups, and question-option choices in a heterogeneous graph.
- Notation and uncertainty: The notation defines graphs, neighbors, nonadjacent nodes, latent entities, candidate queries, responses, and member-response vectors.Entropy and conditional entropy provide the uncertainty quantities used for predictive inference.
3 Group Adaptive Elicitation Framework
The framework combines individual predictive inference with graph-based relational imputation in a closed loop that adaptively selects both questions and representative respondents. An LLM estimates uncertainty and expected information gain, while a heterogeneous GNN propagates observations to improve missing-response inference.
- Problem setting: Sparse group elicitation seeks latent properties from limited queries and responses collected under time, cost, privacy, and participation constraints.Political surveys are given as an example of this setting.
- Overview: At each iteration, the framework jointly selects a question and respondent subset using interaction history, inferred latent entities, and learned relational structure.The goal is to reduce group-level uncertainty through informative queries directed to representative respondents.
- Individual prediction: A meta-trained LLM predicts individual responses from interaction histories and estimates expected information gain for candidate queries.Autoregressive likelihood training supports predictive inference without a parametric prior over the complex latent space.
- Graph imputation: A heterogeneous GNN propagates observed responses and attributes across member, feature, and choice nodes to impute missing responses and update group uncertainty.The graph uses link prediction with relation-specific message passing and masked member-choice edges.
- Test-time adaptive inference: At test time, held-out questions evaluate inference while the system selects questions and respondents by expected information gain from current interaction histories.The graph is initialized from demographic attributes, and member embeddings support subgroup selection through representative cluster centers.
4 Theoretical Results
The paper provides theoretical support for greedy joint question–respondent selection, establishing near-optimality under standard structural assumptions. It also justifies the predictive training objective and explains why greedy selection is preferable to costly multi-step planning for real-time elicitation.
- The optimal joint selection sequence is computationally intractable because it searches combinatorially over respondents and queries.
- The two-stage greedy algorithm selects respondent–question pairs by maximizing immediate information gain from the current interaction history.
- Under submodularity, monotonicity, and alignment assumptions, joint and two-stage greedy strategies achieve solutions within a constant factor of the optimum.
- The utility function measures information gained about group latent properties, including expected information gain, with marginal gain defined by the change in utility after adding new queries.
- Greedy selection is near optimal while more complex multi-step planning provides only marginal empirical improvements at substantial computational cost.
- The predictive training objective is justified through a de Finetti-style perspective linking latent entities to predictive distributions over induced observations.
5 Experiments
The experiments evaluate adaptive elicitation across three opinion datasets, geographic splits, respondent budgets, and ablations of selection, imputation, and planning. The method consistently improves target-question inference, with gains concentrated among sensitive respondents and graph-based imputation contributing substantially.
- Experimental Setup: Evaluation uses CES, OpinionQA, and Twin-2k with South-region meta-training and West-region adaptive inference.Methods select questions and respondents over multiple rounds under fixed observation budgets, evaluating accuracy on disjoint target questions.
- Overall Gains: Across all datasets and respondent budgets, Ours consistently outperforms existing baselines.At a 10% respondent budget on CES, relative gains over the strongest baseline range from 17.1% at round 1 to 12.6% by round 4.
- Sensitivity Analysis: Up to 20% relative recovery occurs for highly sensitive respondents under a 50% budget across both reported datasets.The selection strategy concentrates limited observations on respondents for whom ground-truth responses yield the largest performance improvements.
- Ablations: Greedy query selection achieves higher accuracy than random querying, with gaps widening over interaction rounds across respondent budgets.The gains indicate reliable benefits from selecting more informative questions once imputation is enabled.
- Ablations: GNN-based imputation yields substantially larger gains than disabling imputation across respondent budgets and interaction rounds.It propagates partial observations across the group, enabling faster improvement and more sustained gains.
- Ablations: Strategically observing a small number of respondents produces consistent gains beyond imputation alone.The comparison indicates complementary roles for direct observation and graph-based propagation.
- Multi-Step Planning: Multi-step planning provides at most marginal, unstable gains over greedy selection while incurring substantially higher computational cost.Global-population differences are negligible, and sensitive-subset improvements largely disappear or reverse at the 50% budget.
6 Related Work
Related work spans graphical inference, predictive Bayesian perspectives, sequential elicitation, and LLM-mediated group interaction. This paper connects these strands through adaptive group elicitation with heterogeneous graph propagation.
- Inference for Graphical Models: Earlier graphical models, including CAR models, predict node outcomes using neighboring-node outcomes.These approaches model relational structure in data with inherent graphical properties.
- Predictive Inference: Predictive inference work links uncertainty about latent entities to forecasting induced observables, including through LLM-based Bayesian interpretations.The paper builds on de Finetti-style predictive perspectives and prior LLM elicitation work.
- Multi-Turn Elicitation: Multi-turn elicitation research uses sequential Bayesian experimental design to gather information about unobservable characteristics.Examples include student ability, political intentions, and patient health status.
- Group Interaction with LLMs: Prior LLM group-interaction frameworks target collaborative decision-making but do not directly provide decentralized predictions from partial observations.The paper addresses this setting with heterogeneous GNNs for group evaluation.
7 Conclusion
The paper concludes that effective adaptive group elicitation must jointly choose questions and respondents while propagating partial information across population structure. Its greedy strategy is theoretically near-optimal under stated assumptions.
- Conclusion: Adaptive elicitation jointly selects questions and respondent subsets under query and participation budgets.The objective is to reduce uncertainty about latent group properties.
- Conclusion: Graph-based imputation propagates partial observations across the population to infer missing responses.The framework combines adaptive query selection, group-relational respondent selection, and graph-based imputation.
- Assumptions: The utility function is assumed monotone and submodular, so adding observations does not decrease utility and marginal benefits diminish with history.A query-budget alignment condition excludes queries whose expected information gain is concentrated on arbitrarily small respondent subsets.
- Theoretical Guarantees: Under these assumptions, the paper establishes approximation guarantees for greedy and two-stage greedy selection.The guarantees are stated with constants independent of the respondent budget and interaction horizon.
A.2 Generalization of de Finetti Theorem and Related Justification
The appendix generalizes de Finetti-style predictive reasoning through conditionally identically distributed sequences and martingale predictive distributions. It establishes convergence and consistency results supporting uncertainty quantification and latent-entity recovery.
- Definitions: A conditionally identically distributed sequence has future observations that are identically distributed conditional on the past according to the current predictive distribution.This formulation supports martingale-based predictive inference beyond exchangeability.
- Martingale Justification: For exchangeable latent-entity models, predictive distributions form a martingale with respect to the observed-history filtration.The paper uses this property to justify its predictive-distribution assumption in survey and group-elicitation settings.
- Convergence: For a c.i.d. sequence, predictive distributions converge weakly almost surely to a limiting random probability measure.This extends de Finetti-style representation to the broader c.i.d. setting.
- Latent-Entity Recovery: When a latent entity governs data generation, the limiting distribution recovers the induced distribution associated with that entity.The result connects predictive convergence with latent-entity recovery.
- Copula-Based Scheme: Copula-based updates provide a constructive way to generate predictive sequences satisfying the martingale condition.The Gaussian copula is presented as a practical choice because of its parametric simplicity and flexibility in modeling dependence.
- Consistency: Under stated learning-rate, dependence, and initialization-tail conditions, the copula-updated predictive density is Hellinger consistent at the true density.Together with convergence results, this supports predictive sampling and uncertainty quantification.
B.1 Datasets
The evaluation uses three real-world opinion datasets with demographic attributes and multiple-choice responses, alongside constrained population-level inference and respondent-sensitivity analyses.
- Datasets: The study evaluates adaptive elicitation on CES, OpinionQA, and Twin-2k, spanning political attitudes, social values, and economic preferences.All datasets contain real human responses and support population-level inference under constrained observation budgets.
- Datasets: CES uses the 2024 wave and retains 3,326 respondents after filtering missing demographic or selected opinion-question values.The retained data include eight demographic variables.
- Datasets: OpinionQA draws on three American Trends Panel waves covering family, inequality, and political-typology topics.The panel is designed to be nationally representative of the U.S. population.
- Datasets: Twin-2k contributes 40 questions focused primarily on economic preferences and behavioral decision-making for latent-trait inference.The source is a four-wave, nationally representative U.S. panel fielded on Prolific.
- Metrics: Accuracy, perplexity, and Brier score evaluate discrete-response prediction, uncertainty, and squared probability error, respectively.Perplexity here measures uncertainty over discrete choices rather than standard token-level language-model perplexity.
- Respondent sensitivity: Respondent sensitivity measures the accuracy improvement from observing a respondent’s responses relative to group-level imputation alone.The analysis ranks respondents by this accuracy gap and studies top 50%, 30%, 10%, and 5% sensitivity tiers.
C.1 Calibration Results
Across CES and OpinionQA, adaptive elicitation produces progressively better-calibrated predictions across rounds and observation budgets, while indiscriminate propagation can degrade calibration.
- Calibration: Across CES and OpinionQA, the method achieves the lowest Brier scores and perplexities after the first round across 10%, 30%, and 50% budgets.The calibration gap widens as additional elicitation rounds are performed.
- Calibration: Meta-Greedy-Imp often shows worsening calibration with more rounds, particularly at low budgets, with the strongest degradation in perplexity.The passage attributes this pattern to overconfident predictions from uncertain or weakly informative propagated signals.
- Calibration: Group-relational respondent selection improves calibration by prioritizing highly informative respondents whose responses propagate reliably across the population.The reported trends are consistent across datasets and budgets.
C.2 Ablation on Respondent Selection
Respondent-selection ablations show that group-relational selection outperforms random selection at matched budgets, especially for highly sensitive respondents.
- Selection by sensitivity: Group-relational selection consistently outperforms random selection at the same budget on round-4 CES and OpinionQA accuracy.The comparison is stratified by respondent sensitivity, from Global through Extreme tiers.
- Selection by sensitivity: Under a 50% budget, gains become more pronounced as respondent sensitivity increases, concentrating in the Hard and Extreme tiers.These tiers capture respondents whose observed responses yield the largest accuracy improvements over imputation-only prediction.
- Selection by sensitivity: Hard and Extreme tiers recover a substantial fraction of full-observation performance through group-relational selection.Global and Broad tiers show smaller gains because many low-sensitivity respondents are already well explained by group-level propagation.
C.3 Results with Different Base Model
Using the smaller Llama-3.2-1B backbone, the method retains its advantage across datasets, budgets, and rounds, with early gains especially under low budgets.
- Smaller backbone: With Llama-3.2-1B, the method consistently outperforms all baselines across CES and OpinionQA, query budgets, and interaction rounds.The observed trends resemble those obtained with the Llama-3.1-8B backbone fine-tuned with LoRA.
- Smaller backbone: The method achieves rapid gains in early rounds, especially under low-budget settings, with the smaller backbone.The passage links these trends to group-relational respondent selection and propagation.
- Smaller backbone: Full fine-tuning of Llama-3.2-1B yields performance generally comparable to LoRA-based fine-tuning of Llama-3.1-8B.
C.4 Results with Different Region
Under geographic distribution shift, the adaptive elicitation method remains effective on regions excluded from meta-training. It consistently outperforms baselines across query budgets and achieves substantial early-round gains, including under low-budget constraints.
- The model is trained on South respondents and evaluated on held-out West and Midwest regions.
- Across both held-out regions, adaptive elicitation consistently outperforms baseline approaches across interaction rounds and query budgets.
- Adaptive elicitation achieves substantial gains in early rounds, even under low-budget constraints.
- Relative improvements over Meta-Random, Meta-Greedy, and Meta-Greedy-Imp remain stable despite regional differences in demographic composition and opinion distributions.
- The results indicate that group-relational respondent selection and propagation transfer effectively across geographic regions.