Source-linked AI summary
Hybrid Panels: Toward Human-AI Collaboration in Survey Research
Julia Romberg, Tobias Gummer, Gabriella Lapesa, Tanja Kunz, Claudia Wagner
TL;DR
Population surveys face persistent problems including declining response rates and nonresponse bias, while synthetic panels raise concerns about transparency and readiness. This research note defines hybrid panels, outlines a framework, and pilots one combining human participants with LLMs. The discussion argues that selective integration and iterative validation can strengthen validation, simulation quality, transparency, accountability, and trust, while respondent effort may decrease as LLM alignment improves.
Problem
Population surveys face declining response rates and nonresponse bias, while fully synthetic approaches raise concerns about transparency, accountability, ethical compliance, and readiness.
Method
The paper defines a hybrid panel as a longitudinal survey infrastructure combining human participants and LLMs, and outlines a framework for iterative development, validation, and selective integration.
Results
Selective integration and iterative feedback strengthen validation, improve simulation quality, promote transparency and accountability, and may reduce respondent effort as LLMs become more aligned with humans.
Takeaways & Limitations
Hybrid panels offer a practical framework for combining human participation with AI in survey research while retaining human involvement for more complex or less predictable cases.
Takeaways & Limitations
Participants must clearly consent to AI partially taking over aspects of the survey.
Abstract
from arXiv · showhide
Large-scale population surveys are essential for generating robust social and scientific insights, yet they face significant challenges, including declining response rates, increasing data collection costs, long delays between data collection and data provision, and the risk of nonresponse bias. Advances in artificial intelligence (AI) have opened up new opportunities for AI-supported survey infrastructures where the goal is to overcome these challenges without limiting the data quality. A promising AI-enabled survey infrastructure for which we build a first pilot is a hybrid panel. A hybrid panel is a longitudinal AI-enabled survey which allows to iteratively improve the alignment between large language models (LLMs) and the population they aim to simulate and use the errors to inform the design and implementation of the next survey wave (e.g., inform the participant recruitment, assignment of questions to participants). It incorporates both human participants and LLMs as fundamental elements of its design. In this research note, we introduce the concept of a hybrid panel by providing a definition and outlining an overarching framework, spanning data collection to data validation. We detail results from a first pilot study to illustrate (open) challenges that we identify for hybrid panels.
1 Introduction
Population surveys provide essential longitudinal evidence but face declining response rates, rising costs, delays, and nonresponse bias. The paper proposes hybrid panels that combine human participants and LLMs within an iterative infrastructure for developing and testing AI-enabled survey research.
- Motivation: Population surveys support measuring attitudes, values, and behavior over time, using established samples, standards, protocols, and quality controls.These infrastructures provide methodological support and access to already recruited probability-based samples.
- Challenges: Declining response rates, increasing data collection costs, long provision delays, and nonresponse bias threaten large-scale survey research.The bias risk is particularly relevant for important population subgroups and hard-to-reach populations.
- AI-enabled survey research: AI applications span the survey lifecycle, including questionnaire design, interviewing, and automated coding of open-ended responses.The paper argues that realizing these applications also requires attention to the infrastructures embedding them.
- AI-enabled survey research: Synthetic panels may enable faster and less costly data collection but face severe challenges involving transparency, accountability, ethical compliance, and readiness for full synthesis.These concerns are especially salient because scientific survey research requires open science, reproducibility, and independent data-quality evaluation.
- Hybrid panel: A hybrid panel is a longitudinal survey infrastructure combining human participants and AI models, with human responses and AI inputs iteratively informing development and validation.The design can use planned missingness, AI augmentation, or human validation of AI-generated responses, while comparing AI outputs with repeated human data.
- Contribution: The research note outlines a hybrid-panel framework and reports an ongoing pilot covering human participant recruitment and associated methodological and practical challenges.The framework is intended to support systematic development and testing of AI-enabled survey infrastructures.
2 Implementing a Hybrid Panel for Social Science Research
The hybrid panel combines human participants and AI models in a longitudinal survey infrastructure, enabling them to contribute, supplement, and validate one another’s input over time. The framework spans participant recruitment, LLM selection and calibration, adaptive data collection, and iterative validation, while the pilot identifies participation and bias challenges.
- Definition: A hybrid panel is a longitudinal group of human participants and AI models that repeatedly contribute to surveys or annotation tasks over time.The design treats humans and AI as collaborators rather than replacing human participants.
- Definition: Human responses or annotations can train, calibrate, and validate AI models that subsequently support survey completion or annotation.AI-generated responses can also be iteratively validated against human input.
- Adaptive data collection: Planned-missingness designs can shorten each wave while rotating modules across waves and using AI to answer missing items for later validation.Over time, respondents may answer all modules, allowing comparison with AI-generated answers.
- Potential contributions: Hybrid designs may redistribute data-collection resources toward hard-to-reach populations and sharpen concepts such as imputation, simulation, and data quality.The framework positions hybrid panels between human-only and AI-only panels while advancing LLM alignment and survey research.
- Annotation: The framework extends traditional survey tasks to annotation, capturing how respondents apply abstract concepts to concrete data snippets.Annotation can provide behaviorally grounded representations and may be more engaging than scales or open-ended answers.
3 Discussion
The discussion presents hybrid panels as a framework for integrating human participants and AI in survey research, with iterative human validation and selective delegation. It also identifies unresolved challenges involving simulation quality, recruitment, representation, and evaluation.
- Research agenda: AI-based survey research remains at an early stage and requires a long-term research agenda spanning human subjects, AI models, and continuous evaluation.The authors emphasize that standards for data quality and robust generalization require community involvement and diverse perspectives.
- Open challenges: Accurately reproducing genuine human behavior remains difficult, and discrepancies from ground truth are consequential when survey results inform the public or policymakers.Recruiting hard-to-reach and underrepresented groups remains an open challenge, although hybrid panels could augment their responses if sufficient data support alignment and validation.
- Selective integration: Hybrid panels selectively combine human responses with LLM estimators, delegating predictable subgroup or question responses while retaining complex or less predictable cases for humans.This design treats AI as complementary to human participants rather than as their replacement.
- Validation: Human-in-the-loop validation enables iterative feedback on AI performance, strengthens simulation quality, and supports transparency, accountability, and trust.Human perspectives are especially important when modeling opinions, attitudes, or values for which ground truth is typically unavailable.
- Governance: Direct human involvement can give participants greater control over the data AI may learn from and the responses it may impute.The framework therefore addresses both technical integration and participant preferences and rights.
- Respondent burden: As LLMs become more aligned with human respondents, lower respondent effort could improve participation, measurement error, and panel retention.Validating proposed responses may require less effort than generating original responses, particularly for complex free-text items; initially, all AI-generated responses should receive human validation.
- Open challenges: Hybrid panels may augment human responses from small or hard-to-reach subgroups when recruitment provides enough data and participants for model alignment and validation.The authors frame this possibility as conditional on successful implementation and investment of survey effort and budgets.
Ethics Statement
The ethics discussion addresses risks of using hybrid panels, including unclear participant consent, model inaccuracies and bias, and possible effects on public trust. It presents the infrastructure as a response to survey challenges while emphasizing ongoing ethical oversight and validation needs.
- Current models can produce hallucinated answers and uncontrolled or hidden bias, limiting the use of synthesized results for societal or policy questions.
- Using synthesized results may diminish public acceptance and reduce trust in research, although the discussion does not quantify these effects.
- The proposed approach targets declining response rates, rising data collection costs, and nonresponse bias while preserving the underlying human subject of observation.
- The hybrid panel is framed as a community-driven effort involving ethics-board monitoring, shared evaluation, and empirical reflection on acceptable LLM robustness.
- Participants may not fully understand that AI could partially take over their answers, potentially reducing their compensation.
- Participants who are accurately predictable on one attribute may still be recruited for validation, other constructs, or annotation, with fairness in request allocation required.
4 Appendix A
The appendix describes a German-language web survey of 1,201 German residents recruited through Prolific using a non-probability quota-based sample. It reports recruitment, data-quality procedures, analyses, and a key sample-composition limitation.
- The questionnaire used nine questions described with instructions and response options in Appendix 5.1, and analyses used Python with pandas and scipy.
- Data collection used an incentivized German-language web survey conducted from June 1 through June 6, 2026, without weighting.
- Quality procedures required a 95% Prolific acceptance rate, monitored response times, screened free-text answers for bot usage, and limited respondents to one completion.
- 6,032 eligible participants were invited; 1,260 started, 1,201 completed, 45 exited early, and 14 were filtered out.
- The study recruited 1,201 completed responses from German residents fluent in German through Prolific using a non-probability method.
- Because recruitment was non-probability based, the Prolific sample is likely biased toward young, left-leaning, and highly educated respondents.
5.1 Survey Questions
The survey asks about Prolific workers’ use of AI assistance, including its purposes and framing, and includes an open-ended question about reasons for using AI.
- Respondents report whether they usually use AI support in their Prolific work and can select among several usage categories.
- Four question framings present AI use as common, as one of several online-work aids, or as relevant to accurately describing respondents’ work.
- Instructions state that honest answers do not affect participation or compensation and are not reported to Prolific.
- The response options distinguish AI use for improving text, generating complete answers, selecting predefined answers, or other purposes.
- The survey also includes an open-ended question asking respondents why they usually use AI assistance.
5.1.3 Panel consent
The consent materials introduce panels and hybrid panels, then ask about participation under differing AI, data-sensitivity, compensation, and privacy conditions. Hybrid panels combine human responses with AI assistance, information supplementation, or delegated answering.
- A panel repeatedly surveys the same people, while a hybrid panel uses AI to help answer, supplement information, or answer instead of participants.
- Participants are asked how likely they would be to join a conventional panel and a hybrid panel.
- The hybrid-panel description states that AI processes information from earlier answers and learns from participants’ responses.
- The scenarios vary data sensitivity from general information such as age and hobbies to sensitive information about health, sexuality, religion, and political views.
- They also vary AI autonomy: low autonomy requires direct instructions and participant approval, whereas high autonomy lets AI answer and choose questions independently.
- Compensation conditions range from 13.90 euros per hour, described as minimum wage, to 20.85 euros per hour, described as 50% more.
- The consent text promises high data-protection standards, information about storage and use, and the ability to end participation and request data deletion.
5.1.6 Attitudes towards artificial intelligence scale (ATTARI-12)
The ATTARI-12 section introduces artificial intelligence, asks respondents to rate agreement with attitude statements, and organizes the scale by affective, cognitive, and behavioral facets with reverse-coded negative items.
- Concept and instructions: Respondents indicate how strongly they agree with statements about their attitudes toward artificial intelligence using a five-point agreement scale.The response options range from completely disagree to completely agree, with a neutral midpoint.
- Concept and instructions: Artificial intelligence is described as technology that can perform tasks usually requiring human intelligence, including perceiving, acting, learning, and adapting.It may be part of computers or online platforms and may also appear in devices such as robots.
- Scale structure: Negative affective, cognitive, and behavioral items are reverse-coded.Examples include negative emotions, perceiving artificial intelligence as causing problems, and preferring technologies without artificial intelligence.
5.1.7 Political leaning
The survey asks respondents to place themselves on the political left–right spectrum.
- Political self-placement: The item is presented as a survey instruction asking respondents to identify their own political location.
5.1.8 Gender
The survey asks respondents to report their gender and provides male, female, and diverse response options.
- Gender item: Respondents are asked directly which gender they have.
- Gender item: The gender question is implemented as a categorical self-report item with three listed response categories.
- Gender item: The response options are male, female, and diverse.