PaperlayerView exact revision on arXiv

Source-linked AI summary

SCBO: Semantically Coherent Batching and Ordering for LLM-Based Social Surveys

Yuanzi Li, Lingjie Wang, Zihang Tian, Lei Wang, Xu Chen

arXiv:2609.35250v1cs.CLcs.CY

TL;DR

One-question-per-prompt survey prediction repeatedly encodes shared context, limits each target’s references, and prevents answers from conditioning on earlier predictions. SCBO combines semantic batching and curriculum-based ordering to address these limits; across four datasets and four LLMs, it generally improves accuracy while reducing token costs and inference time.

  • Problem

    One-question-per-prompt prediction repeatedly encodes shared context, restricts each target to a narrow reference set, and predicts answers independently.

    Source: §1 · p. 2

  • Method

    SCBO extracts compact question representations, groups semantically cohesive questions with a shared reference bank, and orders questions and references to align with autoregressive prediction.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    SCBO generally outperforms the non-batched baseline across diverse datasets and LLMs, reducing token costs by over 50% and accelerating inference by 3.3–14.0×.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    SCBO provides a training-free approach that generally improves accuracy while reducing token consumption and inference time in the evaluated survey-prediction setting.

    Source: §1 · p. 3

    Model-based predictions may inherit biases from survey data and LLMs and should not substitute for real respondents or be used for high-stakes decisions about individuals or demographic groups.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) offer a scalable way to simulate survey respondents using demographic profiles and observed reference responses. However, the conventional approach of predicting one question per prompt repeatedly encodes the same context, limits each target to a narrow set of reference responses, and prevents later predictions from using information in earlier answers. Predicting multiple questions in one prompt can reduce these costs, share a broader pool of references, and let later predictions build on earlier ones. This requires forming coherent batches, selecting shared references, and ordering questions and references effectively. We propose Semantically Coherent Batching and Ordering (SCBO), a training-free framework that addresses these challenges. SCBO first uses an LLM to extract compact semantic representations from survey items and filter out template noise. It then groups related questions into batches and builds a shared reference bank using target-specific retrieval and centroid-based completion. Finally, it orders target questions from easy to hard and arranges references according to their semantic alignment with those questions. Experiments on four large-scale survey datasets and four LLMs show that SCBO substantially reduces token consumption and inference time while generally improving prediction accuracy over a non-batched baseline. Code is available at https://anonymous.4open.science/r/SCBO-41D8.

1 INTRODUCTION

SCBO batches survey questions to reduce repeated context encoding, share a broader pool of reference responses, and let later predictions use earlier answers. Across four survey datasets and four LLMs, it reduces token cost and inference time while generally improving accuracy over one-question-per-prompt prediction.

1 INTRODUCTION

  • The one-question-per-prompt approach repeats shared context, limits each target to a small set of references, and predicts answers without conditioning on other answers.The repeated task instruction and respondent profile make this approach expensive; the paper cites WVS and ANES exhaustive-prediction token estimates in Table 1.Source: §1 · p. 2, Table 1 · p. 2
  • Batching makes questions share references and lets later predictions condition on earlier answers, but unrelated questions can make a shared reference set unhelpful.Because generation is autoregressive, the paper proposes ordering reliable, easier predictions before harder ones and aligning references with the target sequence.Source: §1 · p. 2
  • SCBO uses semantic batching to group cohesive questions and select a shared reference bank, then curriculum-based ordering to arrange questions and references.Question Instantiation extracts compact semantic representations to filter template noise; easier questions are placed earlier so their predictions may provide context for harder questions.Source: §1 · p. 2
  • Over 50% lower token cost and 3.3–14.0× faster inference, with generally higher accuracy than the one-question-per-prompt baseline.Experiments cover four large-scale survey datasets and four LLMs.Source: §1 · p. 3

2 PROBLEM FORMULATION: LLM-BASED SOCIAL SURVEYS

The paper formulates survey prediction as predicting a respondent’s answers to unasked questions from their profile and previously recorded responses. It contrasts separate prompts with shared-bank batches, which change context reuse, reference access, and answer dependencies.

2 PROBLEM FORMULATION: LLM-BASED SOCIAL SURVEYS

  • For respondent u, the task is to predict answers to N unasked target questions using a textual demographic profile and previously answered survey questions with their recorded answers.The observed question-answer pairs form the candidate in-context examples.Source: §2 · p. 3
  • The baseline predicts each target answer from its own prompt, combining the task instruction, respondent profile, retrieved references, and target question.Predictions are evaluated against withheld answers using ACC, MAE, and F1.Source: §2 · p. 3
  • The one-question-per-prompt formulation repeats shared context across N predictions, restricts each target to its separately retrieved references, and predicts answers independently.On WVS, shared context averages 491.1 tokens per prediction versus 70.6 tokens for the target question itself.Source: §2 · p. 3
  • Batches share a reference bank across their questions, with budget m = nC, and order both questions and references within the prompt.The formulation partitions N targets into K = N/C disjoint batches of size C.Source: §2 · p. 3
  • Batching reduces repeated shared-context tokens by a factor of C, broadens each target’s access to respondent history, and lets later answers condition on earlier ones.The formulation’s partition, shared bank, and question/reference orderings determine which questions are grouped, which history is shared, and what each answer is conditioned on.Source: §2 · pp. 3–4

3 THE SCBO FRAMEWORK

SCBO maps survey items to semantic representations, forms batches with shared reference banks, and orders target questions and references to address the choices introduced by batched prediction.

3 THE SCBO FRAMEWORK

  • SCBO fixes semantic batches and shared reference banks, then sets question and reference orderings for batched prediction.The framework addresses batch composition and ordering as the three quantities left unspecified by the batched prediction formulation.Source: §3 · p. 4

3.1 PREPROCESSING: QUESTION INSTANTIATION FOR SEMANTIC DENOISING

  • Each survey item is reduced to its topic, core intent, and key entities, removing template wording while retaining topical scope.The resulting embeddings are L2-normalized for subsequent clustering, retrieval, and ordering.Source: §3.1 · p. 4

3.2 SEMANTIC BATCHING AND SELECTION

  • Target questions are clustered by semantic proximity, and each batch shares a bank initialized from per-question top-n references.When overlapping references leave bank slots open, centroid-based retrieval fills them; each target is also associated with its relevant subset of the shared bank.Source: §3.2 · p. 5

3.3 CURRICULUM-BASED ORDERING

  • Questions are ordered from easy to hard using reference similarity, while references are arranged to align with that target sequence.This places references most relevant to earlier targets first, so each question is preceded by its needed evidence.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate SCBO across four survey datasets and four LLMs, examining predictive effectiveness, efficiency, batch-size sensitivity, and component contributions.

4.1 EXPERIMENTAL SETUP

  • Evaluation covered WVS, GSS, ANES, and BSA using four LLMs; effectiveness used ACC, MAE, and F1, while efficiency used TPQ and SPQ.The non-batched baseline used one question per prompt, and comparisons used the same reference budget.Source: §4.1 · p. 6

4.2 IMPLEMENTATION DETAILS

  • Implementation instantiated questions to filter template noise, embedded questions for clustering and retrieval, and formed fixed-size batches using KMeans++ followed by capacity-constrained assignment.The shared instantiation and embedding costs were excluded from reported efficiency metrics.Source: §4.2 · p. 6

4.3 EFFECTIVENESS OF SCBO (RQ1)

  • Across 48 dataset–model–budget configurations, SCBO reduced MAE in all, improved ACC in 46, and improved F1 in 45.Table 2 compares SCBO with the non-batched baseline across datasets, models, and reference budgets.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4 EFFICIENCY OF SCBO (RQ2)

  • Token-cost reductions ranged from 30.1% to 71.1% (about 58.0% on average), while inference speedups ranged from 3.3× to 14.0×.SCBO reduces both token cost and inference time across evaluated configurations.Source: §4.4 · pp. 7–8, Table 3 · p. 8

4.5 IMPACT OF BATCH SIZE (RQ3)

  • Across tested batch sizes, SCBO outperformed the non-batched baseline in accuracy and token cost; TPQ fell monotonically, while ACC peaked around sizes 8–16 for DeepSeek-V4-Flash.At very large batch sizes, accuracy plateaued or declined slightly; moderate sizes offered the best reported balance.Source: §4.5 · pp. 8–9

4.6 ABLATION STUDY (RQ4)

  • SCBO improved performance over all ablation variants, with removing semantic batching causing the largest decline, followed by removing ordering; removing instantiation caused a smaller decline.The ablation evaluated WVS across three models and three reference budgets.Source: Figure 3 · p. 9, §4.6 · p. 9

5 CONCLUSION

SCBO combines semantic batching and curriculum-based ordering to address repeated context encoding, narrow reference pools, and isolated predictions in LLM-based survey prediction. Across diverse datasets and LLMs, it generally outperforms the non-batched baseline while reducing token costs and inference time.

5 CONCLUSION

  • SCBO batches questions to amortize shared context, share a broader reference pool, and let earlier answers inform later predictions.It uses compact question representations filtered for template noise, semantic batching with a shared reference bank, and curriculum-based ordering.Source: §5 · p. 9
  • SCBO generally outperforms the non-batched baseline, reduces token costs by over 50%, and accelerates inference by 3.3–14.0×.The experiments span diverse datasets and LLMs.Source: §5 · p. 10
  • Future work will explore model-adaptive batching based on feedback from the LLM rather than heuristic clustering.Source: §5 · p. 10

AI USE STATEMENT

The statement identifies AI models used in question instantiation, semantic encoding, and response generation. It also reports AI assistance with translation and editing while assigning responsibility for the work and its outputs to the authors.

AI USE STATEMENT

  • GPT-5.5 extracts compact topic, intent, and entity representations from raw survey items for Question Instantiation.Source: AI USE STATEMENT · p. 10
  • text-embedding-3-small encodes representations into semantic vectors used for clustering, reference retrieval, and ordering.Evaluated LLMs generate predicted responses under both the non-batched baseline and SCBO settings.Source: AI USE STATEMENT · p. 10
  • Generative AI tools assisted with English translation and language editing, and the authors reviewed and verified all AI-assisted content.The authors take responsibility for the final text, claims, code, figures, and reported results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The study analyzes publicly released survey data using de-identified records and reports aggregate results. It frames model-based predictions as methodological research, not substitutes for real respondents or tools for high-stakes decisions about people or groups.

ETHICS STATEMENT

  • The study conducts secondary analysis of publicly released WVS, GSS, ANES, and BSA datasets using de-identified records.Source: ETHICS STATEMENT · p. 10
  • No new human participants were recruited, and results are reported only in aggregate.Source: ETHICS STATEMENT · p. 10
  • Because predictions may inherit biases from survey data and LLMs, they should not replace real respondents or inform high-stakes decisions about individuals or demographic groups.The study evaluates methodological feasibility and efficiency of LLM-based social survey prediction.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points to the framework, experimental setup, procedures, and prompts, with source code and supporting materials available online. The related work situates SCBO among LLM social simulation, survey respondent simulation, and batch prompting.

REPRODUCIBILITY STATEMENT

  • The framework, experimental setup, capacity-constrained batching procedure, dataset and pool construction, evaluation statistics, and prompt templates are documented across the paper and appendices.Source code and supporting materials are available at the stated project repository.Source: REPRODUCIBILITY STATEMENT · p. 10

A RELATED WORK

  • LLM-powered social simulation replaces hand-coded behavioral rules with language-grounded agents, while this work focuses on predicting individual survey answers as a foundation for downstream collective modeling.Source: A RELATED WORK · p. 13
  • Prior LLM survey respondent methods condition predictions on demographic profiles and observed references, while most use one question per prompt and incur high token costs at scale.Source: A RELATED WORK · p. 13
  • Unlike batch-prompting methods for independent instances, SCBO uses survey questions' semantic relatedness and shared respondent evidence to build a broader reference bank and order questions from easy to hard.Prior batch prompting research reports sensitivity to instance position and order; Lin et al. (2024) propose permutation, ensembling, and early stopping.Source: A RELATED WORK · p. 13

B IMPLEMENTATION DETAILS

The implementation handles capacity-constrained batches, including a smaller final batch when the target count is not divisible by capacity. It defines respondent- and batch-specific limits for batches and shared reference banks.

B.1 NOTATION AND BOUNDARY CASES

  • The main text assumes the target-question count is divisible by capacity; otherwise, the final batch can contain fewer questions.The first Ku −1 batches have capacity C, while the final batch has capacity Nu −C(Ku −1).Source: B IMPLEMENTATION DETAILS · p. 13, §B.1 · p. 14

B IMPLEMENTATION DETAILS

  • KMeans++ semantic centers are expanded into capacity-sized slots, and the Hungarian algorithm assigns questions one-to-one to form batches.Source: Figure 4 · p. 14

B.1 NOTATION AND BOUNDARY CASES

  • The number of batches and shared reference-bank budget depend on respondent and batch; C is maximum batch size, n the per-question reference budget, and r the retrieval depth, with r ≥n.Source: §B.1 · p. 14
  • If C ≥Nu, all targets form one batch; when Hu < mk, the reference bank contains at most Hu pairs, below its nominal budget.These boundary cases affect realized batch or bank size but do not alter the batching assignment.Source: §B.1 · p. 14

B.2 CAPACITY-CONSTRAINED SEMANTIC BATCHING

Capacity-constrained semantic batching approximates fixed-size clustering through respondent-specific KMeans centers and a linear assignment. The section also describes dataset-scale and evaluation-allocation figures and qualifies how inference time is measured.

B.2 CAPACITY-CONSTRAINED SEMANTIC BATCHING

  • KMeans++ centers are expanded into prescribed-capacity slots, and the Hungarian algorithm assigns questions to slots to produce capacity-constrained batches.With a respondent-specific seed, the procedure gives a deterministic capacity-constrained approximation to the fixed-size clustering objective.Source: §B.2 · p. 14
  • Figure 5 summarizes release scale, target questions allocated per respondent under reference budgets, and target pairs evaluated in each method–model run.WVS uses Wave 7; GSS, ANES, and BSA use their 2024 releases.Source: Figure 5 · p. 15

B.2.1 INFERENCE-TIME MEASUREMENT

  • SPQ is aggregate request latency amortized per target question, not end-to-end wall-clock time under parallel execution.Provider-side region, routing, network conditions, and server load are uncontrolled, so reported speedups apply to the authors’ API environment rather than guaranteeing provider-independent latency.Source: §B.2.1 · p. 15

C DATASET PROCESSING

The dataset pipeline standardizes four survey releases into respondent–question prediction records with disjoint reference and target pools. It filters items and responses, constructs respondent profiles, and assigns eligible respondents separate pools under each reference budget.

C DATASET PROCESSING

  • The pipeline standardizes records from WVS Wave 7, GSS 2024, ANES 2024, and BSA 2024, with compared methods using the same processed records and respondent-specific pool assignments.Source: §C.1 · p. 15, C DATASET PROCESSING · p. 15

C.1 SURVEY SOURCES AND RESPONSE PROCESSING

  • Only closed-ended, single-choice questions with complete option sets are retained; non-substantive response codes and questions without valid answer sets are removed.Retained choices are mapped to consecutive question-specific indices, and unmappable records are discarded.Source: §C.1 · pp. 15–16

C.2 RESPONDENT PROFILES

  • Background and demographic items become third-person natural-language respondent profiles and are excluded from both reference and target pools.Respondents without an available profile are excluded.Source: §C.2 · p. 16

C.3 REFERENCE–TARGET POOL CONSTRUCTION

  • For each budget, valid non-profile responses are randomly ordered; the first Td,k records form targets and the following kTd,k records form references.A respondent needs at least (k + 1)Td,k valid responses, and target responses are never reused as references.Source: §C.3 · p. 16
  • Eligibility can differ across reference budgets, and SCBO retrieves references only from the respondent’s reference pool, embedding question text and options but not respondent answers.Source: §C.3 · p. 16

C.4 EVALUATION DATA STATISTICS

Evaluation uses validation-selected batch sizes and a shared test set. The prompt templates instantiate survey questions into compact semantic representations, present batch predictions with a shared reference bank, and require structured integer-index outputs.

C.4 EVALUATION DATA STATISTICS

  • Respondents are split 15% validation and 85% test; batch size is selected by validation ACC for each dataset–model–budget configuration, and metrics use the shared test set.Figure 5(c) reports test target pairs per method–model run.Source: §C.4 · p. 16

D.1 QUESTION INSTANTIATION PROMPT

  • Question Instantiation prompts an LLM to extract each item’s topic, intent, and entities while removing generic survey wording and repeated answer-scale phrasing.The resulting instance field is embedded; the input includes the question identifier, text, and answer options.Source: §D.1 · pp. 16–17

D.2 BATCH PREDICTION PROMPT

  • The batch prompt supplies a respondent profile, a shared bank of that person’s actual answers, and ordered target questions with relevant-reference IDs.Listed relevant references are primary evidence for each target, while the rest of the bank and profile are secondary context.Source: §D.2 · pp. 17–18
  • The prediction prompt requires valid JSON with exactly one answer object per sample ID and each answer as a zero-based integer option index.Source: §D.2 · p. 17

Related papers

Loading 2609.35250v1…