PaperlayerView exact revision on arXiv

Source-linked AI summary

Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

Patrik Wolf, Thomas Kleine Buening, Andreas Krause, Celestine Mendler-Dünner

arXiv:2607.15277v1cs.CL

TL;DR

LLMs are often treated as conditional estimators, but whether their outputs obey basic aggregation identities remains unclear. This paper tests statistical self-consistency by aggregating persona-prompted estimates across recursively refined population partitions and finds widespread violations, including a macro fallacy in which reconstructed aggregates often outperform direct population estimates.

  • Problem

    Whether contemporary LLMs actually behave as conditional inference systems remains poorly understood, despite the importance of this interpretation for probabilistic reasoning.

  • Method

    The paper recursively partitions populations with binary trees, elicits persona-conditioned estimates and subgroup priors, and compares prior-weighted aggregates across tree levels and with reference data.

  • Results

    Across income, opinion, and synthetic forecasting tasks, LLMs show widespread violations of law-of-total-probability aggregation, with reconstructed subgroup aggregates often outperforming direct estimates.

  • Takeaways & Limitations

    Statistical self-consistency provides a reference-free diagnostic, while the macro fallacy indicates that fine-grained estimates can be better aligned than direct population-level estimates.

  • Takeaways & Limitations

    Distributional-alignment and macro-fallacy evidence relies primarily on ACS data and depends on elicitation, normalization, and demographic-split choices.

Abstract

from arXiv · show

In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Applying this protocol across problem domains and state-of-the-art frontier models, we show widespread violations of basic consistency properties. An in-depth study of persona prompting reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. This effect persists across variations in tree structure and estimation task, and can be partially recovered through implicit prompting. Together, these findings suggest that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.

1 Introduction

The paper tests whether in-context learning behaves as conditional inference by evaluating whether LLM estimates obey the law of total probability across persona-defined population partitions. It finds widespread self-consistency violations, while aggregation over finer subpopulations often improves estimates and can be partly recovered through prompting.

  • Evaluation framework: The study evaluates conditional inference by comparing direct marginal estimates with prior-weighted aggregates of conditional estimates over valid partitions.Binary trees recursively define increasingly specific subpopulations, and each tree level supplies an equivalent aggregate estimator under the law of total probability.
  • Macro fallacy: The macro fallacy is that direct aggregate estimates are systematically less accurate than aggregates reconstructed from finer-grained conditional estimates and corresponding priors.This effect is reported as robust across target quantities, models, and tree structures.
  • Ablations: Increasing specificity improves subgroup-level conditional estimates, but finer partitions make subgroup priors harder to estimate accurately.Ground-truth survey ablations separate the quality of conditional estimates from the quality of subgroup priors.
  • Implicit aggregation: Micro-to-macro prompting partially recovers the aggregation benefit by asking models to reason about relevant subpopulations before producing an aggregate estimate.The intervention adds almost no additional overhead.
  • Self-consistency findings: LLMs show widespread violations of statistical self-consistency across multiple domains, models, aggregation procedures, persona prompting, and synthetic forecasting.The paper introduces split consistency and order consistency as complementary local checks.
  • Evaluation criterion: Statistical self-consistency is proposed as a reference-free evaluation criterion complementary to task-specific metrics such as alignment.Persona prompting additionally enables comparisons with human reference statistics for ablations and investigations of inconsistency sources.

2 Background and related work

Prior work commonly interprets in-context learning as conditional inference, while persona conditioning provides a specific distributional-alignment instantiation. This work distinguishes its probability-axiom notion of self-consistency from other uses of the term in language-model research.

  • In-context learning as conditional inference: In-context learning is widely treated as conditional inference from prompt-provided examples or instructions, without updating model parameters.The paper situates this interpretation within substantial prior work and notes that many theoretical arguments build on it.
  • Persona conditioning and distributional alignment: Persona conditioning augments prompts with possibly incomplete descriptions of people and asks models to respond from those individuals’ perspectives.The passage identifies persona conditioning as a specific instantiation of in-context learning and notes two main operationalizations.
  • Self-consistency: This work defines self-consistency as satisfying basic probability axioms, distinguishing it from sampling-and-voting methods and other context-based consistency studies.Prior literature also uses self-consistency for response stabilization through sampling and voting, or for consistency across equivalent contexts and generation-validation judgments.

3 Expressing aggregates through partitions

This section defines the statistical framework for testing whether LLM-derived subpopulation estimates obey the law of total probability. Binary conditioning trees generate valid, increasingly fine-grained partitions whose elicited conditional estimates and priors can be aggregated and compared with direct population-level estimates.

  • Framework: The law of total probability requires prior-weighted conditional probabilities to yield the same aggregate for every valid partition of a population.Validity requires mutually exclusive, jointly exhaustive subpopulations; γ_S denotes each subgroup’s population fraction.
  • Binary conditioning trees: Binary conditioning trees recursively refine a base population, with nodes at each level forming valid partitions and deeper levels representing finer subpopulations.Each level uses one shared binary split function applied uniformly to all nodes at the preceding level.
  • ACS case study: For the ACS case study, the target is the income tail probability P(Y > τ), elicited conditionally for every tree node.The running example uses the 2024 American Community Survey as a human population reference and asks, for example, whether income exceeds 50 000 USD.
  • LLM elicitation: The evaluation elicits LLM estimates by converting each tree node’s path-based subgroup description into natural-language context.Direct prompting uses only the base context C_0, while additional branch strings express increasingly specific subpopulations.
  • Reconstructed aggregates: LLM-reconstructed aggregates combine elicited conditional estimates with level-wise elicited subgroup priors, then test agreement across tree levels and against the direct root estimate.Priors are obtained through repeated queries, renormalization, and averaging for each partition level.

4 The macro fallacy

The macro fallacy is that reconstructed estimates from explicit subgroup prompts often align better with survey aggregates than direct population-level estimates. Gains arise because decomposition exposes subgroup structure, although increasingly specific priors degrade and eventually limit improvement.

  • 4 The macro fallacy: Reconstructed aggregate estimates are consistently better aligned with survey ground truth at deeper tree levels than direct root-level estimates.This pattern holds across models, problems, and target distributions with varying entropy, despite large alignment-error variations across tree levels.
  • 4 The macro fallacy: Level-dependent alignment errors show that LLM estimates do not compose consistently under the law of total probability.The average aggregation gain follows a concave trend: representation improves with depth before declining as group descriptions become more complex.
  • 4 The macro fallacy: Conditional estimates improve from the root through the first two refinement levels, while level-3 improvements remain lower overall but vary substantially across subgroups.Large errors in negligible-prior nodes contribute little to the prior-weighted average.
  • 4 The macro fallacy: Deeper partitions make subgroup priors harder to estimate, with LLM-estimated priors becoming less aligned with survey-based priors as depth increases.This prior degradation helps explain the concave shape of the average aggregation gain.
  • 4 The macro fallacy: Explicit decomposition reduces the burden of representing heterogeneous populations by separating within-group residual variation from cross-group differences.In the ACS tree, expected within-group variance decreases as the tree is refined, consistent with the law of total variance.
  • 4 The macro fallacy: Micro-to-macro prompting often improves income estimates over direct prompting, but is more model-dependent and less systematic than explicit tree-based aggregation.Its gains cannot be explained by test-time compute alone: the benefit is partly recovered in one query and eventually worsens with deeper partitions.

5 Statistical self-consistency

This section formalizes statistical self-consistency as reference-free checks requiring LLM estimates to obey aggregation identities across valid conditioning partitions and constraint orderings. It shows that local split consistency controls multilevel reconstruction, while self-consistency remains necessary but not sufficient for distributional alignment.

  • Core framework: Self-consistency checks test whether LLM estimates satisfy law-of-total-probability aggregation identities across valid partitions, trees, and problem domains.The checks are instantiated through finite collections of local requirements rather than exhaustive verification.
  • Core framework: Conditional expectations over subpopulations preserve aggregation under any partition, with subgroup estimates weighted by their relative sizes.Tail probabilities, conditional means, moments, and full conditional distributions can be represented through suitable outcome functionals.
  • Split consistency: Split consistency compares a direct estimate for a conditioning event with the prior-weighted aggregate of estimates for its two binary-attribute refinements.The deviation is measured using a metric on the elicited estimate space, and the resulting score is the fraction of satisfied checks across all admissible event–split pairs.
  • Split consistency: Local split consistency controls the discrepancy between a direct root estimate and its recursively aggregated level-ℓ reconstruction, with a worst-case bound linear in ℓ.Observed error grows considerably more slowly than this worst-case bound in the experiments.
  • Order consistency: Combining split consistency with order consistency extends aggregation guarantees across conditioning trees with different split orders and verbalization paths.Order consistency addresses the requirement that equivalent constraint orderings describe the same subpopulation and yield the same estimate.
  • Interpretation: Self-consistency is reference-free and necessary for treating LLM estimates as conditional distributions, but it is complementary to—and does not guarantee—distributional alignment.A model can be internally consistent while misaligned with the target distribution, while external alignment alone does not establish coherence across specificity levels.

6 Evaluation

The evaluation applies reference-free self-consistency checks across ACS prediction, global opinion modeling, and synthetic forecasting tasks. Across domains and tasks, frontier models show widespread and variable violations of probabilistic consistency.

  • ACS prediction: Table 1 reports widespread split and order self-consistency violations across frontier models on ACS income and commute-time prediction at ε = 0.02.The tasks use age and employment status as conditioning attributes.
  • ACS prediction: Across model families, increasing Artificial Analysis Intelligence Index does not systematically improve ACS split or order consistency, with order consistency generally higher.This comparison uses the thresholded income task with τ = 40k USD and ε = 0.02.
  • ACS prediction: Using the same conditioning attributes and subgroup priors, ACS commute-time consistency differs substantially from income consistency, showing dependence on the prediction target.Thus, consistency varies across tasks even when the model and conditioning tree are held fixed.
  • Global opinion modeling: In WVS opinion modeling, models satisfy only a fraction of split and order checks across Canada and Indonesia, with no model uniformly consistent across questions.The evaluation uses five survey questions, age and income splits, and tolerance ε = 0.02.
  • Synthetic forecasting: Synthetic forecasting tasks also show widespread self-consistency failures; in the tennis example, no model satisfies more than half of the split consistency checks.The tasks include Federer–Nadal tennis forecasting and fantasy combat predictions without requiring an external reference distribution.

7 Discussion

The paper finds widespread violations of law-of-total-probability aggregation identities across survey and synthetic tasks, with implications for prompt-dependent estimates and reference-free evaluation. It argues that anchoring consistency constraints at better-aligned fine-grained estimates could improve aggregate predictions, while noting important empirical limitations.

  • Findings: Across ACS income, WVS opinion, and synthetic forecasting tasks, LLM estimates often violate aggregation identities implied by the law of total probability.These findings stress-test the interpretation of in-context learning as conditional inference.
  • Implications: Population-level alignment does not guarantee accurate subpopulation estimates, making predictions depend on prompt granularity, decomposition, or conditioning order.Statistically equivalent specifications can therefore produce incompatible model estimates.
  • Improving consistency: Aggregation identities provide verifiable, external-supervision-free constraints for improving self-consistency even when distributional alignment cannot be measured.The macro fallacy highlights the systematic nature of these consistency violations.
  • Improving consistency: Anchoring constraints at better-aligned lower-level estimates could propagate fine-grained information upward and improve aggregate estimates.Consistency can instead be achieved by adjusting either fine-grained or aggregate estimates, without changing the resulting score.
  • Limitations: The empirical alignment analysis and macro-fallacy findings rely primarily on ACS data, while results also depend on elicitation, normalization, and demographic-split choices.Self-consistency evaluation extends beyond survey settings but remains only a necessary criterion.

Appendices … A Disentangling conditionals and priors

The appendices separate reconstructed estimates into subgroup conditionals and priors, while documenting metrics, analyses, reconstructed distributions, prompt templates, and task-specific elicitation materials.

  • Appendices: The appendices include analyses of conditional alignment across tree levels and LLM-estimated priors.These are listed as A.1 and A.2.
  • Appendices: They define discrepancy metrics for outcome distributions and Wasserstein distance on ordered discrete outcomes.These are listed as B.1 and B.2.
  • Appendices: Further appendices provide normalized alignment error, a complete win-matrix analysis for micro-to-macro prompting, and reconstructed aggregate distributions across tree levels.These materials are listed as F.1–F.3.
  • G Prompt templates: Prompt-template materials cover prior elicitation, sociodemographic prompting, persona prompting, and micro-to-macro prompting.These are listed as G.1–G.4.
  • G Prompt templates: Additional prompt templates cover opinion distribution elicitation, the tennis forecasting task, and the fictional fantasy combat task.These are listed as G.5–G.7.
  • A Disentangling conditionals and priors: Appendix A separates LLM-estimated subgroup conditionals from subgroup priors used to weight reconstructed population estimates.This decomposition is enabled by survey reference statistics for each subgroup induced by the binary conditioning tree.

A.1 Conditional alignment across tree levels · A.2 LLM-estimated priors · B Extensions of self-consistency checks

Oracle-prior aggregation shows that fine-grained subgroup conditionals contain useful population-level information absent from direct aggregate prompting, while LLM-estimated priors become less accurate with finer partitions. The paper also extends self-consistency checks across discrepancy metrics, conditioning trees, and variance contraction.

  • A.1 Conditional alignment across tree levels: Using LLM-estimated subgroup conditionals and ground-truth priors isolates conditional alignment from population-composition error.The original aggregate alignment error conflates errors in subgroup conditionals and priors; replacing estimated priors with survey-based priors measures conditional aggregation accuracy.
  • A.1 Conditional alignment across tree levels: Deeper partitions consistently produce estimates better aligned with the ACS reference statistic than direct root-level prompting.The improvement persists when reconstructing population estimates from increasingly fine-grained subgroup conditionals using ground-truth subgroup priors.
  • A.1 Conditional alignment across tree levels: The oracle-prior analysis extends from threshold probabilities to categorical income distributions using normalized Wasserstein-1 distance.Threshold events reduce to Bernoulli distributions, where Wasserstein-1 recovers the absolute tail-probability difference.
  • A.1 Conditional alignment across tree levels: Reconstructed aggregate income distributions at increasingly refined tree levels show the same alignment pattern as thresholded estimates.Levels 1–3 aggregate LLM-estimated subgroup distributions with ACS ground-truth priors, while the root level directly elicits the aggregate distribution.
  • A.2 LLM-estimated priors: LLM-estimated subgroup priors are compared with survey-based priors using total variation distance across tree levels.Total variation provides a normalized discrepancy in [0, 1] that remains comparable as the number of subgroups grows with depth.
  • A.2 LLM-estimated priors: As partitions become more fine-grained, prior estimates become less aligned with survey ground truth across evaluated models.Coarse splits are estimated relatively accurately, but errors increase when probability mass must be assigned across more specific subpopulations.
  • B Extensions of self-consistency checks: The extensions specify discrepancy metrics for binary and categorical distributions and interpret self-consistency as operations on a binary conditioning tree.They also derive an expected variance contraction criterion from the law of total variance.

B.1 Discrepancy metrics for outcome distributions … B.4 Variance contraction

The appendix defines discrepancy measures for binary and categorical outcomes, formalizes split and order consistency on conditioning trees, and extends self-consistency to an expected variance-contraction check. These constructions align metric choice with outcome structure and derive higher-moment constraints from standard probabilistic identities.

  • B.1 Discrepancy metrics for outcome distributions: For binary outcomes, consistency error is the absolute difference between direct and prior-weighted aggregated success-probability estimates.The target is the Bernoulli event Y > τ, with Tτ(S) = P(Y > τ | X ∈ S) = E[h(Y) | X ∈ S].
  • B.1 Discrepancy metrics for outcome distributions: For ordinal categorical outcomes, normalized Wasserstein-1 distance measures probability-mass shifts along the response order and is comparable across category counts.Normalization maps the distance to [0, 1].
  • B.1 Discrepancy metrics for outcome distributions: For nominal categorical outcomes, total variation provides a permutation-invariant discrepancy between direct and aggregated probability vectors.The factor 1/2 normalizes total variation to [0, 1].
  • B.2 Wasserstein distance on ordered discrete outcomes: On ordered support {1, ..., K}, normalized Wasserstein-1 equals the cumulative-distribution-function distance divided by K − 1.The normalization yields 0 for identical distributions and 1 for maximal endpoint separation, independently of the number of ordered categories.
  • B.2 Wasserstein distance on ordered discrete outcomes: For Bernoulli distributions, the absolute error used in binary experiments is exactly their Wasserstein-1 distance.The ordered support {0, 1} leaves a single cumulative difference.
  • B.3 Tree operations: Split consistency requires each parent estimate to agree, within tolerance, with the prior-weighted aggregate of its two child estimates.Each admissible binary split partitions S into Sk,0 = S ∩ {Ak = 0} and Sk,1 = S ∩ {Ak = 1}.
  • B.3 Tree operations: Order consistency requires equivalent leaf events to receive matching direct estimates when adjacent tree-layer orders are interchanged.Changing the order changes presented constraints and intermediate conditioning events while preserving corresponding leaf events.
  • B.4 Variance contraction: Expected variance contraction requires model-implied conditional variances to satisfy the law-of-total-variance inequality at every internal tree node.For parent S with children SL and SR, child priors γL and γR weight the conditional variances; the constraint is averaged across child membership, not required for each child individually.

C Proofs · C.1 Split consistency · C.2 Order consistency

The proofs establish split consistency by recursively decomposing reconstruction discrepancies across tree levels and using convexity of the distance metric. They establish order consistency by reducing ordering changes to adjacent swaps and combining swap, split, and child-event bounds.

  • C.1 Split consistency: At each tree level, conditioning events partition the base population, enabling level-wise reconstruction from node estimates.The proof denotes the level-ℓ nodes by Lℓ(T) and treats their conditioning events as a partition of S.
  • C.1 Split consistency: Each node’s path weight is the product of elicited relative-size estimates along its root-to-node path.The root has weight one, and child weights factor into parent weights times elicited within-parent relative sizes.
  • C.1 Split consistency: The split-consistency proof bounds total reconstruction error by decomposing it into increments between successive tree levels.The convention for the level-zero reconstruction initiates the telescoping argument.
  • C.1 Split consistency: Grouping child nodes by parent and applying joint convexity shows each level increment is controlled by the corresponding local split discrepancies.Because elicited child weights sum to one, the distance between aggregated child estimates is bounded through convexity.
  • C.1 Split consistency: Summing the per-level bounds completes the proof that the full reconstruction satisfies split consistency.The final inequality invokes ε-split consistency at every node of the preceding level.
  • C.2 Order consistency: For order consistency, any two constraint orderings are connected by adjacent transpositions, with the number of swaps bounded by their inversion count.The triangle inequality then reduces the total ordering discrepancy to the sum of discrepancies from individual swaps.
  • C.2 Order consistency: Under Assumption 1, each adjacent swap has the discrepancy of the corresponding two-attribute conditioning event, so summing the swap bounds proves order consistency.Each swap discrepancy is strictly less than ε before applying the inversion-count bound.
  • C.2 Order consistency: The combined consistency result compares canonical orderings for a population and child events using triangle inequality, convexity, and relative-size normalization.The three resulting bounds combine split-consistency, order-consistency, and child-event discrepancies; relative-size estimates sum to one.

D Additional experimental results … E.3 Alignment error

The appendix tests whether the paper’s findings remain robust across prompting, partition granularity, persona phrasing, and implementation details. It also specifies the ACS-based conditioning tree, prior elicitation procedure, and alignment-error evaluation used in the experiments.

  • D.1 Micro-to-macro prompting: Micro-to-macro prompting asks models to identify relevant subpopulations, estimate their prevalence and outcomes, and combine them into an aggregate estimate.It is evaluated against direct aggregate prompting as a lightweight alternative to externally specified tree-based aggregation.
  • D.1 Micro-to-macro prompting: Micro-to-macro prompting partially recovers decomposition’s benefit, often improving over direct prompting on ACS income, but gains are model-dependent and less systematic than explicit tree aggregation.The comparison spans selected GlobalOpinionQA question–country pairs and ACS income thresholds.
  • D.2 Level-wise self-consistency: Level-wise self-consistency reconstructs aggregate income distributions from increasingly fine partitions and compares them with direct root estimates using normalized Wasserstein-1 distance.Lower W1 values indicate stronger agreement with the root estimate.
  • D.3 Prompt sensitivity: Reconstructed estimates are closer to the diagonal than direct aggregate estimates for most thresholds under both sociodemographic and first-person persona formulations.The model, temperature, repeated generations, outcome, thresholds, and aggregation weights are held fixed across formulations.
  • E Experiment details: The appendix documents the experimental setup supporting the paper’s reported results.The implementation note states that the code is available in the cited repository.
  • E.1 Binary conditioning tree: The binary conditioning tree uses 2024 ACS demographic attributes, recursively applying two binary splits selected from candidate attributes and income-distribution differences.Branches are grouped to have approximately equal survey-weighted probability mass before the final attributes are qualitatively selected.
  • E.1 Binary conditioning tree: Income distributions use survey-weighted yearly income in USD, approximately uniform root bins for the four-bin setting, and 0 USD for respondents below age 15.The same binning is applied across all tree subpopulations.
  • E.2 Prior elicitation: For each induced partition P_ℓ, the prior-elicitation procedure prompts the model n = 50 times to estimate each subgroup’s probability mass.Fully model-induced aggregates require eliciting both subgroup-conditional estimates and subgroup priors.

E.4 Disentangling the sources of aggregation error … F Additional plots

The appendix decomposes aggregation error into conditional and prior alignment, shows how refinement reallocates variance across subgroups, and evaluates self-consistency across survey, forecasting, and fictional tasks. It also includes additional larger plots with more detailed breakdowns.

  • E.4 Disentangling the sources of aggregation error: Conditional estimates are evaluated node by node with normalized Wasserstein distance, while prior estimates are evaluated with total variation distance across tree levels.Figure 4a uses the four-bin income distribution; Figure 4b compares survey-based and LLM-estimated priors.
  • E.4 Disentangling the sources of aggregation error: Prior estimation becomes harder on deeper partitions, creating a trade-off between more accurate subgroup outcomes and less accurate population weights.Coarse partitions are estimated comparatively accurately, whereas refined trees distribute probability mass across more specific subpopulations.
  • E.5 Variance decomposition across tree levels: The law of total variance decomposes aggregate income variance into within-subgroup and cross-subgroup components at every tree level, up to finite-sample error.Refinement shifts variation from residual within-subgroup heterogeneity toward explicit cross-subgroup heterogeneity.
  • E.7 Self-consistency evaluation on the thresholded ACS income setting: The thresholded ACS income comparison uses τ = 40k USD, computes SCε(A) and OCε(A) at ε = 0.02, and reports the AAI Index as a capability proxy.Because the target is binary, normalized Wasserstein distance becomes the absolute difference between exceedance probabilities.
  • E.8 Self-consistency evaluation on the ACS commute time setting: The ACS commute-time analysis reuses the age-and-employment conditioning tree and evaluates SCε(A) and OCε(A) at ε = 0.02 with normalized Wasserstein distance.The four bins cover [0, 1), [1, 15), [15, 30), and [30, 195] minutes, spanning the observed range.
  • E.9 Self-consistency evaluation on WVS opinion distributions: WVS opinion evaluations in Canada and Indonesia use age and model-interpreted low/high income splits, testing SCε(A) and OCε(A) across five questions at ε = 0.02.Split checks compare direct estimates with prior-weighted refinements, while order checks compare predictions under both attribute orderings.
  • E.10 Self-consistency evaluation of tennis forecasting example; E.11 Self-consistency evaluation of fantasy example: Tennis and fictional fantasy forecasting apply the same SCε(A) and OCε(A) checks at ε = 0.02 to binary win probabilities, including a setting without external ground truth.The tennis task forecasts Federer versus Nadal with progressively revealed context; the fantasy task tests consistency in an imaginary game world.

F.1 Normalized alignment error · F.2 Complete win-matrix analysis for micro-to-macro prompting · F.3 Reconstructed aggregate distributions across tree levels

Additional analyses confirm that reconstructed subgroup-based estimates often outperform direct aggregate prompting, while extending evaluation across models, questions, countries, and tree levels. They also visualize how LLM-estimated subgroup distributions and priors reconstruct population-level income distributions.

  • F.1 Normalized alignment error: Reconstructed aggregate estimates are often better aligned with ACS reference statistics than direct aggregate estimates across broader fixed-income thresholds.The gains vary across thresholds, models, and tree levels.
  • F.1 Normalized alignment error: The normalized-alignment-error analysis shows that the main text’s qualitative pattern is robust across a broader range of fixed-income thresholds.Evaluated thresholds include τ = 1k, 10k, 50k, and 80k USD.
  • F.1 Normalized alignment error: The aggregation gain is defined as 1 − AErr(ℓ)/AErr(0), with positive values indicating improved alignment and negative values indicating worse alignment.Error bars represent 90 % confidence intervals from bootstrapping with 1000 samples.
  • F.2 Complete win-matrix analysis for micro-to-macro prompting: Figure 15 provides the complete micro-to-macro win-matrix analysis on GlobalOpinionQA across all evaluated models.The analysis extends the GPT-5.4 example from Figure 11b and compares subgroup prompting with direct prompting across selected questions and countries.
  • F.2 Complete win-matrix analysis for micro-to-macro prompting: In the win matrices, green cells indicate lower absolute error for micro-to-macro prompting than direct prompting, while red cells indicate the opposite.Rows represent selected binary GlobalOpinionQA questions and columns represent selected countries.
  • F.3 Reconstructed aggregate distributions across tree levels: Figure 16 visualizes population-level ACS income distributions reconstructed from each binary-tree layer using LLM-estimated subgroup priors and distributions.Survey-based ground truth is shown consistently across panels alongside the corresponding LLM-induced reconstructed aggregate.

G Prompt templates … J GlobalOpinionQA questions

The appendix specifies prompt templates for eliciting priors, conditional probabilities, survey distributions, and forecasts across demographic, opinion, tennis, and fantasy-combat tasks. It also documents the ACS attributes, WVS questions, and GlobalOpinionQA questions used in the experiments.

  • G.1 Prior elicitation: Prior elicitation presents a mutually exclusive, exhaustive U.S. population partition and requests one probability per group whose masses sum to 1.The prompt requires valid JSON, probabilities in [0, 1], exactly one entry per group, and exact group names.
  • G.1 Prior elicitation: Groups are represented by structured sociodemographic attributes defining each partition subgroup.An example partitions the population by sex into female and male groups.
  • G.2 Sociodemographic prompting: Sociodemographic prompting conditions an income-threshold probability on a brief individual profile without asking the model to role-play.The answer is requested as a single probability formatted [[p]].
  • G.3 Persona prompting: Persona prompting asks the model to answer from the described person’s perspective while preserving the same income-probability question and [[p]] output format.The persona is expressed as a free-text, first-person role-playing description of the demographic profile.
  • G.4 Micro-to-macro prompting: Micro-to-macro prompting explicitly asks models to identify relevant subpopulations, assess prevalence and subgroup probabilities, and combine them into a population estimate.The subpopulations are chosen by the model rather than specified in advance, unlike tree-based aggregation prompts.
  • G.5 Opinion distribution elicitation: Opinion-distribution prompting requests a complete categorical probability distribution over survey answer options for a demographic subpopulation.The output must be JSON with fixed keys, decimal probabilities in [0, 1], and values summing to 1.
  • H ACS attributes: The appendix lists the ACS conditioning attributes, five WVS questions with ordinal scales, and binary GlobalOpinionQA questions with answer options.GlobalOpinionQA question IDs are generated row-index identifiers rather than original survey question IDs.
Loading 2607.15277v1…