PaperlayerView exact revision on arXiv

Source-linked AI summary

Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

Patrik Wolf, Thomas Kleine Buening, Andreas Krause, Celestine Mendler-Dünner

arXiv:2607.15277v1cs.CL

TL;DR

Although in-context learning is often treated as conditional inference, how well contemporary LLMs satisfy its implied probabilistic identities remains poorly understood. The paper aggregates persona-prompted estimates across tree partitions and finds widespread consistency violations, with reconstructed subgroup estimates often more reliable than direct population estimates.

  • Problem

    Although in-context learning is often treated as conditional inference, how well contemporary LLMs satisfy its implied probabilistic consistency identities remains poorly understood.

    Source: §1 · pp. 1–2

  • Method

    The paper uses persona prompts to elicit conditional estimates over tree partitions, aggregates them with subgroup priors, and compares aggregates across levels and against ground truth.

    Source: §1.1 · p. 3

  • Results

    Across ACS income prediction, WVS opinion modeling, and synthetic forecasting, estimates often violate aggregation identities, while reconstructed subgroup aggregates are systematically more reliable than direct estimates.

    Source: §7 · p. 18

  • Takeaways & Limitations

    Self-consistency offers verifiable, reference-free constraints, while prompt granularity and decomposition can yield incompatible estimates for the same target.

    Source: §7 · p. 18

    Empirical alignment and macro-fallacy findings rely primarily on ACS and design choices, and self-consistency alone cannot establish faithful conditional inference.

    Source: §5.3 · p. 14, §7 · p. 19

Abstract

from arXiv · show

In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Applying this protocol across problem domains and state-of-the-art frontier models, we show widespread violations of basic consistency properties. An in-depth study of persona prompting reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. This effect persists across variations in tree structure and estimation task, and can be partially recovered through implicit prompting. Together, these findings suggest that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.

1 Introduction

The paper tests whether in-context learning behaves like conditional inference by checking whether prior-weighted estimates over valid subpopulation partitions match direct aggregate estimates. Across its studies, it finds widespread self-consistency violations and a macro fallacy: fine-grained estimates can yield more accurate aggregates than direct estimates.

1 Introduction

  • In-context learning is often treated as conditional inference, and persona responses are interpreted as distributions over individuals in the described subpopulation rather than estimates for one specific person.Distributional alignment is a popular evaluation metric for persona prompting.Source: §1 · p. 1
  • The law of total probability requires a marginal distribution to equal the prior-weighted aggregate of conditional distributions over any valid partition, making partition-based comparisons a self-consistency check.Partitions can use attributes such as age and employment status or complementary forecast scenarios.Source: §1 · p. 2

1.1 Our work

  • Binary conditioning trees create valid partitions at increasing specificity, allowing persona-prompted conditional estimates to be aggregated with prior weights and compared across levels, against direct estimates, and against ground truth.Persona prompting also enables human reference statistics from conditioning attributes, supporting ablations of self-consistency violations.Source: §1 · p. 2, §1.1 · p. 3
  • Direct aggregate estimates are systematically less accurate than aggregates reconstructed from fine-grained conditional estimates and priors, an effect robust across target quantities, models, and tree structures.The authors call this pattern the macro fallacy.Source: §1.1 · p. 3
  • Greater specificity improves subgroup conditional estimates, but finer partitions make priors harder to estimate because they produce smaller, more numerous subpopulations.Ground-truth survey ablations disentangle conditional-estimate quality from subgroup-prior quality.Source: §1.1 · p. 3
  • Micro-to-macro prompting, which asks a model to reason about relevant subpopulations before estimating an aggregate, partially recovers the aggregation benefit with almost no additional overhead.This tests whether aggregation gains can be elicited within a single prompt.Source: §1.1 · p. 3
  • Across persona prompting and synthetic forecasting, models—including frontier models—show widespread self-consistency violations, and consistency scores show no evidence of correlation with standard benchmark metrics.The authors introduce split consistency and order consistency as complementary local checks across aggregation procedures.Source: §1.1 · p. 3
  • The authors frame statistical self-consistency as a reference-free criterion complementary to task-specific metrics and suggest explicit or implicit aggregation helps propagate subpopulation knowledge into aggregate estimates.The criterion evaluates whether direct marginal estimates agree with estimates reconstructed from valid partitions.Source: §1 · p. 2, §1.1 · p. 3

2 Background and related work

Prior work commonly treats in-context learning outputs as conditional distributions, especially under persona conditioning, while evidence raises doubts about that interpretation. This study situates partition-based probability-axiom checks as reference-free tests of it.

In-context learning as conditional inference

Persona conditioning and distributional alignment

Self-consistency

  • Related consistency work tests probabilistic coherence, while this study uses binary conditioning trees to impose law-of-total-probability and law-of-total-variance constraints without external reference data.The constraints vary in conditioning specificity and apply when split criteria can be verbalized as conditioning events.Source: §2 · p. 4

3 Expressing aggregates through partitions

The section uses the law of total probability to frame aggregate estimates as prior-weighted combinations of conditional estimates over valid population partitions. Binary conditioning trees provide a recursive scaffold for eliciting and comparing direct and reconstructed estimates.

3 Expressing aggregates through partitions

  • The law of total probability requires prior-weighted conditional probabilities to recover the same aggregate for every valid partition of a population.Validity requires subpopulations to be mutually exclusive and jointly exhaustive.Source: §3 · p. 5

3.1 Binary conditioning trees

  • Binary conditioning trees recursively refine a base population, with each level forming a valid partition for comparing direct and reconstructed estimates.A shared binary split function is applied at each level, assigning every member of each node to exactly one child.Source: §3.1 · p. 5

3.2 Human population as a running example

  • The running example uses 2024 ACS data from 3.8 million U.S. individuals as a human baseline and partitions by age and class of worker.The splits are chosen to inform yearly-income estimates, aiming to reduce prior-weighted within-subgroup income variance while keeping subgroup priors reasonably balanced.Source: §3.2 · p. 5

3.3 Verbalizing split criteria for in-context learning

  • LLMs receive natural-language descriptions of each node formed by the split restrictions along its path, while the base context alone defines direct prompting.For the ACS experiments, folktexts [Cruz et al., 2024] implements the descriptions; the base context can also be empty, leaving the population implicit.Source: §3.3 · p. 6

3.4 Constructing LLM estimates

  • For each tree node, the study elicits conditional tail probabilities and subgroup priors, then combines them into reconstructed aggregate estimates.For the ACS case study, a tail event asks whether yearly income exceeds a threshold; priors are elicited repeatedly at each level, renormalized, and averaged. Under conditional inference, reconstructed aggregates should agree across levels and with the direct root estimate.Source: §3.4 · p. 7

4 The macro fallacy

Direct population-level prompts are often less aligned with human reference data than aggregates reconstructed from subgroup estimates, revealing a failure of statistical self-consistency. Refinement can improve elicitation but also degrades subgroup-prior estimates, while implicit micro-to-macro prompting recovers some of the benefit less consistently.

4.1 Alignment of reconstructed aggregate estimates

  • Reconstructed aggregates are consistently better aligned at deeper tree levels than direct prompts, even when LLMs supply both subgroup priors and conditionals.The improvement is non-monotonic with depth and holds across target distributions of varying entropy; it is achieved by changing the prompting strategy without external knowledge.Source: §4 · p. 7, §4.1 · p. 8, Figure 3 · p. 8

4.2 Disentangling error sources in aggregate estimates

  • Conditional estimates improve over the root at early refinement levels but not uniformly, while LLM-estimated subgroup priors become less aligned as partitions deepen.The resulting aggregation gain follows a concave trend, reflecting a trade-off between more specific conditionals and increasingly difficult population weighting.Source: §4.2 · pp. 8–9, Figure 4 · p. 9

4.3 Refinement reduces residual uncertainty

  • Tree refinement shifts variation from residual within-group heterogeneity to explicit cross-group differences, and can expose subgroup information missed by aggregate prompting.In the ACS income task, direct prompting appears to underweight young individuals and overestimate income; making this subgroup explicit improves the reconstructed estimate.Source: Figure 4 · p. 9, §4.3 · p. 10

4.4 Can implicit prompting improve aggregate estimates?

  • Implicit micro-to-macro prompting often improves ACS income estimates over direct prompting, but its gains are more model-dependent and less systematic than explicit aggregation.It asks the model to identify and combine subpopulations within one query, and the gains are not explained by increased test-time compute alone.Source: §4.4 · pp. 10–11, Figure 5 · p. 10

5 Statistical self-consistency

The section formalizes two reference-free checks of whether LLM conditional estimates remain coherent under population refinement and equivalent prompt descriptions. Split consistency tests probabilistic aggregation, while order consistency tests invariance to constraint order.

5.1 Split consistency

  • Split consistency compares a direct estimate with the prior-weighted estimates of two attribute-defined refinements; local checks bound reconstruction error across tree levels.The worst-case bound grows linearly with depth, while observed error grows considerably more slowly (Table 4). The split consistency score is the fraction of satisfied checks across all admissible event-and-split pairs, covering different tree structures while using a fixed canonical verbalization order.Source: §5.1 · p. 12

5.2 Order consistency

  • Order consistency compares estimates of the same conditioning event under different constraint orderings, using pairwise checks on two-attribute events.These checks control order consistency for larger events only under the assumption that swapping adjacent constraints has a context-independent effect; the evaluation restricts checks to two-attribute events because testing every ordering quickly becomes infeasible.Source: §5.2 · p. 13

5.3 Interpreting the scores

  • The checks are reference-free diagnostics: split consistency tests probabilistic composition, while order consistency tests invariance to constraint order.They compare a model’s estimates with other estimates from that model, making them applicable where reliable external labels are expensive, unavailable, or ill-defined.Source: §5.3 · p. 14
  • When pairwise order checks hold and relative-size estimates are order-invariant, split consistency under a canonical ordering extends to trees with different split and verbalization orders.The guarantee applies to every conditioning tree induced by the attribute set, with the split discrepancy extended by order-consistency terms.Source: §5.3 · p. 14
  • Self-consistency is necessary but not sufficient for faithful inference, so it complements rather than replaces alignment with the target distribution.A model can be internally consistent yet misaligned, and external alignment alone does not establish coherence across specificity levels unless all relevant ground-truth conditionals are matched exactly.Source: §5.3 · p. 14

6 Evaluation

The evaluation finds widespread self-consistency failures across ACS prediction, international opinion modeling, and synthetic forecasting, with consistency varying by task and remaining unsaturated as model capability increases.

6.1 Self-consistency in persona prompting

  • ACS evaluations reveal widespread violations of both split and order consistency, with scores differing substantially between income and commute-time prediction.The checks compare direct estimates with prior-weighted subgroup aggregates and assess whether estimates agree when conditioning attributes are reordered, using tolerance ε = 0.02.Source: Table 1 · p. 15, §6.1 · p. 15
  • Neither split nor order consistency improves systematically as model capability rises, and statistical self-consistency remains far from saturated among current models.This pattern holds in the thresholded ACS income task across a broader set of models and families, despite large increases in the Artificial Analysis Intelligence Index within families.Source: §6.1 · p. 15, Figure 6 · p. 16
  • Across Canada and Indonesia, models satisfy only a fraction of WVS split and order checks, and no model is uniformly consistent across questions.Even the strongest models show substantial variation across survey questions and countries, extending the ACS failures beyond U.S. income prediction.Source: §6.1 · p. 16, Table 2 · p. 17

6.2 Self-consistency in forecasting problems

  • The evaluation applies the same binary conditioning-tree framework to tennis and fictional fantasy-combat forecasts, partitioning predictions by contextual event information rather than demographics.The tennis task uses court surface and first-set outcome; the fantasy task uses time of day and wind conditions.Source: §6.2 · p. 17
  • In synthetic forecasting, no model satisfies more than half of the tennis split-consistency checks, while fantasy-task scores range from complete failure to a perfect score.Order consistency is consistently higher than split consistency in tennis; despite lacking external ground truth, the fantasy task still permits principled self-consistency assessment.Source: §6.2 · p. 17, Table 3 · p. 18

7 Discussion

Across ACS, WVS, and synthetic tasks, LLM estimates often violate aggregation identities; direct aggregate prompts are less reliable than reconstructing population estimates from subgroup responses, a pattern called the macro fallacy.

7 Discussion

  • Across ACS income prediction, WVS opinion modeling, and synthetic forecasting, LLM estimates widely violate law-of-total-probability aggregation identities.Direct aggregate prompting is systematically less reliable than reconstructing population estimates from subgroup estimates, even when models contain useful subgroup knowledge.Source: §7 · p. 18
  • Population-level alignment does not ensure accurate subgroup estimates, so prompt granularity, decomposition, or conditioning order can produce incompatible estimates of the same target.This challenges applications that assume outputs approximate conditional probabilities or samples from learned posterior beliefs: locally plausible predictions must also compose consistently.Source: §7 · p. 18
  • Because consistency constraints are directionless, enforcing them around better-aligned lower-level estimates could propagate subgroup information upward and improve aggregate estimates.The proposed aggregation identities provide verifiable constraints without external supervision, including when distributional alignment cannot be measured.Source: §7 · pp. 18–19
  • Evidence for distributional alignment and the macro fallacy relies mainly on ACS data and depends on elicitation, normalization, and demographic-split choices.Self-consistency is necessary but not sufficient for faithful inference; synthetic examples do not establish real-world forecasting or decision accuracy. Future work should test broader domains, richer conditioning, and alternative elicitation, and develop interventions optimizing alignment and consistency.Source: §7 · p. 19

Appendices · A Disentangling conditionals and priors

The appendix separates reconstructed population estimates into subgroup conditionals and priors, using survey reference statistics to isolate conditional-estimation errors with oracle priors and examine LLM-estimated subgroup proportions.

Appendices

A Disentangling conditionals and priors

  • Survey reference statistics allow the appendix to isolate conditional-estimation errors using oracle priors, then compare LLM-estimated subgroup priors with survey proportions as partitions become finer.The decomposition separates the two ingredients in reconstructed population estimates: subgroup conditionals and the priors used to weight them.Source: A Disentangling conditionals and priors · p. 24

A.1 Conditional alignment across tree levels · A.2 LLM-estimated priors · B Extensions of self-consistency checks

Using survey-based subgroup priors, the paper finds that subgroup estimates can improve aggregate alignment, while LLM-estimated priors become less aligned as partitions grow finer. It also details extensions of the self-consistency checks based on discrepancy metrics, conditioning trees, and variance contraction.

A.1 Conditional alignment across tree levels

  • Replacing LLM-estimated subgroup priors with survey-based priors isolates conditional alignment error by removing uncertainty about population composition.The overall aggregate alignment error otherwise combines errors in subgroup conditionals and subgroup priors.Source: §A.1 · p. 24
  • With ground-truth subgroup priors, estimates reconstructed from deeper partitions are consistently better aligned with the ACS reference than direct aggregate estimates.Figure 7 reports conditional aggregation gain across income thresholds; positive values indicate improvement over direct aggregate prompting.Source: §A.1 · pp. 24–25, Figure 7 · p. 25
  • For categorical income distributions, subgroup aggregation also improves alignment, although the level-3 reconstruction is slightly less accurate than level 2.Alignment uses normalized Wasserstein-1 distance, which generalizes the thresholded case; noisier fine-grained subgroup estimates may offset further refinement gains.Source: §A.1 · p. 25, Figure 8 · p. 26

A.2 LLM-estimated priors

  • The prior analysis measures the discrepancy between LLM-estimated and survey-based subgroup distributions with total variation distance, normalized to [0, 1] across tree levels.The compared distributions are defined over the subgroups induced by each level’s partition.Source: §A.2 · pp. 25–26
  • LLM-estimated priors become less aligned with survey ground truth as partitions grow finer, a pattern consistent across evaluated models.Coarse population splits are estimated relatively accurately, while errors increase when probability mass must be assigned across more specific subpopulations.Source: §A.2 · p. 26

B Extensions of self-consistency checks

  • The extensions discuss discrepancy metrics for binary and categorical distributions, interpret self-consistency through binary conditioning trees, and derive variance contraction from the law of total variance.Source: B Extensions of self-consistency checks · p. 26

B.1 Discrepancy metrics for outcome distributions … B.4 Variance contraction

The appendices specify outcome-appropriate distances for comparing direct and aggregated estimates, interpret consistency checks as operations on conditioning trees, and extend them to an expected variance-contraction constraint.

B.1 Discrepancy metrics for outcome distributions

  • For the ACS binary income target, each estimate is a Bernoulli success probability, and discrepancy is the absolute difference between direct and prior-weighted aggregated estimates.The success event is yearly income above threshold τ.Source: §B.1 · pp. 26–27
  • For categorical outcomes, ordinal responses use normalized Wasserstein-1 distance, while unordered nominal responses use total variation distance.Normalization places the distances in [0, 1], supporting comparison across questions with different numbers of response options.Source: §B.1 · pp. 27–28

B.2 Wasserstein distance on ordered discrete outcomes

  • On ordered support {1, . . . , K}, Wasserstein-1 is normalized by K −1 to range from 0 to 1, and binary absolute error equals Bernoulli Wasserstein-1 distance.The range is independent of the number of ordered categories; 0 denotes identical distributions and 1 maximal separation.Source: §B.2 · p. 28

B.3 Tree operations

  • Split consistency requires each parent estimate to match the prior-weighted aggregate of its child estimates within tolerance, across all admissible next splits.A fixed tree exposes only its prescribed attribute order, so the split score considers pairs of conditioning events and admissible splits beyond its parent–child relations.Source: §B.3 · p. 29
  • Order consistency requires corresponding leaf estimates to agree within tolerance when adjacent tree layers are interchanged, although the conditioning events at the leaves remain the same.The reordered trees can present attribute constraints in different orders and expose different intermediate conditioning events.Source: §B.3 · p. 29, Figure 10 · p. 29

B.4 Variance contraction

  • Expected variance contraction requires each parent’s estimated variance to be at least the prior-weighted average of its children’s estimated variances.This is an averaged constraint derived from the law of total variance; it does not require conditional variance to decrease for every child individually.Source: §B.4 · p. 30

C Proofs · C.1 Split consistency · C.2 Order consistency

The proofs establish split-consistency bounds by aggregating child estimates and establish order- and child-event bounds by combining local discrepancies with the triangle inequality and convexity.

C.1 Split consistency

  • Path weights are products of elicited relative-size estimates along root-to-node paths, and the proof defines a reconstruction at each tree level.At each level, the conditioning events partition the base population.Source: §C.1 · p. 30
  • Grouping each level’s child nodes by parent uses the factorization of child weights into parent weights and elicited relative sizes.The proof uses this grouping to bound the increments between successive reconstructions.Source: §C.1 · p. 30
  • Joint convexity and ε-split consistency at every node make each reconstruction increment strictly less than ε; summing the increments completes the proof.The elicited weights at each node sum to one.Source: §C.1 · p. 30

C.2 Order consistency

  • Any two constraint orderings are connected by adjacent transpositions, with the sequence length equal to their number of inversions.The proof bounds this length by a quantity not specified in the supplied passage.Source: §C.2 · p. 31
  • The triangle inequality bounds the discrepancy between orderings by the sum of swap discrepancies, each strictly less than ε under Assumption 1.Each swap has the discrepancy of the corresponding two-attribute conditioning event.Source: §C.2 · p. 31
  • The child-event bound combines canonical-ordering discrepancies below εOC and εSC with a convexity bound using order consistency of child events.The relative-size estimates sum to one, completing the combined bound.Source: §C.2 · p. 31

D Additional experimental results … E.3 Alignment error

Appendix D probes robustness across prompting choices, while Appendix E documents experimental details and alignment evaluation.

D Additional experimental results

D.1 Micro-to-macro prompting

  • Micro-to-macro prompting partially recovers explicit decomposition’s benefit, reducing average error for most models but worsening it for some across ACS income and GlobalOpinionQA.The implicit strategy asks models to identify subpopulations, assess their prevalence and outcomes, and combine them; its gains are less systematic than explicit tree-based aggregation.Source: §D.1 · p. 32, Figure 11 · p. 32

D.2 Level-wise self-consistency

  • The level-wise evaluation compares each partition-reconstructed income distribution with direct root estimates using normalized Wasserstein-1 distance, where lower values indicate stronger agreement.It uses partitions from a depth-four binary tree and reports discrepancies for each level.Source: §D.2 · pp. 32–33, Table 4 · p. 33

D.3 Prompt sensitivity

  • Subgroup aggregation usually moves estimates closer to the diagonal under both prompt formulations, but persona prompts overestimate probabilities and sociodemographic prompts are more accurate.The two formulations keep the model, sampling, outcome, thresholds, and aggregation weights fixed; the authors use sociodemographic prompting in their experiments.Source: §D.3 · p. 33, Figure 12 · p. 34

E Experiment details

E.1 Binary conditioning tree

  • The experimental tree uses ACS demographic attributes, with AGEP and COW splits selected to distinguish income groups and approximately balance branch probabilities.The age split groups respondents younger than 31 or older than 68 versus ages 31–68; COW values 1–7 are labeled employed, while 8, 9, and missing values are unemployed. Root income bins are reused across subpopulations, and under-15 respondents are assigned income of 0 USD.Source: §E.1 · pp. 33–34

E.2 Prior elicitation

  • At each tree level, subgroup priors are estimated from 50 model responses, renormalized, and averaged into a probability distribution.Source: §E.2 · p. 34

E.3 Alignment error

  • Across levels 0–4, subgroup-reconstructed aggregate estimates consistently improve over direct root estimates, with gains measured as 1 − AErr(ℓ)/AErr(0).The analysis normalizes alignment error by the root-level error; larger levels correspond to increasingly fine-grained subgroup predictions.Source: §E.3 · p. 36

E.4 Disentangling the sources of aggregation error … F Additional plots

The appendices decompose aggregation error into conditional- and prior-alignment components, show how variance shifts across tree levels, and document consistency checks across survey, forecasting, and synthetic settings.

E.4 Disentangling the sources of aggregation error

  • More-specific subgroups improve conditional alignment heterogeneously, while prior estimates worsen as partitions deepen, creating a trade-off between outcome estimates and population weights.Some fine-grained subgroups with negligible prior mass remain poorly aligned.Source: §E.4 · pp. 36–37

E.5 Variance decomposition across tree levels

  • By the law of total variance, aggregate income variance equals within- plus between-subgroup variance, with refinement shifting variation from residual heterogeneity to differences between subgroup means.The decomposition holds at every tree level up to finite-sample estimation error.Source: §E.5 · p. 37

E.6 Self-consistency evaluation on the binned ACS income setting; E.7 Self-consistency evaluation on the thresholded ACS income setting

  • The thresholded ACS income task uses τ = 40k USD with age and employment conditioning; split and order consistency scores use ε = 0.02, and binary probability discrepancy is absolute difference.Models are grouped by family and ordered by AAI Index; the binned experiment covers the full ACS-observed income range.Source: §E.6 · p. 37, §E.7 · p. 38

E.8 Self-consistency evaluation on the ACS commute time setting

  • The commute-time check uses the same age/employment tree and priors as the income experiment, testing four-bin distributions with split and order consistency scores at ε=0.02.The bins cover the full observed ACS commute range, up to 195 minutes.Source: §E.8 · p. 38

E.9 Self-consistency evaluation on WVS opinion distributions

  • WVS checks cover five questions in Canada and Indonesia using age and model-interpreted low/high income splits, with low/high required to form complementary categories.Split and order consistency scores measure the fraction of checks within ε=0.02 using normalized Wasserstein distance.Source: §E.9 · pp. 38–39

E.10 Self-consistency evaluation of tennis forecasting example

  • The tennis example checks Federer–Nadal win-probability consistency across court-surface and first-set conditions, using split and order scores at ε=0.02.The estimated prior probabilities serve as aggregation weights, and binary discrepancies are absolute probability differences.Source: §E.10 · p. 39

E.11 Self-consistency evaluation of fantasy example

  • The fictional fantasy example applies split and order checks to win probabilities despite having no external ground-truth distribution, illustrating that consistency checks require only direct and conditional estimates.The conditions are day versus night and calm versus windy, and binary discrepancies are checked against ε=0.02.Source: §E.11 · p. 39

F Additional plots

F.1 Normalized alignment error · F.2 Complete win-matrix analysis for micro-to-macro prompting · F.3 Reconstructed aggregate distributions across tree levels

Across these appendix analyses, reconstructed aggregates from subgroup estimates often align better with the ACS reference than direct aggregate estimates. Additional figures examine micro-to-macro win rates and reconstructed distributions across tree levels.

F.1 Normalized alignment error

  • Across broader fixed income thresholds, subgroup-based reconstructed estimates often align better with the ACS reference than direct aggregate estimates, with gains varying by threshold, model, and tree level.Figure 14 reports aggregation gain across models and tree levels.Source: Figure 14 · p. 40, §F.1 · p. 40
  • Figure 14 defines aggregation gain as 1 − AErr(ℓ)/AErr(0), where positive values indicate improved alignment relative to direct prompting.Error bars show 90% confidence intervals computed by bootstrapping 1,000 samples.Source: Figure 14 · p. 40

F.2 Complete win-matrix analysis for micro-to-macro prompting

  • The complete GlobalOpinionQA win-matrix analysis covers all evaluated models and compares micro-to-macro prompting with direct prompting across question–country pairs.Each cell indicates which approach has lower absolute error; the percentage above each matrix is the fraction of pairs improved by micro-to-macro prompting.Source: Figure 15 · p. 41, §F.2 · p. 41
  • In the win matrices, models with win rates above 50% appear in the top row, while models below 50% appear in the bottom row.The win rate is the fraction of question–country pairs where micro-to-macro prompting improves over direct prompting.Source: Figure 15 · p. 41

F.3 Reconstructed aggregate distributions across tree levels

  • As the reconstruction moves down the tree, the aggregate income distribution becomes better aligned with the ground-truth ACS distribution.Figure 16 compares the direct root estimate with aggregates reconstructed from increasingly fine-grained conditional estimates.Source: Figure 16 · p. 42, §F.3 · p. 42
  • For each tree layer, the reconstructed population-level ACS income distribution aggregates LLM-estimated subgroup distributions using LLM-estimated subgroup priors.Figure 16 shows the same survey-based base-population distribution in each panel alongside the reconstructed aggregate.Source: Figure 16 · p. 42, §F.3 · p. 42

G Prompt templates … J GlobalOpinionQA questions

The appendix documents the prompts used for demographic and opinion estimation and forecasting, along with the ACS attributes and survey questions used in the experiments.

G.1 Prior elicitation

  • The prior prompt defines a full U.S. sociodemographic partition and requires probabilities across groups to sum to 1 in a strict JSON response.The group list is specified by structured sociodemographic attributes.Source: §G.1 · p. 43, G Prompt templates · p. 43

G.2 Sociodemographic prompting; G.3 Persona prompting

  • Sociodemographic prompts estimate an income-threshold probability from a third-person profile, while persona prompts ask the model to answer from that person’s perspective.Both request a single probability in the format [[p]].Source: §G.2 · pp. 43–44, §G.3 · p. 44

G.4 Micro-to-macro prompting

  • The micro-to-macro prompt asks the model to choose relevant U.S. subpopulations, consider their prevalence and threshold probabilities, and combine them into a population estimate.Unlike tree-based aggregation, the subpopulations are not specified in advance.Source: §G.4 · pp. 44–45

G.5 Opinion distribution elicitation

  • The opinion prompt elicits a complete categorical probability distribution over survey answer options for a demographic subpopulation, returned as JSON with probabilities summing to 1.The WVS prompt uses fixed answer-option keys and a fixed output schema.Source: §G.5 · p. 45

G.6 Tennis forecasting task; G.7 Fictional fantasy combat task

  • The tennis and fantasy-combat prompts ask for a win probability under specified match or encounter attributes and require a single decimal probability in the format [[p]].The tennis example forecasts Roger Federer against Rafael Nadal; the fantasy scenario concerns a main character fighting an enemy.Source: §G.6 · p. 46, §G.7 · p. 46

H ACS attributes

  • Table 5 lists ACS variables and their descriptions and values for demographic conditioning, adding a n/a class to each value list for missing survey data.Continuous variables use observed ranges, while categorical variables use ACS response codes.Source: Table 5 · p. 47

I WVS questions

J GlobalOpinionQA questions

Related papers

Loading 2607.15277v1…