Source-linked AI summary
KITE: Scaling Jev Population Experiments with Sparse Flagship Calibration
Hengyu Li
TL;DR
KITE addresses how to run large population experiments cheaply without mistaking computational precision for agreement with people. It combines a typed behavioral kernel, sparse flagship anchors, tabulated execution, and shared discrepancy; sparse calibration improves reported effect and decision estimates, though hybrid human-referenced testing remains limited.
Problem
Population experiments need models of how responses change with situations, while cheap inference alone cannot establish agreement with people.
Source: §1 · p. 1Method
KITE queries a typed behavioral kernel for unique states, applies sparse paired flagship corrections, executes populations from a table, and propagates measured human–model discrepancy.
Source: §1 · pp. 1–2, §3.1 · p. 3Results
Epstein effect MAE falls from 0.0305 to 0.0180 (−41%; paired error reduction 0.0125, 95% CI [0.0064, 0.0167]); SocSci210 captured gain rises from 0.268 to 0.386 with 0.5–1.5% anchor coverage.
Source: §1 · pp. 1–2Takeaways & Limitations
The reported evidence supports using the cheap kernel for content response and sparse corrections for demonstrated intervention families, with property-specific records linking outputs to validation scope and uncertainty.
Source: §1 · p. 2Only Epstein and SocSci210 independently test the hybrid against human references, and their effect structures and prior exposure differ.
Source: §11 · p. 12
Abstract
from arXiv · showhide
KITE queries a typed behavioral kernel once per unique state, then executes populations of any size from the table with event-keyed randomness and common random numbers. An expensive flagship model is reserved for sparse paired anchors that estimate intervention effects. Measured human-model discrepancy is propagated as shared error into every conclusion. Population-experiment cost thus scales with unique states and anchors, while uncertainty is governed by evidence about people rather than Monte Carlo noise. On Epstein experiments with 9,070 participants, anchors covering 1.7% of states reduced effect error by 41% (absolute MAE reduction 0.0125). On 37 held-out SocSci210 experiments, 0.5-1.5% anchor coverage raised captured decision gain from 0.27 to 0.39. The kernel passed content-fidelity criteria in all 15 new countries of a 16-country study. Shared discrepancy yielded retrospective coverage of 93% and 96% at nominal 80% and 90%, versus 29% and 36% from human sampling uncertainty alone. A million agents executed 20 tabulated steps in 0.9 seconds on a laptop. This architecture offers a route to screening candidate interventions before human trials, multi-country content audits, and uncertainty-aware policy comparison at the cost of a few thousand kernel calls with sparse flagship anchors. Property-specific evidence records connect each use to its validation scope, correction provenance, and uncertainty, making these applications auditable.
1 Introduction
KITE separates cheap population execution from evidence about human agreement by combining a typed behavioral kernel, sparse flagship corrections, and shared discrepancy. Its contributions span calibrated intervention estimates, explicit validation scopes, and auditable execution, while several behavioral extensions remain unvalidated.
1 Introduction
- KITE addresses the distinction between computational precision and agreement with people through a typed kernel, sparse flagship corrections, tabulated execution, and measured human–model discrepancy.Cheap inference helps computationally but cannot establish agreement with people.Source: §1 · p. 1
- Epstein effect MAE falls from 0.0305 to 0.0180 (−41%; paired error reduction 0.0125, 95% CI [0.0064, 0.0167]) with sparse flagship calibration.In SocSci210, 0.5–1.5% anchor coverage raises captured gain from 0.268 to 0.386 (gain 0.119, [0.014, 0.224] at three anchors).Source: §1 · pp. 1–2
2 Related work
- The related-work framing distinguishes agent execution scale from human intervention validity and separates marginal fit from condition sensitivity, intervention choice, and within-person dependence.The paper adapts multifidelity and discrepancy modeling to behavioral effects, while evidence records track properties individually rather than collapsing them into a scalar trust score.Source: §2 · pp. 2–3
3.1 Kernel contract and population
- The kernel predicts a distribution over answers to separately specified states, treatments, and outcomes; wording, option labels, field order, and model version are part of its contract and cache key.Personas come from eligible microdata covariates, while population weights remain separate; demographic descriptions are inputs, not validated identities.Source: §3.1 · p. 3
3.2 Prediction and execution
- The tabulated tier queries each unique state once, then reuses its response distribution for event-keyed execution with common random numbers; weighted expectations remove response-draw noise.The measured scale workload included 29.2% repeated rendered states, and its large-population setting used fixed content exposure without an evolving network.Source: §3.2 · p. 3
3.3 Sparse correction and its provenance
- Sparse correction estimates paired flagship effects over anchor states and replaces, rather than adds to, the kernel effect; cells with unreachable targets or fewer than six anchors retain uncorrected predictions.A shared tilt matches a cell mean, not individual effects or a full human response distribution; SocSci210 tests cell-mean correction in study–task blocks, not distributional tilts on every survey scale.Source: §3.3 · pp. 4–5
- The correction-cell rule is developmental evidence from an in-sample Arechar prototype: global country-level tilting worsened discernment, while finer baseline cells improved approximation without necessarily improving human fit.The passage characterizes this as representation-choice evidence, not another independent hybrid test.Source: §3.3 · pp. 4–5
3.4 Discrepancy, behavioral tiers, and implementation status
- The discrepancy model shares study- and arm-level error across affected agents, so behavioral uncertainty does not vanish as population size grows; network conditioning and integrated workflows remain designs, not validated capabilities.Measured corrections are flagship-sourced and must not be labeled experimentally estimated human effects.Source: §3.3 · p. 5, §3.4 · p. 5
4 Materials and evaluation protocol
The evaluation combines held-out social-science experiments, cross-country survey judgments, and a multi-wave intervention dataset. It defines separate measures for calibration, condition sensitivity, decision choice, effect error, and interval coverage, with criteria committed before corresponding held-out analyses.
4 Materials and evaluation protocol
- SocSci210 contains 210 TESS experiments, including 40 unseen studies and 37 supporting held-out intervention comparisons; Arechar spans 45 headlines and 16 countries, while Epstein includes 9,070 participants across five waves.Epstein covers eight intervention types and 20 cards, equally divided between true and false headlines.Source: §4 · p. 5
- The protocol measures centered condition sensitivity, reliable-contrast sign accuracy, captured decision gain, condition-minus-reference effect MAE, and interval coverage of human effect estimates including sampling error.Captured gain compares choices of the model’s highest and lowest conditions with random selection and the available human optimum.Source: §4 · p. 5
- Criteria were committed before each corresponding held-out run or newly resolved outcome analysis; Appendix A contains the protocol, exposure disclosures, retrospective qualifications, and errata.Source: §4 · p. 5
5 Validity as calibration
The section shows that marginal fit alone does not establish intervention validity, and that decision performance remains below the human pilot on the reported comparison. It also reports limited demographic-persona information and a separately evaluated memory tier with gains in sequence-level structure but incomplete coherence.
5 Validity as calibration
- The condition-blind pooled-human oracle has distance 0.0339 versus Jev’s 0.1519, but it is a diagnostic using the evaluation study’s answers and has no condition variation.Its centered correlation is undefined, illustrating why a marginal leaderboard can reward outcome-distribution knowledge without useful intervention predictions.Source: §5 · pp. 5–6
- Jev sign accuracy is 0.671 [0.500, 0.803] and captured gain is 0.268 [0.050, 0.454] on G-A’s test analysis; the stronger declared criterion is not met against the half-sample human pilot.Human sign accuracy is 0.702 against 0.985 for the pilot, and gain is 0.237 against 0.628; increasing simulated respondents from 5 to 200 per cell shows no rising trend.Source: §5 · p. 6
- The cross-validated demographic-group predictor has individual-response correlation 0.013 for these representations and outcomes, without implying that all respondents are interchangeable or richer measurements lack information.Source: §5 · p. 6
- In six clean within-subject studies, three-answer self-memory with rank-preserving marginal correction passes coherence and marginal criteria; median correlation-pattern agreement rises from 0.01 to 0.52, but distant-pair coherence reaches only 40% of human coherence.These sequence-level results do not establish individual prediction or persistent network behavior.Source: §5 · p. 6
6 Content fidelity across 16 countries
All three Arechar held-out parts pass the frozen item-and-person gate, including both non-US parts across all 15 countries. Translated-questionnaire results test transfer across participant populations and human instruments, not Jev’s multilingual inference; intervention sensitivity remains a separate capability.
6 Content fidelity across 16 countries
- All three Arechar held-out parts pass the frozen item-and-person gate, and both non-US parts pass sharing, accuracy, and truth-discernment gates in all 15 countries.Table 1’s upper panel reports kernel gates and its lower panel compares models on the same 30-person-per-country-condition subset in the both-new part.Source: §6 · p. 6, Table 1 · p. 7
- Among nine translated-questionnaire countries, new-country median sharing/accuracy correlations are 0.736/0.852, versus 0.711/0.832 for English; this tests population and instrument transfer, not multilingual inference.Jev reads English in both groups.Source: §6 · p. 6
- The kernel fails both intervention-sensitivity gates, while Astra passes; Astra’s prompt advantage over Jev is 0.077 [0.055, 0.099] and tips advantage is 0.030 [0.008, 0.051].Astra’s pooled effects reach approximately 40–60% of human effects; tips evidence passes the per-intervention rule but not a two-test Bonferroni threshold.Source: §6 · p. 6
- Cross-country comparisons are not supported; familiar accuracy interventions also do not establish inference on novel interventions, and recall versus inference remains unresolved.The A3c publication-status test finds no detectable interaction in flagship advantage: 0.044 [-0.078, 0.170].Source: §6 · pp. 6–7
7 Sparse flagship calibration
Sparse flagship anchors reduce intervention-effect error in Epstein and improve approximation and captured gain on held-out SocSci210, while policy-value, magnitude, and pairing conclusions remain qualified.
7.1 Epstein: lower intervention-effect error
- Epstein effect MAE falls 41%, with paired absolute-error improvement 0.0125 (95% CI [0.0064, 0.0167]) and lower hybrid error out of wave.The evaluation used 1,080 anchors, approximately 1.7% of 64,000 application states.Source: §7.1 · p. 7
- With the same Epstein anchors, increasing personas from 10 to 200 per wave leaves hybrid MAE essentially flat at about 0.0180–0.0181.The approximate perfect-model sampling floor is 0.0117; the observed hybrid MAE is not equal to that floor.Source: §7.1 · p. 7
- The preregistered primary policy-value gain is not shown: hybrid minus kernel is 0.0027 [-0.0118, 0.0169].The best-minus-worst policy range is 0.039 across five decisions, and numerical proximity to the human pilot is not an equivalence test.Source: §7.1 · p. 8
- Epstein’s kernel selects the human worst arm in wave 3, while correction is fallible even when average error improves.In wave 3 it predicts +0.032 against −0.011 observed; wave 4 partisan norms are overpredicted (+0.056 versus +0.006).Source: §7.1 · p. 8
- Headline heterogeneity is not established: hybrid within-arm r is 0.157 [-0.001, 0.241], below direct audit and kernel control-sharing estimates.A control-distribution tilt is proposed for a future frozen test, not validated as a repair.Source: §7.1 · pp. 8–9
7.2 SocSci210: improved captured decision gain
- Across 37 held-out SocSci210 studies, the primary approximation correlation rises from 0.608 to 0.709 [0.550, 0.832], meeting both frozen requirements narrowly.This shows movement toward the flagship, not exact reproduction.Source: §7.2 · p. 9
- Captured gain rises from 0.268 to 0.386, an improvement of 0.119 [0.014, 0.224] at three anchors, but hybrid superiority to the flagship is not established.One anchor per condition yields gain 0.114 [0.014, 0.215], and two yield 0.118 [0.006, 0.224].Source: §7.2 · p. 9
- At three SocSci210 anchors, raw effect MAE rises from 0.052 to 0.083; equal-budget pairing is not shown to outperform unpaired prediction, and agreement is not a calibrated runtime confidence rule.On reliable test contrasts, kernel and hybrid agree on 78.9%, with accuracy 0.742 there.Source: §7.2 · p. 9
8 Discrepancy propagation
Shared human–model discrepancy remains a source of behavioral uncertainty even as persona-sampling noise falls. Sparse calibration and tabulated execution reduce the need for expensive predictions, but model agreement and scale do not remove validation limits.
8 Discrepancy propagation
- Retrospective coverage on 37 held-out studies was 0.67/0.93/0.96, versus 0.14/0.29/0.36 for human-sampling-only intervals at nominal 50%/80%/90%.At 90%, mean interval width was 0.324 and interval score 0.422, compared with 0.064 and 0.746 without discrepancy; lower interval score is better.Source: §8 · p. 9
- The discrepancy scale, τ^2 + σ^2 = 0.088, exceeds persona-sampling standard deviation at five respondents per cell (0.0235) and 100 respondents (0.0033).Astra also required discrepancy modeling: β = 0.579 [0.32, 0.66], with test coverage 0.63/0.86/0.94.Source: §8 · p. 9
- Jev’s discrepancy slope shifts from 0.925 on selected studies to 0.661 on 20 unselected development studies and approximately 0.50 on test reliable contrasts.Residual estimates also depend on simulation sample size; separating excess discrepancy from Monte Carlo variance and using unselected, deployment-matched data remain planned improvements.Source: §8 · pp. 9–10
9 Cost and scale
- On SocSci210, Jev, Luna, and Astra captured gains of 0.272/0.305/0.336; Astra’s gain advantage over Jev was 0.064 [-0.048, 0.168].All three err together on 19.4% [9.6%, 32.9%] of reliable contrasts, versus 3.5% under error independence; higher distributional fit does not establish uniform decision advantage.Source: §9 · p. 10
- Common random numbers reduced effect standard deviation from 0.00303 to 0.00072 across 100 seeds at 104 agents, an approximately 18-fold variance reduction.The exact conditional effect was 0.0221; the S1 execution timings used one laptop and excluded table construction.Source: §9 · p. 11
- Reproducing vendor outputs is not guaranteed despite reproducible code, because measurements depend on proprietary model access and recorded versions.The 20,000-state S1 workload completed in 920.7 seconds; 29.2% of requested states were exact cache hits.Source: §9 · p. 11
10 Licensed uses and evidence records
KITE scopes each reported property to its evidence, correction provenance, and uncertainty. The supported applications are limited to specified screening and comparison uses, not general claims about correctness.
10 Licensed uses and evidence records
- An evidence record links each reported property to its reference population and stimuli, model and configuration hashes, operator provenance, uncertainty method, and unsupported assumptions.Correction records also expose requested and achieved means, feasibility, affected weight, and cells where the original kernel effect was replaced.Source: §10 · p. 11
- Supported uses are content screening, direction screening for literature-known intervention families with calibration, and cost–quality comparison with explicit discrepancy.A narrow interval from repeated simulation supplies no additional license.Source: §10 · p. 11
- A property supported by one evidence-record row does not automatically transfer to another row.The integrated user-facing report remains a design.Source: §10 · p. 11, Table 4 · p. 12
- Neither agreement between models nor a provider’s confidence field is a calibrated probability that a conclusion about people is correct.Source: §10 · p. 12
11 Limitations
The paper limits its claims to a narrow set of human-referenced tests, intervention settings, and supported model uses. It also identifies constraints on fidelity, reproducibility, and appropriate interpretation of demographic and content-ranking evidence.
11 Limitations
- Only Epstein and SocSci210 provide held-out human-referenced hybrid tests, and their effect structures and prior exposure differ.D1’s policy-value primary and later headline-heterogeneity primary were not shown; only ten wave-relative effects support its error result.Source: §11 · p. 12
- Sparse calibration can import flagship errors, as in Epstein wave 5, while the kernel overpredicts norms messages; neither parent is universally better.D3’s primary approximation gain passes narrowly, within-task sensitivity is unchanged within uncertainty, and raw effect MAE worsens.Source: §11 · p. 12
- Effect magnitudes need further calibration because discrepancy slopes vary across study selections, and broad Epstein coverage does not validate cross-domain uncertainty.The experiments do not license prediction of interventions absent from the literature.Source: §11 · p. 12
- Local memory coherence and cohort-level marginal correction may erase real persistence or treatment pathways when exposure is endogenous.Large-scale tabulated execution does not validate the mapping as an online transition process, network diffusion, or cascade scale.Source: §11 · p. 12
- Stimulus reconstruction, omitted card images, representative within-cell stimuli, demographic compression, and supported-scale exclusions constrain fidelity.Proprietary access and unavailable response caches mean reproducible code does not guarantee independent regeneration of the same predictions.Source: §11 · p. 12
12 Ethics, terms, and availability
- The demonstrations concern misinformation harm reduction, not persuasion optimization, election forecasting, or replacement of human participants.Simulated demographic profiles may reproduce stereotypes; content-ranking evidence must not infer individual beliefs or rank national populations by susceptibility.Source: §12 · pp. 12–13
- The full Jev response cache is not released, and outputs are not used for distillation or training an imitation model.Correction operates on task-specific distributions and does not train a surrogate.Source: §12 · p. 13
A.1 Datasets, sampling, and stimuli
The appendix specifies the datasets, sampling choices, prior-exposure qualifications, and metrics underlying the evaluations. These details define the scope of the reported comparisons and uncertainty estimates.
A.1 Datasets, sampling, and stimuli
- For SocSci210, shared-scale comparisons leave 37 studies, 162 blocks, 532 reference-condition effects, and 304 reliable pairwise contrasts for D3.The marginal analysis includes 39 studies.Source: §A.1 · p. 14
- The SocSci210 kernel table has 136,901 predictions, with up to 200 respondent rows per cell, not every original human respondent.D3 and D2 development uses 91 seen studies and 2,195 effects selected for reliable contrasts; the earlier unselected set contains 20 studies, and these sets are not independent replications. D3 anchors are the first three respondents in deterministic hash order, queried under every condition; one representative stimulus is used per condition although 24% of cells contain variants.Source: §A.1 · p. 14
- Arechar calibration used the United States and odd-indexed items; test parts were US/even, other countries/odd, and other countries/even.Epstein randomization was within wave; the kernel used 200 application and 30 disjoint audit personas per wave, and the original cards included images.Source: §A.1 · p. 14
A.2 Commitment, prior exposure, and errata
- Criteria files specify representations, seeds, endpoints, nulls, and verdicts before corresponding held-out runs or newly resolved outcome analyses.These are version-controlled preregistrations with differing prior exposure, not a claim that all test information was unseen; D2 test coverage is retrospective.Source: §A.2 · p. 14, Table 5 · p. 15
- D1 errata correct the sharing-effect count to ten, replace forecast costs with actual 10,080 flagship items and $1.60, and document a serialization-only rerun.The broad D2 coverage interval is recorded as uninformative without changing the endpoint; D1 headline criteria and D3 report no errata.Source: §A.2 · p. 14
A.3 Metric definitions and uncertainty
- Marginal distribution fit is Wasserstein distance on the stated [0, 1] answer scale, not L1 distance between probability vectors.Captured gain compares model-selected human condition means with available gain, weights blocks by available gain, and uses a study bootstrap; reported intervals are generally 95% bootstrap intervals, with D3 flagship comparisons using declared 90% intervals.Source: §A.3 · pp. 15–16
B Epstein: choices, effects, and robustness
Epstein results compare human choices and discernment with kernel, hybrid, and direct-audit estimates, while documenting uncertainty and robustness conditions. Reduced discernment error does not improve both response channels.
B Epstein: choices, effects, and robustness
- Table 6 reports locked within-wave choices and observed discernment; human choices are descriptive optima from the full sample, and wave 1 has no sharing intervention.Controls are wave-specific.Source: Table 6 · p. 16
- Table 7 reports ten wave-relative discernment effects on the binary sharing scale, with human intervals bootstrapped within wave and arm using 4,000 resamples.The table also reports treated and control participant counts.Source: Table 7 · p. 16
- No system exceeded its chance baseline at p < 0.05 in the policy-value results.Difference intervals use 2,000 participant bootstraps; chance references use 5,000 within-wave permutations.Source: Table 8 · p. 17
- Below 200 personas, Table 9 entries are means ± standard deviations across 20 subsamples; the full-table row is deterministic conditional on recorded predictions.These are not confidence intervals.Source: Table 9 · p. 17
- Kernel true/false channel MAEs were 0.0139/0.0289, versus hybrid 0.0235/0.0224 and direct audit 0.0222/0.0231.The lower discernment error combines improved false-card response with worse true-card response.Source: B Epstein: choices, effects, and… · p. 17
- Paired calibration’s descriptive effect correlation with people was 0.60, versus 0.05 for direct audit; differing panels and budgets preclude a causal comparison.These correlations are descriptive, not evidence of a causal benefit from pairing.Source: B Epstein: choices, effects, and… · p. 17
C SocSci210: full paired-anchor results
SocSci210 reports paired-anchor comparisons on test alongside development diagnostics, then details how discrepancy intervals and model fits are evaluated. Test agreement limits extrapolation from developmental diagnostics.
C SocSci210: full paired-anchor results
- Table 10 covers 37 studies, 162 blocks, 532 effects, and 304 reliable contrasts; F3 and F5 share respondents, inflating their approximation correlation.Fk denotes unpaired flagship respondents per cell; Hk denotes paired anchors per condition plus the kernel table.Source: Table 10 · p. 17, Table 11 · p. 18
- Development H3−K sign improvement was 0.038 [0.008, 0.084], gain 0.076 [−0.017, 0.179], and within-task r difference 0.008 [−0.087, 0.078].The test approximation primary is an improvement criterion, not the earlier proposed equivalence test.Source: C SocSci210: full paired-anchor results · pp. 17, 19
- At 90%, H3–F5 differences were −0.006 [−0.041, 0.007] for sign, −0.033 [−0.087, 0.028] for gain, and −0.047 [−0.101, −0.009] for within-task r.These are the declared 90% comparisons; other paired-difference rows use 95%.Source: Table 12 · p. 18, C SocSci210: full paired-anchor results · p. 19
- Kernel–flagship effect agreement was 0.863 on development and 0.608 on test, limiting extrapolation from developmental diagnostics.Table 13 contains development results, not independent confirmation.Source: C SocSci210: full paired-anchor results · p. 19, Table 13 · p. 19
D Discrepancy model: full table
- Table 14’s selected fit uses five model respondents per cell; its unselected fit uses 20 development studies with 50 respondents per cell.Table 15’s test discrepancy intervals include parameter uncertainty, while no-discrepancy intervals include human sampling uncertainty only.Source: Table 14 · p. 19, Table 15 · p. 19
- The interval score penalizes width and misses, while the selected-study versus unselected-fit comparison cannot isolate study selection from model-sampling noise.Larger coverage alone can be achieved with excessive width.Source: D Discrepancy model: full table · p. 19
E Additional calibration and comparison tables
The additional tables and figures report variance reduction and define comparisons, while provenance notes qualify how recorded outputs and uncertainty should be interpreted. Several execution costs depend on recorded runs and probe-level accounting.
E Additional calibration and comparison tables
- Table 16 reuses nested subsets of one recorded kernel run, so its entries are not independent experiments; costs use an average of 4.3 conditions.Costs are per simulated experiment.Source: Table 16 · p. 20
- Table 17’s intended common sample has 3,615 items, but the historical Jev five-per-cell file lacks two; its upper- and lower-panel correlations use different scorer populations.Lower-panel paired differences therefore do not equal differences between all upper-panel entries.Source: Table 17 · p. 20
- Table 18 defines magnitude as mean absolute inter-item Spearman correlation relative to humans, and structure as correlation between simulated and human correlation patterns.“Distant/human” compares corrected distant-pair coherence with people’s.Source: Table 18 · p. 20
F Provenance and interpretation notes
- Human uncertainty in Table 7 and Figure 4 comes from participant-level bootstrap results resolved in RESOLUTIONS.md, not intervals inferred from rounded means.The tables and figures use the included records and supplied vector figures.Source: F Provenance and interpretation notes · p. 20
- SocSci210’s 456,587 prepared observations are response rows, with simulated sampling capped per cell; three-model and D3 scorer populations differ.The distinct scorer populations explain their different condition-sensitivity correlations and magnitude ratios.Source: F Provenance and interpretation notes · pp. 20–21
E Additional calibration and comparison tables
- At 10^4 agents, common random numbers reduced treatment-effect SD from 0.00303 to 0.00072 across 100 seeds, an approximately 18-fold reduction.The vectorized-engine timings exclude table construction and fresh inference.Source: Figure 5 · p. 21
F Provenance and interpretation notes
- Cache-hit and cache-miss counts refer to option probes, not predictions; the D1 cost was recovered from console output, and B2’s roughly $3 cost is estimated.The B2 estimate is not a measured cost claim.Source: F Provenance and interpretation notes · p. 21