PaperlayerView exact revision on arXiv

Source-linked AI summary

The AI Evaluation Ecosystem

Yash Dave, Sang T. Truong, Serena Wang, Sanmi Koyejo

arXiv:2610.09296v1cs.AIcs.CY

TL;DR

Benchmark design is often studied as a fixed instrument, even though evaluation outcomes shape ecosystem decisions and future measurement. The paper builds a hybrid generative agent-based simulation and applies it to benchmark holdout design. In the modeled setting, private holdouts shrink score–satisfaction gaps on most benchmarks but widen them on a few, depending on how holdout weights shift scoring credit.

  • Problem

    Benchmark-design methods often assess instruments while holding the surrounding ecosystem fixed, despite feedback among actors and difficult-to-study counterfactual dynamics.

    Source: §1 · p. 1

  • Method

    The paper develops a hybrid GABM with LLM-driven strategic actors, structurally partitioned information, and rule-based market dynamics to study benchmark holdout design.

    Source: §1 · p. 3, §4 · p. 5

  • Results

    From public_only to private_only, the score–satisfaction gap shrinks on 10 of 13 benchmarks and widens on three in LLM runs, in every seed.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The framework offers a hypothesis-generating sandbox for studying how evaluator and policy choices reshape ecosystem dynamics.

    Source: §1 · p. 3, §6 · p. 11

    The finding is scoped to a leaderboard-driven market with mixed-population gap measures, and it requires testing on real leaderboards.

    Source: §5.3 · p. 11

Abstract

from arXiv · show

AI evaluation shapes the decisions of model providers, users, funders, and regulators. We argue that designing valid benchmarks requires contextualizing design choices in the dynamics of this ecosystem of actors. We develop a simulation architecture that combines rule-based market dynamics with LLM-driven strategic actors, building on advances in Generative Agent-Based Modeling (GABM). We model benchmarks, consumer needs, and provider capabilities as vectors over a six-dimensional capability space (reasoning, coding, knowledge, safety, communication, agentic), with structural information partitions across actors. As a case study, we apply this stylized simulation to explore benchmark holdout design. We find that moving from public benchmarks to private holdout benchmarks shrinks the gap between benchmark scores and user satisfaction on most benchmarks but widens it on a few, depending on where holdout weights shift scoring credit. We stress-test our findings at both the instrument and case-study level, drawing on the V&V framework of Sargent (2013) and GABM-specific evidence criteria. Beyond holdout design, our simulation is a hypothesis-generating sandbox for studying how evaluator and policy choices, in turn, reshape the ecosystem.

1 Introduction

The paper argues that benchmark design must account for an ecosystem in which evaluation outcomes reshape actor decisions, rather than treating benchmarks as fixed, one-shot instruments. It develops a GABM framework and uses benchmark holdout design as a hypothesis-generating case study, with findings stress-tested at both the instrument and case-study levels.

1 Introduction

  • Benchmark design often treats calibration, validity, and contamination resistance as fixed-instrument properties, although benchmarks also steer research and model selection and can distort what they measure when optimized.The paper frames evaluation as a closed-loop system in which benchmarks affect ecosystem behavior after adoption.Source: §1 · p. 1
  • Ecosystem questions about provider incentives, evaluator independence, and feedback channels are difficult to study because formal models become intractable and counterfactual empirical data may be unavailable.The paper notes that policy designs may address symptoms rather than causes when these dynamics are not understood.Source: §1 · pp. 1, 3
  • The paper develops an ecosystem framework and hybrid GABM in which LLM-driven actors reason over partitioned information, while a heuristic mode isolates structural from behavioral dynamics.The simulation varies a benchmark’s publication of items and dimension weights, which the paper calls holdout design.Source: §1 · p. 3
  • Private-holdout benchmarks do not always reduce the gap between benchmark scores and user satisfaction in the simulated ecosystem.The study treats the simulation as a hypothesis generator, not a predictor.Source: §1 · p. 3
  • The authors stress-test the simulation as a laboratory and separately test whether the holdout-design finding holds up, drawing on V&V and GABM evidence criteria.The cited frameworks include Sargent [2013], Zhou et al. [2025], and Vezhnevets et al. [2023].Source: §1 · p. 3

2 Related Work

The related work spans AI evaluation and benchmarking, governance of information systems, and generative agent-based modeling. This paper positions ecosystem-level dynamics as distinct from individual-benchmark failure modes and uses a hybrid simulation design.

2 Related Work

  • Benchmarking frameworks and preference evaluations provide coverage, standardized measurement, and dynamic rankings, while related work documents pathologies including gaming, contamination, construct-validity critiques, and adoption concentration.These studies primarily address failure modes at the individual-benchmark level.Source: §2 · p. 3
  • The paper treats evaluation as shared measurement infrastructure, distinct from supply-chain flows or a firm’s value-creating activities, and studies how competition, capital allocation, and consumer behavior can produce failures.This ecosystem framing is presented as complementary to adjacent supply- and value-chain accounts.Source: §2 · pp. 3–4
  • The simulation restricts LLM reasoning to strategic planning while deterministic rules govern market mechanics, scoring, and state transitions, combining agent expressiveness with causal transparency.The related work also notes validation methods and robustness checks for generative-agent simulations, alongside concerns about black-box behavior, bias, stochasticity, and variance underrepresentation.Source: §2 · p. 4

3 Mapping the AI Evaluation Ecosystem

The ecosystem consists of actors with different goals, information access, and decision rights, whose responses to evaluation outcomes reshape future evaluations. The section maps the actors and the channels through which evaluation information flows.

3 Mapping the AI Evaluation Ecosystem

  • The framework maps six actor types and their primary information flows, then turns those channels into simulation mechanisms that can be altered one at a time.The channels span development, market, and regulatory dynamics.Source: §3 · p. 4
  • Providers use internal and external signals to adjust training and releases, while evaluators design, publish, and retire benchmarks and may face conflicts of interest.Third-party evaluators have proposed minimum operating conditions covering model access, contingent compensation, and recusal.Source: §3 · p. 4
  • Consumers use leaderboard scores and news as capability proxies when choosing models, while regulators act on public signals, incidents, and media because they cannot directly observe provider capabilities or consumer satisfaction.Adoption decisions affect provider revenue and funder attention; regulators can mandate evaluations, enforce disclosure, or issue voluntary frameworks.Source: §3 · p. 4
  • Funders allocate capital using scores, adoption, media sentiment, and incidents, reinforcing benchmark optimization, while media attention amplifies or attenuates signals received by other actors.The section describes capital flows as performative and media influence as selective coverage.Source: §3 · p. 4

4 Generative Agent-Based Model of the Evaluation Ecosystem

The model represents evaluation as interactions among institutional actors with distinct information and decision roles. Its stylized mechanics connect benchmark scoring, provider investment, consumer satisfaction, and sequential ecosystem responses across a round-based simulation.

4 Generative Agent-Based Model of the Evaluation Ecosystem

  • The model treats institutions as strategic actors and consumers as a rule-based population, using a stylized structure to surface candidate dynamics rather than prove real-world outcomes.Six-dimensional capability vectors, hand-calibrated consumer needs, linear scoring, perturbed holdout weights, and a safety lever are deliberate modeling choices.Source: §4 · p. 5

4.1 Benchmark Model and Structural Misalignment

  • Benchmark weights are hidden from actors, and the modeled suite is structurally misaligned with consumer needs, exceeding average needs on reasoning and coding while falling below them on communication early and safety late.The simulation uses 13 active benchmarks over the 40-round horizon, beginning with four benchmarks anchored to real evaluations.Source: §4.1 · p. 5

4.2 Provider Investment and Emergent Gaming

  • Providers allocate budgets across R&D, safety, and product levers; the score–satisfaction gap emerges when benchmark weights diverge from consumer needs and providers learn benchmark-focused investment yields faster score gains.The gap compares a benchmark’s published score with matched consumer satisfaction weighted by segment size and alignment with the benchmark’s dimensions.Source: §4.2 · pp. 5–6

4.3 Ecosystem Actors and Feedback Channels

  • Information access is partitioned into public, private, and ground-truth tiers, while a heterogeneous market of 51 consumer segments assesses delivered capability rather than published scores.Incidents reach downstream actors; media influences exploration, regulators use an escalation ladder, and funders allocate capital into provider budgets.Source: §4.3 · p. 6

4.4 Round Protocol and Experimental Setup

  • Sequential round phases let later actors observe earlier outcomes, with providers, regulators, and funders LLM-driven while consumers and media remain rule-based; heuristic runs provide a non-LLM baseline.The heuristic mode supports distributional claims, while LLM runs provide strategic reasoning traces and directional cross-checks; the lower LLM seed count limits distributional inference.Source: §4.4 · pp. 6–7
  • In one 40-round baseline run, a critical incident cost the market leader 36 percentage points of share in one round, and its recovery reached about 68% by round 40.Figure 2 also displays incidents and regulator interventions, provider R&D and safety allocations, and end-of-run benchmark scores against matched satisfaction.Source: §4.4 · p. 7, Figure 2 · p. 7

5 Results: Case Study of Benchmark Holdout Design

Private holdouts narrow the benchmark-score/user-satisfaction gap on most benchmarks but widen it on some, depending on how scoring credit shifts. The simulation checks this mechanism and frames broader ecosystem studies as hypothesis generation, not quantitative prediction.

5.1 Holdout design: private holdouts shrink the gap unevenly across benchmarks

  • ∆g rises as dominant-dimension capability surplus falls (r=−0.69 LLM; −0.71 heuristic), because holdout weights shift scoring credit between stronger and weaker dimensions.Matched satisfaction uses true weights and is unaffected; therefore, the score shift changes the gap.Source: §5.1 · p. 8
  • From public_only to private_only, gaps shrink on 10 of 13 benchmarks in LLM mode and 9 of 13 in heuristic mode, but widen on selected benchmarks.In LLM mode, gaps widen for Instruction Following, Clinical Reasoning, and Legal Reasoning; Adversarial Robustness is the lone cross-mode disagreement, with small shifts.Source: §5.1 · p. 8, Figure 3 · p. 8

5.2 Validation

  • Cross-mode signs agree on 12 of 13 benchmarks; rescoring end-of-run capability under both weight sets reproduces the shift direction on all 13 and most of its size.The checks include an iid_holdout null, cross-vendor replication, within-benchmark rescoring, and a fitted null recovering 97% of ∆g variance; no real-world counterfactual ecosystem is available.Source: §5.2 · p. 9

5.3 Limitations

  • The six-dimension ontology is a modeling choice: a flatter ontology or different investment-axis separability could change which benchmarks widen or eliminate the asymmetry.The mechanism is robust within the chosen ontology.Source: §5.3 · p. 10
  • The simulated setting is leaderboard-driven, public-benchmark-dominated, multi-firm competition; engagement in procurement or enterprise-B2B contexts is unclear.The model also omits providers’ internal evaluation infrastructure, another pressure on R&D direction.Source: §5.3 · p. 10
  • Consumer need-weights are researcher judgments not validated against user-reported preferences, and four of 13 classifications flip when measured against safety-heavy segments alone.Need-weights set the starting gap, whose sign determines whether a privacy-induced shift widens or shrinks it.Source: §5.3 · pp. 10–11

5.4 A catalog of simulation case studies

  • The architecture can host policy and institutional-design case studies by adding actor levers, channels, or information-flow perturbations to trace ecosystem pathways.Its role is to reveal pathways and candidate dynamics for empirical study, not predict real-world effects quantitatively.Source: §5.4 · p. 11

6 Conclusion

The conclusion treats AI evaluation as an interacting system and uses a stylized simulation to examine benchmark holdout design. Private holdouts usually narrow the score–satisfaction gap, but the direction depends on the benchmark suite and where scoring weights shift.

6 Conclusion

  • The paper frames evaluation as interacting actors and feedback loops that jointly shape measurement, use of results, and emerging failure modes.It maps those loops, instantiates them in an agent-based simulation, and studies holdout design as one case.Source: §6 · p. 11
  • Private holdouts shrink score–satisfaction gaps on 10 of 13 benchmarks but widen them on three in every seed; which benchmarks widen depends on the suite.The widening is associated with holdout weights shifting credit toward dimensions where provider R&D has concentrated capability.Source: §6 · pp. 11–12
  • Heuristic and LLM planners agree on the gap-change direction for 12 of 13 benchmarks, and rescoring end-of-run capability under holdout weights reproduces all 13 directions.The iid_holdout null removes the stable pattern, while rescoring starting capability gives the wrong direction on three benchmarks.Source: §6 · pp. 11–12
  • The simulation generates hypotheses for empirical and policy tests, but predicts no real market because capability and R&D allocation are unobservable there.No natural experiment holds the benchmark suite fixed while changing its holdout policy.Source: §6 · p. 12
  • The simulation can also examine evaluator and policy choices, including conflicts of interest, transparency mandates, audit verification, media coverage, and benchmark sponsorship.Holdout design is the first of these questions worked through in the paper.Source: §6 · p. 12

A Extended Related Work

  • The extended related-work appendix covers governance and performativity, AI supply and value chains, GABM validation, simulation methods, and governance in other industries.Its public-goods discussion draws on information-security economics and AI governance work.Source: A Extended Related Work · p. 21, §A.1 · p. 21

A.1 Governance, public goods, and performativity

  • Existing model, dataset, system, and evaluation obligations do not model feedback loops that can undermine evaluation integrity when evaluation becomes a competitive signal.Systems-thinking and performativity perspectives provide a way to study measurement and optimization co-evolving at ecosystem scale.Source: §A.1 · p. 21

A.2 AI supply chains, value chains, and the evaluation layer

The appendix contrasts evaluation with linear supply- and value-chain framings, then reviews hybrid GABM design, simulation’s role alongside other methods, and governance analogues. It also specifies the simulation’s round protocol and capability dimensions.

A.2 AI supply chains, value chains, and the evaluation layer

  • Evaluation is shared measurement infrastructure, not a product flowing downstream or a firm activity; actors use it in parallel, and failures can arise when measured actors optimize against it.The paper positions this layer as one on which supply-chain actors depend and against which value-chain steps are priced.Source: §A.2 · pp. 21–22

A.3 Generative agent-based modeling with LLMs: extended discussion

  • The design delegates strategic planning to LLM actors while rules govern market mechanics and state transitions, and it limits claims to collective patterns rather than point estimates.The appendix notes concerns about black-box behavior, stochasticity, bias, and insufficient behavioral variance, and describes coherence checks and validation against Sargent’s framework.Source: §A.3 · p. 22

A.4 Why a simulation, and what other methods offer

  • Simulation permits structural-hypothesis tests by changing one ecosystem component at a time, while game theory, system dynamics, observational studies, and expert elicitation offer complementary strengths.The appendix says the multi-actor, multi-benchmark, multi-round structure resists closed-form solution, and system dynamics cannot represent provider differences.Source: §A.4 · pp. 22–23

A.5 Governance analogues in other industries

  • Clinical research and aviation offer governance mechanisms the simulation can test, including independent evaluation, mandatory disclosure, liability routing, and information sharing.The analogies are imperfect because AI capability changes quickly and evaluation targets are more contested than clinical endpoints or airworthiness standards.Source: §A.5 · p. 23

B Full Simulation Architecture

  • The appendix specifies a 40-round simulation in which each round represents one month, with tables defining actors, parameters, and capability dimensions.It identifies separate tables for benchmarks, providers, consumers, regulator levers, and funder parameters.Source: B Full Simulation Architecture · p. 23

B.1 Round Protocol (Pseudocode)

  • Each round proceeds from provider planning and evaluator scoring through belief updates, incidents, and media coverage, then consumer, regulator, and funder responses.Providers plan from prior public signals; downstream actors respond in that order to the round’s public state.Source: §B.1 · p. 23, Figure 4 · p. 24

B.2 Capability Dimensions

  • The model uses six capability dimensions that providers can invest in separately and public benchmarks measure, without assuming statistical independence.Table 3 links the dimensions to real-world anchors and primary investment pathways.Source: §B.2 · p. 24, Table 3 · p. 24

B.3 Benchmark Pool and Introduction Schedule

This section specifies how benchmarks and other ecosystem actors are represented and updated, and how public and private holdout scores differ, are observed by providers, and are varied. It identifies stipulated design choices and the roles of score asymmetry and holdout fraction, while noting that benchmark coverage does not fully match consumer needs and conclusions are conditional on modeling assumptions.

B.3 Benchmark Pool and Introduction Schedule

  • 0.26 vs 0.18 for reasoning and 0.18 vs 0.13 for coding, benchmark mean loadings exceed population-average need weights; communication shows the reverse, at 0.17 vs 0.27.By round 40, safety need is about 0.22 while mean benchmark loading is about 0.15; the four initial benchmarks also weight coding and safety more than consumers do.Source: §B.3 · p. 25
  • Four benchmarks are active at round 0, with nine scheduled every four rounds or two rounds after a score stall; all 13 are active by round 36.At least one benchmark enters early in about two thirds of runs, and final per-benchmark results are the same with or without early introductions.Source: §B.3 · pp. 25–26, Table 4 · p. 25

B.5 Provider Investment Model

  • Provider R&D targets blend benchmark beliefs, weighted 0.80, with a consumer-need signal whose reliability depends on product investment; safety spending separately raises safety capability.The simulation also applies diminishing returns to funding and capability, and smooths portfolio allocations over the previous three rounds.Source: §B.5 · pp. 26–27

B.6 Heuristic Provider Planning

  • Heuristic provider planning is a structural baseline: fixed benchmark focus levels combine with rules that shift portfolio shares in response to incidents, share trends, crises, and regulatory interventions.The rules clip and renormalize allocations within bounds; unlike LLM mode, heuristic providers do not change benchmark priorities during a run.Source: §B.6 · pp. 27–28

B.7 Consumer Market

  • Consumer satisfaction reflects experienced capability, incidents, and cost, while switching responds to expected satisfaction; a segment’s expected quality blends leaderboard scores with experienced quality.The market has 51 segments, and organizational consumers’ share rises from 0.25 to 0.55 over a run.Source: §B.7 · p. 28

B.8 Media Actor; B.9 Regulator; B.10 Funders

  • Media coverage can be triggered by incidents, score gains, leaderboard changes, saturation, or divergence; one regulator uses fixed incident-rate response thresholds, while four funder types allocate capital across providers.Media outputs include sentiment, provider attention, risk signals, and a narrative state; funder allocations feed into provider budgets.Source: §B.8 · p. 29, §B.9 · p. 29, §B.10 · p. 30

B.11 Open-Source Provider Mechanics; C.2 Assumption status

  • OpenCore differs from closed providers through funding, cost, safety-floor, product-signal, and information advantages, including revealing benchmark-weight information in proportion to its market share.The assumptions behind the holdout-design result are rated calibrated, anchored, or stipulated, and the conclusions are conditional on those assumptions.Source: §B.11 · p. 30

C.2 Assumption status

  • The 13-benchmark suite has three benchmarks whose highest-loading dimension is reasoning and two for each other dimension, but its stipulated vectors may over-represent coding, under-represent knowledge, and omit multimodal coverage.The comparison is informal and concerns coverage only; the text-only ontology excludes multimodal benchmarks, and the per-benchmark loading vectors are not externally fitted.Source: §C.2 · p. 32

C.4 Private Benchmark Score Reporting

  • The reporting cadence is K = 3 rounds, calibrated to frontier-lab score advances every 2.5 to 3.3 months; all private benchmarks publish together every K rounds.Source: §C.4 · p. 34
  • The cosine between public and holdout weights abstracts provider-observable score asymmetry from contamination, training on test-task structure, and adversarial holdout construction.The simulation does not model these sources separately; the iid_holdout ablation sets cos θ = 1 to remove the composite distance and isolate reporting lag and noise.Source: §C.4 · p. 34
  • Realized public–holdout cosine medians were 0.98 for partial benchmarks and 0.91 for private benchmarks, with ranges of 0.94–0.99 and 0.76–0.94, respectively.Partial benchmarks use hand-authored holdout weights; private benchmarks double their distance from public weights, clipping at zero and renormalizing. The perturbation is milder than the target values for most benchmarks and varies across benchmarks.Source: §C.4 · p. 34
  • A partial holdout (h = 0.3) is approximately 1.8× noisier than a private holdout (h = 1.0), slowing providers’ belief updates.Providers update beliefs from published scores using a delta rule, with a noisy-public-weights prior at benchmark introduction; the model excludes learning from the public fraction of items.Source: §C.4 · p. 34
  • The five suite conditions range from public_only to private_only, with baseline assigning 8 public, 3 partial, and 2 private benchmarks.The other conditions are private_dominant (all partial) and iid_holdout (holdout weights equal public weights).Source: §C.4 · pp. 34–35

C.5 Private-Benchmark Cadence Calibration (K)

This section calibrates a default private-benchmark reporting cadence of K = 3 using benchmark histories tied to discrete release events, and evaluates the simulation through model-behavior, comparison, and operational checks. The evidence is limited: qualitative patterns and privacy-condition ordering are reported, but counterfactual ecosystem trajectories are unobservable, and the single-seed shock probe does not establish consistent strategic responses.

C.5 Private-Benchmark Cadence Calibration (K)

  • The analysis uses eight frontier labs and 24 benchmarks, excludes four high-cadence or live-updating benchmarks, and truncates MMLU, GSM8K, and BBH at their last frontier advance.The exclusions avoid checkpoint and community resubmission signals or continuous score drift; truncation avoids ceiling-bound submissions after saturation inflating Kadvance.Source: §C.5 · p. 35
  • Benchmarks are grouped into seven analytical dimensions, but none directly anchors Communication; simulated instruction-following and long-context evaluations inherit Reasoning and Knowledge cadence by analogy.Math is separated from Reasoning because math benchmarks saturate at different rates.Source: §C.5 · p. 35
  • Median-of-medians Kadvance spans 2.47–3.28 months across seven dimensions; the weighted-mean average is 3.39 months, supporting the K = 3 default.The median-of-medians is the statistic anchoring the calibration; K = 3 is within 0.4 months of the weighted-mean average.Source: §C.5 · p. 36, Table 13 · p. 36
  • Restricting calibration to five tier-1 labs preserves the K = 3 anchor and 3.39-month weighted-mean average, with dimension-level median shifts no greater than 0.22 months.The tier-1 median-of-medians range is 2.47–3.10 months; Composite shifts from 3.28 to 3.10 and Agentic from 3.25 to 3.03.Source: §C.5 · p. 36
  • Across tier-1 providers, median Kadvance values cluster at 2.57–4.75 months; OpenAI and Anthropic advance fastest, while Meta’s longer median partly reflects coverage of only 9 of 20 benchmarks.Table 14 reports per-provider aggregate Kadvance across the primary benchmark set.Source: §C.5 · p. 36, Table 14 · p. 36
  • The main ablation synchronizes all private benchmarks at K = 3 rounds to isolate reporting lag, leaving provider-specific asynchronous release heterogeneity outside scope.Provider-specific Bernoulli release rates matched to per-provider medians are identified as a separable sensitivity question.Source: §C.5 · p. 36

D.4 Apparatus-vocabulary audit

  • Prompts label benchmark types neutrally and state their mechanics without framing them as a policy lever, supporting the documented Minimal-Control and Unawareness assumptions.Private benchmarks use holdout-only scoring with sample fraction h=1.0; the prompt vocabulary omits terms such as “gaming” and “Goodhart.”Source: §D.4 · p. 42

E Generative ABM Validation

  • Validation uses independent replications with paired seeds for conditions; heuristic claims use N=50 seeds, while LLM claims use N=10 Sonnet seeds and N=3 each for Opus and GPT-5.5.Server-side LLM sampling is not controlled by the shared seeds, limiting the variance reduction for behavioral comparisons.Source: E Generative ABM Validation · p. 43

E.1 Data validity

  • Five parameters use external data, including private-benchmark reporting lag K=3; these values are not tuned to produce results and are held constant across conditions.Where calibration data is unavailable, the appendix records the value as a modeling choice and explains its rationale.Source: §E.1 · p. 43

E.2 Conceptual model validation

  • The simulation exhibits three design-target patterns and two emergent patterns, including incident-driven path dependence with a provider ending at about 7%–74% share across seeds.The emergent patterns lack external referents; their co-occurrence provides qualitative support for the architecture, not statistical fit to any single output.Source: §E.2 · p. 43

E.3 Computerized model verification

  • The code enforces a three-tier visibility partition, keeping GroundTruth inaccessible to prompt builders and ruling out leakage across information tiers.Per-run instrumentation records LLM reasoning traces; invalid JSON responses are retried up to three times before actor-specific fail-safe actions.Source: §E.3 · pp. 43–44

E.4 PIMMUR audit

  • The PIMMUR audit addresses six validity threats through agent diversity, interaction structure, persistent memory, minimal-control, unawareness, and empirical grounding.The design excludes apparatus vocabulary from prompts and structurally restricts information sharing; consumer behavioral coefficients are hand-constructed and noted as a calibration limitation.Source: §E.4 · pp. 44–45

E.5 Operational validation

  • Because counterfactual ecosystems are unobservable, operational validation focuses on exploring model behavior and comparison with other models, with statistical comparison only against a fitted null.The stated accuracy criterion is the sign of each effect and its rank ordering along the privacy ladder, not prediction of Δg magnitudes.Source: E Generative ABM Validation · p. 43, §E.5 · p. 45
  • Removing the incident stream made the market concentrate further on the leader in every run; other structural ablations showed no stable direction.A capability-release shock appeared in provider reasoning, but portfolio responses differed across reruns, so the authors draw no conclusion from them.Source: §E.5 · p. 46
  • Privacy-condition ordering agreed across heuristic and LLM planners on 12 of 13 benchmarks and replicated across Sonnet, Opus, and GPT-5.5.The cross-model comparison used N=3 per condition for Opus and GPT-5.5; this seed count does not rule out variance underrepresentation or model-version sensitivity.Source: §E.5 · p. 46
  • The null model recovered 97.4% of ∆g variance, while the 40-round simulation added +0.2 pp r2; held-out r2 was 0.955 with +2.0 pp from simulation.Most variance lies between benchmarks, where both approaches use the same fixed weight differences, so these fits do not test the trajectory’s contribution.Source: §E.5 · p. 46

E.6 Shock-propagation smoke test (single seed)

  • The N=1 probe showed injected industry-news framing reaching actor reasoning, but did not establish consistent strategic responses or fidelity to the real ecosystem.At round 25, OpenCore’s reasoning, knowledge, and coding capabilities were perturbed to 95% of leader; the probe rules out prompt-to-reasoning failure only.Source: §E.6 · pp. 46–47
  • Across three released re-runs, provider R&D-share changes ranged from −8 to +4 pp and differed in sign across providers and seeds.In the single reported run, four of five closed-source providers raised R&D share by 3 to 9 pp; this contrast supports drawing no conclusion about portfolio responses.Source: §E.6 · p. 47

E.7 Cross-model reasoning traces

The appendix compares cross-model reasoning about holdout mechanisms, runtime and cost, and the validation evidence supporting the simulation. Across planners, privacy framing differs more than the operational response to flat scores.

E.7 Cross-model reasoning traces

  • Planning entries were tagged with fixed keyword and regular-expression patterns; each entry counts once per frame, not once per keyword occurrence.The patterns were applied consistently across models and checked against sampled excerpts for overmatching.Source: §E.7 · p. 47
  • Privacy framing appeared in ≈32% of Sonnet and Opus planning rounds and ≈14% of GPT-5.5 rounds, which still cited the lag as a reason not to overreact.GPT-5.5 used less mechanical detail, but the cited example shows substantive reasoning about the lag.Source: §E.7 · pp. 47–48, Figure 6 · p. 48
  • All three planners treated flat scores as expected under the reporting lag and declined to over-respond, although GPT-5.5 explained the mechanism less explicitly.The endpoint behavior replicates across planners despite differences in reasoning detail.Source: §E.7 · pp. 47–48
  • Incident framing appeared in ≈62% of Sonnet, ≈33% of Opus, and ≈21% of GPT-5.5 planning rounds.The incident-frame rate was about 3× higher for Sonnet than GPT-5.5; the lower-rate model still covered incidents in shorter paragraphs.Source: §E.7 · p. 48, Figure 6 · p. 48

E.8 Cross-model runtime and cost

  • A 40-round run averaged ≈63 minutes under Sonnet, ≈79 under Opus, and ≈87 under GPT-5.5; estimated billed cost was roughly 1.7–2× higher for Opus or GPT-5.5.Token counts were not logged, so the wall-clock ratio is the firmest reported quantity; Sonnet was used for breadth and the other models for paired-seed checks.Source: §E.8 · pp. 48–49

E.9 Coverage summary

  • Table 17 maps Sargent’s four verification-and-validation activities to the paper’s evidence and its locations, alongside the PIMMUR audit for LLM multi-agent simulations.The appendix also identifies an evidence ceiling for the validation claims.Source: Table 17 · p. 49, §E.9 · p. 49

F Simulation Study of Benchmark Holdout Design

  • Appendix F extends the holdout-design study with distributional results, robustness checks, qualitative reasoning excerpts, ablations, a null-model test, and cross-LLM replication.It also lists methodological caveats applying across the LLM layer.Source: F Simulation Study of Benchmark… · p. 49

F.1 Heuristic layer: role and methodology

The heuristic mode supplies a low-cost structural baseline for comparing ecosystem outcomes with LLM-driven planning. Its simpler rules enable larger samples but omit several strategic behaviors represented in LLM mode.

F.1 Heuristic layer: role and methodology

  • Heuristic runs are three orders of magnitude cheaper than LLM runs, enabling 50 seeds per condition for the privacy-ladder distributional claims.LLM runs use 10 paired seeds per primary Sonnet condition and 3 per cross-vendor condition, so the paper reports directional agreement and matched-seed comparisons rather than tight continuous-outcome intervals.Source: §F.1 · pp. 49–50
  • Agreement between heuristic and LLM privacy-ladder results supports a structural mechanism, while differences can flag LLM-specific behavior such as label-reading.The heuristic planner uses the same observed context but does not read public, partial, or private benchmark labels.Source: §F.1 · p. 50
  • Heuristic providers cannot model identity-driven signal weighting or changing per-benchmark priorities, and benchmark_orientation remains fixed at 0.80.Provider identity affects only initial portfolio and learning rate in the heuristic mode; focus levels remain fixed throughout the run.Source: §F.1 · p. 50
  • The paper does not compare dynamic heuristic and LLM evaluator selection, so the evaluator-mode results do not establish a difference between them.The heuristic evaluator uses a coverage-gap rule, whereas the LLM evaluator selects from the pool itself.Source: §F.1 · p. 50

F.2 Privacy ladder: extended heuristic results

The extended results show that privacy effects vary across benchmarks and conditions, while rescoring evidence attributes most of the heuristic gap change to holdout weights rather than capability changes. Reliability also varies by benchmark, privacy condition, and leader identity.

F.2 Privacy ladder: extended heuristic results

  • Heuristic and LLM modes agree on the sign of ∆g for 12 of 13 benchmarks; the Adversarial Robustness disagreement is within sampling noise, and early introductions leave the LLM shrink count at 10 of 13.Long Context is near the boundary in heuristic mode, and its classification can depend on the run subset.Source: §F.2 · pp. 50, 52, Figure 7 · p. 51
  • Instruction Following widens across every tested suite, while Clinical and Legal Reasoning widen rates vary with roster composition and disappear in the domain-expertise suite.Clinical and Legal Reasoning widen in 97% and 93% of default-suite seeds, but in none under the suite adding a domain-expertise benchmark.Source: §F.2 · pp. 51–52
  • Across 87,617 market-leader rows, overpromise rates range from low single digits to the mid-teens; public_only has the highest rate, while baseline and private_only sit closer to the diagonal.The scatter plots benchmark score against matched consumer satisfaction, with below-diagonal points indicating overpromising.Source: §F.2 · p. 52, Figure 8 · p. 52
  • The reliability pattern persists across providers holding the market-leader role, so the aggregate calibration drift is not driven by one dominant provider.Figure 9 colors market-leader scoring rows by provider identity.Source: §F.2 · p. 52, Figure 9 · p. 53
  • Randomized benchmark assignment yields ≈0.013–0.017 gap compression, and rescoring partial benchmarks with public weights gives a within-benchmark coefficient of ≈0.014 per benchmark.The within-benchmark counterfactual holds capability, cadence, and benchmark identity fixed while varying weight asymmetry.Source: §F.2 · p. 53
  • The scoring-layer effect S has the sign of ∆g on all 13 benchmarks and accounts for 0.963 of summed absolute effects, while capability effects are indistinguishable from zero on 11.End-of-run capabilities differ little between conditions; the decomposition attributes most heuristic ∆g to how the same capabilities are scored under different weights.Source: §F.2 · p. 54
  • The realized-capability identity matches ∆g’s sign, but evaluating it at round-0 capability matches only 10 of 13 benchmarks; the decomposition covers heuristic runs only.Mean capability rises from 0.36 to 0.58 over the run, and two stable wideners change sign along the way.Source: §F.2 · p. 54

F.3 Qualitative check: LLM reasoning about the benchmark-type label

The qualitative check finds that LLM providers reason explicitly about private benchmark labels and reporting lag. That label-reading sits alongside, rather than producing, the gap outcome observed across modes.

F.3 Qualitative check: LLM reasoning about the benchmark-type label

  • In the private_only examples, providers interpret flat scores as lagged reporting and recommend staying the course, while the public_only excerpts track visible scores and rivals.The quoted public_only reasoning includes Orion Labs’ benchmark lead and funding; private_only examples cite investments not yet surfaced.Source: §F.3 · pp. 54–55
  • Across the sampled matched excerpts, providers under private_only name the K=3 reporting lag, while public_only excerpts focus on competitor scores, funding, and R&D gains.The three providers span frontier-lab, safety-oriented, and open-source archetypes; the excerpts were selected from matched seed and round conditions.Source: §F.3 · pp. 54–55
  • LLM providers read benchmark-type labels, but their portfolio decisions land in the same region as heuristic decisions, which cannot read those labels.The cross-mode agreement supports the narrower claim that label-reading does not produce the gap outcome.Source: §F.3 · pp. 54–55

F.4 Structural ablation: removing incidents

Removing incidents is the only reported structural ablation with a consistent direction across available runs: Orion Labs gains market share and concentration versus matched runs, while other ablation effects are small or inconsistent.

F.4 Structural ablation: removing incidents

  • Without incidents, Orion Labs ends at 78% to 82% market share versus 59% to 74% in matched runs, with HHI higher by 0.09 to 0.23.The result holds across seeds 42, 43, and 44.Source: §F.4 · p. 55
  • The no-incidents result agrees with Appendix C.8, where the timing of the first critical incident separates baseline runs.Source: §F.4 · p. 55
  • Other ablations produced small effects or effects that changed direction across seeds or repeated runs, so the authors draw no conclusion from them.LLM sampling is nondeterministic, and one run per condition does not fix the sign of effects of this size.Source: §F.4 · p. 55

F.5 Null-model capability test

The null model explains nearly all pooled LLM holdout-gap variation, while simulation adds little overall but more on a uniform-capability test set. Cross-model checks use matched runs, with limited seed coverage for Opus and GPT-5.5.

F.5 Null-model capability test

  • The simulation adds +0.2 pp r2 over the null on the cross-sequence LLM dataset, where the null explains 97.4% of ∆g variance.The comparison uses 208 paired-prediction cells and five-fold cross-validation by seed.Source: §F.5 · p. 55, Figure 10 · p. 56, Table 19 · p. 56
  • The aggregate cross-provider σ2 falls from 0.000491 under the regular baseline to 0.000037 under uniform capability, a ratio of 0.075.This attributes about 7.5% of cross-provider gap variance to simulation dynamics and about 92.5% to initial-capability heterogeneity; most per-benchmark ratios are 0.01–0.13, with Adversarial Robustness at 0.451 and Advanced Math at 0.185.Source: §F.5 · pp. 56–57, Figure 11 · p. 57
  • Path B tests generalization by training on 62 heterogeneous-cap LLM runs and evaluating on six held-out uniform-capability runs, paired into 39 prediction cells.Initial-capability heterogeneity is zero in every dimension in the test runs.Source: §F.5 · p. 57, Table 20 · p. 57
  • On the held-out uniform-capability test set, the null reaches r2=0.955 and simulation r2=0.975, making the simulation’s marginal contribution +2.0 pp.The null’s r2 is about 2 pp below the heterogeneous-cap reference of 0.974; the absolute simulation contribution remains small.Source: §F.5 · p. 57, Table 20 · p. 57, Figure 12 · p. 58
  • The three tests converge on a small simulation residual, but it is concentrated on benchmarks such as Adversarial Robustness, where strategic safety-investment divergence may differentiate providers.The pooled fit is dominated by between-benchmark variance; evaluating the identity with round-0 rather than realized capability mis-signs three benchmarks.Source: §F.5 · pp. 57–58

F.6 Cross-model setup and scope

  • Cross-model coverage pairs Sonnet, Opus, and GPT-5.5 at seeds 42–44 across five privacy conditions; Opus and GPT-5.5 have only three seeds per condition.The comparison supports outcome direction and ordering, while small single-cell differences remain noisy and are not tested statistically.Source: §F.6 · pp. 58–59

F.7 Privacy-ladder robustness

The privacy-condition ordering in per-benchmark gap replicates across three planners, while model-level market outcomes vary and cross-model coverage is narrow. The appendix also bounds the mechanism to the simulator’s abstractions and modeled actor choices.

F.7 Privacy-ladder robustness

  • On most matched conditions, GPT-5.5 gaps sit closer to Sonnet’s than Opus’s; within-vendor Sonnet–Opus differences can match cross-vendor differences.The authors therefore do not treat vendor identity as the dominant axis of variation in these data.Source: §F.7 · p. 59
  • Opus has lower mean safety allocations, while HHI has no consistent model ordering; at baseline seed 43, HHI is 0.528 for Sonnet, 0.480 for Opus, and 0.374 for GPT-5.5.At three seeds per condition, these model differences support direction only.Source: §F.7 · p. 59, Table 21 · p. 61
  • Across Sonnet, Opus, and GPT-5.5, public_only has the largest gap and private_only compresses high-gap benchmarks nearest to zero, with intermediate conditions between them.The ordering also holds on underpromise-side benchmarks, where public_only has the negative signature documented in the main text.Source: §F.7 · p. 59, Figure 13 · p. 60

F.8 Path dependence across models

  • Under baseline, within-model seed dispersion and fixed-seed cross-model dispersion are comparable, each spanning roughly 25 to 30 percentage points in final-round market share.Opus clusters tightly across three seeds, but the authors cannot distinguish a model-specific incident response from a small-sample artifact.Source: §F.8 · p. 60, Figure 15 · p. 61
  • Two additional frontier LLMs replicate the privacy-mechanism finding behaviorally, though reasoning-level support is partial and explicit invocation of the K-lag mechanism varies by model.Opus also has lower mean safety allocation, and incident-framing rates fall from Sonnet to Opus to GPT-5.5.Source: §F.8 · p. 62

F.9 Methodological caveats

  • LLM comparisons use paired seeds to control simulation random streams, but server-side response sampling remains stochastic; continuous-outcome variance estimates are wider than in the heuristic layer.Distributional claims rely on the heuristic N=50 layer rather than tight LLM inter-run confidence intervals.Source: §F.9 · p. 62

G.1 Depth on the four main-body modeling-choice limits

  • The privacy mechanism’s cosine-weight channel combines contamination, training-on-test-task effects, and adversarial holdout construction, which the simulator does not resolve separately.Item-level randomization and distribution shift with unchanged dimension weights also lie outside this cosine abstraction.Source: §G.1 · pp. 62–63
  • Results depend on the six-dimension ontology, a competitive public-leaderboard setting, and modeled consumer need weights whose exact values are judgment-based rather than directly elicited from users.The authors also note that gatekeeping, defense procurement, enterprise evaluation, and alternative need weights may change which settings or benchmarks show shrinking versus widening gaps.Source: §G.1 · p. 63

G.3 Methodological bounds of GABMs

The simulation produces qualitative candidate dynamics rather than point predictions, and its validation is constrained across several partially independent dimensions. The paper also frames extensions as new levers or actor-to-actor channels, with two extended case studies remaining unrun.

G.3 Methodological bounds of GABMs

  • Simulation outputs are directional patterns, effect orderings, and conditions for mechanisms to fire, not point predictions or claims about real-market magnitudes.Those magnitudes depend on the parameterization; the paper targets findings such as whether privacy shifts scoring credit toward dimensions where provider R&D is concentrated.Source: §G.3 · p. 67
  • Validation spans four partially independent dimensions; parameter grounding varies, and exhaustive joint sensitivity analysis is computationally intractable.Core drivers are anchored to external data, while peripheral settings include modeling choices and some numerical-stability choices.Source: §G.3 · p. 67
  • Prompt audits rule out only covered demand characteristics, not prompt invariance; neutral provider prompts also mean the holdout result does not test providers prompted to chase holdout weights.A provider instructed to infer and target those weights could change which benchmarks widen.Source: §G.3 · pp. 67–68
  • The six-dimensional capability representation, scalar benchmark orientation, and separate safety lever shape findings; a different representation could change which benchmarks widen.The weight-asymmetry mechanism rests on this asymmetric architecture.Source: §G.3 · p. 68
  • The privacy-ladder ordering replicates across Claude Sonnet 4.6, Claude Opus 4.6, and GPT-5.5, but cross-vendor checks do not remove concern about model-specific bias.GPT-5.5 invokes the K-lag mechanism in its reasoning at roughly half the Anthropic rate; validation across other LLM families remains a tractable extension.Source: §G.3 · p. 68
  • No decisive external-validity test is available because true capabilities, consumer satisfaction, and provider rationales are unobservable, while observable proxies are downstream outcomes.An exogenous-event probe and parameter grounding provide checks but do not substitute for out-of-sample prediction against real market data.Source: §G.3 · p. 68
  • Because agent-based models have no natural stopping rule, the ecosystem frame bounds extensions to new levers or channels, recording limits when the architecture cannot support them.This avoids rebuilding the architecture for every addition.Source: §G.3 · p. 68

H Extended Case Studies

  • The appendix works through evaluator capture and sketches media-shadow and benchmark-sponsorship studies, neither of which has been run.Both sketches add a lever to an existing actor without adding an actor or information flow.Source: H Extended Case Studies · p. 68

H.1 Evaluator capture: cross-mode results

The evaluator-capture case produces a suggestive reversal in market-concentration direction between heuristic and LLM runs, while the LLM runs also widen the score–satisfaction gap. The section describes the mechanism and calibration, then outlines media-shadow and benchmark-sponsorship studies.

H.1 Evaluator capture: cross-mode results

  • Evaluator capture engages best-of-N submissions, early access, discretionary provider budgets, and evaluator revenue from submission fees and funder allocations.The reported LLM planner chooses 1–10 submissions per round, with each submission costing 0.05 simulation units.Source: §H.1 · p. 69
  • The case uses an early-access blend of 0.7 and evaluation_lag=0, exposing a stronger best-of-N score-inflation channel than the canonical lag of 3.The submission cap is 12, below the 27 variants tested by Meta; the blend factor and provider budgets are modeling choices.Source: §H.1 · p. 69
  • Heuristic ∆HHI = +0.041; LLM ∆HHI = −0.029 at N=2, reversing concentration direction, though neither effect is individually established.The LLM result compares matched baseline and evaluator_capture runs at two paired seeds; the sign difference is suggestive at these sample sizes.Source: Table 23 · p. 69, §H.1 · p. 69
  • Across two seeds, evaluator_capture widened the score–satisfaction gap by mean ∆gap +0.010, with both paired deltas positive.Apex share rose by +0.081 while its safety allocation fell by 0.022; the authors trace reallocation to throughput, which benefits providers best placed to fill extra leaderboard slots, regardless of safety posture.Source: §H.1 · pp. 69–70

H.2 Media shadow

  • The media-shadow study would vary coverage of incidents versus leaderboard movement to test whether provider investment follows media attention independently of evaluator publications.The MediaActor already weights leaderboard changes, safety incidents, regulatory escalations, and provider announcements; those weights are currently constant.Source: §H.2 · p. 70

H.3 Benchmark sponsorship

  • Benchmark sponsorship could give co-funders earlier access to benchmark weights, but the simulation lacks sponsor tags and a pre-access information channel.The proposed study distinguishes rent extraction from sponsorship as a signal of commitment.Source: §H.3 · p. 70

Related papers

Loading 2610.09296v1…