Source-linked AI summary
MatrAIx: Simulating the World with 8.3 Billion Persona Agents
Xiaomin Li, Yuexing Hao, Jianheng Hou, Jintao Huang, Qianfeng Wen, Shirley Huang, Yifan Liu, Xiaoyi Liu, Yilan Fan, Yijun Wang, Koutian Wu, Ruoqi Gao, Muhammad Ahmed Mohsin, Jing Tang, Brihi Joshi, Heming Liu, Zheyuan Deng, Zonglin Di, Sankalp Jajee, Jiuyao Lu, Zhiwei Zhang, Saksham Kapoor, Ishan Gupta, Yunhan Zhao, Chanwoo Park, Yucheng Lu, Bing Hu, Weihang Xiao, Aravind Mohan, Hanwen Xing, Runyu Zhang, Mihir Kulshreshtha, Yuanda Xu, Qianyu Zhu, Dianzhuo Wang, Yuxin Xiao, Bowen Jiang, Yongye Su, Wenhao Chai, Zuxin Liu, Lawrence Yunliang Chen, Xuandong Zhao, Ethan Ye, Shivam Patel, Jason Xie, Alex Martin Richmond, Weixiang Ding, Emre Okcular, Diya Mathew, Ziheng Wang, Rana M. Shahroz Khan, Zhejian Peng, Fang Wu, Fan Nie, Xinyang Han, Yubin Kim, Jiawei Zhang, Zhenting Qi, Huangyuan Su, Xu Pan, Abinitha Gourabathina, Hyewon Jeong, Hemanth Neelgund Ramesh, Kumail Alhamoud, Kimia Hamidieh, Zidi Xiong, Samuel Schmidgall, Pengrui Han, Yepeng Huang, Yongheng Wang, Bowen Yang, Alex Gu, Yuchu Wang, Akshay Paruchuri, Brenna Li, Hejie Cui, Jiayuan Ding, Chaosheng Dong, Jiahao Wang, Yixuan He, Chi Wang, Pamela Bhattacharya, Tianyi Peng, Paul Pu Liang, Mitchell Gordon, Yilun Du, Marinka Zitnik, James Zou, Prasanna Tambe, Philip Torr, Emily Fox, Asu Ozdaglar, Dawn Song
TL;DR
Human evaluation is costly, while offline benchmarks often omit how diverse users interact with systems. MatrAIx provides an end-to-end infrastructure for evaluating AI systems and digital products with heterogeneous simulated users. In a 400-trial adherence study, assigned behaviors were expressed or correctly suppressed in 91.5% of trials.
Problem
Human evaluation is costly, and offline benchmarks often measure task outcomes without modeling how diverse users interact with systems and judge results.
Source: §1 · p. 2Method
MatrAIx combines an 8.3-billion-record persona population, four persona-agent interaction environments, and a library of 1,010 application tasks.
Source: §7 · p. 11Results
366 of 400 trials (91.5%) expressed or correctly suppressed assigned behavior in a controlled study spanning all four environments.
Source: §6 · p. 11Takeaways & Limitations
The application studies expose variation across persona groups and user needs that aggregate scores alone may miss.
Source: §1 · p. 4Persona-agent models are part of the evaluation configuration and should be reported, with important findings checked using more than one model.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Human evaluation of AI systems and digital products is costly, slow, and difficult to scale. Offline evaluations are more scalable but often abstract away human diversity and interactive behavior. We therefore introduce MatrAIx, a population-scale simulated-user evaluation infrastructure for testing AI systems and digital products with heterogeneous users. MatrAIx has three core components: First, Persona 8B contains 8.3 billion persona records represented by 1,290 categorical dimensions. Records are either sampled from a dependency graph that preserves correlated attributes or derived from human-authored profiles. We release a quality-filtered coreset of approximately 1 million personas, comprising 599,847 human-grounded and 400,000 synthetic records. Second, the MatrAIx Playground provides four environments in which diverse users evaluate and interact with digital products: Survey, AI Chatbot, Web, and App. Third, MatrAIx provides 1,010 application tasks spanning more than 25 domains, including Commerce, Software, Finance, and Healthcare. We conducted 18,189 evaluation trials across eight representative tasks. Persona agents were powered by three LLMs: Claude Opus 4.8, GPT 5.5, and Claude Haiku 4.5. The resulting feedback captures how decisions and preferences vary across persona backgrounds, including hesitation after a price increase, willingness to continue after an AI assistant fails, and latency tolerance. We conducted two main validation studies: First, a 400-trial controlled study evaluated persona adherence across ten behavioral attributes and all four environments. The declared behavior was expressed or correctly suppressed in 366 trials (91.5%). Second, human and LLM judges evaluated the extraction quality of human-grounded personas. Overall, MatrAIx provides an end-to-end infrastructure for evaluating AI systems and digital products with diverse simulated human users.
1 Introduction
MatrAIx addresses the limits of costly human studies and outcome-focused offline benchmarks with an end-to-end infrastructure for evaluating AI systems and digital products through diverse simulated users.
- Persona 8B combines dependency-graph synthetic and human-grounded profiles in a 1,290-dimension schema, with 8.3 billion records and an approximately 1-million-persona coreset.The coreset comprises 599,847 human-grounded and 400,000 synthetic records.Source: §1 · pp. 2, 4
- The Playground supports Survey, AI Chatbot, Web, and App studies, recording persona-agent interactions and applying task-specific verification for cohort-level reporting.Studies cover system behavior, feature usefulness, latency, user experience, and privacy and security controls.Source: §1 · pp. 3–4
- MatrAIx Applications offers 1,010 reusable tasks across more than 25 domains; 18,189 trials across eight representative tasks reveal differences across persona groups and user needs.Domains include Commerce, Software, Finance, and Healthcare; the task count does not mean all 1,010 studies were executed.Source: §1 · pp. 3–4
- In 400 trials, agents expressed or correctly suppressed assigned behavior in 366 trials (91.5%) across ten attributes and all four environments.Human ratings of extracted personas averaged 4.135/5; GPT 5.5 and Claude Opus 4.8 scores were within one point of the human mean in 79.2% and 93.8% of comparisons, respectively.Source: §1 · p. 4
2 Related Work
Prior work covers persona resources, user simulation, and interactive agents, while MatrAIx combines population variation with interactive evaluation of a fixed digital-product target.
Persona data and populations
- Population persona collections need typed profiles that preserve marginals, dependencies, and structural constraints, while existing resources stop short of end-to-end sampling, interaction, and outcome analysis.Free-text profiles are flexible, whereas typed schemas make missing values, contradictions, population queries, and sampling decisions easier to inspect.Source: §2 · p. 4
User simulation
- Simulators can flatten within-group variation, amplify stereotypes, or ignore persona fields; validation spans persona adherence, human–simulation agreement, ranking reliability, and sim-to-real transfer (Li et al., 2025a; Xie et al., 2024a; Chang et al., 2025; Dou et al., 2025; Zhou et al., 2026).Persona-conditioned models support controlled profile comparisons, but fluent or plausible responses alone do not establish human-like behavior.Source: §2 · pp. 4–5
Agents and evaluation
- Agent benchmarks test acting systems with task outcomes and rubric-based LLM judges, calibrated against human annotations; MatrAIx instead varies simulated-user populations while fixing a website, application, survey, or other product target (Liu et al., 2023; Wang et al., 2024a; Angelopoulos et al., 2023).Unlike agent benchmarks, MatrAIx uses agents as simulated participants rather than requiring the target itself to be an agent.Source: §2 · p. 5
3 Persona 8B: A Population-Scale Persona Dataset
Persona 8B provides a shared 1,290-dimension schema and dependency-aware sampling for an 8.3-billion-record population, complemented by human-grounded profiles. Its public 1M coreset combines filtered records and best-effort calibration to four population marginals.
3.1 Representation and Schema
- Persona 8B represents 8.3 billion records in a shared 1,290-dimension categorical schema for describing variation, querying records, and sampling cohorts, not reconstructing identifiable people.Records may be synthetic assignments or partial human-grounded profiles, with unsupported attributes left null.Source: §3.1 · p. 5, §3.2 · p. 6
3.2 Synthetic Persona Generation with DAG Sampling
- Dependency-aware DAG sampling conditions each of 1,290 dimensions on relevant predecessors, combining source-informed adjustments with hard compatibility constraints.The approach preserves rare but valid profiles while excluding contradictory combinations; dimensions are sampled in topological order from grounded priors and local conditional distributions.Source: §3.2 · pp. 6–7
3.4 Quality Control and the Public 1M Coreset
- The deterministic 1M coreset contains 599,847 human-grounded and 400,000 synthetic records, calibrated to four published marginals rather than representative joint coverage.Missing fields are not imputed, and the split is a release design choice rather than an estimate of real-world source proportions.Source: Table 2 · p. 7, §3.4 · p. 7
3.3 Real-Persona Extraction and Volunteer Collection
- Human-grounded profiles map biographies, review histories, coded surveys, and consented self-reports into the same schema, retaining only source-supported assignments.The six sources include 355 consented MatrAIx Persona Survey self-reports; records contain no direct identifiers, and constrained LLM extraction is used for free text from Wikipedia, Amazon Reviews, and PRISM Alignment.Source: §3.3 · p. 7
3.4 Quality Control and the Public 1M Coreset
- Quality control checks synthetic records for cross-attribute conflicts and human-grounded records for unsupported assignments and provenance, then deduplicates each record type.Human-grounded records use exact hashing and MinHash fuzzy detection; synthetic records are deduplicated over 14 high-information attributes.Source: §3.4 · p. 7
4 Evaluation Infrastructure
MatrAIx runs independent persona trials across four product-interaction environments, preserving trial artifacts and verifier findings for multidimensional evaluation and reruns.
4.1 Simulation Configuration and Execution
- Each run manifests its population, sampling, task, agent, and model, then launches independent persona trials that produce canonical artifacts and task-typed verifier findings.The manifest retains both the requested population and realized cohort; typed findings distinguish persona fidelity, product outcomes, and execution failures. Trials share no state and can run in parallel.Source: §4.1 · p. 8
4.2 Four Evaluation Environments
- Four environments capture distinct interactions: surveys record answers and rationales, chatbots conversations and goal resolution, web browsing actions and screenshots, and apps interaction sequences and resulting state changes.The app environment supports Linux, macOS, or iOS applications operated through mouse, keyboard, or touch actions.Source: §4.2 · p. 8
4.3 Execution, Verification, and Reporting
- Trials support local or remote parallel execution, outcome-specific programmatic or human/LLM verification, and reports that aggregate evidence by task, cohort, and subgroup.Remote workers receive only approved, non-secret configuration fields, while model-provider credentials remain on the worker. Evaluators can change the cohort, scenario, or verifier and rerun while keeping the task version and other settings fixed.Source: §4.3 · p. 8
5 Application Tasks
MatrAIx provides a 1,010-task library across four interaction environments and more than 25 domains, with task specifications linking products, persona cohorts, scenarios, outcomes, and retained evidence. A meal-planning chatbot study illustrates this design, while its observed subgroup differences remain descriptive rather than statistically confirmed.
5.1 Task Library
- The release contains 1,010 task specifications: 621 Survey, 371 AI Chatbot, 12 Web, and 6 App, spanning four anchor domains and more than 25 others.Tasks are either grounded in public instruments, existing systems, products, or interfaces, or synthetic designs for recurring studies such as price sensitivity and retention.Source: Table 3 · p. 9, §5.1 · p. 9
5.2 Application Task Specification
- Each task specifies its evaluation target, persona cohort, scenario and objective, and outcome evidence, with cohort queries and seeds enabling repeatable sampling or population comparisons.The task contract keeps product credentials and scoring rules out of persona instructions.Source: §5.2 · p. 9
- Tasks retain environment-specific artifacts for evaluation, and verifiers combine observable programmatic checks with human or LLM judgments while keeping task success separate from persona fidelity.Reports summarize completion, outcome distributions, uncertainty, and subgroup differences while linking back to traces and evidence.Source: §5.2 · p. 9
5.3 Case Study: Meal-Planning Chatbot
- In the meal-planning study, persona agents using Opus 4.8, GPT 5.5, or Haiku 4.5 queried the same GPT-4o mini assistant with the same task protocol.The design compares persona-agent models while holding the assistant fixed and examines persona attributes alongside stated likelihood of following the resulting plan.Source: §5.3 · p. 9, Figure 2 · p. 10
- 66% of empty nesters versus 46% of career changers reported higher likelihood of following the meal plan, but no association across 82 persona dimensions remained significant after correction (best q=0.51).The cohort was not weighted to represent a real population, so these differences are descriptive rather than confirmed persona effects.Source: Figure 2 · p. 10
6 Validation of Simulated-User Evaluation
MatrAIx validation tests execution, cross-model recovery of persona effects, behavioral adherence, and source-grounded persona extraction. Results show strong behavioral adherence and extraction quality, while findings depend on task relevance and should be checked across models.
Execution coverage
- Eight representative tasks, two per environment, ran with GPT 5.5, Claude Opus 4.8, and Claude Haiku 4.5; Survey, AI Chatbot, and Web used about 1,000 personas per model.The two App tasks used 24 and 20 personas because native interaction was more expensive; analyses used actual completion denominators and Benjamini–Hochberg correction across eight primary outcomes.Source: §6 · p. 10
Application-level consistency
- In the OpenBB task, trust level separated subgroups under all three persona-agent models (Cramér’s V=0.228–0.363, all q<10^-8), which ordered the four trust groups identically.Persona effects were clearest when the assigned attribute was directly relevant to the task.Source: §6 · p. 10
Controlled behavioral adherence
- 366 of 400 trials (91.5%) expressed or correctly suppressed the declared behavior across Survey, AI Chatbot, Web, and App.33 of 40 attribute-by-environment cells reached at least four of five successes in both arms; Survey, AI Chatbot, and Web did so for 9 of 10 attributes each, versus 6 of 10 for App.Source: §6 · p. 11, Figure 3 · p. 11
Extraction quality
- On 100 source-matched personas rated by six humans, mean quality was 4.135/5; GPT 5.5 and Claude Opus 4.8 were within one point of the human mean in 79.2% and 93.8% of comparisons, respectively.Two LLM judges scored 1,000 extracted personas, with 89.1% of paired metric scores within one point.Source: §6 · p. 11
Interpretation
- The persona-agent model should be reported with each evaluation result, and important findings should be checked with more than one model before guiding product decisions.The studies validate execution, persona adherence, and source-grounded extraction quality; remaining validation scope is discussed in Appendix M.Source: §6 · p. 11
7 Conclusion … D.4 Normalization and Validation
This section summarizes MatrAIx’s evaluation infrastructure and validation for pre-deployment screening, while documenting how its persona records and coreset are constructed, calibrated, and validated. It also details the taxonomy, source mapping, dependency-aware sampling, and volunteer consent and data-handling procedures.
7 Conclusion
- 18,189 trials across eight application tasks accompanied 366 of 400 assigned behaviors expressed or correctly suppressed (91.5%), while six raters scored source-matched personas 4.135 out of 5 on average.The authors recommend checking important findings across persona-agent models and tracing them to interactions; human studies remain necessary before conclusions are applied to real populations or consequential decisions.Source: §7 · p. 11
A.4 Competing Interests
- OpenAI and Anthropic supported the work and produce evaluated agent models; the authors disclose the relationship and state that supporters had no role in study design, analysis, interpretation, or submission decisions.Per-trial records were retained, and analysis code and task definitions were released to support independent verification.Source: §A.4 · p. 23
B.1 Three-Layer Persona Taxonomy
- The persona taxonomy organizes 1,290 attributes into five groups, 16 subgroups, and 55 conceptual categories, alongside a separate 43-category storage schema.The layers are dataset design choices intended to support cohort selection by concepts while keeping record formats stable; the complete attribute mapping supports auditing and regrouping.Source: §B.1 · p. 23, Figure 4 · p. 24
B.2 Complete Schema Category Index
- Table 4 indexes all 43 schema categories, and release materials provide the complete attribute mapping and a 1,290-item questionnaire supplement with prompts and allowed values.The mapping includes stable indices, field identifiers, labels, schema categories, and conceptual taxonomy assignments.Source: §B.2 · p. 23, Table 4 · p. 25
B.3 Detailed Schema-to-Source Grounding Map
- The source map distinguishes inputs used for measurement conventions, population priors, selected dependencies, and validation; listing a source does not mean every field is estimated from it.Table 5 records these source families and their roles in schema design, prior estimation, dependency construction, compatibility rules, or downstream validation.Source: §B.3 · p. 25, Table 5 · p. 26
C.1 DAG Construction and Sampling Details
- The DAG sampler combines source-informed dependency factors with hard compatibility masks, excluding invalid assignments without additionally penalizing rare but admissible combinations.Examples of rules include excluding adult work histories for young children, reconciling language with proficiency, and aligning accessibility states with relevant health attributes.Source: §C.1 · p. 27
- The dependency graph combines 1,290 emitted persona dimensions with 18 latent roots, separating estimated statistical factors from hard compatibility rules and sampling dimensions in topological order.Edges are added through source, schema, and LLM-assisted coverage reviews, with candidates checked by hand; deterministic seeds support reproducible, parallel shard generation.Source: §C.1 · p. 28
C.2 Detailed Persona Post-Processing Pipeline
- The 21-rule quality filter rejected 239,310 of 10,002,288,277 original records, and the accepted snapshot contains 8,399,989,719 rows.Post-processing preserves rejection provenance, deduplicates human records using exact hashes and MinHash, and reduces synthetic records through entropy-based projection buckets; the materialized snapshot is 10,281 rows below target.Source: §C.2 · pp. 28–29
C.3 Coreset Candidate Selection and Calibration
- The coreset uses retained human-source records and selects 400,000 synthetic records from 2,187,354 candidates using seeded shard sampling.Calibration iteratively shares positive weights across four target dimensions; deterministic sampling without replacement yields an exact-size, order-independent sample, while missing fields are not imputed to satisfy targets.Source: §C.3 · p. 29
D.1 Extraction Engine and Output Contract
- Free-text extraction combines literal regex matches with retrieved dimensions judged by an LLM, preserving separate outputs, disagreements, confidence, and quoted evidence.The judge may abstain and sees only retrieved dimensions, their questions, and allowed-value sets; extracted fields retain provenance and distinguish direct evidence, structured claims, summary inferences, and unsupported fields.Source: §D.1 · pp. 29–30
D.2 Source-Specific Preprocessing
- Source preprocessing uses distinct record definitions and extraction pipelines: Wikipedia profiles, Amazon reviewer histories, coded survey crosswalks, hybrid PRISM extraction, and direct volunteer submissions.Wikipedia and Amazon batches use category-based dimension chunks and resumable, idempotent processing; GSS, Afrobarometer, and ConvAI2 use crosswalk or conservative phrase-matching approaches.Source: §D.2 · p. 30, §D.3 · p. 30
D.3 Observed, Hybrid, and Direct Modes
- The volunteer instrument preserves native None/N/A answers, exports skipped items as null, and gives all 1,290 items a native or synthetic decline option for sensitive attributes.It requests no names or contact details and collects no address, fingerprint, or analytics; exported-file return is separate from answering and is the act of consent, under three-year MIT-managed retention and CC BY 4.0 release terms.Source: §D.3 · p. 30
D.4 Normalization and Validation
- Normalization restores schema order, nulls disallowed values, demotes ungrounded evidence, and validates records, allowed values, confidence, null semantics, and quotation grounding when source profiles are available.Exact observed values override inferred values with direct provenance and confidence 1.0; checks are designed to expose unsupported inference rather than silently fill missing human attributes.Source: §D.4 · p. 30
D.5 Volunteer Survey Cohort … E.3 Execution Lifecycle and Environment Routing
The appendix characterizes the released volunteer cohort and describes MatrAIx’s runtime: configurable, reproducible trials connect personas and tasks to product environments and verification.
D.5 Volunteer Survey Cohort
- The 355-person opt-in cohort is unweighted, not a probability sample: 5.8% were North American, 39.5% South Asian or Sub-Saharan African, 61.8% low or lower-middle socioeconomic, and 33.2% rural.Recruitment-channel composition and response propensities were unavailable, so population weights could not be justified.Source: §D.5 · p. 31
- 91.3% of records answered the median item; item coverage ranged from 87.3% to 95.8%, while category coverage ranged from 89.3% to 92.4%.272 of 355 records answered all 1,290 items; the remaining 83 answered between 520 and 1,186.Source: §D.5 · p. 30, Figure 6 · p. 31
E Runtime and Environments
- MatrAIx extends Harbor with persona-conditioned cohort sampling, four product-facing environment adapters, task-owned verification, and population-level reporting.Source: E Runtime and Environments · p. 32
E.1 Trial, Job, and Cohort Formalization
- Trials bind a persona, task, agent, model, and seed; task-owned verifiers score artifacts, while seeded cohort jobs preserve rerun inputs and parallelize stateless trials.Persona, agent, model, and task remain independently configurable for controlled comparisons that hold other factors fixed.Source: §E.1 · p. 32
E.2 Launch Surfaces and Application Attachment
- The same resolved recipe runs from the playground, CLI, or REST API; users attach surveys, chatbots, web targets, or native apps without reimplementing the product.Applications are supplied as stimuli and questionnaires, REST or Model Context Protocol sidecars, browser or container targets, or native platform backends.Source: §E.2 · pp. 32–33
E.3 Execution Lifecycle and Environment Routing
- Each isolated trial binds its recipe, renders persona instructions, acts through the task interface, writes a canonical submission, and persists the verifier-scored record.Execution mode and location are independent: defaults vary by environment, container mode strengthens isolation, smoke mode avoids model calls, and local or remote HTTP workers share task and artifact definitions.Source: §E.3 · p. 33
E.4 Scaling and Reproducibility … F.7 Case Study: Meal-Planning Interaction Telemetry
The appendices describe how MatrAIx scales and records evaluations, defines and accounts for task specifications and cohorts, and verifies and reports trials. Case studies show model-dependent price-sensitivity responses and looping meal-planning conversations with subgroup differences.
E.4 Scaling and Reproducibility; E.5 The MatrAIx Playground
- Scaling replicates independent trial units across bounded local concurrency or stateless remote workers, while the Playground supports browsing personas, configuring studies, running evaluations, and viewing population reports without code.Seeded sampling permits cohort redraws, but model-behavior reproducibility depends on provider versions and backend determinism; remote dispatch excludes credentials, and interactive tasks can use isolated containers or desktop backends.Source: §E.4 · p. 33, §E.5 · p. 33, Figure 8 · p. 34
E.6 Telemetry, Verification, and Reporting Schema; F.5 Verification and Reporting Contract
- Trial telemetry preserves identifiers, prompts, actions, tool calls, timing, state changes, verifier findings, and artifacts; task-owned verification is rules-first, with judges reserved for subjective properties.Batch reports first aggregate counts, distributions, categorical and Boolean findings, and outcome-grouped text without a language model; optional judge scans follow, with links to trial artifacts retained.Source: §E.6 · p. 35, §F.5 · pp. 36–37
F Application Task Details; F.1 Library Accounting and Status Definitions
- The task inventory contains 1,010 unique available specifications, assembled from curated tasks and three template-based collections after duplicate resolution, not a count of implemented or executed tasks.A task is a versioned recipe declaring population, persona-facing goal, product attachment, runtime requirements, verifier, and aggregation policy; implementation, execution, and empirical reporting are tracked separately.Source: §F.1 · p. 35, F Application Task Details · p. 35
F.2 Declarative Task Contract; F.4 Environment-Specific Artifacts
- Task contracts separate scenarios from environment-specific resources and backends, while environment adapters retain distinct primary artifacts under a common trial envelope.Task families can reuse fixtures but keep separate identifiers, scenarios, product metadata, cohort queries, and manifest entries.Source: §F.2 · p. 35, §F.4 · p. 36
F.3 Cohort Selection Contract
- Cohort strategies query Persona 8B dimensions without changing records, while launch manifests preserve both the requested strategy and the resolved cohort, allocations, weights, seed, and selected personas.Failed or missing trials remain in job accounting and are not replaced after results are observed.Source: §F.3 · p. 36
F.6 Case Study: Candy Land Price Sensitivity
- Hesitation responses were 98.3% for GPT 5.5, 27.0% for Claude Opus 4.8, and 83.3% for Claude Haiku 4.5 in the Candy Land study.Under Opus, 73.0% selected fair_buy; the declared economic-motivation stratification showed no significant outcome difference after correction, and several response fields were nearly degenerate.Source: §F.6 · p. 37
F.7 Case Study: Meal-Planning Interaction Telemetry
- Across 500 cost-sensitive conversations, Request plan returned to State constraints 482 times and constraints self-looped 279 times, showing repeated constraint elicitation (Figure 9).Conversations averaged six exchanges and fourteen moves; cost-sensitive personas routed more transitions from Reject as unrealistic to Ask substitution than premium-seeking personas (283 versus 204).Source: §F.7 · pp. 37–38, Figure 9 · p. 38
F.8 Case Study: News+ Subscription Decision · G Experimental Setup
In the News+ case study, all trials produced verified evidence of live app browsing, while persona-specific free-text reasons revealed catalog-fit concerns that subscription counts could not assess precisely. Dimension-level tests found no corrected associations with the modal experience rating.
F.8 Case Study: News+ Subscription Decision
- 72/72 trials returned schema-valid submissions, named at least one publication seen on the page, and recorded the same live price: $12.99 per month.This was the load-bearing check that the OS-app environment was functioning; nothing was mocked and no arm hallucinated a price.Source: §F.8 · p. 38, Figure 10 · p. 39
- Subscriptions were 1/24 for GPT 5.5, 5/24 for Claude Opus 4.8, and 0/24 for Claude Haiku 4.5, leaving each arm’s rate imprecisely estimated.The 95% Wilson intervals were [0.7, 20.2], [9.2, 40.5], and [0.0, 13.8] percent, respectively; the omnibus test cleared correction (q=0.025), but Haiku’s zero subscriptions also left no usable rank correlation across the six segments.Source: §F.8 · p. 38
- In each model arm, 23 of 24 reasons cited persona-specific factors; declines turned on catalog fit, while price appeared in nearly every reason.Price appeared in 24, 23, and 24 reasons for GPT 5.5, Claude Opus 4.8, and Claude Haiku 4.5, respectively, and was named by subscribers and decliners alike.Source: §F.8 · p. 38
- Candy Land found no detectable segment structure, whereas News+ yielded persona-specific free-text reasons but lacked the cohort size to test modest behavioral differences.The authors suggest structured free-text facets may be more informative than a binary conversion flag alone for expensive OS-App studies.Source: §F.8 · p. 38
- 44 of 82 dimensions had enough personas for testing, but none of their associations with the modal experience rating survived Benjamini–Hochberg correction (best q=0.11).The descriptive gradient was 89% modal ratings among personas strongly preferring reading over watching versus 14% among those with no preference; the declared subscription outcome was one in 24.Source: Figure 10 · p. 39
- Each News+ trial browsed the real Apple News app in an iPhone-17 simulator on iOS 26.4, returning a structured decision for a cohort of 24.The submission contract required the price read from the screen and a non-empty list of noticed titles; two persona dimensions were fully crossed, with other dimensions unconstrained.Source: §F.8 · p. 38, Figure 10 · p. 39
G.1 Tasks, Models, and Cohorts · G.2 Statistical Tests and Artifact Integrity
The sections describe the three-model, shared-cohort evaluation design, its statistical procedures, and explicit artifact-integrity exceptions. Table 12 summarizes primary outcomes across eight tasks, while the text qualifies cohort coverage and interpretation.
G.1 Tasks, Models, and Cohorts
- Each of eight tasks was intended to run once with GPT 5.5, Claude Opus 4.8, and Claude Haiku 4.5 on a shared sampled cohort.Survey, Chatbot, and Web tasks use approximately 1,000 personas per model; the two App tasks use cohorts of 24 and 20.Source: §G.1 · p. 39
- Reports retain response coverage, use field-specific answered denominators, test MIT OpenCourseWare at course level, and caution that small App cohorts cannot resolve modest subgroup effects.For example, the OpenBB Claude Opus 4.8 run has 980 usable trials out of 1,000.Source: §G.1 · p. 39
G.2 Statistical Tests and Artifact Integrity
- Categorical outcomes use an omnibus three-model chi-square test, numeric outcomes use a three-arm mean test, and proportions use exact 95% Wilson intervals.Benjamini–Hochberg controls multiplicity within each declared test family; subgroup ordering uses Spearman’s ρ with exact permutation p-values.Source: §G.2 · p. 39
- The meal-planning GPT 5.5 arm matches the shared cohort manifest on only 43 of 370 identifiers, unlike the completely matching Opus and Haiku arms, so its cross-model result is not interpreted.The exception is retained in the run record rather than silently repaired after inspection.Source: §G.2 · pp. 39–40
- Table 12 reports the complete primary-outcome results for eight validation tasks, with three-arm chi-square q-values corrected across the eight tasks.The table defines range as the largest minus smallest model share in percentage points; its caption does not provide the individual task values.Source: Table 12 · p. 40
- For Stocks, two artifacts write sentiment and disagree on 1, 2, and 4 trials across the three arms; reported primary outcomes use the declared decision file.This exception remains in the run record rather than being silently repaired after inspection.Source: §G.2 · p. 40
H Application-Level Validation Results … I.1 Probe Design
The validation sections report product-level outcomes, cross-model consistency in persona effects, and persona-fidelity statistics. They also document an attribute-level adherence probe across four environments, including its matched-cohort design and judging procedure.
H.1 Complete Product-Level Outcomes
- Table 13 reports one product-facing measure per task, with each rate’s numerator and answered-trial denominator exposed.The table compares three agent models on identical cohorts, aggregates Notion’s three paid tiers against its free tier, and flags the meal-planning GPT 5.5 arm as incomparable because it did not run the declared cohort.Source: Table 13 · p. 40, §H.1 · p. 40
H.2 Cross-Model Persona-Effect Consistency
- 14 of 22 defined pair-by-task comparisons point in the same direction, with median ρ = +0.29; OpenBB trust level is the only dimension significant under all three models.All three models order its four trust groups identically; the chance of three independent random orderings coinciding is 0.0017. Correlations without a detected subgroup effect are not interpreted as evidence.Source: §H.2 · p. 40, Table 14 · p. 41
H.3 Persona-Fidelity Statistics
- GPT and Opus age-band matches are indistinguishable from uniform guessing (q = 0.84 and q = 0.85), while Haiku matches 1,000 of 1,000 trials.Table 15 reports persona fidelity across all three model pairs using Cohen’s κ to correct raw agreement for each model’s answer distribution.Source: Table 15 · p. 41
I Persona Adherence Validation
- The appendix documents the matched-cohort design, complete per-cell results, and verbatim evidence cited by the judge for the attribute-level adherence probe.The probe is reported in section 6.Source: I Persona Adherence Validation · p. 41
I.1 Probe Design
- The probe tests ten observable behavioral attributes: seven cognitive or register attributes and three coding attributes that surface only when the agent writes code.The attributes include emoji use, humor, politeness, storytelling, jargon use, verbosity, register, code comment style, code naming verbosity, and code-summary documentation.Source: §I.1 · p. 41
- Across Survey, AI Chatbot, Web, and OS-App on Linux, each attribute-environment cell uses matched positive and negative cohorts of five personas each, totaling 400 trials.Stratified sampling varies only the tested attribute while holding other persona fields to the population distribution; the environments test both direct text output and behavior passing through an execution layer.Source: §I.1 · p. 41
- Claude Opus 4.8 judges each completed trajectory and any artifact with a binary verdict on whether the target behavior was expressed or correctly suppressed.For positive personas, expression is success; for negative personas, success means the opposite value is expressed, so both counts use a higher-is-better direction.Source: §I.1 · p. 41
I.2 Per-Attribute Results and Backbone Sensitivity … J.1 Extraction-Quality Rubric
The 400-trial probe found 91.5% adherence with Opus 4.8, with failures concentrated in suppression and environment- or attribute-specific limits. A second backbone scored 79.2%, while extraction quality was evaluated separately using LLM and human ratings on a five-dimension rubric.
I.2 Per-Attribute Results and Backbone Sensitivity
- Opus 4.8 achieved 366/400 = 91.5% adherence, compared with GPT-5.6-sol at 317/400 = 79.2% under the same protocol and judge.Opus had 33/40 strong cells; the comparison changes the acting model, with the Chat environment also differing in adapter for the GPT run.Source: §I.2 · p. 42, Table 16 · p. 42
- Opus failures clustered in OS-App and suppression, with Web under-production and attributes conflicting with model defaults explaining key weak cells.OS-App accounted for 17 of 34 failures; its negative politeness arm failed in all five trials, which the authors interpret as a boundary of persona conditioning rather than a measurement artifact.Source: §I.2 · p. 42
- GPT-5.6-sol’s deficit was structured: coding attributes were near parity at 91% vs 100%, while soft-style cognitive attributes scored 73% vs 88%.Its positive and negative arms scored 154/200 = 77% and 163/200 = 82%, respectively; the Chat comparison also reflects an adapter difference.Source: §I.2 · pp. 42–43
- GPT-5.6-sol scored 0/5 on positive verbosity in all four environments, as its concise prior overrode the declared attribute.Prompt elaboration and personas with verbosity-aligned correlated dimensions did not change the outcome; terser replies also amplified Web under-production.Source: §I.2 · p. 43
I.3 Cited Judge Evidence
- The 400-trial probe relied on Claude Opus 4.8 as its sole judge, requiring each verdict to cite verbatim behavior from the trajectory.Table 17 pairs positive- and negative-persona behavior in Survey, showing behavioral contrasts rather than mere restatement of persona attributes.Source: §I.3 · p. 43, Table 17 · p. 43
- In the separate human-label study, six raters scored 100 personas, averaging 4.135/5; Claude and GPT judges were within one point of the human mean in 93.8% and 79.2% of comparisons.The human labels belong to extraction-quality validation, not the binary behavioral probe; 97.2% of rater–rater comparisons were within one point.Source: §I.3 · p. 43
J Extraction Quality Validation
- Extraction quality was evaluated through paired dual-LLM ratings of all 1,000 extractions and six-rater human ratings of a 100-persona subset, using five shared dimensions.Both used a 1–5 scale, but LLM judges received the full quantitative rubric while human raters received concise descriptions and a filtered field view; human results calibrate rather than replace LLM judgments.Source: J Extraction Quality Validation · pp. 43–44
J.1 Extraction-Quality Rubric
- The extraction-quality rubric covers five metrics, with higher scores indicating better quality.The metrics use a 1–5 scale.Source: J Extraction Quality Validation · p. 43, Table 18 · p. 44
J.2 LLM-Judge Evaluation … L Future Directions
LLM judges usually scored extractions within one point, and human ratings were favorable overall. The paper also compares system capabilities and outlines future extensions that require stronger validation against longitudinal human data.
J.2 LLM-Judge Evaluation
- The judges independently scored 1,000 extractions from four sources using byte-identical prompts, producing 1,000 paired records and 5,000 paired metric scores.The set comprised 500 Wikipedia, 250 Stack Overflow, 200 Amazon, and 50 PRISM extractions; all outputs parsed successfully.Source: §J.2 · p. 44
- 89.1% of 5,000 paired judge scores differed by at most one point; agreement was 99.2% for coverage (M3) and 97.8% for overall fidelity and plausibility (M5).Overall mean scores were 3.773 for GPT and 3.931 for Claude.Source: Table 19 · p. 44, §J.2 · p. 44
J.3 Human Evaluation on the 100-Persona Subset
- The human subset comprised 100 personas matched to the full set’s source composition, with six human ratings per persona averaged into the reference scores.Six raters completed four 25-persona packets, yielding 600 persona-level rating records and 3,000 metric scores.Source: §J.3 · pp. 44–45
- Human ratings averaged 4.135/5, with 84.7% scoring 4–5; within-one agreement was 97.2% for H–H, 79.2% for GPT–H, and 93.8% for Claude–H comparisons.H–H covers pairs of human raters; GPT–H and Claude–H compare each judge with the six-rater human mean.Source: Table 20 · p. 45, §J.3 · p. 45
K System Capability Comparison
- The capability matrix compares published system statements rather than ranking systems; a cross means a capability is unstated, not demonstrated absent.Its dimensions include product-as-subject evaluation, persona conditioning, behavioral grounding, native-device execution, and reproducible cohorts; multi-agent social worlds are a deliberate MatrAIx non-goal.Source: K System Capability Comparison · p. 45, Table 21 · p. 45
L Future Directions
- Future work could add user-centered outcomes, diverse task generation, robustness tests, and expanded open-source environments for reproducible evaluation across products and domains.Proposed outcomes include satisfaction, trust, retention, and failure recovery in interactive and long-horizon settings; tasks could express the same goal in different ways.Source: L Future Directions · p. 46
- Further work could improve persona fidelity and consistency, add long-term memory, compare simulation methods, and develop tools to interpret how attributes shape model behavior.Potential applications include red teaming, personalized assistants, and social-science studies; stronger validation against longitudinal human data is needed before use in personalization, safety assessment, or decisions about real populations.Source: L Future Directions · p. 46
M Responsible Use, Release, and Limitations
MatrAIx releases a filtered Persona 1M coreset for simulation, not a representative sample of people, and its results are model-dependent and require human validation for claims about people. The section also warns about shared-backbone bias and identifies behavioral realism as an unmeasured priority for future validation.
Responsible use
- Personas are simulation instruments, not people or a probability sample, and the release does not support impersonation, attributing behavior to real people or communities, individual targeting, or discriminatory persuasion, exclusion or pricing.The volunteer instrument requests no name, contact detail or account identifier and allows declining all 1,290 items; work using the release should preserve that posture and avoid reidentification. Simulations do not replace consultation with affected people, especially in consequential regulated settings.Source: M Responsible Use, Release, and… · p. 46
Release
- The public release is a filtered, provenance-checked and deduplicated Persona 1M coreset, not the full internal population, and only sampled records paired with a model, interface and task become agents.Its synthetic component is calibrated toward four supported marginals; the release includes a manifest, an audit of achieved shares and infeasible residuals, and per-file hashes.Source: M Responsible Use, Release, and… · p. 46
Interpreting results
- Every result is a persona-agent result, not direct evidence of how people behave, so claims about people require human validation on the same task and instrument.The passage points to a 100-record study in Appendix J as an example design.Source: M Responsible Use, Release, and… · p. 46
Shared-backbone warning
- Using the same model backbone for the persona and system under test makes favorable results ambiguous and may inflate satisfaction by suppressing friction a person would express.Agreement could reflect either a well-served user or a model recognizing and preferring its own output.Source: M Responsible Use, Release, and… · p. 46
Model dependence
- The paid-plan share spans 23.2% to 93.9% across three persona models on identical cohorts, while median pairwise Cohen’s 𝜅 across 88 joinable fields is approximately zero.The experiments do not cross persona and system backbones, so they cannot separate self-preference from ordinary model-to-model variation.Source: M Responsible Use, Release, and… · p. 46
Shared-backbone practice
- MatrAIx records the agent model, and consequential evaluations should also test the cohort with at least one persona model that does not share the system’s backbone.Properly isolating shared-backbone effects requires crossing persona and system models on one task, an experiment the authors have not run.Source: M Responsible Use, Release, and… · p. 46
Behavioral realism
- The adherence probe measures whether a declared style appears in a trajectory, not whether personas disclose, correct, refuse or abandon as real users do.The experiments do not establish that personas withhold context, push back or give up like people.Source: M Responsible Use, Release, and… · pp. 46–47
Future validation
- Two proposed validation designs compare simulated turns with matched real-log slices or continue held-out conversations from extracted personas, with results reported per agent model.The population-level design compares turn length, question type, volunteered or withheld context, correction and abandonment rates, and response-option entropy; a real-versus-simulated classifier targets AUC near 0.5. The individual-level design scores simulated next turns against what the person wrote, using shuffled-persona and no-persona baselines; public logs require screening for memorization.Source: M Responsible Use, Release, and… · p. 47
Related papers
This paper cites
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?2023
- Generative Agents: Interactive Simulacra of Human Behavior2023
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs2023
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments2024
- $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains2024
- Personalizing Dialogue Agents: I have a dog, do you have pets too?2018
Cited by
- Markets of Minds: A 51-Billion-State Economy of Intelligence2026
- PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers2026
- PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas2026
- Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation2026