PaperlayerView exact revision on arXiv

Source-linked AI summary

MatrAIx: Simulating the World with 8.3 Billion Persona Agents

Xiaomin Li, Yuexing Hao, Jianheng Hou, Jintao Huang, Qianfeng Wen, Shirley Huang, Yifan Liu, Xiaoyi Liu, Yilan Fan, Yijun Wang, Koutian Wu, Ruoqi Gao, Muhammad Ahmed Mohsin, Jing Tang, Brihi Joshi, Heming Liu, Zheyuan Deng, Zonglin Di, Sankalp Jajee, Jiuyao Lu, Zhiwei Zhang, Saksham Kapoor, Ishan Gupta, Yunhan Zhao, Chanwoo Park, Yucheng Lu, Bing Hu, Weihang Xiao, Aravind Mohan, Hanwen Xing, Runyu Zhang, Mihir Kulshreshtha, Yuanda Xu, Qianyu Zhu, Dianzhuo Wang, Yuxin Xiao, Bowen Jiang, Yongye Su, Wenhao Chai, Zuxin Liu, Lawrence Yunliang Chen, Xuandong Zhao, Ethan Ye, Shivam Patel, Jason Xie, Alex Martin Richmond, Weixiang Ding, Emre Okcular, Diya Mathew, Ziheng Wang, Rana M. Shahroz Khan, Zhejian Peng, Fang Wu, Fan Nie, Xinyang Han, Yubin Kim, Jiawei Zhang, Zhenting Qi, Huangyuan Su, Xu Pan, Abinitha Gourabathina, Hyewon Jeong, Hemanth Neelgund Ramesh, Kumail Alhamoud, Kimia Hamidieh, Zidi Xiong, Samuel Schmidgall, Pengrui Han, Yepeng Huang, Yongheng Wang, Bowen Yang, Alex Gu, Yuchu Wang, Akshay Paruchuri, Brenna Li, Hejie Cui, Jiayuan Ding, Chaosheng Dong, Jiahao Wang, Yixuan He, Chi Wang, Pamela Bhattacharya, Tianyi Peng, Paul Pu Liang, Mitchell Gordon, Yilun Du, Marinka Zitnik, James Zou, Prasanna Tambe, Philip Torr, Emily Fox, Asu Ozdaglar, Dawn Song

arXiv:2608.04205v1cs.AI

TL;DR

Human evaluation is costly and slow, while offline benchmarks often omit diverse users’ interactive behavior and preferences. MatrAIx addresses this gap with a population-scale persona dataset, four interactive environments, and reusable tasks, and validates the infrastructure through large-scale demonstrations and adherence studies. The authors report 91.5% correct expression or suppression of assigned behaviors in a 400-trial controlled study, alongside explicit limits on interpreting persona-model effects.

  • Problem

    Human evaluation is costly and slow, while offline benchmarks often omit how diverse users formulate requests, interact with systems, and judge results.

  • Method

    MatrAIx combines 8.3 billion dependency-aware or source-grounded personas, four interaction environments, and reusable application tasks into an end-to-end simulated-user evaluation infrastructure.

  • Results

    91.5% of trials in a 400-trial controlled study expressed or correctly suppressed the assigned behavior across ten attributes and four environments.

  • Takeaways & Limitations

    MatrAIx supports evaluation of AI systems and digital products with heterogeneous simulated users and can expose variation across persona groups and user needs.

  • Takeaways & Limitations

    Persona model and application backbone are not crossed, so self-preference effects cannot be separated from ordinary model-to-model variation.

Abstract

from arXiv · show

Human evaluation of AI systems and digital products is costly, slow, and difficult to scale. Offline evaluations are more scalable but often abstract away human diversity and interactive behavior. We therefore introduce MatrAIx, a population-scale simulated-user evaluation infrastructure for testing AI systems and digital products with heterogeneous users. MatrAIx has three core components: First, Persona 8B contains 8.3 billion persona records represented by 1,290 categorical dimensions. Records are either sampled from a dependency graph that preserves correlated attributes or derived from human-authored profiles. We release a quality-filtered coreset of approximately 1 million personas, comprising 599,847 human-grounded and 400,000 synthetic records. Second, the MatrAIx Playground provides four environments in which diverse users evaluate and interact with digital products: Survey, AI Chatbot, Web, and App. Third, MatrAIx provides 1,010 application tasks spanning more than 25 domains, including Commerce, Software, Finance, and Healthcare. We conducted 18,189 evaluation trials across eight representative tasks. Persona agents were powered by three LLMs: Claude Opus 4.8, GPT 5.5, and Claude Haiku 4.5. The resulting feedback captures how decisions and preferences vary across persona backgrounds, including hesitation after a price increase, willingness to continue after an AI assistant fails, and latency tolerance. We conducted two main validation studies: First, a 400-trial controlled study evaluated persona adherence across ten behavioral attributes and all four environments. The declared behavior was expressed or correctly suppressed in 366 trials (91.5%). Second, human and LLM judges evaluated the extraction quality of human-grounded personas. Overall, MatrAIx provides an end-to-end infrastructure for evaluating AI systems and digital products with diverse simulated human users.

1 Introduction

MatrAIx addresses the limits of generic-user and offline evaluations by combining a population-scale persona dataset, interactive environments, and reusable application tasks. The infrastructure supports heterogeneous simulated-user studies and reports persona adherence and evaluation differences across user groups.

  • Core infrastructure: MatrAIx combines Persona 8B, the MatrAIx Playground, and MatrAIx Applications into an end-to-end simulated-user evaluation infrastructure.The components cover persona representation, interactive execution, and versioned task specifications.
  • Core infrastructure: Persona 8B represents human variation with 8.3 billion records under a shared 1,290-dimensional categorical schema.Synthetic records use dependency-aware sampling, while human-grounded records map source material into the same schema.
  • Core infrastructure: The Playground supports Survey, AI Chatbot, Web, and App environments for questionnaires, conversations, browsing, and native-application interaction.These environments let persona agents interact with evaluated products in different ways.
  • Core infrastructure: MatrAIx Applications provides 1,010 reusable task specifications spanning more than 25 domains, including Commerce, Software, Finance, and Healthcare.Tasks define the target product, cohort, scenario, user goal, outcome measures, and verifier.
  • Validation and demonstrations: The paper investigates population structure, extraction quality, persona adherence, and task outcomes to reveal differences across persona groups and user needs.The authors treat population coherence, persona adherence, and evaluation sensitivity as distinct properties.
  • Validation and demonstrations: 18,189 trials across eight tasks and a separate 400-trial study demonstrate the complete pipeline and evaluate persona adherence.Assigned behaviors were expressed or correctly suppressed in 366 of 400 trials, or 91.5%.

2 Related Work

Prior work develops persona representations, user simulation, and interactive-agent benchmarks, but these strands address different aspects of diverse-user evaluation. MatrAIx builds on them by combining population-conditioned simulation with interactive environments and task evaluation.

  • Persona data and populations: Persona resources represent users as dialogue descriptions, structured profiles, persistent states, or large synthetic collections.Existing resources may be crowd-authored, generated at scale, grounded in population statistics, reconstructed from documents, or updated from interaction histories.
  • User simulation: User simulation supports controlled comparisons across profiles and can reveal differences in preferences and failure modes.Interactive simulations also need to capture disclosure, correction, refusal, and abandonment.
  • User simulation: Recent simulator evaluations examine persona adherence, behavioral chains, human–simulation agreement, ranking reliability, and sim-to-real transfer.These are distinct validation targets for simulated users.
  • Agents and evaluation: Agent benchmarks evaluate planning, tool use, and action across APIs, websites, applications, and operating systems, often combining executable outcomes with rubric-based LLM judgments.This literature focuses on agent capabilities and evaluation procedures across interactive environments.

3 Persona 8B: A Population-Scale Persona Dataset

Persona 8B is a population-scale dataset for describing human variation, querying cohorts, and conditioning persona agents. It combines dependency-aware synthetic generation with source-grounded records, quality controls, and a calibrated one-million-record public coreset.

  • Representation and schema: Persona 8B contains 8.3 billion records represented by 1,290 categorical dimensions covering background, psychology, capability, behavior, and lifestyle.A fully specified synthetic persona assigns a value to every dimension, while human-grounded records may retain null unsupported fields.
  • Representation and schema: The schema supports population queries and model-facing persona conditioning through a shared typed coordinate system.Queries can select attributes such as age, region, language, expertise, or accessibility needs.
  • Synthetic persona generation: Synthetic personas are sampled from a dependency graph so conditional relationships and compatibility constraints are preserved across attributes.Dimensions are sampled in parent-first order; examples include education conditioned on age and English proficiency conditioned on language and region.
  • Human-grounded records: Human-grounded records map biographies, behavioral histories, coded surveys, and consented self-reports into the same schema, leaving unsupported dimensions null.The population draws from six sources, including Wikipedia, Amazon Reviews, Stack Overflow, the GSS, PRISM Alignment, and consented survey responses.
  • Quality control and coreset: Quality control checks conflicts, unsupported assignments, provenance, and duplicates before release.Synthetic records use cross-attribute checks and high-information-attribute deduplication, while human-grounded records use provenance checks and exact or fuzzy deduplication.
  • Quality control and coreset: The public coreset contains 599,847 human-grounded and 400,000 synthetic records calibrated to four published marginal distributions.The split is a release design choice, and the result is not representative joint coverage over all 1,290 dimensions.

4 Evaluation Infrastructure

MatrAIx turns a population query and application task into independently tracked persona-agent trials across four interaction environments. It preserves trial artifacts, verifies outcomes, and aggregates evidence for cohort and subgroup reporting while supporting scalable execution and reruns.

  • Trial execution: A simulation records the population query, eligible pool, sampling procedure, task version, agent, model, and seed in a run manifest.Each sampled persona receives an independent trial represented by persona, task, interface, model, and seed.
  • Evaluation environments: Survey, AI Chatbot, Web, and App environments record different interaction evidence suited to questionnaires, conversations, browsing, and application use.Recorded evidence includes answers, conversations, calls, pages, actions, screenshots, submissions, interaction sequences, final states, and side effects.
  • Scaling and reruns: Local or remote workers can execute multiple trials in parallel, and adding workers increases throughput without changing task or output formats.Only approved non-secret configuration fields are sent remotely; provider credentials remain on workers.
  • Evidence and verification: Each trial stores the persona, task, agent, model, seed, trajectory, final state, verifier results, and environment-specific artifacts.Programmatic verifiers check observable outcomes, while human or LLM judges handle outcomes requiring interpretation.
  • Evidence and verification: Reports aggregate results by task, cohort, and subgroup while preserving links to underlying trials and evidence.This supports population-level and subgroup-level reporting from retained artifacts.
  • Scaling and reruns: Evaluators can change the cohort, scenario, or verifier and rerun a study while keeping the task version and other settings fixed.This enables controlled iteration after reviewing a report.

5 Application Tasks

MatrAIx Applications organizes reusable evaluation tasks across four interaction environments and more than 25 domains. Each task specifies the target, persona cohort, scenario, objective, outcomes, evidence, and verification procedure.

  • Environments: The four environments are Survey, AI Chatbot, Web, and App, with the environment determining how persona agents interact with a product.Tasks define the system or product under test, persona cohort, scenario, user objective, and outcome measures.
  • Task library: 1,010 unique task specifications span more than 25 domains, including Commerce, Software, Finance, and Healthcare.The library includes grounded tasks based on public instruments, models, products, or interfaces, alongside synthetic tasks for controlled study designs.
  • Evidence and verification: Task-specific artifacts include answers, conversations, viewed pages, actions, submissions, exported files, permission changes, and final application states.These artifacts support measures such as simulated-user purchase or continuation behavior while preserving the underlying interaction evidence.
  • Case study: The meal-planning chatbot study holds the assistant fixed while varying the persona-agent model across Opus 4.8, GPT 5.5, and Haiku 4.5.This design compares how different persona agents use the same persona information when interacting with the same assistant.

6 Validation of Simulated-User Evaluation

MatrAIx validation tests execution, cross-model persona effects, behavioral adherence, and source grounding. The studies find consistent task-relevant subgroup patterns, 91.5% controlled adherence, and strong agreement for human-grounded persona extraction, while emphasizing model-specific reporting and cross-model checks.

  • Application-level consistency: Trust level separates OpenBB subgroups under all three persona-agent models, with Cramér’s V= 0.228–0.363 and all q< 10^-8.All three models order the four trust groups identically, indicating a consistent subgroup pattern when the attribute has a clear behavioral channel.
  • Controlled behavioral adherence: 366 of 400 controlled adherence trials succeed (91.5%), with assigned behavior expressed or correctly suppressed.Across 40 attribute-by-environment cells, 33 reach at least four of five successes in both arms.
  • Controlled behavioral adherence: Survey, AI Chatbot, and Web each meet the strong-attribute threshold for 9 of 10 attributes, compared with 6 of 10 for App.Each attribute uses five personas declaring one pole and five declaring the opposite pole.
  • Extraction quality: On 1,000 extracted personas, 89.1% of paired LLM-judge metric scores are within one point; six human raters score a source-matched subset 4.135/5 on average.GPT 5.5 and Claude Opus 4.8 are within one point of the human mean in 79.2% and 93.8% of comparisons, respectively.
  • Interpretation: The persona-agent model is part of the evaluation configuration, so important findings should be checked with more than one model before product decisions.The reported validation scope covers execution, persona adherence, and source-grounded extraction quality.

7 Conclusion

MatrAIx combines population-scale personas, interactive environments, and reusable application tasks to evaluate AI systems and digital products with heterogeneous users. The paper reports broad execution and validation results while framing its taxonomy, constraints, and provenance as auditable design choices.

  • Infrastructure: MatrAIx combines Persona 8B, the MatrAIx Playground, and 1,010 versioned tasks across more than 25 domains.The infrastructure includes approximately 8.3 billion persona records, four interaction environments, and 18,189 trials across eight representative tasks.
  • Validation: The controlled adherence study succeeds in 366 of 400 trials (91.5%), while six human raters assign human-grounded personas a mean quality score of 4.135 out of 5.These findings support pre-deployment screening, subgroup analysis, stress testing, and comparison across product versions.
  • Schema design: The three-layer taxonomy is a dataset-specific, auditable organization of 1,290 attributes that can be regrouped from the released attribute-level mapping.The design comprises five groups, 16 subgroups, and 55 conceptual categories, alongside a separate 43-category storage schema.
  • Population construction: Compatibility masks exclude invalid assignments while leaving rare but admissible combinations unpenalized by the mask.Representative rules align work histories with age, language with proficiency, and accessibility states with relevant health attributes.
  • Missingness: The questionnaire records unsupported human-grounded fields as null, distinguishing native None or N/A answers from synthetic opt-outs.Among 1,290 items, 434 have native None or N/A options and the remaining 856 receive a synthetic opt-out.
  • Data scope: Recruitment materials document open eligibility, unverified responses, platform-specific variants, and three-year retention on MIT-managed storage.The instrument requests no name, contact detail, or account identifier and collects no address, fingerprint, or analogous identifiers.

D.5 Volunteer Survey Cohort

The volunteer survey cohort exercises most of the 1,290-item instrument, but it is an opt-in sample whose composition cannot support population weighting. MatrAIx’s execution pipeline binds personas to independent, reproducible trials across multiple environments.

  • Cohort: 355 released volunteer records provide the documented cohort for the survey analysis.The records are identified as real_human_survey data and match the public dataset.
  • Coverage: 91.3% is the median item-answer coverage, with individual items ranging from 87.3% to 95.8%.Across 43 interface categories, the range narrows to 89.3%–92.4%.
  • Coverage: 272 of 355 records answer all 1,290 items, while the remaining 83 answer between 520 and 1,186.
  • Composition: The cohort is geographically and socioeconomically uneven, including 5.8% in North America, 39.5% in South Asia or Sub-Saharan Africa, and 61.8% in low or lower-middle socioeconomic bands.The released cohort also includes 33.2% of records from rural settings.
  • Limitation: Population weights are not justified because recruitment-channel composition and response propensities are unavailable.The cohort is therefore reported as released rather than treated as a probability sample.
  • Execution: Each trial binds a persona, task, agent interface, model, and seed, then produces an artifact for task-owned verification.The same recipe can be launched interactively, from a command line, or through an API.

E.4 Scaling and Reproducibility

MatrAIx scales evaluation by replicating independent, seeded trials across workers while preserving a common task and artifact contract. Its task library distinguishes discoverability, implementation, execution, and empirical reporting, and its case studies expose both model variation and sparse-app-study limits.

  • Scaling: Independent trial work units scale through configurable local concurrency or remote stateless workers without changing the trial contract.Seeded sampling redraws cohorts, while provider versions and backend determinism constrain strict model-behavior reproducibility.
  • Execution: Remote execution keeps provider credentials on workers and uses containers or desktop backends to isolate higher-risk interactive tasks.Sensitive fields are minimized or redacted before telemetry crosses an execution boundary.
  • Playground: The Playground lets users browse personas, assemble cohorts, configure studies, observe simulated users, and read aggregate reports without code.The same interface is described as the interactive front end to MatrAIx.
  • Task inventory: Task-library accounting reports 1,010 available specifications, while implementation, execution, and empirical reporting are tracked separately.Available means discoverable; implemented, executed, and empirically reported require progressively stronger evidence.
  • Model variation: 98.3% of GPT 5.5 personas hesitated, compared with 27.0% under Claude Opus 4.8 and 83.3% under Claude Haiku 4.5.Under Opus, fair_buy was selected by 73.0%, yielding opposite product conclusions from the same brief and cohort.
  • Subgroups: The economic-motivation stratification produced no detectable outcome difference after correction across the three model arms.Subgroup rank correlations were not distinguishable from chance over four groups.

F.8 Case Study: News+ Subscription Decision

The News+ case study tested a live iOS app through multi-step browsing and obtained valid structured submissions from every trial. Binary subscription rates were too sparse for precise estimation, but free-text reasons consistently tied decisions to persona-specific catalog fit.

  • Execution validity: 72 of 72 trials returned schema-valid submissions and listed at least one publication seen on the page.Every arm read the same live price: $12.99 per month; nothing was mocked and no arm hallucinated a price.
  • Subscription outcome: One GPT 5.5 persona, five Claude Opus 4.8 personas, and zero Claude Haiku 4.5 personas subscribed.The corresponding 95% Wilson intervals were [0.7, 20.2], [9.2, 40.5], and [0.0, 13.8] percent.
  • Limitation: The subscription outcome is too sparse for precise single-arm rate estimates or usable subgroup rank correlation.Haiku subscribed nobody in any of the six segments, leaving no ordering for another arm to agree with.
  • Free-text feedback: 23 of 24 reasons in every arm cited something specific about the persona rather than the product in general.Examples included language, regional-news, and field-specific catalog fit.
  • Decision rationale: Price appeared in 24, 23, and 24 reasons across the three arms, while catalog fit distinguished the declines.Price was named by subscribers and decliners alike.

G.1 Tasks, Models, and Cohorts

The validation suite ran eight tasks across three language models, using large shared cohorts for Survey, Chatbot, and Web tasks and much smaller App cohorts. Analyses preserve response coverage, use preregistered-style multi-arm tests, and explicitly document integrity exceptions.

  • Design: Each of eight tasks was intended to run once with GPT 5.5, Claude Opus 4.8, and Claude Haiku 4.5 on a shared sampled cohort.Survey, Chatbot, and Web tasks used approximately 1,000 personas per model; App tasks used 24 and 20.
  • Coverage: 980 usable OpenBB trials out of 1,000 under Claude Opus 4.8 illustrate field-specific answered denominators.The App cohorts are too small to resolve modest subgroup effects.
  • Analysis: Categorical outcomes use one omnibus three-arm χ2 test, numeric outcomes use a three-arm mean test, and multiplicity is controlled with Benjamini–Hochberg correction.Proportions use exact 95% Wilson intervals, while subgroup ordering uses Spearman’s ρ with exact permutation p-values.
  • Integrity: The meal-planning GPT 5.5 arm matches only 43 of 370 cohort identifiers, so its cross-model result is not interpreted.The Opus and Haiku arms match completely.
  • Integrity: Stocks discrepancies across two artifact files are retained in the run record, while reported primary outcomes use the declared decision file.The files disagree on 1, 2, and 4 trials across the three arms.

H Application-Level Validation Results

Application-level validation separates persona fidelity from task outcomes and tests whether persona subgroups show consistent outcome rankings across models. Across 22 defined pair-by-task comparisons, subgroup ordering agreed in 14, with a median Spearman correlation of +0.29.

  • Subgroup consistency: 14 of 22 defined pair-by-task comparisons point in the same direction, with a median Spearman ρ of +0.29.Subgroups are ranked separately within each task and model, allowing outcome levels to differ while testing whether persona effects point in the same direction.
  • Subgroup consistency: Trust level in OpenBB is the only dimension significant under all three models, whose four groups receive identical orderings.Three independent random orderings of four groups coincide with probability (1/4!)^2 = 0.0017.
  • Persona adherence: The validation probe measures whether ten observable behavioral attributes are expressed or correctly suppressed across Survey, AI Chatbot, Web, and OS-App environments.Each attribute is evaluated in every environment using matched positive and negative persona cohorts.
  • Persona adherence: Each trial records a completed trajectory or artifact, and an LLM judge returns a binary verdict on whether the target persona value appeared in actual behavior.The judge reads the trajectory together with the persona’s target attribute value.

I.2 Per-Attribute Results and Backbone Sensitivity

Under the matched 400-trial protocol, Opus 4.8 adheres to declared behavioral attributes more often than GPT-5.6-sol, with failures concentrated in suppression, execution limits, and attributes opposed to model priors. The comparison also audits judge grounding separately from the behavioral result.

  • Overall adherence: 366/400 = 91.5% of Opus 4.8 trials expressed or correctly suppressed the declared value, versus 317/400 = 79.2% for GPT-5.6-sol.The protocol holds personas, tasks, environments, and the Opus 4.8 judge fixed while changing the acting model, except for the flagged Chat adapter difference.
  • Opus failure concentration: 17 of 34 Opus failures occurred in OS-App, whose environment-level score was 83/100; suppression was weaker than expression at 181/200 versus 185/200.OS-App negative arms account for 11 failures, including cog-politeness at 0/5 and cog-storytelling at 1/5.
  • Opus failure concentration: Seven non-strong cells cluster around Web under-production and attributes that conflict with the model’s default behavior, rather than random judge disagreement.Truncated trajectories limit opportunities for attributes to surface, while concrete-example defaults weaken negative storytelling instructions.
  • Backbone sensitivity: Coding attributes remain near parity at 91% versus 100%, while soft-style cognitive attributes fall to 73% versus 88% for GPT-5.6-sol versus Opus 4.8.Both positive and negative arms decline for GPT-5.6-sol, indicating a general steerability deficit rather than a one-directional bias.
  • Backbone sensitivity: GPT-5.6-sol records 0/5 for positive cog-verbosity in every environment, as its concise, structured prior overrides the declared long-winded behavior.Prompting for elaboration and aligning correlated persona dimensions changed the outcome only marginally or not at all.
  • Judge grounding: The top-line 91.5% uses only Claude Opus 4.8 as judge, while human labels belong to a separate extraction-quality study rather than calibrating this probe.The probe’s direct grounding comes from cited behavioral evidence in Table 17.

J.2 LLM-Judge Evaluation

The extraction-quality evaluation compares GPT-5.5 and Claude on 1,000 persona extractions and separately benchmarks both against six-rater human means on 100 personas. Agreement is generally high, but the human study is used for calibration rather than replacement of LLM judgments.

  • LLM-judge comparison: 1,000 extractions produced 5,000 paired metric scores after independent GPT-5.5 and Claude evaluation, with all outputs parsing successfully.The dataset covers Wikipedia, Stack Overflow, Amazon, and PRISM records, and both judges received byte-identical prompts.
  • LLM-judge comparison: 89.1% of 5,000 paired metric scores differed by at most one point, with strongest agreement for coverage at 99.2%.Overall means were 3.773 for GPT and 3.931 for Claude; overall fidelity and plausibility reached 97.8% within one point.
  • Human reference: The human subset contains 100 personas rated by six raters, yielding 600 persona-level rating records and 3,000 individual metric scores.The source composition matches the full set, and six human scores are averaged for each persona and metric.
  • Human reference: LLM judgments are compared with the six-rater human mean using the proportion of scores differing by at most one point.The study analyzes human and LLM evaluation streams separately because their interfaces and rubric presentation are not identical.
  • Interpretation: The human evaluation calibrates rather than replaces LLM judgments, and the reported comparisons position the systems rather than rank them.This framing preserves the distinct roles of human reference ratings and paired LLM evaluation.
  • Scope boundary: Future extensions require stronger validation against longitudinal human data before simulated behavior informs personalization, safety assessment, or decisions about real populations.The paper also identifies persona fidelity and consistency across models and repeated runs as methodological areas for improvement.

M Responsible Use, Release, and Limitations

MatrAIx is released with explicit boundaries: personas are simulation instruments, not probability samples or direct substitutes for human behavior. The experiments also leave shared-model bias and realistic interactive behaviors insufficiently isolated, motivating careful reporting and further validation.

  • Responsible use: Persona 8B is not a probability sample of a real population and must not be used to impersonate individuals, attribute behavior to identifiable communities, or target people.The population conditions on attributes including age, region, income, employment, and health.
  • Release scope: The public release is a filtered Persona 1M coreset, while execution scale depends on pairing sampled records with a model, interface, and task.The coreset includes provenance, deduplication, calibration, audit, and hashing artifacts.
  • Interpretation: Every reported result is a persona-agent result rather than a direct claim about human behavior, and people-related claims require human validation on the same task and instrument.The text recommends a design such as the 100-record study in Appendix J.
  • Model dependence: Shared backbones can make favorable outcomes ambiguous because a persona may recognize and prefer the system model’s output rather than reflect human acceptance.This can inflate satisfaction and suppress friction that the evaluation is intended to surface.
  • Model dependence: 23.2% to 93.9%: paid-plan share varied across three persona models on identical cohorts, while median pairwise Cohen’s κ across 88 joinable fields was approximately zero.The experiments vary persona models but do not cross persona and system backbones, so the self-preference term remains inseparable from ordinary model variation.
  • Behavioral scope: The experiments do not directly measure whether simulated users disclose context, correct systems, refuse, push back, or abandon interactions as people do.The adherence probe tests whether declared styles appear in trajectories, which addresses conditioning rather than resemblance to real users.
  • Future validation: Comparing simulated turns with matched real-log slices could assess turn length, question type, context disclosure, correction, abandonment, and response-option entropy.A real-versus-simulated classifier is proposed as an additional summary of these differences.
Loading 2608.04205v1…