PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Driven Persona-Conditioned Agents for A/B Test Simulation

Ziyad Benomar, Weronika Łajewska, Leonardo Perelli, Saab Mansour

arXiv:2609.01038v1cs.AI

TL;DR

Live A/B tests require substantial traffic, engineering effort, and weeks of measurement. The paper proposes structured-question simulation with LLM agents conditioned on personas grounded in anonymized behavioral data, and evaluates it across 40 tests. Its findings identify question format, domain alignment, depth–diversity balance, and subsampling as central factors, with directional accuracy varying by metric and efficient pre-screening as the supported application.

  • Problem

    A/B testing is costly because experiments require sufficient user traffic, engineering effort, and typically weeks of data collection to reach statistical significance.

  • Method

    The framework uses persona-conditioned LLM agents whose structured-question responses simulate treatment and control outcomes, with personas built from anonymized behavioral data.

  • Results

    Across 40 A/B tests, pairwise rating achieved 0.75–0.80 directional accuracy, while public e-commerce data rivaled platform-specific personas and 500-persona subsampling caused negligible quality loss.

  • Takeaways & Limitations

    The results support using data-driven persona simulation as a fast, low-cost pre-screening tool rather than a replacement for human A/B tests.

  • Takeaways & Limitations

    The benchmark covers 40 tests from one e-commerce domain and two metric types, leaving generalization to other domains, metrics, and multi-step journeys untested.

Abstract

from arXiv · show

A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data-activity patterns, engagement signals, and inferred demographics-enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75-0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.

1 Introduction

The paper proposes using LLM agents conditioned on data-driven personas to simulate A/B test outcomes and study the design, persona, and sampling factors that determine simulation quality.

  • Motivation: Online A/B tests require substantial user traffic, engineering effort, and weeks of data collection before reaching statistical significance.These costs limit how many product ideas teams can evaluate.
  • Approach: The framework constructs LLM agents from anonymized behavioral signals rather than synthetic personas derived from screenshots or rule-based descriptions.The personas are grounded in months of user activity to support population modeling.
  • Research agenda: A/B test simulation is framed as a structured question task spanning question format, persona source alignment, behavioral depth, population diversity, and subsampling.The benchmark studies these factors systematically across the simulation pipeline.
  • Evaluation: The benchmark contains 40 A/B tests covering click-through rate and subscription metrics.Experiments are organized around four research questions concerning elicitation, persona construction, depth–diversity trade-offs, and population sampling.
  • Contributions: The paper reports four contributions: structured-question evaluation, domain-alignment analysis, depth–diversity characterization, and 2× cost reduction through subsampling without measurable quality loss.These contributions position data-driven persona simulation as an approach for evaluating experiment candidates before live testing.

2 Related Work

Prior work shows that LLMs can model human responses and behavior, while A/B test simulation remains constrained by persona realism, traffic-log requirements, and evaluation design.

  • LLM-based user simulation: LLMs can reproduce response distributions across demographic groups and approximate population-level behavioral patterns.Persona conditioning can partially mitigate systematic biases, especially when personas are grounded in observed data.
  • A/B test simulation: SimAB uses persona-conditioned agents with synthetic screenshot-derived personas and reports 67% agreement on 47 tests.Unlike the present framework, SimAB does not ground personas in real user behavior.
  • A/B test simulation: SimGym replays real traffic traces through browser agents but requires detailed session logs and does not separate persona construction from the simulation task.The paper instead studies persona composition independently as a factor in simulation fidelity.
  • Persona generation from data: Prior persona-generation work finds that behavioral grounding and richer descriptions can improve population-level or role-playing behavior, while large-scale synthetic profiles may lack real user traces.These findings motivate comparing data sources and behavioral depth in A/B test simulation.
  • Evaluation of simulation alignment: Simulation evaluation can target sign prediction or effect-magnitude alignment, with this paper emphasizing directional accuracy, sign overlap, and sign Bhattacharyya distance.The metrics assess alignment from individual experiment outcomes to distributional patterns.

3 A/B Testing Simulation

The framework turns each A/B test into persona-conditioned preference elicitation, aggregates variant scores into an estimated treatment-effect distribution, and evaluates sign alignment.

  • Problem formulation: The objective is to predict whether a treatment improves, harms, or negligibly affects a metric without running the experiment on real users.Agents score control and treatment variants to produce a simulated relative-effect distribution.
  • Framework overview: The pipeline has three stages: persona construction, simulation through structured questions, and aggregation with evaluation.Each stage corresponds to a distinct modeling or measurement choice.
  • Question design: Question formats vary by isolation versus comparison and binary versus rating responses, with pairwise presentation randomizing variant order per persona.The format determines how variants are shown and how preferences are elicited.
  • Persona construction: Personas synthesize anonymized behavioral records into structured profiles containing inferred demographics, behavioral statistics, activity patterns, and shopping characteristics.The profiles are statistical conditioning variables for population-level simulation, not representations of identified individuals.
  • Persona construction: Behavioral depth and population diversity are competing persona-pool objectives: richer histories improve profile detail, whereas broader sampling covers more user segments.Highly active users provide deeper data but may be less representative of the broader population.
  • Population sampling: Subsampling reduces inference cost by selecting a smaller persona subset that preserves the full pool’s distributional properties.The study compares uniform random sampling and Kernel Herding among its subsampling strategies.
  • Evaluation: The evaluation reports accuracy, sign overlap, and sign Bhattacharyya distance to measure binary correctness and confidence alignment.Ground-truth and simulated positive-effect probabilities are derived from their respective effect distributions.

4 Experimental Setup

Experiments use a curated e-commerce benchmark of 40 visual-design test pairs, contrasting deep and representative persona pools with public-data baselines under controlled batch inference.

  • Benchmark: The benchmark comprises 40 experimental test pairs drawn from historical e-commerce interaction data and evaluating visual design variants.The sample is curated rather than representative of a platform’s full user base or testing infrastructure.
  • Benchmark: Each test compares a control widget with one or more treatments, using confidence intervals for the estimated Treatment − Control metric difference.These intervals define the ground-truth effect distribution used in evaluation.
  • Persona pools: The deep pool contains 935 highly engaged personas with dense behavioral histories, whereas the representative pool contains 935 stratified users spanning broader demographic and behavioral segments.The two pools test behavioral depth against population coverage.
  • Persona pools: Additional pools of 1000 personas are built from survey data, movie ratings, and open e-commerce transactions as public-data baselines.These pools test whether public data can rival platform-specific persona sources.
  • Implementation: All experiments use Claude Sonnet 4.5 with cloud batch inference, temperature 0, JSON-constrained responses, and randomized variant order per persona.Persona generation uses sequential flows for demographics and behavioral narratives.

5 Results

The results evaluate question design, persona sources, persona-pool composition, subsampling, and component contributions across the benchmark. Pairwise rating, domain-aligned data, behavioral personas, and population-aware sampling emerge as strong configurations, with trade-offs between depth and diversity.

  • Question Design Comparison: Pairwise rating achieves 0.75 accuracy on CTR and 0.80 on subscription tests, becoming the format used in subsequent experiments.It significantly outperforms several alternatives, although pairwise binary remains competitive on some metrics.
  • Persona Sources: 0.90 versus 0.80 subscription accuracy favors open e-commerce personas over platform data, while domain-aligned sources outperform out-of-domain sources overall.In-domain behavioral data achieves 0.70–0.90 accuracy, compared with 0.57–0.69 for out-of-domain sources.
  • Persona Pool Comparison: 0.75 versus 0.60 CTR accuracy favors the deep pool, but subscription accuracy is 0.80 for both deep and representative pools.The representative pool’s broader demographic coverage largely compensates for sparser individual behavioral histories at the population level.
  • Population Sampling Efficiency: 500-persona subsamples preserve near-full-pool accuracy within 1pp on CTR and match or exceed it on subscriptions, enabling a potential 2× cost reduction.Kernel Herding performs best for subscription tests and Greedy Farthest for CTR, while random sampling also remains strong.
  • Component Contributions: +25–30pp CTR and +15–20pp subscription accuracy lifts over unconditioned baselines show the value of full behavioral personas.Demographics-only conditioning degrades substantially on subscriptions, reasoning mainly benefits CTR, and a single generic persona offers no advantage over no persona.
  • Question Design Comparison: Pairwise formats consistently outperform independent formats across three models, while the preferred pairwise variant depends on the model.Pairwise rating offers greater interpretability at comparable accuracy.

6 Discussion

The framework is positioned as a practical complement to live A/B testing, with cost-efficient simulation and guidance on when its predictions are reliable. Its usefulness is bounded by data requirements and sensitivity to effect size.

  • Potential applications: With batch inference, full simulations could complete in hours at a fraction of multiweek experiment costs, supporting broader design-space exploration.The proposed use is pre-screening clearly inferior treatments and prioritizing experiments by predicted impact.
  • When to trust simulation: Simulations are most reliable for large effect sizes and least trustworthy for near-zero effects, where small perturbations can flip the predicted direction.The discussion therefore suggests using outputs as a ranking signal rather than a binary decision criterion.
  • Cost–quality trade-off: 500-persona subsampling performs within 1pp of the full pool for subscription tests and matches it on CTR.Even n=100 remains competitive at 0.77–0.80 CTR and 0.82–0.86 subscriptions.
  • Data requirements: Domain alignment matters more than data volume or source exclusivity, with public e-commerce data rivaling platform-specific personas.Below ∼20 recorded transactions, simulation degrades as the LLM defaults to generic reasoning.
  • Enabling reproducible research: The methodology is reproducible with public preference datasets by reframing item pairs as control and treatment variants with ground-truth preference directions.The benchmark itself relies on proprietary A/B test outcomes.

7 Conclusions

The paper presents LLM-agent A/B test simulation using data-driven personas constructed from real behavioral data. Across 40 tests and two metric types, it identifies question design, domain alignment, persona depth, diversity, and subsampling as central factors.

  • 7 Conclusions: The framework simulates A/B tests with LLM agents conditioned on data-driven personas constructed from real behavioral data.The evaluation covers 40 A/B tests across two metric types.
  • 7 Conclusions: Pairwise rating is the most effective question format, achieving 0.75–0.80 directional accuracy.The conclusion lists question format as the first of four established findings.
  • 7 Conclusions: Domain alignment of persona source data is the primary driver of simulation quality, with public e-commerce data rivaling platform-specific personas.This finding concerns source-data alignment rather than source exclusivity.
  • 7 Conclusions: Demographic diversity can largely compensate for reduced per-persona behavioral depth.

Limitations

The evaluation is bounded by isolated screenshots, a single e-commerce domain, two metric types, inferred demographics, and known LLM biases.

  • Scope: The framework evaluates isolated screenshots rather than full page contexts, omitting browsing intent, session history, and surrounding content.Generalization to multi-step user journeys remains untested.
  • Scope: The benchmark contains 40 tests from one e-commerce domain and two metric types, leaving broader generalization untested.The unsupported settings include other domains, metric types, and multi-step user journeys.
  • Persona construction: LLM-inferred demographics may introduce systematic biases because they are not self-reported or validated against users’ actual characteristics.These inferred attributes may distort population-level predictions.
  • Model biases: Positivity and anchoring biases may produce optimistic treatment evaluations that sign-based metrics partially mask when both variants are equally inflated.The pairwise format partially mitigates this by canceling uniform positivity bias in relative treatment effects.

Ethical Considerations

The paper describes anonymized behavioral-data processing and frames simulation as a structured evaluation of variant preferences. It also specifies the experimental quantities and question formats used in the pipeline.

  • Data handling: Behavioral data are anonymized and de-identified before processing, with no personally identifiable information included.The persona pipeline does not retain explicit links to source records.
  • Inferred attributes: Inferred age, gender, and income attributes are approximate statistical priors generated by the LLM rather than validated self-reports.They serve as conditioning signals for population-level simulation and are not used for individual profiling or targeting.
  • A/B testing: A/B testing compares control and treatment groups on metrics such as CTR by testing equality of their true means against a two-sided alternative.The design prespecifies Type I error α, Type II error β, and the minimum detectable effect to determine sample sizes.
  • Ground truth: The observed treatment effect estimator is ˆδ = ˆµt − ˆµc, and its sampling distribution represents uncertainty across repeated identical experiments.The simulation benchmark centers this distribution on the observed effect.
  • Simulated A/B testing: Persona-conditioned agents view control and treatment stimuli and predict treatment-effect distributions without requiring actual human participants.The stimuli can include advertisement designs or website layouts, and metrics can include click-through rate.
  • Question design: The simulation frames preference elicitation as structured questions whose format controls how variants are presented and evaluated.The studied designs include independent binary, independent rating, pairwise binary, and pairwise rating formats.
  • Question design: Showing variants in isolation or together interacts with anchoring and positional bias, making question design consequential for simulation accuracy.

C Evaluation Metrics: Intuition via Toy Examples

The evaluation represents ground-truth and simulated effects as distributions and emphasizes whether they agree on effect direction. Sign metrics capture directional agreement more appropriately than magnitude-focused distances when ship/no-ship decisions depend on sign.

  • Distributional setup: Ground truth and simulated effects are represented as normal distributions, with the simulation aggregating persona-level relative effects into a predicted distribution.The simulated mean and standard error come from persona-level predicted relative scores.
  • Distributional setup: p and q denote the probabilities that the ground-truth and simulated relative effects are positive, respectively.Both probabilities are computed using the standard normal CDF.
  • Sign metrics: Accuracy records whether the two distributions have the same effect sign, while SignOv and SignBC provide smoother probabilistic alignment measures.Sign overlap equals 1 − |p − q|, whereas the Bhattacharyya-based metric penalizes disagreement differently.
  • Toy-example intuition: Accuracy becomes noisy near zero means because small perturbations can flip signs, whereas SignOv and SignBC account for distributional uncertainty more smoothly.This makes the latter metrics less abrupt than binary accuracy in ambiguous cases.
  • Operational interpretation: Sign alignment prioritizes the direction needed for ship/no-ship decisions, while KL and Wasserstein distances can penalize magnitude errors despite correct direction.Magnitude metrics remain useful when precise effect-size estimation matters, such as cost-benefit analysis.

E Open Datasets and Personas

The paper evaluates reproducibility on public rating datasets by constructing pairwise comparisons and testing deep and public e-commerce persona configurations. Accuracy increases with effect size, while public datasets and personas provide a reproducible benchmark without proprietary infrastructure.

  • Datasets: Three public datasets—Book Crossing, Jester Jokes, and MovieLens—supply 100 sampled items each for reproducibility evaluation.Their rating scales range from 1–10, −10–10, and 0.5–5.0, respectively.
  • Benchmark construction: Each item pair forms an A/B scenario whose ground-truth effect is the difference in mean ratings, while personas generate predicted rating differences.The benchmark contains 4,950 pairwise comparisons per dataset.
  • Persona configurations: The evaluation compares deep personas from the main experiments with a public persona set based on online shopping profiles.Both configurations estimate the probability that one item is better than another under Gaussian assumptions.
  • Results: Accuracy increases monotonically with absolute effect size across all three public datasets and both persona configurations.Larger rating differences therefore yield more reliable predictions in this benchmark.
  • Reproducibility: Public datasets and personas enable benchmarking, reproduction, and extension without access to proprietary experimentation infrastructure.The authors present this as a concrete path for reproducibility and direct comparison.
  • Comparison boundary: Direct numerical comparison with SimAB is precluded because its exact test subset, code repository, and processed datasets are not specified or published.Methodological differences further complicate a fair comparison.

H Population Subsampling: Extended Results

The subsampling experiments test whether smaller persona pools preserve simulation quality. Results show that small samples remain competitive, while diversity-oriented methods can reduce variance and subsampling can sometimes improve uncertainty alignment.

  • Subsampling methods: Persona embeddings support random, Greedy Farthest, and Kernel Herding subsampling strategies over the persona pool.Greedy Farthest spreads selections across embedding space, while Kernel Herding minimizes discrepancy from the full distribution.
  • Evaluation setup: The figures compare sign metrics against sample size for marketing and subscription tests, averaging 1,000 independent trials with standard-error bars.The experiments evaluate how closely smaller samples reproduce the full-pool behavior.
  • Extended results: At 100 personas, samples recover full-pool CTR performance, with Greedy Farthest slightly outperforming Random and Kernel Herding.For subscription tests, Kernel Herding performs best, but the gap among algorithms is small.
  • Extended results: Random sampling remains strong for both use cases, while Greedy Farthest and Kernel Herding provide lower variance.Thus, algorithm choice affects stability more than the broad quality conclusion.
  • Interpretation: Subsamples can outperform the full pool because added variance may correct excessive certainty or a directionally incorrect full-pool prediction.This effect matters particularly for uncertainty-aware metrics such as SignOv and SignBC.
Loading 2609.01038v1…