PaperlayerView exact revision on arXiv

Source-linked AI summary

Proxy-Validated LLM UX Micro-Simulations: An Artifact-First Protocol for Early-Stage Decision Support

Alexandre Cristovão Maiorano

arXiv:2608.13563v1cs.HCcs.AIcs.SE

TL;DR

Early-stage teams need UX decision signals despite limited access to users and usability studies. This paper develops an artifact-first LLM micro-simulation and proxy-validation pipeline, finding higher weighted-Jaccard alignment for embeddings than lexical or TF-IDF baselines on app-review and support-tweet proxies, while large-k top-k Jaccard overstates alignment.

  • Problem

    Early-stage teams need UX feedback for decisions, but usability evaluation is costly, slow, and often deferred, leaving the meaning and validation of synthetic feedback unresolved.

  • Method

    The paper generates structured UX micro-simulation feedback from versioned prompts and validates friction themes against noisy public proxy distributions using weighted and top-k Jaccard metrics and lexical, TF-IDF, and embedding baselines.

  • Results

    Embedding-based alignment exceeded lexical and TF-IDF baselines on app-review and support-tweet proxies, while top-k Jaccard reached 1.0 at k = 10 although W remained 0.017 on Amazon.

  • Takeaways & Limitations

    The pipeline provides ranked friction signals and an auditable, versioned iteration loop for early-stage teams deciding what to fix next and which risks to accept.

  • Takeaways & Limitations

    The token-overlap grounding heuristic underestimates true overlap, producing hallucination-proxy values near 1.0 that are conservative lower bounds rather than evidence of widespread fabrication.

Abstract

from arXiv · show

Early-stage teams often lack users, time, and budget to run repeated UX studies, yet still need decision-oriented signals to iterate safely. We study an LLM-driven UX micro-simulation pipeline that generates structured customer-experience feedback (walkthrough steps, friction points, micro-survey signals) from versioned prompts, personas, tasks, and UI snapshots. Because public usability datasets with task outcomes are scarce, we validate simulated friction themes using multiple public proxy corpora (app reviews, support tweets, and open-source software issues). We propose a lightweight proxy-validation protocol with two alignment metrics: top-k Jaccard and distributional weighted-Jaccard (W), and compare lexical, TF-IDF, and multilingual embedding baselines across six proxy datasets. Embedding-based alignment yields higher W than lexical baselines on primary app-review and support-tweet proxies (e.g., W=0.128 vs 0.000 on Gojek), while top-k Jaccard is shown to overstate alignment at large k. We ablate four agent strategies (single-pass, best-of-N, hybrid, and a proposed score-then-select judge) across Azure OpenAI deployments and report bootstrap confidence intervals over 8 method-dataset pairs; these intervals reveal that the embedding W point estimate is systematically unstable under resampling at our subsample size. We also provide a failure-mode analysis of grounding and fabrication proxies, with documented calibration caveats and worked examples of outputs flagged as fabricated by an adversarial judge. Our artifact-first pipeline produces reproducible tables and figures from versioned run artifacts, supporting iterative prompt and taxonomy refinement before final paid-model calibration.

1 Introduction

The paper addresses early-stage UX uncertainty with LLM-driven micro-simulations that generate structured feedback and validate friction themes against noisy public proxy distributions. Its artifact-first protocol supports reproducible, auditable iteration through versioned inputs, alignment metrics, bootstrap intervals, and error analyses.

  • Introduction: Early-stage teams must make product decisions under UX uncertainty because conventional evaluation is costly, slow, and often deferred.This delay can allow regressions and friction to accumulate until they become expensive to unwind.
  • Introduction: The pipeline generates structured, role-conditioned UX feedback from versioned prompts and validates friction signals against app reviews, support tweets, and open-source issues.These proxies are treated as noisy friction-theme distributions rather than ground truth for task success.
  • Introduction: The proxy-validation protocol compares lexical and multilingual embedding baselines using top-k and weighted-Jaccard alignment metrics.The study asks which method is most robust and whether conclusions remain stable under sampling, prompt, and taxonomy changes.
  • Introduction: The artifact-first evaluation suite combines reproducible versioned inputs, structured outputs, bootstrap confidence intervals, and error analyses before stronger paid models are introduced.This design makes the pipeline reproducible and supports iterative refinement of prompts and taxonomies.
  • Introduction: Practitioners receive ranked friction themes, compact failure-risk summaries by locale or domain proxy, and an auditable trail explaining changes and prioritization.The intended output is an actionable iteration loop rather than a single UX score.

2 Background and Related Work

Prior work spans inspection-based UX evaluation, large-scale mining of public user feedback, and LLM-based simulation and judging, while highlighting limits in data coverage, validity, and rubric reliability. This work positions itself as an artifact-first proxy-validation protocol that provides reproducible alignment evidence and structured iteration signals before costly human studies or model calibration.

  • Inspection-based UX evaluation: Cognitive walkthroughs identify usability issues without user studies, while HEART and SUS connect goals or perceived usability to lightweight measurable signals.These established approaches cover theory-driven inspection, metrics frameworks, and a widely used usability instrument.
  • Mining user feedback at scale: Public reviews, support conversations, and software issues enable reproducible feedback-mining studies but rarely provide task completion times or controlled experimental conditions.Prior surveys also document established use cases and evaluation gaps in requirements engineering and software maintenance.
  • LLMs as simulators and the “human surrogate” debate: LLM-based simulators and personas support scalable HCI simulation and ideation, but studies report validity risks and misalignment with human response distributions.The paper frames simulation as decision support for early iteration rather than a replacement for human studies.
  • LLM-based evaluation and rubric reliability: LLM judges offer scalable approximations to human evaluation but can show position and verbosity biases and inconsistent rubric application.The proposed safeguards emphasize schema-constrained outputs, artifact-linked decision trails, and small gold-labeled calibration sets when available.
  • Positioning: The contribution is a lightweight proxy-validation loop that quantifies simulation–corpus alignment with reproducible metrics, bootstrap confidence intervals, and artifact-first reporting.It supplies structured go/no-go signals for prompt and taxonomy iteration before expensive human studies or large-scale model calibration.

3 Method

The method generates schema-constrained, persona-based UX micro-simulations with structured friction and survey outputs, then compares those friction themes with proxy corpora using explicit alignment metrics and baselines. It separates candidate generation from selection and bounds its claims about what proxy alignment can establish.

  • Simulation pipeline: The pipeline samples a persona, journey, and task, renders a UI-context snapshot, and prompts an LLM to produce structured customer-experience feedback.Outputs include walkthrough steps, taxonomy-mapped friction points, sentiment, suggested fixes, micro-survey signals, and SUS-like answers under a versioned schema contract.
  • Agent strategies: Four schema-compliance strategies are compared: single-pass, best-of-N, hybrid fallback, and score-then-select.Score-then-select independently scores each candidate with a fixed rubric before deterministic selection, reducing ambiguity about cross-candidate anchoring.
  • Proxy alignment: Proxy validation compares simulated and proxy friction distributions over a shared, versioned taxonomy using top-k Jaccard and distributional Weighted-Jaccard W.Domain stopwords and optional category weights reduce spurious keyword matches while preserving explicit taxonomy and weighting rules.
  • Proxy alignment: The alignment baselines are Lexical keyword matching, TF-IDF category assignment, and multilingual Embedding similarity using BGE-M3 via Ollama.Top-k Jaccard is treated as coarse and human-auditable, but can become artificially high when k is large relative to taxonomy size.
  • Claim boundaries: The method claims usefulness for early-stage iteration, detection of egregious proxy mismatches, and stronger single-shot taxonomy matching from embeddings than lexical baselines on primary proxies.It treats bootstrap stability of embedding W as a separate empirical question and does not claim individual-level human-behavior matching, task success, or user preference.

4 Datasets and Proxies

The study validates UX micro-simulations against diverse public proxies spanning app reviews, customer-support tweets, and open-source software issues. These datasets indicate friction themes rather than ground-truth task outcomes, with explicit sampling and locale/domain constraints for computational feasibility and cautious transfer claims.

  • Public proxy datasets: 664k Tinder Google Play reviews provide an English, rated app-review proxy.The Tinder corpus contains approximately 664,000 records.
  • Public proxy datasets: 1.67M Gojek reviews and 2.81M customer-support tweets expand coverage to Indonesian app feedback and unrated support conversations.The Gojek corpus contains approximately 1.67 million rated reviews, while the Twitter corpus contains approximately 2.81 million records without star ratings.
  • Public proxy datasets: OSS issues from Grafana, Prometheus, OpenTelemetry, Jaeger, Zipkin, and Sentry provide a sampled B2B observability proxy.The listed repositories include OpenTelemetry Collector and OpenTelemetry JS alongside the other observability tools.
  • Public proxy datasets: Amazon Appstore proxies include 79,531 low-rated reviews and 8,971 annotated reviews with issue-type categories.Empty review bodies were dropped from the low-rated corpus; the annotated subset supports LLM-based label evaluation using categories as noisy gold labels.
  • Proxy interpretation and feasibility: Proxy datasets represent friction themes, not task-outcome ground truth, and locale and domain remain explicit to avoid overclaiming transfer.Embedding alignment used a pre-cap of n = 10,000 texts for Tinder and Gojek, while Twitter and low-rated Amazon reviews lacked that pre-cap; BGE-M3 used n = 200 per corpus.

5 Experiments

The experiments compare agent strategies under matched schema constraints, quantify cost, quality, and proxy alignment, and assess metric stability and taxonomy classification. Artifact-first exports keep tables and figures traceable to versioned runs.

  • Agent ablations: Three strategies—single-pass, best-of-N, and hybrid—are ablated under matched schema constraints, with hybrid falling back to best-of-N when validation fails.Runs use Azure OpenAI deployments with n = 60 per condition/model, and multi-candidate methods default to N = 2.
  • Agent ablations: Calls, Prefix ok, and a weighted Quality composite quantify inference cost, taxonomy-prefix compliance, and output quality.Quality weights content completeness most heavily through 0.35 rwords and 0.20 rsteps.
  • Proxy alignment: Proxy alignment uses top-k Jaccard and weighted-Jaccard, with defaults k = 8, TF-IDF threshold t = 0.05, embedding threshold t = 0.35, and embedding subsample n = 200.Large Tinder and Gojek app-review corpora are capped at n = 10,000 before embedding subsampling.
  • Proxy alignment: Bootstrap confidence intervals evaluate whether alignment point estimates of W remain stable under resampling across lexical, TF-IDF, and embedding method–dataset pairs.The evaluated proxies include Amazon, Tinder, and Gojek.
  • Taxonomy classification: A fixed n = 500 Amazon subset tests taxonomy classification with Azure-hosted gpt-4.1 under single-pass structured output, reporting accuracy and macro-F1 against category labels.The LLM result is contrasted with IDF and embedding logistic-regression baselines computed on the same subsample.
  • Reproducibility: Every run emits CSV, JSON, and LaTeX artifacts, while a build script automatically exports tables and figures from versioned run artifacts.This keeps the manuscript synchronized with the underlying experiments.

6 Results

Embedding generally aligns better than lexical baselines on primary proxies, but distributional overlap remains modest and embedding W point estimates are unstable under bootstrap resampling. Results also show that Jk can overstate alignment, model choice dominates protocol differences, and proxy outputs support comparative iteration rather than UX-quality proof.

  • Practical use: The pipeline’s value is comparative and diagnostic: teams can compare prompt or taxonomy versions, select top-k friction themes, ship changes, and rerun artifacts while checking bootstrap stability.These proxies are not controlled UX studies, and the stated goal is early-stage iteration with auditable artifacts and bounded claims.
  • Proxy alignment: Lexical Jk reaches 1.0 at k = 10 on Amazon Annotated while weighted-Jaccard remains unchanged, supporting W as the primary distributional signal.The result motivates keeping k smaller than the taxonomy size because large-k top-k Jaccard can overstate alignment.
  • Model sensitivity: Prefix rate 0, grounding 0, and fabrication proxy 1.0 show that gpt-5.2 collapses across every tested condition, the only reproducible model-level breakdown observed.By contrast, gpt-4.1 reports HQ hybrid-LLM quality 0.707 and CQ hybrid-auto and best-of-N auto both at 0.726.
  • Category reliability: Accuracy 0.556 and macro-F1 0.438 for gpt-4.1 on annotated Amazon contrast with accuracy 0.635 for an embedding-plus-logistic-regression baseline.The LLM still beats majority, lexical, and TF-IDF baselines on macro-F1, while functionality_features mapped to ui_ux is the largest single confusion.
  • Proxy alignment: Wembed = 0.128 vs Wlex = 0.000 on Gojek, while embedding exceeds lexical alignment across primary app-review and support-tweet proxies.Amazon Low-rated likewise reports Wembed = 0.119 vs Wlex = 0.009; Sentry is an exception, with Wlex = 0.138 > Wembed = 0.072.
  • Uncertainty and validation: W = 0 across all 50 embedding bootstrap iterations on Amazon Low-rated, Tinder, and Gojek, despite single-shot point estimates of W = 0.119, 0.087, and 0.128.The embedding W estimates are therefore systematically unstable at n = 200, making bootstrap CIs necessary before treating them as targets.

7 Limitations and Threats to Validity

The pipeline faces construct, internal, external, evaluation, taxonomy, and calibration threats that limit how simulation alignment should be interpreted. These limitations motivate treating outputs as audited decision support rather than definitive evidence of usability or task success.

  • Construct validity: Proxy alignment measures distributional similarity of friction themes, not task success, satisfaction, usability, or causal impact.Noisy mixtures of domains and user populations make the construct approximate despite multiple metrics, sensitivity analyses, and confidence intervals.
  • Internal validity: Prompt framing, persona-induced bias, unrealistic behavior, and divergence from human synthetic-survey distributions threaten simulation fidelity.The authors therefore emphasize versioned prompts, schema-constrained outputs, and artifact trails for auditability.
  • Taxonomy coverage of simulations: Essentially every emitted friction point in gpt-4.1 runs carries the navigation: taxonomy prefix, narrowing the surfaced friction-theme distribution.This concentration may reflect both the taxonomy-prefix instruction and the navigation-heavy product UI; broader coverage requires more diverse personas and tasks.
  • External validity: Results depend on the chosen taxonomy, locales, and proxy sources, while OSS issues remain indirect and may over-represent power users or issue reporters.Future work should evaluate domain-specific proxies and compare against controlled studies when available.
  • Evaluation validity: The annotated Amazon subset is a weak, potentially biased gold label, and LLM evaluators or labelers may apply rubrics inconsistently.The labeled subset is used primarily for calibration and error analysis, not as a definitive performance target.
  • Grounding heuristic calibration: Token-overlap grounding and fabrication proxies provide conservative lower bounds because shared UI vocabulary can distort their values.Teams should recalibrate the stop list to their UI vocabulary; the proxy remains useful for relative comparisons when stop-list bias is consistent.
  • Cost and model choice: Stronger paid-model calibration and broader sweeps over embedding models and subsample sizes are deferred to control cost and avoid iterative overfitting.The study uses Azure-hosted gpt-4.1, gpt-4.1-mini, and gpt-5.2 models, plus BGE-M3 embeddings via Ollama.

8 Conclusion

The paper concludes that its reproducible UX simulation and proxy-validation pipeline provides directional, artifact-traceable signals for early-stage iteration. Results support embedding-based alignment on most app-review and support-tweet proxies, while limitations constrain interpretation and motivate broader calibration and decision integration.

  • Conclusion: The pipeline compares simulated friction signals with multiple public corpora using top-k Jaccard and weighted-Jaccard.It is designed as a reproducible proxy-validation protocol.
  • Summary of findings: W = 0.128 vs 0.000 lexical on Gojek, with BGE-M3 outperforming lexical and TF-IDF baselines on app-review and support-tweet proxies.Sentry OSS issues were an exception, where product-specific vocabulary favored lexical matching.
  • Summary of findings: Top-k Jaccard reached 1.0 at k = 10 while W remained at 0.017 on the Amazon annotated proxy.This illustrates why the two alignment metrics provide complementary signals.
  • Limitations recap: Proxy corpora are noisy, do not measure task success, and support only directional signals rather than claims about individual user behavior.Grounding depends on stop-list choices, and embedding stability is bounded by one subsample size and one anchor construction despite bootstrap CIs across 8 method–dataset pairs.
  • Future work: Future work will broaden locale and domain matching, add adversarial controls, calibrate stronger paid models at scale, and connect alignment to product decisions and release gating.The paper also plans bootstrap CIs across alignment methods and datasets.
  • Product perspective: The proposed repeatable UX check produces prioritized friction themes, confidence bounds, and traceable artifacts before and after changes.This is intended to help teams focus on high-leverage issues, communicate risk, and detect newly introduced friction.

A Additional Tables

The appendix includes a sensitivity analysis of selected settings on the Amazon annotated proxy.

  • A Additional Tables: The Amazon annotated proxy is examined under selected settings in a sensitivity analysis.

B Friction Taxonomy

The paper uses a versioned friction taxonomy that defines categories, default weights, representative keywords, multilingual variants, and domain-specific overrides.

  • Multilingual coverage: The complete taxonomy configuration preserves Portuguese, Spanish, and Indonesian keyword variants for multilingual proxy datasets.These multilingual keyword lists are maintained in the versioned configuration rather than fully displayed in Table 15.
  • Weighting: Domain-specific weight overrides are stored in the versioned taxonomy configuration.Table 15 reports default weights, while the configuration contains domain-specific adjustments.
  • Taxonomy definition: Table 15 defines the friction categories used throughout the paper, alongside default weights and representative keywords.The displayed keyword set is truncated to six examples per category.

C Prompt Excerpts

The appendix presents the core prompts for structured UX micro-simulation, grounding and taxonomy control, candidate selection, scoring, and app-review friction labeling. These prompts impose strict JSON, schema, grounding, and single-category constraints.

  • C Prompt Excerpts: The appendix provides short excerpts of the key prompts used in the UX micro-simulation pipeline.
  • Simulation prompt (v1): The v1 simulation prompt requires structured JSON, ASCII, no PII or secrets, strings of at most 12 words, exactly two steps, one friction point, and one HEART GSM item.
  • Simulation prompt (v1): The v1 prompt also requires non-empty feedback and fixes for each simulated output.
  • Simulation prompt (v2) — grounding + taxonomy prefix: The v2 additions restrict friction claims to UI elements named in the snapshot and require exactly one taxonomy category prefixed by its category_id.
  • Judge prompt (best-of-N selection): The best-of-N judge receives multiple candidate JSON outputs, selects the best candidate, and returns a selected index with scores.
  • Judge prompt (score-then-select): The score-then-select judge independently evaluates one candidate on schema compliance, UI grounding, actionability, and overall quality, returning scores and rationale.
  • Appstore label prompt (LLM evaluation): The appstore label prompt routes each review to exactly one friction-taxonomy category and requests a short justification in valid JSON.
Loading 2608.13563v1…