PaperlayerView exact revision on arXiv

Source-linked AI summary

The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations

Dmitrij Żatuchin

arXiv:2609.04047v1cs.IRcs.CL

TL;DR

Repeated-query audits of LLM brand recommendations lack standardized iteration, metric, reliability, and drift practices. This paper formalizes the Dice Roll Method with conditional mixed-model inference, dependence-preserving resampling, generalizability analysis, and drift diagnostics. External validation supports the reliability machinery but not fixed iteration tiers, favoring a pilot-then-solve approach.

  • Problem

    Repeated-query auditing lacks standardized guidance for iteration counts, stability metrics, reliability thresholds, and cost-efficient design.

  • Method

    The paper combines a generative variance model with negative-binomial mixed modeling, Cliff’s delta, dependence-preserving bootstrap, simulation-based power, generalizability theory, and drift diagnostics.

  • Results

    The D-study estimates G = 0.58 at n = 5, G = 0.74 at n = 10, and G = 0.81 at n = 15, while external validation reproduces reliability predictions in 37 of 39 cells with no failures.

  • Takeaways & Limitations

    The protocol provides a statistically principled footing for repeated-query brand-recommendation audits while supporting pilot-based iteration planning rather than fixed universal tiers.

  • Takeaways & Limitations

    Domain-specific effect sizes and thresholds remain based on this group’s brand-recommendation data, and fixed iteration tiers do not transfer across external corpora.

Abstract

from arXiv · show

Background: Researchers increasingly use repeated identical prompts to audit stochastic variation in large language model (LLM) brand recommendations, yet no standardized protocol exists for setting iteration counts, selecting stability metrics, or establishing reliability thresholds. Objective: We formalize the Dice Roll Method as a reusable protocol for repeated-query auditing of LLM brand recommendations, grounded in a generative model of temperature-scaled nucleus sampling. Methods: Total response variance is decomposed into sampling, prompt-phrasing, run-to-run, and model-version components. The stack: a negative-binomial mixed model with iterations as repeated measures; Cliff's delta as the distribution-free effect size; dependence-preserving bootstrap; simulation-based power; a generalizability-theory decomposition; drift diagnostics on pinned snapshots. We reanalyse five brand-recommendation auditing studies: approximately 190,000 observations, 270+ brands, 6 languages, iteration counts 5 to 40. Results: Three tiers of iteration guidance emerge from the D-study: exploratory (n = 5, G = 0.58), confirmatory (n = 10, G = 0.74), and rigorous (n = 15, G = 0.81), tied to effect-size and generalizability targets. The four metric families (count, set, embedding, fairness-adjusted PASOR) are complementary, motivating a compact metric battery over single indicators. A pre-registered external validation on three independent corpora (Motoki et al., 100-round; Rozado, 24 models; llm-stability) reproduces the D-study reliability prediction in 37 of 39 cells with no failures and the n = 5 power value to two decimals; the fixed tiers do not transfer, supporting a pilot-then-solve reading. Conclusion: The protocol gives repeated-query auditing of LLM brand recommendations a statistically principled footing under the conditional, non-Gaussian structure of real autoregressive generation.

1 Introduction

The paper formalizes repeated identical querying as the Dice Roll Method and addresses the lack of standardized iteration, metric, reliability, and cost-efficiency guidance. It replaces an earlier Gaussian, independence-based stack with a protocol tailored to conditional, non-Gaussian LLM generation.

  • Motivation: Repeated identical queries produce variable brand recommendations because stochastic token generation is intrinsic to LLM output.The paper treats this variability as a measurement property rather than noise to average away.
  • Motivation: The Dice Roll Method names repeated querying and aggregation for estimating distributional properties of LLM recommendations.Each query samples from the model’s output distribution, while aggregation yields more stable estimates.
  • Standardization gap: Published studies vary from 5 to 40 iterations and use heterogeneous stability metrics without formal guidance for power or cost-efficient design.Examples include coefficient of variation, cosine similarity, entropy, and Gini coefficients.
  • Contribution: The paper combines five-study reanalysis with Monte Carlo power simulation to formalize a standardized auditing protocol.The protocol addresses power, metric selection, reliability, convergence, and cost-efficiency questions.
  • Contribution: The revised framework replaces independence-based Gaussian analyses with conditional, non-parametric methods and drift, embedding-variance, and PASOR diagnostics.The revision responds to dependence, over-dispersion, small-sample uncertainty, and omitted model-version variance.
  • Contribution: The protocol’s five contributions cover generative modeling, simulation-based GLMM power, complementary metrics, generalizability theory, and stationarity diagnostics.It uses a negative-binomial GLMM, Cliff’s delta, bootstrap procedures, reliability decomposition, and drift testing.

2 Stochastic Structure of LLM Outputs

The paper models LLM response variability as conditional, multi-component stochastic generation rather than independent Gaussian noise. This structure motivates mixed models, dependence-preserving resampling, distribution-free effect sizes, and explicit drift diagnostics.

  • Generative structure: The inferential framework begins with a generative model of temperature-scaled softmax and nucleus sampling.At each decoding step, the model samples from a temperature-adjusted distribution restricted to a cumulative-probability nucleus.
  • Generative structure: At temperature 0.3, nucleus sampling concentrates probability without becoming deterministic, preserving residual response variability.The effective support remains non-degenerate for most contexts.
  • Dependence: Autoregressive token dependence makes response-level metrics compound objects rather than independent, identically distributed Gaussian draws.Each token depends on preceding tokens, and downstream entities inherit this correlated sequence structure.
  • Variance decomposition: Total variance is decomposed into token sampling, prompt phrasing, model-version and time, and interaction components.The decomposition separates variation that repeated queries can average from prompt, systematic model, drift, and interaction effects requiring other designs.
  • Inference: Conditional dependence requires mixed models with repeated-measure random effects or block bootstrap of intact iteration sequences.Resampling individual observations can destroy within-condition dependence and inflate apparent information.
  • Inference: Over-dispersed, zero-inflated brand counts motivate negative-binomial mixed models and Cliff’s delta instead of Gaussian summaries and Cohen’s d.The coefficient of variation can also behave poorly when means are low or zero.
  • Inference: Potential temporal non-stationarity requires explicit drift diagnostics because infrastructure changes can alter outputs without version-string changes.The revised methodology addresses dependence, non-Gaussianity, and non-stationarity together.

3 Related Work

Related work establishes that repeated-query auditing is widely used but lacks principled iteration guidance and unified measurement practice. The paper positions its protocol against this variability and against analytical methods that ignore dependence and non-Gaussian output distributions.

  • LLM variability: LLM output variance can persist even at temperature 0.0 because of floating-point, batching, and hardware-level non-determinism.This makes apparent determinism an inadequate basis for single-query auditing.
  • LLM variability: Repeated aggregation is needed to distinguish systematic recommendation patterns from stochastic fluctuations, but prior work did not establish sufficient observation counts.Single responses can differ substantially in brand counts, company inclusion, and evaluative framing.
  • Prior auditing studies: Prior studies used iteration counts from 5 to 40 and reported substantial variation in brand-count and concentration measures.One study used 40 iterations for an adult-recipient subset and 10 for a Valentine’s Day subset.
  • Prior auditing studies: A reputation-sourcing study used n = 5 iterations, found mean cosine similarity of 0.54, and reported 14% cross-model agreement.The comparison suggests that cross-model differences can exceed within-model stochastic variation in that study.
  • Prior auditing studies: Prior category and gender-bias studies used Gini, entropy, and related measures to characterize concentration and distributional diversity.The category study reported mean Gini = 0.28, while the companion analysis treated entropy and Gini as complementary.
  • Protocol motivation: The present study formalizes pragmatic prior iteration choices through statistical analysis rather than intuition alone.Its focus is the statistical adequacy of repeated-query designs.
  • Power analysis: Classical power analysis targets sample sizes for detecting effects at specified significance and power levels.The cited conventional framework commonly uses 80% power and Cohen’s d benchmarks.
  • Power analysis: Classical power analysis fails here because repeated queries are dependent and output metrics are non-Gaussian, causing miscalibrated effect sizes and power calculations.Observed issues include zero-inflation, over-dispersion, heavy tails, and floor or ceiling effects.

4 Methodology

The methodology reanalyses five brand-recommendation auditing studies using shared repeated-query conditions and a statistical framework designed for over-dispersed, dependent LLM outputs. It combines mixed models, robust effect sizes, dependence-preserving convergence analysis, simulation-based power, and generalizability theory.

  • Data sources: Five source studies provide reanalysis data spanning different iteration counts, languages, models, and stability metrics.S1 includes n = 40 and n = 10 subsets; S2 and S5 use n = 5; S4 includes 9,577 responses across 20 brands, 6 languages, and 3 models.
  • Metric framework: The metric framework treats information-theoretic measures and embedding stability as complementary rather than interchangeable indicators.The revised protocol replaces a single embedding estimate with ensemble-based uncertainty propagation.
  • Primary inferential model: Substantive group contrasts are estimated with a negative-binomial GLMM containing fixed effects for prompt type, gender, language, and interactions.Random intercepts for prompt, model, and prompt–model interaction address conditional dependencies, while the negative-binomial likelihood handles over-dispersed counts.
  • Data sources: All source studies use identical prompts repeated at temperature 0.3 with structured brand extraction from responses.The shared conditions define the methodological basis for comparing stability and reliability across studies.
  • Effect size reporting: Cliff’s δ with BCa bootstrap confidence intervals is the primary effect size, supplemented by trimmed-mean Cohen’s d and quartile coefficient of dispersion.Cliff’s δ is distribution-free and robust to skewness, heavy tails, and ordinal data; QCD replaces coefficient of variation where low means make scale-sensitive dispersion unstable.
  • Power and convergence: Power simulations generate responses from fitted conditional models across iteration counts n ∈ {3, 5, 7, 10, 12, 15, 20, 30, 40}.Sensitivity analyses vary the mean and dispersion parameters across observed study ranges so recommendations are paired with their distributional conditions.
  • Power and convergence: Moving-block bootstrap and empirically selected convergence models estimate stability while preserving dependence across iterations.Logarithmic, power-law, Michaelis–Menten, and linear models are compared using AIC, BIC, and leave-one-prompt-out cross-validated log-likelihood.
  • Reliability via generalizability theory: Generalizability theory decomposes variance across prompts, iterations, models, and interactions, then projects reliability across candidate iteration–model designs.The D-study identifies whether improving an audit requires more iterations, more models, or both; ICC is retained only for continuity.

5 Results

The results show that per-cell power, convergence, reliability, and metric behavior support iteration guidance that is useful but conditional on audit design. The analyses also show that pooled simulations can overstate power, while model-specific effects and metric correlations vary substantially.

  • Power and observed effects: The pooled-prompt sensitivity analysis saturates near n = 3, so it does not provide tight iteration recommendations; the per-cell D-study supplies the binding constraint.The pooled design aggregates 8 prompts × 3 models × n iterations, producing 72 observations at n = 3.
  • Power and observed effects: At n = 5, per-cell power is 0.44 for large effects (δ = 0.474) and 0.23 for medium effects (δ = 0.33), with the 80% threshold first met at n = 15.The source studies’ observed effects therefore imply n = 10–15 iterations for many audits.
  • Power and observed effects: Observed effects produce model-specific regimes: Grok’s δ = 0.26 is detectable at 80% power by n ≈20, GPT’s δ = 0.18 requires n ≈40, and Gemini’s δ = 0.04 is statistically indistinguishable from null at practical iteration counts.At n = 10, Grok reached 48% power and the Valentine’s subset reached 31% power for GPT.
  • Convergence: The block-bootstrap standard error falls from 3.52 at n = 2 to 1.67 at n = 10, with 80% of asymptotic precision at n = 7, 90% at n = 10, and 95% at n ≈15.The trajectory follows a diminishing-returns pattern and is consistent with an approximate 1/√n rate after dependence-preserving resampling.
  • Metric behavior: Metric relationships are data-dependent: Gini and Shannon correlate at r = −0.98, while CV and Gini correlate at r = 0.98 on the Valentine’s subset.These structural count-based metrics therefore carry largely the same information in this subset, although the broader four-family battery remains defensible in principle.
  • Reliability: Within-cell noise accounts for 78.2% of total variance, compared with 21.8% for the model-by-prompt cell, making iteration count the main lever for reducing sampling noise.The variance decomposition treats the model-by-prompt cell as the measurement object.
  • Reliability: G = 0.80 is crossed between nI = 12 (G = 0.77) and nI = 15 (G = 0.81), while nI = 10 yields moderate generalizability (G = 0.74).At nI = 20, G = 0.85, whereas G ≥0.90 would require more than 40 iterations or design changes that reduce within-cell noise.
  • Reliability: Reliability increases with iteration count, crossing ICC = 0.74 at n = 8 and reaching ICC = 0.81 at n = 10, although confidence intervals remain wide.The single-study analysis uses k = 15 prompt-model combinations.

6 External Validation on Independent Corpora

External validation on three independent corpora largely reproduced the protocol’s reliability and power predictions, while exposing horizon- and variance-dependent limits on fixed iteration tiers and drift diagnostics.

  • 37 of 39 pre-registered D-study prediction cells replicated, with 2 partial replications and no failures.
  • Variance components estimated from early rounds predicted longer-horizon reliability with maximum absolute error 0.038 on C1, median absolute error 0.008 on C2, and errors at or below 0.002 on C3.
  • The n = 5 power calibration transferred: C1’s direct subsampling produced median power 0.43 for large effects.
  • Fixed iteration constants did not transfer because reliability varied with variance ratios; pilot variance estimation followed by solving for target G was supported instead.
  • Convergence guidance was horizon-relative: 80% of asymptotic precision arrived by n = 8 on short grids but by n = 16 against C1’s n = 50 asymptote.
  • The drift battery required amendments because regression-window and PSI components generated widespread or mechanically saturated flags, whereas KS diagnostics separated placebo from primary-arm behavior.

7 Discussion

The discussion translates the validation and reanalysis into iteration, metric-selection, and protocol guidance while emphasizing that recommendations depend on effect sizes, variance structure, temperature, model coverage, and prompt diversity.

  • Minimum Iteration Recommendations: G = 0.58 at n = 5, 0.74 at n = 10, and 0.81 at n = 15 define exploratory, confirmatory, and rigorous iteration tiers.
  • Minimum Iteration Recommendations: n = 15 first meets 80% per-cell power for large effects, while n > 40 would be required to reach the 0.90 individual-prompt decision threshold under the observed variance ratio.
  • Operating Bounds: Higher temperatures increase token-level sampling variance and shift iteration requirements upward; different analytical tasks may require fewer iterations at the same effect-size target.
  • Metric Selection: A parsimonious metric core combines one structural metric, ensemble semantic cosine similarity, and PASOR-Gini, with mean brand count as a descriptive summary.
  • Metric Selection: Shannon entropy is redundant with Gini on narrow-prompt audits but becomes potentially useful when pilot correlations show structural metrics decouple across diverse prompts.
  • Implications for Existing Published Findings: At n = 5, power was approximately 0.44 for large effects and 0.23 for medium effects, limiting resolution of within-condition variability in published audits.
  • Implications for Existing Published Findings: The revised framework found the GPT-5.2 gender contrast statistically indistinguishable from zero: Cliff’s δ = 0.16, CI [−0.04, 0.34].
  • Standardized Protocol: The protocol formalizes pinned configuration logging, tiered iteration selection, multi-model coverage, and the compact metric battery under conditional, non-Gaussian, potentially non-stationary data.

7. Drift

The protocol treats drift, temperature, model version, domain, language, and embedding coverage as boundaries on generalizability. External validation supports some machinery but shows that fixed iteration tiers and some diagnostics require recalibration.

  • Operational constraints: At n = 10 with 3 models, a 100-query study requires 3,000 API calls, costs approximately $15, and takes 2–4 hours.Running GLMM and drift diagnostics adds negligible cost compared with collecting additional iterations.
  • Domain and language scope: The protocol’s observed effect sizes, convergence rates, and metric correlations may differ outside brand-recommendation auditing.Stability properties were largely consistent across six European languages, but non-European languages with greater structural distance from English remain untested.
  • Evidential circularity: The domain-specific effect sizes and thresholds remain based on this group’s data because no independent repeated-prompt brand-recommendation corpus is publicly available.External validation independently checks parts of the statistical machinery, but not the brand domain itself.
  • Temperature dependence: At τ = 0.3, the analytical framework transfers across temperatures, but specific iteration thresholds require recalibration because higher temperatures increase σ2_tok.Temperature dependence limits direct transfer of the reported iteration guidance.
  • Provider-side drift: The drift analysis found 0 of 15 S1 cells flagged, while multi-week S2, S4, and S5 collection windows remain untested.Audits requiring stationarity must use short collection windows or fixed-window time slicing because the protocol cannot prevent provider-side drift.
  • Model dependence: Future model architectures may have different stochastic properties, and the nucleus-sampling derivation does not automatically transfer to fundamentally different decoding schemes.The framework therefore requires longitudinal assessment across model updates.
  • Convergence and embedding scope: The convergence guidance is horizon-relative: 80% of asymptotic precision arrives at n = 16 against an n = 50 asymptote.The external corpus confirms a 1/√n rate, while the three-embedding ensemble does not cover all current embedding architectures.

8 Conclusion

The revised paper formalizes repeated-query auditing with a dependence-aware statistical stack and empirical variance-component analysis. Its findings support calibrated iteration planning and external validation of some protocol components, while favoring pilot-then-solve over fixed tiers.

  • Conclusion: The protocol combines a negative-binomial GLMM, Cliff’s δ with BCa intervals, dependence-preserving bootstrap, simulation-based power, generalizability analysis, and drift diagnostics.These components are designed for the conditional, non-Gaussian structure of repeated autoregressive generation.
  • Conclusion: G = 0.58 at n = 5, G = 0.74 at n = 10, and G = 0.81 at n = 15, crossing the 0.80 group-level threshold between n = 12 and n = 15.These values come from the generalizability-theory D-study on pooled S1+S5 data.
  • Conclusion: 37 of 39 external-validation cells reproduced the D-study prediction with no failures, while fixed iteration tiers and two drift-battery components failed to transfer.The revised prescription is pilot-then-solve with amended diagnostics.
  • Conclusion: The protocol enables researchers to justify iteration counts through simulation-based power, match generalizability targets to budgets, and preregister drift diagnostics.An open-source Python notebook automates GLMM fitting, bootstrap variants, D-study projections, and drift tables.
  • Conclusion: The paper aims to strengthen the methodological foundations of repeated-query audits whose findings carry increasing policy and commercial significance.This consequence is framed as the paper’s broader motivation rather than as a demonstrated empirical effect.

Declarations

The declarations identify the work as a secondary analysis of five prior studies and disclose a relevant author affiliation with Rankfor.AI.

  • Declarations: The manuscript reanalyses raw iteration-level data from five prior empirical studies to answer a distinct methodological question about iteration counts and metric choice.The authors characterize the overlap as a secondary analysis rather than republication of prior substantive conclusions.
  • Declarations: D. ˙Zatuchin is Chief Executive Officer of Rankfor.AI, which develops AI brand intelligence tools and implements the Dice Roll Method protocol.The paper reports no external funding.

Appendix A Power Analysis Lookup Tables

Appendix A provides a per-cell iteration-count lookup table intended for practitioners and reviewers.

  • Appendix A Power Analysis Lookup Tables: Table A1 provides a per-cell iteration-count lookup table for practitioner and reviewer reference.The passage identifies the table’s purpose but gives no individual lookup values.

Appendix B Metric Definitions

Table B2 provides formal definitions for all six metrics used in the Dice Roll Method.

  • Table B2 formally defines all six metrics used in the Dice Roll Method.

Appendix C Hypothesis Test Summary

Table C3 summarises formal hypothesis tests under the revised inferential framework, including two hypotheses added in the present revision.

  • Table C3 summarises the formal tests of hypotheses evaluated under the revised inferential framework.
  • H5 and H6 were added in the present revision.

Appendix D Reproducibility Checklist

The reproducibility checklist specifies preregistration, design selection, model configuration, drift diagnostics, data handling, metric batteries, and reporting requirements for Dice Roll audits.

  • Design selection: Use Table A1 for per-cell iteration guidance and Table 8 for pooled-prompt upper bounds when selecting an audit design.
  • Design selection: Select an audit design from the D-study projection that meets a pre-registered G target: 0.80 for group-level or 0.90 for individual-prompt decisions.
  • Model configuration: Pin temperature = 0.3, max tokens = 1,024, system prompt, model identifier, API endpoint, and snapshot date.
  • Drift and matching: Plan drift diagnostics by splitting collection windows and applying KS, PSI, and NB-drift tests, adding window as a fixed effect when flagged.
  • Data collection: Use checkpointing every 25–50 API calls, automatic resume capability, and a word-boundary-aware brand alias dictionary.
  • Metric battery: Use Gini or CV, cosine-similarity ensembles, and PASOR-Gini; add Shannon entropy only when pilot correlations show decoupling from Gini.
  • Metric battery: Report across-model embedding standard deviation rather than point estimates alone, using an ensemble such as BGE-M3, Nomic, and MiniLM.
  • Iteration guidance: The per-cell lookup table reaches 80% power at n = 15 for δ = 0.474, approximately n = 20 for δ = 0.26, and approximately n = 40 for δ = 0.18.
Loading 2609.04047v1…