PaperlayerView exact revision on arXiv

Source-linked AI summary

Using simulation studies to evaluate statistical methods

Tim P Morris, Ian R White, Michael J Crowther

arXiv:1712.03198v3stat.ME

TL;DR

Simulation studies are widely used but often lack careful design and reporting. This tutorial introduces a structured framework and practical guidance, showing through examples how thoughtful analysis and visualisation clarify method performance.

  • Problem

    Simulation studies are widely used in statistical research, yet some researchers lack confidence while others report studies poorly, motivating clearer practice.

  • Method

    The tutorial proposes ADEMP for planning and reporting, alongside guidance on terminology, coding, performance measures, and result presentation.

  • Results

    Worked examples show that visualisations can reveal bias, estimator differences, and confidence-interval behaviour that summary performance measures may obscure.

  • Takeaways & Limitations

    Transparent reporting of simulation design and execution is critical, with aims, data-generating mechanisms, methods, estimands, performance measures, and rationale documented.

  • Takeaways & Limitations

    No simulation study is definitive, so fully documented designs and publicly released code are needed to support replication or extension.

Abstract

from arXiv · show

Simulation studies are computer experiments that involve creating data by pseudorandom sampling. The key strength of simulation studies is the ability to understand the behaviour of statistical methods because some 'truth' (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analysed and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting and presentation. In particular, this tutorial provides: a structured approach for planning and reporting simulation studies, which involves defining aims, data-generating mechanisms, estimands, methods and performance measures ('ADEMP'); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine that included at least one simulation study and identify areas for improvement.

1 INTRODUCTION

Simulation studies use pseudo-randomly generated data from known distributions to obtain empirical evidence about statistical-method performance, especially when analytic results are difficult or assumptions fail. This tutorial addresses weaknesses in their design, analysis, presentation and reporting by providing unified practical guidance and the ADEMP planning framework.

  • Simulation studies are computer experiments that create data by pseudo-random sampling from known probability distributions.They are used particularly to evaluate new methods and compare alternative methods.
  • They provide empirical performance results for specified scenarios, including situations where analytic results are difficult or methods make wrong assumptions.Simulation studies can assess method resilience when data are messy.
  • The paper identifies poor practice in simulation-study design, analysis, presentation and reporting, including limited acknowledgement of Monte Carlo uncertainty.Monte Carlo procedures can produce different results from different sets of random numbers.
  • The tutorial provides unified practical guidance through structured planning and reporting, coherent terminology, coding guidance, and discussion of performance measures and their estimation.It introduces the ADEMP structure for planning simulation studies.
  • The tutorial also reviews simulation studies reported in Statistics in Medicine Volume 34 and presents guidance on computational considerations, performance measures, and results presentation.Its sections cover review, planning, coding, performance-measure estimation, and presentation.

2 SIMULATION IN PRACTICE: A REVIEW OF STATISTICS IN MEDICINE, VOLUME 34

The authors reviewed simulation-study practice in Statistics in Medicine Volume 34, identifying 199 of 264 research articles (75%) that included at least one simulation study. They randomly sampled 100 such articles for detailed review.

  • Review scope: The review covered research articles published in Statistics in Medicine Volume 34 (2015), excluding non-research content.Excluded items included biostatistics tutorials, commentaries, book reviews, corrections, letters to the editor, and authors’ responses.
  • Review scope: 199 of 264 research articles (75%) included at least one simulation study.The review restricted attention to research articles in the volume.
  • Sampling and review process: The reviewers randomly selected 100 articles involving simulation studies to estimate binary-answer proportions with maximum standard error 0.05.The maximum standard error occurs when the proportion is 0.5.

3 PLANNING SIMULATION STUDIES USING ADEMP

The ADEMP framework structures simulation-study planning around aims, data-generating mechanisms, estimands, methods and performance measures. Planning should align these components with the properties and scenarios the study is intended to investigate.

  • ADEMP framework: ADEMP comprises Aims, Data-generating mechanisms, Estimands, Methods and Performance measures.The framework provides a structured approach to planning simulation studies.
  • Aims: Simulation aims may target bias, precision, variance estimation, robustness to misspecification, or method viability and failure under challenging settings.Studies may provide proof of concept or deliberately stretch methods to identify settings where they fail.
  • Data-generating mechanisms: Data-generating mechanisms should be chosen to match the aims and may be simple, realistic, unrealistic, parametric, or resampling-based.Simulation studies often use multiple mechanisms or vary factors such as sample size to cover different scenarios.
  • Data-generating mechanisms: Factorial variation of factors such as sample size and true parameter values is generally more informative than changing them one at a time.When computation prevents a fully factorial design, predictions beyond initially simulated mechanisms can support a more careful non-factorial design.
  • Estimands and methods: The estimand is the population quantity being estimated, and methods should be included only when they target the specified estimand or estimands.Estimands may be model parameters or other quantities such as prognostic ability or a fitted outcome mean.
  • Estimands and methods: Method definitions and performance measures require care because non-convergence, perfect prediction and unclear or secondary targets can complicate evaluation.The review found studies evaluating between 1 and 33 methods, while performance-measure denominators varied by applicability.

4 COMPUTATIONAL AND PROGRAMMING ISSUES IN SIMULATION STUDIES

Simulation studies require careful control of pseudorandom-number generation, coding, parallelisation and error handling because minor implementation problems can compromise results. Reproducible random-number states, staged development and built-in checks help diagnose failures and prevent dependence between simulated datasets.

  • Random-number generation: Modern pseudo-random-number generators are deterministic and reproducible when correctly used, allowing simulations to continue and failed repetitions to be rerun from recorded states.The starting state for a failed repetition can be noted and the repetition rerun to understand when a method does not work.
  • Random-number generation: Set the seed once at the beginning and store the random-number-generator state at each repetition and after the final repetition.The recommended pseudocode stores the repetition and state before generating each dataset, then stores n_sim+1 and the final state.
  • Random-number generation: Deriving repetition seeds from successive draws can create dependence between simulated datasets, compromising performance estimates and Monte Carlo SEs.In the example, repeated values appear across rows, and draws become independent of the first repetition only when i > n_obs.
  • Parallelisation: Parallelise different data-generating mechanisms rather than repetitions within one mechanism when using a single seed, unless independent random-number streams are explicitly managed.The same seeds across different data-generating mechanisms are less problematic because performance is typically estimated separately for each mechanism.
  • Coding and validation: Minor coding errors can completely change simulation results, so development should begin with small, specific tests and expand through explored datasets, multiple seeds, few repetitions and built-in checks.Known-property methods should be included where possible, and anticipated convergence problems or zero bias can be checked before running the full simulation.
  • Error handling: When analyses fail, continue the simulation by capturing the error and recording the failed analysis as a missing estimate with its reason when possible.The passage recommends Stata’s capture command and R’s try command for this purpose.

5 ANALYSIS OF ESTIMATES DATA

The section recommends checking missing estimates and exploring estimates graphically before analysing performance measures. It explains key measures, their Monte Carlo uncertainty, and cautions about interpreting comparisons when missingness, bias, or sample size affect results.

  • Exploratory analysis: Exploratory plots should precede formal analysis to inspect distributions, outliers, method correlations, systematic differences, agreement, and confidence-interval coverage.Recommended displays include univariate and bivariate plots, method-versus-method comparisons, limits-of-agreement plots, and zip plots.
  • Missing estimates: Missing estimates are the first performance measure to assess, because non-convergence can create non-random missingness and undermine comparisons of other measures.Further analysis should be tentative when missingness is substantial or differs across methods, and failures should be investigated to improve code robustness.
  • Performance measures: Performance measures should usually be interpreted jointly, since optimizing one property can conceal another, such as zero variance concealing bias.Bias assesses targeting of the true value, while empirical SE assesses precision over repetitions.
  • Monte Carlo uncertainty: Monte Carlo standard errors quantify simulation uncertainty arising from the finite number of repetitions, yet 93 reviewed studies did not mention them.The standard-error formulas assume normally distributed estimates for several measures, although robust alternatives exist for non-normal estimates.
  • Performance measures: MSE combines squared bias and variance but can yield sample-size-dependent method rankings, so its comparisons require caution.In the example, root MSE favored method B for n_obs < 60 and method A for n_obs > 60; relative precision also requires caution when methods are biased.
  • Coverage: Bias-eliminated coverage separates coverage problems caused by bias from those caused by incorrect interval width by evaluating coverage for the mean estimate rather than the true value.Bias can cause under-coverage, while interval-width errors can cause under- or over-coverage.

6 REPORTING

The reporting of simulation studies should follow a reproducible structure and present results in tables or graphs designed for readability, clarity and relevant comparisons. Tables and graphical displays each offer advantages, but their limitations and Monte Carlo uncertainty must be considered.

  • Methods reporting: ADEMP provides an appropriate reporting order, with methods details sufficient to support study reproduction, including the value and determination of n_sim.Planning and writing the simulation study before execution can largely complete the methods section and help other researchers replicate it.
  • Tabular displays: Tables should arrange data-generating mechanisms, methods, estimands and performance measures to make important comparisons—especially between methods—easy to inspect.Rows and columns can be swapped according to the relative number of methods and data-generating mechanisms, while estimands and performance measures should be placed where the table is easiest to digest.
  • Tabular displays: Monte Carlo SEs should not clutter tables, may be reported in captions, and should constrain the displayed numerical precision.Results should not be presented more accurately than justified by the Monte Carlo SE, such as reporting coverage to three decimal places.
  • Graphical displays: Graphical displays make patterns easier to spot across dimensions not shown side-by-side and can present raw estimates alongside performance summaries.Plots can also summarize estimates and model SEs, but they are less space-efficient, do not provide exact numbers readily, and may require separate plots for different performance measures.
  • Graphical displays: Plots can show Monte Carlo SEs directly, such as 95% confidence intervals, while methods should usually occupy horizontal rows and performance measures columns.For large full-factorial studies, nested-loop plots are one possible presentation, and no single format is universally correct; readability and clarity should guide the choice.

7 WORKED ILLUSTRATIVE EXAMPLE

The worked example applies the ADEMP framework to compare exponential, Weibull, and Cox methods for estimating a hazard ratio under two survival-data-generating mechanisms. It illustrates model misspecification, exploratory visualisation, performance assessment, and the need for broader scenarios.

  • Design: The simulation compares exponential, Weibull, and Cox proportional-hazards models for estimating a treatment hazard ratio in a randomised trial with survival outcomes.The aims include assessing baseline-hazard misspecification, unnecessary model complexity, and the use of a semiparametric model.
  • Design: Two mechanisms use 500 patients with equal randomisation and hazard ratio 0.607, with γ = 1 supporting both exponential and Weibull models and γ = 1.5 supporting only Weibull.The exponential model is correctly specified only for the first mechanism, whereas Weibull is correct for both and Cox assumes no baseline-hazard form.
  • Results: There were no missing estimates or standard errors, and separation was not an issue.The analysis assessed convergence, bias, coverage, and empirical and model-based standard errors for the log-hazard ratio.
  • Results: When γ = 1, the three methods had similar means and variances, with means close to the true θ = −0.5; under γ = 1.5, empirical standard errors were slightly higher.The higher standard errors under γ = 1.5 were attributed to fewer events among the 500 observations.
  • Results: Under γ = 1.5, the misspecified exponential model showed bias toward the null, approximately 10% of the true value, while Weibull and Cox behaved well.Coverage remained above the nominal 95% despite the bias, and model-based standard errors appeared to overestimate empirical standard errors.
  • Limitations: The example’s data-generating mechanisms lacked breadth, motivating exploration of varying n_obs, λ, and θ across ranges to identify when problems arise.The authors conclude that further research is needed because misspecified exponential models can bias hazard-ratio estimates, probably not by much.

8 CONCLUDING REMARKS

Simulation studies are powerful but often poorly reported, making transparent design, execution, uncertainty assessment, and reproducibility essential. The tutorial advocates ADEMP, simulation protocols, public code, and collaborative studies to improve credibility and resolve contradictory findings.

  • Simulation studies are invaluable for understanding statistical methods, but transparent reporting of their design and execution is critical because findings may guide medical and scientific research.
  • ADEMP structures planning and reporting around aims, data-generating mechanisms, methods, estimands, performance measures, and the rationale for decisions.
  • Reports should estimate Monte Carlo uncertainty, supported by formulas and software packages including simsum and rsimsum for common performance measures.
  • Detailed protocols written before coding, public code release, and consortia of authors can reduce selective reporting, support replication, and help investigate contradictory simulation results.
  • Benchmarking must account for implementation skill because poor implementation can make a superior method appear weak.
  • No simulation study is definitive, so researchers should interpret results cautiously and use the tutorial’s guidance to improve planning, execution, understanding, and reporting.

APPENDIX A: REVIEW: FURTHER INFORMATION

The review found high agreement between reviewers and substantial variation in data-generating mechanisms, while most studies evaluated few estimands and methods.

  • Reviewer agreement: Reviewers agreed on 132 of 140 answers (94%) across 14 key variables.Agreement was measured between TPM and each of the other authors; IRW and MJC did not review any of the same articles.
  • Data-generating mechanisms: Most simulation studies used under 20 data-generating mechanisms, although the largest number was ∼41 billion.Studies generally varied few factors, and fully factorial designs were the most common approach among studies varying more than one factor.
  • Estimands and methods: Simulation studies generally evaluated few estimands and methods, with a single estimand and four methods being the most common counts.These patterns suggest that many studies provide proof-of-concept or address new problems for which few methods are available.
Loading 1712.03198v3…