SafeLLM4SE: Statistical Evaluation and Reporting for LLM-based Software Engineering Systems

1 Introduction

• Repeated invocations can produce different software engineering outcomes, and LLM non-determinism can affect correctness and scientific conclusions.Agentic systems compound randomness across multi-step trajectories, yet are commonly evaluated with one run per task; model services may also change over time.Source: §1 · pp. 1–2
• Single outputs, averages, best-of-N, and pass@k capture different aspects of performance but may incompletely characterize stochastic behavior.Best-of-N and pass@k alone do not describe the distribution of performance across complete executions.Source: §1 · p. 2
• SafeLLM4SE treats outputs as stochastic observations and distinguishes expected quality, execution-to-execution stability, and estimation uncertainty.Its principles recommend confidence intervals, design-appropriate tests, effect sizes, and adaptive sampling that stops at a precision target or budget limit.Source: §1 · pp. 2–3
• SafeLLM4SE proposes a practical methodology and minimum reporting standard for statistically principled evaluation of stochastic LLM-based software engineering systems.It integrates established statistical techniques, offers default recommendations through an open-source application, and allows justified alternatives.Source: §1 · pp. 2–3
• In the hypothetical HumanEval example, C’s average success rate is 0.36 at N=10 and 0.54 at N=100, while a single-output evaluation ranks A best.At N=100, C’s SD is 0.344 versus B’s 0.175, and C’s 95% CI is wider despite its higher average success rate than B’s.Source: §1 · p. 2, Table 1 · p. 3
• The authors demonstrate SafeLLM4SE using its Python implementation to compare two LLMs on HumanEval.The article proceeds from related work to methodology, a HumanEval case study, and conclusions.Source: §1 · pp. 3–4

2 Related Work

• Prior LLM evaluation work addresses non-determinism and standardized multi-metric assessment, but HELM does not prescribe sampling, aggregation, or statistical comparison of repeated generations.HumanEval’s pass@k summarizes the chance of at least one correct solution among k samples, not execution-level variability or estimation uncertainty.Source: §2 · p. 4
• SafeLLM4SE adapts variance-aware benchmark evaluation and run-count planning to LLM-based SE, where API usage and token consumption dominate evaluation costs.The cited work includes variance-aware procedures [3], power analysis for run counts [7], and interval estimates with robust aggregates [1].Source: §2 · p. 4
• Compared with statistical guidelines for randomized SE algorithms [2] and NLP evaluation protocols [8], SafeLLM4SE adds CI-driven adaptive sampling and SE-specific reporting.Its scope includes artifacts such as functional correctness and costs such as token budgets.Source: §2 · p. 4
• SafeLLM4SE complements general reproducible-ML checklists with an SE-specific methodology and an open-source tool intended to lower adoption barriers.The methodology integrates adaptive sampling, distribution-aware comparison, and a minimum reporting standard.Source: §2 · p. 4

3 SafeLLM4SE

• SafeLLM4SE models each output as a stochastic-process realization and specifies statistical evidence for reliable, reproducible, and comparable experimental conclusions.For prompt, model, and configuration conditions, outputs are mapped through a task-appropriate metric to a performance distribution whose properties are estimated from repeated observations.Source: §3 · pp. 4–5, §3.1 · p. 5
• The framework treats outputs as realizations of stochastic processes rather than deterministic artifacts, grounding its evaluation and reporting protocol in repeated observations.The workflow begins with a metric-defined performance variable and targets a property of its distribution.Source: §3 · pp. 4–5, §3.1 · p. 5

3.2 Adaptive Sampling

• Adaptive sampling continues until the confidence-interval width reaches a target or the token budget is exhausted, avoiding arbitrary fixed execution counts.The target width adapts sample size to observed variability; the procedure uses metric-appropriate intervals, including t-based or bootstrap intervals for continuous metrics and Wilson intervals for binary metrics.Source: §3.2 · pp. 5–6

3.3 Statistical Characterization

• Evaluations characterize quality, stability, and estimation uncertainty as complementary dimensions of system performance.Recommended indicators include central tendency or success probability for quality, dispersion for stability, and confidence intervals—95% by default—for estimation precision.Source: §3.3 · pp. 6–7

3.4 Reporting And Visualization

• The reporting standard requires estimates and confidence intervals for primary indicators, alongside experimental details needed to assess reproducibility and practical reliability.It calls for model identifiers or versions, execution dates and times, available snapshot information, sampling procedures, and, where possible, visualizations of observed outcomes.Source: §3.4 · p. 7

3.5 Statistical Comparison

• Statistical comparisons use procedures suited to independent or paired samples and report uncertainty and effect magnitude alongside test results.For independent samples, the default combines Mann–Whitney U, Cliff’s δ, and a bootstrap interval for the difference; paired data use a design-appropriate test, paired effect size, and paired bootstrap interval.Source: §3.5 · pp. 8–9

3.6 Decision Making

• SafeLLM4SE combines a statistical methodology with a minimum reporting standard to support reproducibility, comparability, and evidence-based decisions.It does not prescribe a universal ranking rule: superiority judgments should consider quality, uncertainty, statistical evidence, practical relevance, and stability together.Source: §3.6 · pp. 10–11

4.1 Experimental Setup

• At temperature 2.0, Gemini and Qwen-Coder each generated solutions for all 164 HumanEval problems per complete run.The relatively high temperature was chosen to amplify stochastic variation.Source: §4 · p. 11, §4.1 · p. 11
• Sampling began with 30 complete runs and stopped when the 95% bootstrap CI width for mean success was at most 0.01, with no execution cap.Each bootstrap sample resampled complete executions with replacement.Source: §4.1 · pp. 11–12
• Each run’s score is the proportion of problems passing all tests; the mean estimates expected benchmark success, while SD summarizes run-to-run stability.Source: §4.1 · p. 11

4.2 SafeLLM4SE Reporting

• Gemini averaged 0.9335 success versus 0.8238 for Qwen-Coder, with observed SDs 0.0110 vs. 0.0204; Figure 1 shows higher, tighter repeated-run scores.Each plotted observation is one complete execution of all 164 HumanEval problems.Source: §4.2 · p. 12, Table 4 · p. 12, Figure 1 · p. 13

4.3 Statistical Comparison

• The comparison treats execution-level success rates as independent and uses Mann–Whitney U, Cliff’s δ, and a bootstrap CI for the mean difference.Each execution is an unlinked realization of its model’s stochastic process.Source: §4.3 · p. 13

4.4 Decision Making

• 10.97 percentage points higher mean benchmark success rate for Gemini than Qwen-Coder, with a 95% CI entirely above zero; Cliff’s δ=1, lower variability, and fewer tokens.Every observed Gemini benchmark score exceeded every observed Qwen-Coder score.Source: §4.4 · pp. 13–14, Table 5 · p. 14
• The case study used the SafeLLM4SE application to generate the data and Figure 1; the software is freely available on PyPI.Source: §4.4 · p. 14

• Trustworthy evaluation treats LLM outputs as stochastic observations and characterizes quality, stability, and estimation uncertainty rather than relying on one generation or point estimate.Source: §5 · p. 14
• Adaptive sampling links execution counts to a CI-width target, while comparisons consider uncertainty and effect magnitude alongside statistical significance.Fixed sample sizes can provide insufficient evidence or incur unnecessary cost.Source: §5 · p. 14
• SafeLLM4SE provides a practical basis for more reproducible, comparable, and accountable evaluations; its software and case-study data are freely available.Source: §5 · p. 14