Source-linked AI summary
You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements
Xuenan Zhang, Yuqing Yang, Giancarlo Pellegrino
TL;DR
Web security measurements must sample large domain populations because exhaustive analysis is too costly, yet evidence about sampling bias and generalizability has been limited. This paper combines a literature review with large-scale measurements of Tranco and Common Crawl to compare sampling strategies and datasets. It finds that probability-based methods are generally stable, while Top N and hybrid approaches can remain biased, motivating adaptive probability sampling.
Problem
Evidence is limited on whether common sampling strategies systematically bias web security measurements and affect their generalizability and comparability.
Method
The study reviews prior work and evaluates eight sampling strategies using 500k Tranco domains and 24.8M Common Crawl hosts across security issues and measurement objectives.
Results
Probability-based methods provide stable estimates for many objectives, whereas Top N can remain biased and Random sampling from Tranco can differ from Common Crawl prevalence by up to 19.6 percentage points.
Takeaways & Limitations
Sampling strategy and population should be selected together, with adaptive probability-based sampling offering practical guidance when issue prevalence is unknown.
Takeaways & Limitations
The findings cannot be claimed to generalize fully across all security properties, datasets, or domains, and Common Crawl analysis is limited to static security issues.
Abstract
from arXiv · showhide
Web measurement studies rely on domain datasets such as Tranco to quantify the prevalence and impact of security issues at scale, but exhaustively analyzing these datasets is often infeasible because of the cost of advanced analysis techniques, requiring the use of sampling. Despite its widespread use, sampling remains largely guided by convention---most commonly \emph{Top $N$} domain selection---rather than evidence, and its influence on the validity and generalizability of security findings has received little systematic evaluation. Consequently, it remains unclear whether common sampling strategies introduce systematic bias, distort observed vulnerability rates, or limit comparability across studies. In this work, we undertake, to the best of our knowledge, the first comprehensive investigation into how sampling methodologies affect the measurements and the conclusions. Through a comprehensive literature review and large-scale measurements of 500k Tranco and 24.8M Common Crawl hosts, we perform a comparative evaluation of datasets and sampling strategies. We show that, while Top $N$ sampling may be a rational strategy, the researchers have to bear in mind that Top $N$ does not reflect the overall distribution of the web. Instead, probability-based strategies yield stable, unbiased estimates for prevalence and many impact objectives. Hybrid sampling provides no advantages over pure probability sampling, as its deterministic prefix consistently contributes negatively to accuracy. Building on these results, we provide data-backed guidance for future studies, proposing to use an adaptive probability-based sampling strategy that remains effective even when the prevalence of the target issue is unknown.
1 Introduction
Web security measurements require sampling because exhaustive analysis is often infeasible, but dataset and sampling choices can limit how findings generalize. This study evaluates these choices and finds that probability-based sampling is generally more representative than deterministic Top N selection.
- Motivation: Sampling reduces the cost of analyzing large web datasets, whose security measurements may require expensive crawling, dynamic taint tracking, SMT solving, or static analysis.Common seed sources include Tranco, CrUX, and Common Crawl.
- Measurement Design: Dataset choice should match the objective: popularity-ranked Tranco suits impact studies, whereas broad Common Crawl coverage better supports prevalence analyses.
- Main Findings: Top N can deviate from the web’s overall distribution by up to 10.6 percentage points, whereas Random, Systematic, and Stratified sampling remain close to the true value on the 500k-domain Tranco dataset.Increasing the Top N sample size does not remove this bias, and hybrid methods inherit the deterministic prefix’s bias.
- Main Findings: Random sampling from Tranco can remain up to 19.6 percentage points from the Common Crawl prevalence estimate, showing that sampling correctly from the wrong population still produces a different answer.The discrepancy persists as sample size increases, with a systematic average underestimation rate of 30%.
- Study Scope: The paper provides the first systematic empirical evaluation of how sampling strategies affect web security measurements across datasets, issues, objectives, and metrics.The evaluation combines a literature review with measurements of 500k Tranco domains and 24.8M Common Crawl hosts.
3 Problem Statement
The paper frames sampling as a source of uncertainty in web security measurement and evaluates strategies, datasets, and practical choices under resource constraints. Its study combines a literature review with large-scale crawling and analysis of Tranco and Common Crawl.
- Research Goal: The study asks how sampling choices shape measurement results and the scientific conclusions drawn from them.
- Research Questions: The evaluation compares deterministic Top N, probability-based Random, Systematic, and Stratified methods, plus hybrid designs, across accuracy, stability, convergence, and structural assumptions.The study also examines how hybrid methods decompose into deterministic and probabilistic components.
- Datasets and Objectives: The study compares impact and prevalence estimates obtained by applying the same sampling strategy to popularity-ranked Tranco and unranked Common Crawl.Tranco represents popular domains, while Common Crawl is used to approximate broader web coverage.
- Literature Review: The literature review analyzes 107 papers and identifies Tranco as the most popular domain list and Common Crawl as the most popular non-popularity-based list.
- Data Collection: The experiments crawl Tranco domains live and analyze static Common Crawl artifacts, using depth two and at most 250 pages per domain for the Tranco crawl.Common Crawl supports header-based prevalence analysis, while runtime analyses such as XSS and TLS measurements use Tranco.
6 Assessing Impact
Impact estimation separates Top N from probability-based sampling: Top N remains structurally biased, while probability methods stay accurate and stable across full and Top K baselines. Hybrid accuracy comes from the probabilistic tail, whereas the deterministic prefix contributes negatively.
- Full-list and Top K impact estimation: Top N remains biased across full and Top K baselines, while probability-based strategies stay centered near the true impact and become more accurate as the baseline grows.Top N errors depend on ranking-prefix structure; probability-based variation reflects sampling noise rather than persistent estimator bias.
- Full-list and Top K impact estimation: Systematic sampling provides the strongest overall performance on the full 500k baseline, with probabilistic RMS below 0.3 versus 2.6 for Top N.Random, Systematic, and Stratified estimates oscillate around the true impact, while Top N reaches nearly ±9 percentage points at small sampling ratios.
- Hybrid sampling: Hybrid estimators inherit deterministic-prefix bias, so increasing the probabilistic tail primarily spends sample budget correcting Top N rather than improving over pure probability sampling.The bias persists longer for Top 5K than Top 10K prefixes, although sufficiently large tails can pull Top 10K hybrids close to the true impact.
- Hybrid sampling: Tail-sampler choice matters less than prefix size: Random, Stratified, and Bucket differ by less than 0.3% in peak errors for a fixed prefix, while prefixes differ by up to 100%.The dominant factor is the deterministic prefix rather than the particular probabilistic tail sampler.
- Contribution analysis via Shapley values: Shapley analysis assigns strictly negative contributions to Top N blocks, while probabilistic tails become beneficial once their size is large enough.Stratified and Bucket tails cross into positive contribution slightly earlier than Random, consistent with their lower variance.
- Impact across vulnerability classes: Probability sampling remains well-behaved across vulnerability classes, but rarer issues require larger samples to stabilize than common issues.The qualitative estimator behavior remains consistent; prevalence changes the amount of data needed for convergence.
7 Prevalence
Prevalence estimation depends on both the population and the sampling strategy: Tranco and Common Crawl have different vulnerability profiles, and random sampling is stable within either dataset but cannot correct a mismatched baseline. Using Tranco against Common Crawl can produce persistent underestimation.
- Scope of prevalence analysis: Client-side XSS and TLS misconfigurations are excluded from the Common Crawl prevalence analysis because evaluating them would require infeasible dynamic instrumentation or TLS handshakes at scale.The remaining issue classes still reveal substantial differences between the datasets.
- Differences Between Tranco and Common Crawl: Common Crawl and Tranco differ substantially in vulnerability prevalence, with Common Crawl higher for several issues and Tranco higher for CORS and Injection.The mismatch changes direction across issue classes, so Tranco is not representative of broader-web prevalence.
- Random sampling for prevalence estimation: Random sampling is stable and unbiased within each dataset, but sampling Tranco against the Common Crawl baseline produces a persistent -16.5% deviation and 19.6% peak absolute deviation.The gap does not close as sample size increases.
- Random sampling for prevalence estimation: The prevalence-estimate ratio converges to 0.70×, indicating 30% underestimation when Tranco is used instead of Common Crawl despite sufficiently large random samples.The result isolates dataset mismatch as a source of inherited deviation rather than random-sampling instability.
- Random sampling for prevalence estimation: Optimizing sampling strategy or sample size cannot mitigate the deviation inherited from choosing an inappropriate prevalence dataset.Dataset selection therefore remains essential even when random sampling is internally consistent.
8 Practical Guidance for Sampling
Sampling guidance should match the measurement objective and dataset, while adaptive probability sampling determines sample size from observed precision rather than fixing it in advance. In the case study, it achieved accurate estimates for common issues with modest samples and expanded substantially for sparse outcomes.
- 8.1 Adaptive Probability Sampling: Adaptive probability sampling begins with a 1–2% pilot, doubles the cumulative sample, and stops when estimate stability and confidence-interval conditions are met.It can continue to the complete dataset when the observed issue is too sparse for the target precision.
- 8.2 Case Study: At least 98% of stopped estimates were within 5% relative error for common and moderately common issues, with median stopping fractions of 4%–16%.The strategy used 1,000 retrospective trials and summarized typical, expected, and worst-case stopping costs.
- 8.2 Case Study: For sparse CORS, fixed 2% and 4% Top 100K samples were accurate in only 10.4% and 21.9% of runs, whereas adaptive sampling continued to the full population there.In Top 500K, adaptive sampling used 64% for the sparse outcome, making the precision cost explicit.
- 8.3 Recommendation: For impact beyond an exact Top N prefix, probability sampling from the declared population is safer than a deterministic prefix, with adaptive sampling available when resources are constrained.Top N remains appropriate when the research question is explicitly limited to that prefix.
- 8.3 Recommendation: For prevalence, researchers should use a broad dataset such as Common Crawl and probability sampling, while separating prevalence analyses from Tranco-based impact analyses.The recommendation reflects the different populations represented by ranked lists and broad web archives.
9 Discussion
The discussion concludes that sampling strategy and dataset choice jointly determine measurement reliability: probability methods are robust, while Top N and hybrid prefixes can retain bias. The authors also delimit generalizability through dataset, crawling, and analysis-scope constraints.
- 9.1 Lessons Learned: Top N introduces persistent bias when estimating properties beyond its declared prefix, and increasing the sample size does not remove that error.It remains natural when the target is exactly the specified prefix.
- 9.1 Lessons Learned: Probability-based methods had RMS error below 0.3 percentage points on the Tranco Top 500K baseline, versus 2.62 percentage points for Top N.Their estimates remained centered around the baseline and improved predictably as sample size increased.
- 9.1 Lessons Learned: Hybrid strategies inherit bias from their deterministic Top N prefix and therefore provide no inherent advantage over probability sampling.Their performance depends largely on how harmful that prefix is.
- 9.1 Lessons Learned: Using Tranco for prevalence can deviate from the Common Crawl baseline by up to 19.6 percentage points, even when sampling is statistically well-behaved for Tranco.Larger samples reduce variance but do not eliminate error caused by the misaligned population.
- 9.2 Limitations: The study cannot claim full generalizability across all security properties, datasets, or domains because its ground truth, crawling setup, and fixed depth may omit relevant cases.Network locality, instability, bot detection, DNS, TLS behavior, and authenticated or deep content may affect observations.
- 9.2 Limitations: Common Crawl analysis covers static security issues, while dynamic measurements require live data and remain a direction for future sampling research.Selecting domains from Common Crawl and then analyzing them live is described as an existing practice.
10 Related Work
Related work has examined representativeness, measurement configurations, crawling, and reproducibility, but this paper centers the comparatively underexplored role of sampling strategies. Its systematic review and measurements compare eight strategies across large web populations.
- Sample representativeness in web and security studies: Prior studies have linked sampling methods to biased distributions of policy violations and proposed URLSamp, which clusters pages by URL patterns.This work extends representativeness concerns to web security measurement sampling.
- Web measurement experiment setups: Earlier work shows that tools, browser configurations, and crawler choices can change web measurement results.These studies examine pipeline components other than domain sampling.
- Reproducibility of measurement studies: Reproducibility research reports that many studies fail to reproduce and that small setup differences can yield different conclusions.This motivates examining sampling as another potentially consequential design choice.
- Contribution: This paper reviews 107 papers using eight sampling strategies and compares them through large-scale ground-truth measurements on Tranco and Common Crawl.The comparison finds persistent Top N bias, stable probability-based estimates, and no hybrid advantage.
A Ethical Considerations
The study addresses ethical risks through non-intrusive testing, infrastructure safeguards, restricted vulnerability disclosure, and aggregated reporting. It also frames the released methodological insights as supporting research rigor rather than enabling exploitation.
- The researchers reduced infrastructure load by distributing domains across shuffled buckets and rate-limiting DNS queries.
- The methodology avoided server-side payload injection, fuzzing, and exploitation, relying instead on passive analysis and controlled JavaScript execution.
- Because approximately 100k domains exhibited security issues, the study shared only aggregated, non-identifying results and reported confirmed findings to official contacts.
- The authors characterize the residual risk of publishing sampling insights as negligible because the findings concern methodological rigor rather than exploit development.
- The released materials include tools, reviewed literature, case-study data and code, an adaptive-sampling example, and nonidentifying measurement results.
D Breakdown of Error Types at Collection Time
Collection-time failures affected roughly 20% of the Tranco 500k domains, while error rates varied across rank buckets and valid-domain shares remained between 72% and 96%. The observed nonuniform issue distribution supports using the dataset to evaluate sampling behavior.
- Roughly 20% of Tranco 500k domains failed to return a crawling result, with network issues comprising six of the ten most frequent error types.
- Across 50 rank-ordered buckets, valid-domain shares remained between 72% and 96% despite varying error rates.
- The nonuniform distribution of domains with security issues indicates that the dataset retains real-world variation relevant to sampling-strategy evaluation.
- The appendix models hybrid sampling with two components: a deterministic Top N prefix and a randomized rest-sampling mechanism.
- Using negated expected error as coalition value, positive Shapley values denote error reduction while negative values denote increased estimation error.
Experimental Procedure.
The experimental procedure defines hybrid components, evaluates their error contributions across rest-sampling methods and sample sizes, and specifies systematic and stratified implementations. Systematic sampling uses a random start and fractional interval, while stratified sampling allocates observations proportionally across rank-based strata.
- The analysis fixes a Top N component, varies rest-sample sizes, and draws the remainder using Random, Bucket, or Stratified sampling.
- A positive component value indicates reduced estimation error, whereas a negative value indicates that the component increases error through bias or distortion.
- Systematic sampling: Systematic sampling selects elements at a fractional interval k = N/n from a random starting position in an ordered population.
- Systematic sampling: The implementation uses floor to obtain integer indices, modulo to prevent overflow, and a uniformly distributed starting position within the first interval.
- Stratified random sampling: Stratified sampling partitions the population by relevant characteristics and draws samples proportionally to stratum sizes before combining them.
Stratified random sampling.
The adaptive strategy uses nested, uniformly sampled prefixes that expand cumulatively, estimates affected-unit prevalence at each stage, and quantifies uncertainty with a finite-population correction. It continues sampling when no affected units appear.
- The strategy draws nested prefixes from one random permutation without replacement, expanding cumulative sampling fractions from 2% through 100% when necessary.
- At each stage, the procedure estimates the affected-unit rate from cumulative inspected and affected counts.
- When no affected units are observed, the procedure doubles the cumulative sample instead of stopping because prevalence remains insufficiently informed.
- After at least one affected unit is found, the method computes a 95% proportion-confidence half-width to quantify uncertainty.
- The finite-population correction accounts for sampling without replacement, reducing uncertainty as coverage increases and reaching zero at complete-population measurement.
1 Draw one random permutation of F
The adaptive strategy checks both stability across successive estimates and statistical precision before stopping. It uses a user-specified tolerance whose choice trades measurement cost against estimate precision.
- Step 3: Check stability and precision: D_k measures relative change between consecutive estimates, testing whether doubling the sample materially changes the result.
- Step 3: Check stability and precision: R_k measures the relative half-width of the current 95% confidence interval, testing whether the estimate has sufficient statistical precision.
- Step 3: Check stability and precision: The two conditions are complementary because similar consecutive estimates can still be imprecise, while confidence intervals alone do not verify stability.
- Step 4: Stop or expand the sample: The strategy stops only when consecutive estimates change by at most τ and the current 95% confidence-interval half-width is at most τ relative to the estimate.If either condition fails, the cumulative sample size is doubled and the next stage is evaluated.
- Choosing the tolerance: τ is user-specified: smaller values require larger samples for greater precision, whereas larger values permit earlier stopping with less precision.The main evaluation uses τ = 0.05; a looser example is τ = 0.10.
G.2 Adaptive Sampling Strategy: Case Study
The case study evaluates whether adaptive sampling can stop reliably without knowing the population-wide affected-unit rate. Across populations and issue frequencies, common outcomes stop with small samples, while sparse outcomes require substantially more data.
- G.2 Adaptive Sampling Strategy: Case Study: The stopping function uses only observations available at each stage, and the full-population rate is used only afterward to calculate retrospective relative error.Retrospective reliability is the fraction of 1,000 trials with relative error E ≤ 0.05.
- Top 100K results: For Tranco Top 100K, Cookie security reaches 99.0% reliability with a mean sample of 8.14%, while Clickjacking reaches 98.3% reliability with a mean sample of 16.37%.Cookie security stops at 8% in 98.3% of trials, whereas Clickjacking stops at 16% in 97.7%.
- Top 500K results: For Tranco Top 500K, Cookie security stops at 4% in every trial with 99.9% reliability, whereas sparse CORS stops at 64% in 97.2% of trials.The mean sampling fractions are 4.00% for Cookie security and 65.01% for CORS.
- Common Crawl results: On Common Crawl, Cookie security and Clickjacking stop at 4% in every trial, while CORS stops at 4% in 99.4% of trials and achieves reliabilities of 100.0%, 100.0%, and 99.0%, respectively.