PaperlayerView exact revision on arXiv

Source-linked AI summary

Practice Makes (Im)Perfect: A Look Back at Benchmarking Practices for Microarchitectural Side-Channel Attacks

Iliana Fayolle, Antoine Geimer, Daniel De Almeida Braga, Clémentine Maurice

arXiv:2609.03893v2cs.CR

TL;DR

Meaningful comparison of microarchitectural side-channel attacks is difficult because implementations and experimental conditions vary, while reported error rates may use different definitions. The authors survey 83 papers, identify recurring benchmarking flaws, and report that papers in the corpus average 5.5 flaws. They propose evaluation properties and a checklist to support more meaningful benchmarking.

  • Problem

    How to meaningfully evaluate and compare microarchitectural side-channel attacks remains unresolved, with implementation and experimental differences and inconsistent error-rate definitions complicating comparisons.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The authors survey 83 papers from 2014–2024 in leading security and architecture conferences, classify evaluation practices, and develop a taxonomy and benchmarking checklist.

    Source: §1 · p. 2, §3 · p. 2, §3.1 · p. 3

  • Results

    5.5 full or partial benchmarking flaws per paper on average; 77 papers (92.8%) had at least one full flaw and 76 (91.6%) had at least one partial flaw.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The authors identify properties relevant to evaluating new attacks and provide a checklist intended to yield meaningful insights into primitive capabilities.

    Source: §1 · p. 2

    The flaw list is not claimed to be complete or sound, and the selected top-conference corpus excludes transient-execution attacks and implementation-specific constant-time violations.

    Source: §3.3 · p. 3

Abstract

from arXiv · show

Microarchitectural side-channel research has grown at an exceptional pace in recent years, increasing the need for rigorous and meaningful benchmarking. Early attack papers typically relied on indirect proxies, such as covert-channel bandwidth or key-recovery on naive AES and RSA implementations, setting de facto standards that many subsequent works continued to replicate, sometimes by directly comparing against raw numbers from prior work. While these practices offer convenient points of comparison, current benchmarks may not be the most relevant to assess specific properties of new primitives. Even more problematic, microarchitectural attacks are notoriously sensitive to experimental conditions: minimal changes in the target system can significantly alter outcomes and performance. As a result, inadequate evaluation practices undermine reproducibility and cast doubt on the relevance of comparisons, even in top-tier venues where such issues should be identified. This paper tackles the core problem of proper benchmarking for microarchitectural side-channel attacks and examines its broader impact on research quality in the field. We survey 83 attack papers published in top-ranked security and architecture conferences from 2014 to 2024. From this corpus, we identify and define 19 recurrent benchmarking flaws that affect evaluation completeness, relevance, soundness, and reproducibility. These flaws include unfair or absent comparisons, missing code or materials, and the failure to evaluate the key attack properties. On average, each paper exhibits 5.5 such flaws, highlighting how widespread the issue is, even in highly selective venues. Based on our findings, we identify and suggest key properties that are relevant to properly evaluate new attacks. We also highlight trends over time and different practices between security and architecture conferences.

1 Introduction

The paper asks how to meaningfully evaluate and compare microarchitectural side-channel attacks, whose novelty can involve new components, improved techniques, or new architectures. It surveys benchmarking practices and proposes a taxonomy and checklist to support meaningful evaluation.

1 Introduction

  • Historical benchmarks often reuse classical attacks or covert-channel bit rate, although patched vulnerabilities may be poor proxies for modern implementations.The exploited vulnerability may serve only as a proxy for demonstrating sufficient side-channel resolution.Source: §1 · p. 1
  • Published results are difficult to compare directly because implementations and experimental conditions differ, and side channels are sensitive to noise.A controlled lab result is not directly comparable to a result from a more realistic setting such as a cloud environment.Source: §1 · pp. 1–2
  • Reproduction is hindered by unavailable or nonworking code, omitted engineering details, and processor-generation changes that can require attack adaptation.Reimplementing previously published side channels can also impose substantial engineering effort.Source: §1 · p. 2
  • The paper surveys benchmarking practices in 83 microarchitectural side-channel papers published across eight leading security and architecture conferences from 2014 to 2024, and introduces a taxonomy and checklist for evaluating attack capabilities.The authors say the checklist could help establish a unified benchmark for rigorous evaluation.Source: §1 · p. 2

2 Motivation

The section argues that raw benchmark comparisons can obscure differences in implementations, protocols, and experimental settings. Error-rate definitions also vary, making a reported percentage difficult to interpret across papers.

2 Motivation

  • The authors frame their examples as representative recurring practices, not judgments about the merit or quality of individual cited papers.They aim to highlight practices that reduce the value of empirical evidence.Source: §2 · p. 2
  • Raw bandwidth comparisons can be misleading when channels use different protocols, settings, or levels of error correction, or when newer results are compared with older unoptimized work.The passage contrasts a cloud-oriented channel with error correction and no observable error in a realistic setting against controlled evaluations that may prioritize speed.Source: §2 · p. 2
  • “Error rate” may refer to Edit distance, Hamming distance, accuracy, or an unspecified computation, so the same percentage can represent different transmission failures.The passage says fair comparison requires explicitly defining the metric and any error-correcting code.Source: §2 · p. 2

3 Methodology

The authors use a structured survey and codebook to assess benchmarking in papers within a defined scope. They describe their annotation and selection procedures and acknowledge methodological and corpus limitations.

3.1 Survey methodology

  • A structured survey and generic codebook capture recurring evaluation features while providing a shared vocabulary rather than imposing a fixed checklist.The survey covers primitive type, evaluation goals, targets and platforms, metrics, comparisons, and artifact availability.Source: §3.1 · p. 2
  • The authors assess flaws against each paper’s claims, applying criteria conditionally so a platform-specific contribution is not penalized for lacking broader evaluation unless it claims generality.Four experienced researchers iteratively refined the survey and codebook; a second researcher reviewed each annotation, with disagreements discussed and ambiguous cases resolved conservatively.Source: §3.1 · pp. 2–3

3.2 Selection of papers to review

  • The review covers side-channel papers from 2014–2024 in four leading security and four architecture conferences, excluding work centered on specific software implementations or transient-execution attacks.The authors scraped proceedings, screened out papers that did not evaluate a side or covert channel, and refined 114 papers to a corpus of 83.Source: §3.2 · p. 3

3.3 Limitations

  • The authors acknowledge possible expert and self-reporting biases, incomplete or unsound flaw lists, and potential bias from venue selection and corpus imbalances.Their scope also excludes transient-execution attacks and papers whose novelty is exploiting constant-time violations in implementations.Source: §3.3 · p. 3
  • Similar attacks with similar characteristics and targets from one paper were merged into a single occurrence, which the authors say minimally affected the total attack count.This was necessary because some papers did not clearly distinguish the number of attacks used for evaluation.Source: §3.3 · p. 3

4 Taxonomy of flaws

The taxonomy defines evaluation properties and identifies flaws that may affect them. It distinguishes levels of impact, including cases where a criterion is unclear, incomplete, flawed, or not applicable.

4 Taxonomy of flaws

  • A good evaluation should be complete, relevant, sound, and reproducible, assessing key attack properties and enabling readers to reproduce experiments at least on the same device.Completeness means measuring the right things, not everything; soundness requires measuring intended properties rather than unintended side effects or only controlled settings.Source: §4 · p. 4
  • The authors adapt a prior list and define 19 flaws that may affect evaluation properties, summarizing the most impacted properties for each flaw in Table 1.The taxonomy is based on the authors’ experience and their systematic survey.Source: §4 · p. 4
  • The impact labels distinguish met requirements, underspecified information, flawed evaluations, and criteria that are not applicable to a paper.Underspecified information can be unclear or incomplete; a criterion may be not applicable when it does not fit the paper’s contribution.Source: §4 · p. 4

A. Benchmark omissions

Benchmarking omissions hide experimental dimensions that affect attack properties. Principled evaluation states what is measured, quantifies trade-offs, and enables future normalization.

A. Benchmark omissions

  • Microarchitectural evaluations cannot exhaustively cover every layer and parameter, but should state what they measure, quantify key trade-offs, and enable future normalization.Relevant dimensions span hardware, firmware or microcode, operating systems, and applications.Source: A. Benchmark omissions · p. 4
  • Error-rate reports should state the unit and number of trials, since vague claims such as “zero-error” can let throughput mask errors.Without error rates, readers cannot properly assess an attack’s practicality.Source: A. Benchmark omissions · p. 4
  • Spatial resolution identifies the smallest distinguishable unit, while temporal resolution covers time per sample and sustainable sampling rate; omitting either hinders comparison.The paper identifies these as flaws A2 and A3.Source: A. Benchmark omissions · p. 4
  • Noise-resilience testing under realistic events matters because error rates may rise sharply when another process runs, exposing attacks that work only in quiescent setups.The paper identifies this as flaw A4, affecting completeness and especially relevance.Source: A. Benchmark omissions · pp. 4–5
  • Claims about an attack that lack supporting evidence are unsubstantiated and should be disregarded, weakening evaluation completeness and relevance.The paper labels this flaw A5.Source: A. Benchmark omissions · p. 5
  • Restricting evaluation to one platform weakens reproducibility and completeness; even when portability is not the goal, cross-architecture behavior and needed adaptations help characterize the attack surface.The paper identifies single-platform evaluation as flaw A6.Source: A. Benchmark omissions · p. 5
  • Evaluations should vary impactful configurations to find suitable settings; a single configuration lacks completeness, and each distinct primitive should be tested across multiple configurations.The paper identifies this as flaw A7 and does not require every later evaluation of the same primitive to repeat the sweep.Source: A. Benchmark omissions · p. 5

B. Improper comparisons

Comparisons can mislead when they ignore differences in protocols, hardware, threat models, or measurement semantics. Simplified payloads and idealized setups can also weaken soundness and practical relevance.

B. Improper comparisons

  • Comparisons across incompatible protocols, hardware, threat models, or measurement semantics require reproducing the baseline or explicitly normalizing and qualifying differences.Small changes in target or setup can drastically alter observations.Source: B. Improper comparisons · p. 5
  • Raw-number comparisons across different setups can create a false impression of superiority, while no comparison leaves readers unable to position the contribution in the literature.The paper notes that first-of-its-kind attacks may not be comparable and can be labeled NA.Source: B. Improper comparisons · p. 5
  • Repetitive “easy” payloads may trigger extra microarchitectural effects, and extrapolating partial attack stages to end-to-end scenarios may yield incorrect conclusions.These practices affect soundness and may undermine practical relevance.Source: B. Improper comparisons · p. 5
  • Idealized systems or prerequisites inconsistent with the threat model can make an evaluation measure properties of the setup rather than the attack.The paper identifies this as flaw B4, affecting soundness and relevance.Source: B. Improper comparisons · p. 5

C. Missing information

Missing experimental and implementation details can prevent readers from reconstructing evaluations and comparing results. The cited flaws span system versions, artifacts, measurements, protocols, and attack prerequisites.

C. Missing information

  • Omitting operating-system, kernel, software, or CPU/GPU models harms reproducibility because updates and microarchitectural differences can change attack behavior.The paper labels these omissions C1 and C2; hardware details also affect completeness and soundness.Source: C. Missing information · p. 5
  • Available attack code needs documentation of implementation details and adaptations so readers can reproduce experiments and rerun baselines under the same conditions.The paper identifies code availability and documentation as C3 and links good artifacts to improved comparisons.Source: C. Missing information · p. 5
  • Reporting the number of samples required to recover an element or complete the attack indicates signal-to-noise ratio when a single measurement is unreliable.The paper identifies missing sample counts as C4.Source: C. Missing information · p. 5
  • Classification evaluations need model, feature, and training-set details, while covert channels need protocol descriptions covering synchronization, framing, clocking, and shared knowledge.These omissions are C5 and C6; channel protocol details also clarify the attack model for comparison.Source: C. Missing information · pp. 5–6
  • An error-rate report should explain how the metric was computed, since accuracy, error correction, and other methods can yield different reported rates.The paper identifies this as C7, affecting soundness and reproducibility.Source: C. Missing information · p. 6
  • When a paper introduces multiple primitives or variants with different prerequisites, failing to state those differences can obscure the threat model and evaluation requirements.The paper identifies this as C8, affecting completeness and reproducibility.Source: C. Missing information · p. 6

5 Survey results

Across 83 papers, benchmarking flaws were widespread, and the survey examined attack types, evaluation practices, and recurring weaknesses in benchmark omissions, comparisons, and reporting. The results show that many evaluations lacked key properties or information needed to assess and reproduce attacks.

5 Survey results

  • 77 papers (92.8%) had at least one full flaw and 76 (91.6%) at least one partial flaw; the average was 5.5 flaws per paper.Figure 3 shows the distribution of flaws per paper; the median was 3 full flaws and 2 partial flaws.Source: §5 · p. 6, Figure 3 · p. 6

5.1 General observations

  • The corpus spans 218 attacks, and covert-channel evaluations commonly report raw speed and error rate.Control-flow cryptographic attacks appeared in 27 papers (32.5%), fingerprinting and memory-access cryptographic attacks in 17 each (20.5%); 73 papers (88%) evaluated CPU attacks. Victim patterns resembled and Lou et al. [53].Source: §5.1 · pp. 6–7

5.2 Low-prevalence flaws

  • Partial flaws were more common than full flaws; recurrent gaps involved mitigation checks, CPU and classifier details, covert-channel protocols, and attack prerequisites.Among 15 papers with the mitigation-bypass flaw, only 1 (6.7%) evaluated only a subset of claimed bypasses; 19 of 53 (35.8%) reported protocol details only partially.Source: §5.2 · p. 7

5.3 A. Benchmark omissions

  • Benchmark omissions included missing error rates, spatial or temporal resolution, and quantitative noise-resilience evaluations.9 of 79 papers missed spatial and temporal resolution fully or partially; 37% fully omitted noise-resilience evaluation and 25.9% did so partially. Temporal resolution was omitted or reduced to cycles or bandwidth in 31.7%, while 14.6% reported it incompletely or unclearly.Source: §5.3 · pp. 7–8
  • 20.7% of papers evaluated attacks on a single platform, and 31.7% did not test all primitives across multiple configurations.Some papers checked applicability on other platforms without full evaluation or used multiple CPUs from one vendor and generation; single-configuration tests also limited evidence about performance under different settings.Source: §5.3 · p. 8

5.4 B. Improper comparisons

  • 67.3% of papers made unfair comparisons, 35.8% omitted comparisons to prior work, and 33.3% compared only partially.For 2 papers (3.8%), comparison fairness could not be determined because the reporting was unclear or incomplete.Source: §5.4 · pp. 8–9
  • 16.9% benchmarked only highly controlled or simplified systems, and 16.9% had evaluations designed to highlight performance.Examples included fine-tuned settings, unlikely privileges, specially instrumented targets, and attacks demonstrated only partially or on toy cases rather than in realistic environments.Source: §5.4 · p. 9

5.5 C. Missing information

  • Missing artifacts affected 49.4% of papers; target/version, error-processing, and side-channel sample-count information also often lacked detail.24.1% had only part of an artifact available, and 42.8% did not fully report sample counts; absent or incomplete artifacts could prevent full reproduction.Source: §5.5 · pp. 9–10

6 Discussion

The discussion reviews benchmarking changes over time and differences between security and architecture venues, while cautioning that the corpus is imbalanced. It reports improving artifact availability alongside persistent gaps and distinct community evaluation practices.

6 Discussion

  • The community comparisons require caution because the corpus contains more security papers and more recent publications.Few architecture papers predate 2020, so the analysis primarily focuses on 2020–2024.Source: §6 · p. 10

6.1 Benchmarking practices evolution over time

  • Code and documentation availability improved as artifact evaluation spread, but many papers remain partially flawed and venue practices vary.Figure 6 shows the availability trend; the authors say the problem is not fully resolved and AE practices differ by venue.Source: Figure 6 · p. 10, §6.1 · p. 10
  • After 2020, papers evaluated more platforms and used more primitive benchmarks, while the number of benchmarked configurations varied considerably.The increase in primitive benchmarks was driven primarily by side-channel-attack benchmarks; covert-channel interest declined slightly.Source: §6.1 · p. 10

6.2 Communities’ practices

  • Security papers had fewer flaws overall and appeared to improve after 2020, whereas architecture papers appeared to have more flaws without the same improvement trend.The authors note the architecture trend may reflect the smaller number of papers in that group.Source: §6.2 · p. 10
  • Architecture papers more often evaluated covert channels—78.6% versus 56.4% of security papers—while security papers evaluated more platforms.Both communities commonly used speed and error rate for covert-channel assessment; architecture papers more often introduced specialized metrics.Source: §6.2 · p. 10, Figure 7 · p. 11
  • Security papers shared code more often, sometimes incompletely, while more than half of architecture papers did not publish code even after artifact evaluation began.The discussion also notes an increase in benchmarks used for evaluation, mostly attributable to security papers and to a lesser extent architecture papers, with the latter trend qualified by the small corpus.Source: §6.2 · p. 11

7 Recommendations

The authors recommend more complete, relevant, and reproducible benchmarking, with distinct roles for authors, reviewers, and the research community. Their guidance emphasizes measuring primitive properties, documenting experiments, and making comparisons fair and realistic.

7.1 To authors

  • Authors should share code and materials, explain error-rate calculations and correction methods, and assess error rates and noise resilience.If sharing code would be unethical, authors should state that explicitly.Source: §7.1 · p. 11
  • Authors should measure spatial and temporal resolution rather than rely on cryptographic attacks or applications as proxies for a primitive’s capabilities.They should explain applicability using those metrics and provide alternatives when resolution does not apply.Source: §7.1 · p. 11
  • Authors should compare prior methods under the same or similar conditions, use common benchmark protocols where possible, and disclose comparison limits when reimplementation is infeasible.They also recommend realistic evaluation settings, including relevant configurations, noise levels, and attacker privileges.Source: §7.1 · p. 11
  • Authors should test primitives across platforms, configurations, noise levels, and access privileges, and explain where they perform well or struggle.The recommendations frame these evaluations as ways to assess generalization, robustness, and applicability.Source: §7.1 · pp. 11–12
  • Authors should explicitly report target systems and versions, sample counts, error methods, resolution, classifications, protocol details, and attack prerequisites.The authors connect explicit reporting to more complete and reproducible benchmarks.Source: §7.1 · p. 12
  • Claims that a primitive is stealthy or difficult to detect should be supported by empirical detectability evidence and comparisons with existing methods.The authors note that no paper in their corpus exhibited this flaw, but still identify it as an important consideration.Source: §7.1 · p. 12

7.2 To reviewers and program committees

  • Reviewers should encourage benchmarks of primitive properties in realistic settings and across platforms, rather than prioritizing end-to-end attacks or proxy metrics.The recommendation particularly cautions architecture reviewers not to prioritize end-to-end attacks over complete evaluation.Source: §7.2 · p. 12

7.3 To the community

  • The community should standardize artifact evaluation and promote comprehensive reporting of resolution, errors, noise resilience, generalizability, targets, sample counts, and correction methods.The authors suggest mandatory artifact evaluation or stronger incentives where appropriate.Source: §7.3 · p. 12

8 Related Work

The related work includes surveys of attack classes and countermeasures, analyses of evaluation properties and reproducibility, and methodologically close meta-analyses in other fields. The paper distinguishes its broad focus on benchmarking practices from work centered on particular attack types or evaluation domains.

8 Related Work

  • Ge et al. and Lou et al. [53] survey microarchitectural attacks and countermeasures, but focus on attack classes, targets, and defenses rather than benchmarking practices.Ge et al. emphasize shared environments such as cloud settings, whereas Lou et al. take a more environment-independent perspective.Source: §8 · p. 12
  • Rauscher et al. identify overlooked properties for cache attacks, while this paper takes a broader meta-level view and advocates resolution metrics over repeated attack scenarios.Schloegel et al. examine reproducibility and guideline adherence in fuzzing, a field with pre-existing criteria unlike this one.Source: §8 · p. 12
  • Van der Kouwe et al. study benchmarking flaws in systems defense, and De Meulemeester et al. examine hardware-cost evaluation and propose a uniform methodology.The authors adapt recommendations inspired by the SIGPLAN empirical evaluation checklist to microarchitectural side-channel benchmarking flaws.Source: §8 · pp. 12–13

9 Conclusion

The survey finds that all 83 papers had benchmarking flaws, averaging 5.5 issues per paper, and offers recommendations for improving evaluation practices. It argues for attack-agnostic resolution metrics, stronger review standards, and reproducibility through shared artifacts.

9 Conclusion

  • Across 83 papers, the survey identified 19 benchmarking flaws; none was flaw-free, and papers averaged 5.5 issues.The authors recommend improvements to benchmarking practices for authors, reviewers, program committees, and the broader community.Source: §9 · p. 13
  • The authors advocate wider use of resolution metrics and emphasize reviewer and program-committee roles in enforcing sound methods and clear reporting.They also call for sharing code, artifacts, and experimental materials, and hope to encourage a unified benchmark suite.Source: §9 · p. 13

A Open science

  • The study’s collection form, collected data, and analysis code and figures are available for consultation and reuse.The code and materials are hosted on GitHub and Software Heritage.Source: A Open science · p. 15

B Ethical considerations

  • The study reports no ethical concerns because it is a meta-analysis of existing research without human subjects, user data, or real-world vulnerability testing.The evaluation was conducted over an extended period to avoid overburdening authors.Source: B Ethical considerations · p. 15

C Code-book

  • The code-book classifies flaws spanning attack-property evaluation, comparisons, realistic settings, reporting, and reproducibility.Its criteria include omitted error-rate and noise-resilience evaluations, unfair or absent comparisons, single-platform testing, and missing target details or artifacts.Source: C Code-book · pp. 15–17

D List of papers included in the study

E Usual targets

The surveyed attack papers target a range of implementations and software versions. GnuPG and libgcrypt, Alexa’s top websites and Chrome, and OpenSSL recur among the reported targets.

E Usual targets

  • Among 41 control-flow cryptographic attacks, GnuPG and libgcrypt were the most commonly targeted victims.GnuPG v1.4.13 appeared seven times, while libgcrypt v1.6.3 appeared five times and v1.5.2 three times.Source: E Usual targets · p. 17
  • Among 23 fingerprinting attacks, Alexa top 100 websites and Chrome versions were the most commonly targeted victims.Nine of the 23 attacks (39.1%) targeted unknown or unspecified victims.Source: E Usual targets · p. 17
  • OpenSSL, particularly v1.0.1e, was the most commonly targeted victim among 17 memory access-based cryptographic attacks.Source: E Usual targets · p. 17
  • The surveyed papers used diverse software and toolchain versions, including many MbedTLS releases, GCC v7.5–v10.3, and WolfSSL v4.2.0 and v5.6.4.Source: E Usual targets · pp. 17–18

F Tendencies’ figures

The appendix presents figures related to tendencies described in Section 6 that were omitted from the main paper for space. Their captions specify the quantities and groups represented, but do not state the plotted outcomes.

F Tendencies’ figures

Loading 2609.03893v2…