Source-linked AI summary
"They don't care about this": A Systematic Study of TEE Build Reproducibility in the Wild
Annika Wilde, Marco Gutfleisch, Felix Reichmann, Anirban Chakraborty, Yuval Yarom, M. Angela Sasse, Ghassan Karame
TL;DR
TEEs rely on remote attestation to authenticate deployed code, but linking an attested measurement to inspectable source requires reproducible builds. This paper studies 115 TEE deployments and interviews 12 SGX developers, finding that reproducibility is uncommon and hindered by technical, organizational, and ecosystem factors.
Problem
TEE users need to connect an attested binary to source code they can inspect, but reproducibility in TEE builds and developers’ recognition of its importance were insufficiently understood.
Method
The paper combines an empirical analysis of 115 TEE deployments with qualitative interviews of 12 SGX developers.
Results
91% of examined TEE projects were not reproducible, indicating that reproducible builds remain the exception rather than the norm in the TEE ecosystem.
Takeaways & Limitations
The findings support treating reproducibility as a core TEE quality and security requirement and addressing it across developers, project managers, and standardization bodies.
Takeaways & Limitations
Only 12 of 50 contacted SGX project maintainers participated, and interview findings may be affected by recall and social desirability bias.
Abstract
from arXiv · showhide
Trusted Execution Environments (TEEs) have become a cornerstone of modern cloud computing, providing strong confidentiality and integrity guarantees for both code and data. A critical component of this trust model is remote attestation, which enables external entities to verify the authenticity and integrity of code executing within a TEE through cryptographic measurements. However, the effectiveness of remote attestation fundamentally depends on the verifier's ability to trace the reported measurement back to the original source code - a property that can only be guaranteed through reproducible builds. In this paper, we investigate the reproducibility of TEE builds through a technical analysis of 115 TEE deployments. Our analysis spans popular TEEs such as Intel SGX, Intel TDX, and AMD SEV, and reveals that a striking 91% of those deployments were not reproducible, with 80% failing to provide both source code and a reference build, the two essential prerequisites for reproducibility. To explore the root causes, we contacted the maintainers of 50 SGX projects and managed to recruit 12 developers from industry and academia for interviews. Only one of our participants reported that reproducibility is a priority during development, effectively confirming our technical findings. Beyond technical barriers (e.g., timestamps included in the binary) that can be readily addressed, we identify broader ecosystem-level challenges, such as the lack of control over the build environment in projects involving multiple stakeholders. We argue that achieving reproducibility in TEEs requires a holistic development approach that extends beyond individual developers and calls for stronger commitments - rather than treating TEEs as a "security badge".
1 Introduction
TEEs shift trust toward inspectable enclave code, but remote attestation can connect deployed binaries to vetted source only when builds are reproducible. This study finds reproducibility is rare across the TEE ecosystem and is constrained by both technical and ecosystem-level factors.
- 1 Introduction: Remote attestation authenticates the deployed binary but does not establish that its code is trustworthy without source inspection and reproducible builds.Users must compare a locally rebuilt binary with the attested binary bit by bit to connect vetted source code to the deployment.
- 1 Introduction: Reproducibility is critical because users need to verify that the attested enclave binary was genuinely built from source code they have inspected.The paper frames this as a central unresolved question for widely deployed TEEs and for developers’ awareness of reproducibility’s importance.
- 1 Introduction: The study investigates this gap through a comprehensive analysis of TEE builds and a qualitative interview study with SGX developers.The introduction presents these as two complementary approaches.
- 1 Introduction: 91% of 115 examined TEE projects remained unreproducible, while the study combined technical analysis with interviews of 12 SGX developers.The projects covered Intel SGX, Intel TDX, and AMD SEV-SNP; 10 of 12 developers said reproducibility was neither prioritized nor considered.
- 1 Introduction: Build failures arise from familiar nondeterminism, secret material, and procedural mismatches across components that can influence the enclave’s final measurement.The paper argues that reproducibility problems undermine the meaning of remote attestation and require a system-level view of the build process.
2 Background and Related Work
TEEs isolate sensitive computation and use remote attestation to authenticate enclave measurements, while reproducible builds make binary identity independently verifiable. Prior work explains conventional nondeterminism and proposes attested provenance mechanisms, but this paper focuses on reproducibility practices among TEE developers.
- 2.1 Trusted Execution Environments: TEEs isolate either a process or an entire virtual machine to protect sensitive code and data during execution.Examples include SGX and TrustZone for process-based TEEs, and TDX and SEV-SNP for VM-based TEEs.
- 2.1 Trusted Execution Environments: Remote attestation lets a verifier validate an enclave’s signed measurement and compare it with a trusted reference.The measurement is a hash of the enclave’s initial code and data, while signer identity identifies the signing entity.
- 2.2 Reproducible Builds: Reproducible builds produce identical outputs from identical source, compiler, and configuration inputs across environments.Bit-by-bit reproducibility requires identical binaries, whereas semantic reproducibility requires identical behavior despite potentially different representations.
- 2.2 Reproducible Builds: Traditional build nondeterminism can arise from operating-system, filesystem, timestamp, and embedded-path variation.Examples include __DATE__ timestamps and __FILE__ absolute paths recorded in binaries.
- 2. Background and Related Work: Earlier research studied reproducibility barriers in traditional software and developer communities, while TEE-specific proposals assume developer interest in reproducible builds.This paper instead examines SGX developers’ awareness, understanding, and attitudes toward reproducibility.
3 Reproducible TEE Builds in the Wild
Across 115 TEE applications, reproducible builds are rare and depend more on build-system practices than on the underlying TEE technology. Case studies show that missing documentation, design choices, and unavailable dependency information commonly prevent independent verification.
- Results: 91.3% of TEE applications were not reproducible, including 77% that lacked both source code and a reference measurement.Only ten projects provided sufficient information to reproduce the enclave measurement, and three required non-public developer information.
- Results: Reproducibility was 0% for academic projects and 12% for non-academic projects, with only one academic project providing a reference measurement.Because the dataset was imbalanced toward non-academic and SGX projects, subsequent analysis focused on 84 non-academic projects.
- Results: Reproducibility varied across platforms—0% for TDX, 9.2% for SGX, and 28.6% for SEV-SNP—but the technology association was not statistically significant (p = 0.099).The authors characterize the challenge as orthogonal to the underlying TEE technology.
- Results: 42.9% of projects using reproducibility-oriented build tools were reproducible, compared with 22.2% using prebuilt Docker images, 18.2% using Dockerfiles, and 5.3% using unspecified systems.The association between build-system choice and reproducibility was statistically significant (p = 0.012).
- Case Studies: Case studies found that most reproducibility problems were addressable through better build documentation, although some projects remained unreproducible because dependencies or build details were unavailable.Secret Network was unreproducible by design because production binaries included signatures made with an undisclosed key; CCF became reproducible with its CI/CD build path and updated documentation.
4 Interview Study
Interviews show that TEE developers lack a consistent understanding of reproducibility and rarely prioritize it, citing environmental complexity, dependencies, build irregularities, and limited debugging support. Participants also described ecosystem and tooling changes that could improve reproducibility, while noting tensions around signatures, hotfixes, and customer demand.
- 4.2.1 Perception of Reproducibility: Participants linked reproducibility to security, but some considered it difficult for end-users to test and therefore irrelevant to many customers.Five participants called reproducibility essential for security, while two questioned its relevance because most users would rely on service statements or public values.
- 4.2.1 Perception of Reproducibility: Developers held inconsistent views of reproducibility, with some defining it as repeatable research output rather than identical enclave binaries or builds.Seven participants described reproducibility as replicating published results, illustrating the lack of a shared operational definition.
- 4.2.2 Challenges: Six of twelve participants identified limited control over build environments as a major challenge, especially when dependencies, tools, or distributed builds are managed by different stakeholders.Participants emphasized that dependencies may be embedded in the operating system and that distributed builds can produce different binaries depending on how work is distributed.
- 4.2.2 Challenges: Participants reported recurring technical obstacles including changing dependencies, embedded timestamps and paths, limited awareness, enclave complexity, and difficulty debugging nondeterminism.Five participants discussed dependency challenges, five build-induced irregularities, four lack of awareness, four enclave complexity, and two debugging difficulties.
- 4.2.3 Prioritization of Reproducibility: Only one participant treated reproducibility as a project priority, while eight reported little or no priority and others prioritized different development tasks.One participant described reproducibility as a release blocker; another said companies recognize the problem but do not invest manpower.
5 Discussion
Reproducibility is rare across TEE projects and platforms, reflecting both familiar technical obstacles and organizational constraints. Despite recognizing its security importance, developers and organizations seldom prioritize it or control the environments needed to achieve it.
- Only 10 of 115 analyzed projects, or 8.7%, were reproducible, while missing documentation explained failures in three projects that became reproducible after developer input.Reproducible builds remain uncommon despite their importance for remote attestation and secure TEE deployment.
- Reproducibility challenges were TEE-agnostic, with no statistically significant association between reproducibility and SGX, TDX, or SEV-SNP.Interviewees reported similar issues across platforms, including nondeterministic inputs, insufficient documentation, and complex dependency management.
- Developers’ Challenges: Timestamps, filesystem ordering, randomness, undisclosed secrets, and dependency management created recurring technical barriers, while versioned-input tools such as Nix, Bazel, and Yocto improved reproducibility.Many obstacles are known from traditional reproducible-builds research, although secrets may be unavailable to external reproducers.
- Organizational Barriers: Limited developer control over centrally managed or shared CI/CD environments made reproducibility a broader organizational problem rather than an individual developer responsibility.Builds often involve DevOps teams, multiple teams, and external dependencies beyond developers’ direct control.
- Eight participants viewed reproducibility as highly important for security, yet all but two reported that it was generally not treated as a development priority.The findings indicate a gap between perceived security importance and organizational decision-making.
- Study Limitations: Interview evidence was limited by the participation of only 12 developers and by recall and social-desirability bias in self-reported accounts.The technical analysis nevertheless included production-grade industry projects, where poor reproducibility was also prevalent.
6 Recommendations
The paper recommends treating reproducibility as a foundational TEE security and quality requirement supported by developer practices, organizational priorities, standardized environments, and ecosystem-wide incentives.
- TEE stakeholders should raise awareness, provide practical guidance, and establish reproducibility as a foundational concern in development.The recommendations target developers, project managers, decision-makers, customers, and standards bodies.
- Application Developers: Developers should reduce nondeterminism by avoiding unstable build inputs, documenting dependencies and hashes, and recording standardized build metadata.Suggested practices include avoiding __DATE__ and __FILE__, using compiler options such as -ffile-prefix-map, and adopting .buildinfo-style records.
- Project Managers and Decision-Makers: Organizations should make reproducibility a release requirement supported by training, coding guidelines, process audits, and provenance services.The paper recommends organizational enforcement and code-provenance mechanisms rather than relying on manual source-to-binary verification.
- Project Managers and Decision-Makers: Standardized environments such as Docker, Nix, and Bazel can isolate toolchains and dependencies, provided versions and inputs are explicitly pinned.Nix enforces exact dependency sources and hashes, while Bazel tracks input hashes and Docker encapsulates dependencies and configurations.
- End Users and Customers: Customers can strengthen incentives by requiring reproducibility and verifiable build information in procurement specifications.Market pressure can move reproducibility from an optional feature toward a core component of TEE trust and security.
- Standards and Research: Standards groups and researchers can advance the ecosystem through shared standards, tools, best practices, verification mechanisms, and reference implementations.The SCITT Working Group is identified as one potential venue for this work.
7 Conclusion
The study finds that reproducible TEE builds are uncommon and that the barriers are primarily technical-organizational rather than specific to one TEE technology. It responds with recommendations aimed at making reproducibility an early, ecosystem-wide security requirement.
- 91% of the 115 examined TEE projects were not reproducible, showing that reproducible builds remain the exception across the ecosystem.The analysis covered TEE deployments and complemented them with interviews of 12 SGX developers from industry and academia.
- Reproducibility challenges were largely TEE-agnostic and involved nondeterministic builds, inadequate documentation, limited environment control, and low prioritization.The paper reports that organizational and awareness-related factors were the primary obstacles.
- The paper recommends that developers, project managers, and standards bodies establish reproducibility as a core quality and security requirement from project inception.The recommendations aim to provide practical guidance without disrupting existing workflows.
A Open Science
The paper makes its research artifacts publicly available to support evaluation of its contributions and analysis.
- The authors publicly release a detailed results table, reproducibility-test instructions, and scripts for processing data and generating the paper’s tables.
B Ethical Considerations
The study reports that its institution lacked an IRB or ERB but followed institutional and national data-protection requirements for human-participant research.
- The institution had no institutional review board or ethics review board at the time of the study.
- The research team included two Human-Centered Security experts who supervised planning and ethical compliance.
- The study used a consent form previously verified by the institution’s data protection officer.
1. Onboarding
The interview onboarding introduced the study, established reproducibility as reproducing an enclave hash from source code to binary, and organized questions around projects, obstacles, and broader solutions.
- Introduction Reproducibility: The onboarding asked developers about their understanding of reproducibility, how they learned it, and its importance to projects, users, and stakeholders.
- Introduction Reproducibility: Reproducibility was defined as the ability to reproduce the enclave hash linking source code to its binary output.
- Project-Specific Challenges: Project-focused questions covered steps taken, testing practices, investment, and obstacles encountered when building reproducible TEE applications.
- General and Cross-TEE Challenges: The interview expanded beyond individual projects to ask about reproducibility issues across open-source SGX projects and other TEE platforms.
- Wider Context: Recruitment materials framed the study as a conversation about SGX reproducibility and invited discussion of why it is not widely addressed.
4. Improving Reproducibility
The final interview section solicited concrete ideas for improving reproducibility, including containers, information sharing, and stronger research-community support.
- 4. Improving Reproducibility: The interview’s final section explored ideas and solutions for supporting more reproducible enclave builds.
- 4. Improving Reproducibility: Participants were asked whether Docker containers could simplify reproducibility testing and help their projects or others.
- 4. Improving Reproducibility: The discussion also covered where reproducibility information should be shared and how the research community could better support developers.
- 4. Improving Reproducibility: The session concluded with final questions before recording was stopped.
D Emails & Codebook
The supplementary materials include the study’s SGX outreach framing, an interview-recruitment figure, and a codebook mapping themes to the discussion of organizational barriers, known issues, and security-badge thinking.
- Emails & Codebook: The researchers introduced the study as research on reproducibility in open-source projects using Intel SGX.
- Emails & Codebook: Figure 3 presents an email to SGX project developers and maintainers seeking interviews about reproducibility challenges.
- Emails & Codebook: The codebook organizes discussion themes as Organizational Barriers, Known Issues, and Security is a Badge.