"They don't care about this": A Systematic Study of TEE Build Reproducibility in the Wild

1 Introduction

• Remote attestation authenticates the developer-signed enclave binary, not code trustworthiness; reproducible local builds connect inspected source to the attested measurement.That connection lets users check that the deployed binary matches the source they vetted.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Source auditing and recompilation require time and expertise beyond most end users, but skilled community members can publish verified reference measurements for others to rely on.The paper presents this as a collaborative way to establish trust in sensitive services.Source: §1 · p. 2
• Across 115 projects spanning SGX, TDX, and SEV-SNP, 91% remained unreproducible; 94% lacked documentation or tooling, 77% lacked both prerequisites, and 10 of 12 developers said reproducibility was not prioritized.Reproducibility rates varied by primary build system, including prebuilt Docker images and Nix.Source: §1 · p. 2
• Failures stemmed from technical nondeterminism, such as timestamps and filesystem ordering, secret material, and development processes that conflict with pipeline-wide measurement control.Because every build-pipeline component can affect the enclave measurement, the authors argue for TEE-specific practices that prioritize reproducibility and give developers greater control over build environments.Source: §1 · p. 2

2.1 Trusted Execution Environments

• TEEs isolate either a single process or an entire VM; this work calls a TEE instance initialized with user code an enclave.Examples include SGX and Arm TrustZone for process-based TEEs, and TDX and AMD SEV-SNP for VM-based TEEs.Source: §2.1 · p. 2
• The TCB includes the CPU, TEE firmware, and enclave code, while privileged system software is treated as untrusted.Enclaves have a measurement of their initial code and data and a signer identity derived from the developer’s public key; remote attestation signs the measurement for verification against a trusted reference.Source: §2.1 · pp. 2–3

2.2 Reproducible Builds

• Reproducible builds produce identical outputs from the same source, compiler version, and configuration across environments; bit-by-bit reproducibility requires identical binaries, while semantic reproducibility requires identical behavior.Reproducibility supports independent checks that binaries match expected outputs, especially in open-source projects.Source: §2.2 · p. 3
• Timestamps, build paths, file-system ordering, archive metadata, compiler randomness, and uninitialized memory can introduce build nondeterminism.Examples include __DATE__ and __FILE__ macros embedding values that vary across builds or environments.Source: §2.2 · p. 3

2.3 Related Work

• Prior traditional-software studies examine non-determinism in build processes and TrueCrypt [27], while interviews with 24 Reproducible-Builds.org contributors identify technical and ecosystem challenges [27].This study instead focuses on SGX developers’ awareness, understanding, and attitudes toward reproducibility.Source: §2.3 · p. 3
• TEE proposals include attestable builds that verify inputs and attest to binary hashes [22] and a Code Transparency Service for provenance and accountability.Both approaches assume developer interest in reproducibility; this study investigates SGX developers’ awareness, motivation, and challenges.Source: §2.3 · p. 3

3.1 Methodology

• The study selected candidate applications from cloud marketplaces and Awesome SGX Open Source, then assessed eligible projects by compiling releases and comparing enclave measurements.The final dataset contained 115 applications; 88 lacked either source code with build instructions or a reference measurement, leaving 27 eligible for reproduction assessment.Source: §3.1 · pp. 3–4

3.2 Results

• 91.3% of the 115 applications were not reproducible; 77% lacked both source code and a reference measurement, and only 37% of projects providing a reference were reproducible.Ten applications provided sufficient information to reproduce the measurement: seven using public documentation alone and three requiring non-public developer information.Source: §3.2 · p. 4
• Reproducibility was 0% for TDX, 9.2% for SGX, and 28.6% for SEV-SNP projects, but the differences were not statistically significant (p-value = 0.099).The filtered set included 65 SGX, 14 SEV-SNP, and three TDX commercial implementations.Source: §3.2 · pp. 4–5
• 42.9% of projects using reproducibility-oriented build tools were reproducible, versus 22.2% with prebuilt Docker images, 18.2% with Dockerfiles, and 5.3% with unspecified systems (p = 0.012).The study identifies dependency pinning and hermetic build environments as features of the highest-performing tools.Source: §3.2 · p. 5

3.3 Case Studies

• Secret Network’s SGX release is non-reproducible by design because it is signed with an undisclosed key; Nginx-SGX and MongoDB-SGX embed build-varying certificates in measured binaries.Secret VM v0.0.25’s SEV-SNP measurement was reproduced, while for TDX the initial measurement and one of four additional registers matched the release.Source: §3.3 · pp. 5–6
• CCF required its CI/CD build path, Gramine required its full Debian 11 release environment, and Ternoa remained unreproducible because required dependencies were unknown.These cases illustrate how incomplete build-environment information can prevent reproduction, even when compilation succeeds.Source: §3.3 · p. 6
• For 11 of 17 unreproducible projects, the cause remained unknown; manual tests found Mystikos sensitive to build path and filesystem, while MobileCoin’s measurement was unchanged across tested filesystems.The authors conclude that most identified issues could be addressed with more detailed documentation, but pinpointing causes can be difficult without the original build environment.Source: §3.3 · p. 6

4.1 Methodology.

• The exploratory qualitative study used semi-structured interviews with 12 experienced open-source SGX project maintainers recruited from the authors’ dataset.The researchers emailed contributors from identified SGX projects and received 13 responses to 62 invitations; participants came from 11 countries, half held academic positions, and half worked in industry.Source: §4 · p. 6, §4.1 · p. 7, §4.1.3 · p. 7

4.1.4 Interview Structure & Procedure.

• Researchers developed and piloted the interview guide, then transcribed interviews and used thematic analysis with iterative coding and review by a second researcher.The guide was piloted twice internally and twice externally; interviews covered participants’ definitions and practices, challenges, improvements, and offboarding.Source: §4.1.2 · p. 7, §4.1.4 · pp. 7–8

4.2.1 Perception of Reproducibility.

• Eight participants saw reproducibility as security-relevant, but participants differed on its meaning: seven emphasized repeatable research results, three identical binaries, and five expected application behavior.Two participants questioned its relevance to end-users because verifying reproducibility can be difficult.Source: §4.2.1 · p. 8

4.2.2 Challenges.

• Limited control over build environments was the most frequently cited challenge (6/12), while five participants raised dependency challenges and five cited build-induced irregularities.Four participants mentioned timestamps in binaries and two mentioned embedded paths; changing packages, compiler toolsets, and legacy dependencies also complicated reproducibility.Source: §4.2.2 · pp. 8–9
• Participants also cited limited awareness (4), enclave complexity (4), and difficulty debugging failures (2); one warned that early signatures and rapid hotfix disclosure create distinct risks.Debugging difficulties included poor documentation of hash-influencing components and trouble identifying nondeterminism; the hotfix concern was that source disclosure could expose vulnerabilities before users applied patches.Source: §4.2.2 · p. 9

4.2.3 Prioritization of Reproducibility.

• Only one participant treated reproducibility as a project priority, while eight reported little or none; competing work and weak customer valuation also hindered investment.P4 described assigning one full-time employee to spend five to six weeks improving reproducibility; P14 estimated 5% to 10% of effort over the years, varying by project.Source: §4.2.3 · pp. 9–10, §4.2.4 · p. 10

4.2.4 Practices to achieve reproducibility.

• Eight participants discussed tools: all named Docker, while Nix, Bazel, and other practices offered stricter input control; participants also reported trial-and-error learning and no single reference source.Four already used Docker and four considered it valuable, though unpinned inputs were criticized; individual practices included stable environments, static paths, fixed timestamps, and fixed signature entropy.Source: §4.2.4 · p. 10

4.2.5 Suggested improvements.

• Participants recommended raising awareness, improving tools and technical infrastructure, and supporting research-defined practices; they had no consensus on where reproducibility information should be disseminated.Suggestions included Docker build environments, SBOMs, package-manager features, best practices, sample projects, and researchers joining projects as maintainers.Source: §4.2.5 · pp. 10–11

5 Discussion

• Only 10 of 115 analyzed TEE projects (8.7%) were reproducible, and missing documentation was the primary obstacle in three cases requiring developer input.Once the missing information was obtained, achieving reproducibility was straightforward.Source: §5 · p. 11

5.1 Trends Across TEE Technologies

• No statistically significant association linked TEE technology to reproducibility, while interviews described similar challenges across platforms.The authors suggest these challenges generalize across TEE architectures that rely on remote attestation and enclave measurement verification; Arm TrustZone does not natively support remote attestation, though reproducibility can still mitigate supply-chain attacks.Source: §5.1 · p. 11

5.2 Reproducibility Challenges

• Timestamps, filesystem ordering, paths, secrets, signatures, and dependency management complicate reproducible builds, while Nix, Bazel, and Yocto substantially improve reproducibility.Organizational barriers compound these technical issues: developers often lack control over CI/CD environments, and one non-reproducible dependency can compromise the application; only 35% of contacted teams responded, and four cases still lacked sufficient information to reproduce the build.Source: §5.2 · p. 11

5.3 Security Is A Badge

• Eight interview participants considered reproducibility important for SGX security, yet all but two said it was not generally a development priority.One participant reported that reproducibility required five to six weeks of dedicated effort once prioritized; the authors argue for stronger organizational and economic incentives, including customer demand for end-to-end attestation support.Source: §5.3 · pp. 11–12

5.4 Study Limitations

• Developers for all 50 of 180 SGX projects with located contact information were contacted; 12 agreed to interviews, yet technical analysis found poor reproducibility equally prevalent in industrial-scale TEE deployments.Interview-based findings remain subject to recall and social-desirability bias.Source: §5.4 · p. 12

6.1 Application Developers

• Application developers should build reproducibility literacy, reduce known sources of nondeterminism, and document build processes and exact dependencies.The study reports that 33% of developers were unaware of reproducibility’s importance for TEEs. Suggested practices include avoiding __DATE__ and __FILE__, using GCC’s -ffile-prefix-map, and recording versions and cryptographic hashes in standardized metadata.Source: §6.1 · p. 12

6.2 Organizational Managers and Stakeholders

• Managers should make reproducibility a release requirement, support it with training and audits, and use provenance services to verify source-to-binary lineage before release.Developers in large organizations may lack authority to change standardized CI/CD workflows; the paper recommends extending deployment-acceptance requirements to TEE applications. It also describes a public, append-only provenance ledger integrated into CI pipelines.Source: §6.2 · p. 12
• Organizations should standardize build environments with tools that isolate processes and pin toolchains and dependencies, especially in large, multi-team projects.Docker requires explicitly pinned versions; Bazel sandboxes build actions and tracks input hashes, while Nix enforces exact dependency sources and hashes.Source: §6.2 · pp. 12–13

6.3 End Users and Customers

• The main obstacle is inadequate organizational incentive; end users can push adoption by requiring reproducibility in procurement and requesting verifiable build information.The recommendations frame market pressure as a way to make reproducibility a core component of trust and security in TEE-based software.Source: §6.3 · p. 13

6.4 Special Interest Groups and Researchers

• Special interest groups should develop ecosystem standards and raise awareness, while researchers should identify barriers and build verification mechanisms, tools, and reference implementations.The IETF SCITT Working Group is cited as an example of a group that can advance this work.Source: §6.4 · p. 13

7 Conclusion

• 91% of 115 examined TEE deployments were not reproducible, and interviews with 12 SGX developers covered industry and academia.The study found reproducible builds remain the exception rather than the norm in the TEE ecosystem.Source: §7 · p. 13
• The study recommends that developers, project managers, and standardization bodies raise awareness and establish reproducibility as a core quality and security requirement from project inception.It also calls for practical guidance that supports reproducibility without disrupting existing workflows.Source: §7 · p. 13

• The research artifacts needed to evaluate the paper’s contributions are publicly available at.Source: A Open Science · p. 14
• The artifacts include a detailed results table analyzing TEE projects from.Source: A Open Science · p. 14
• The artifacts include reproducibility-test instructions for the 27 applications in Table 3 and scripts for processing data and generating paper tables.Source: A Open Science · p. 14

B Ethical Considerations

• The institution had no IRB or ERB, so two human-centered security experts supervised study planning and compliance with ethical standards, alongside adherence to data-protection and privacy regulations.The consent form had been verified by the data protection officer and covered information normally required for IRB/ERB approval.Source: B Ethical Considerations · p. 14
• Participants gave consent, were told participation was voluntary and withdrawal carried no consequences, and received contact information for data-protection questions.Before recording, participants were reminded of the privacy policy and invited to ask questions.Source: B Ethical Considerations · p. 14
• Researchers contacted project experts using GitHub email addresses, paid €50 for each 25-minute interview, and transcribed interviews locally before anonymizing them.The payment corresponds to €120 per hour; the anonymization aimed to prevent publication of sensitive information.Source: B Ethical Considerations · p. 14

1. Onboarding

• Interviewers introduced the study and institution, explained its motivation, introduced the interviewee, addressed consent and data-protection questions, and began recording with consent.Source: §1 · p. 14

2. Narrative Prompt

• Participants were invited to discuss reproducibility generally, especially in their most recent SGX project, and to explain what it meant to them and how they learned about it.Source: §2 · pp. 14–15
• For the study, reproducibility meant being able to reproduce the enclave hash, linking source code to its binary output.Source: §2 · p. 15
• The guide asked whether participants considered reproducibility in development, how important it was to users or communities, what steps or tests they used, and how much effort they would invest.Source: §2 · p. 15

3. Challenges in Reproducible Enclave Builds

• Participants were asked about reproducibility obstacles in their current projects and whether they had encountered issues in other projects.Source: §3 · p. 15
• The guide raised the study’s observation that reproducibility is often problematic across open-source SGX projects, then asked why it is not widely addressed and who could prioritize it.Source: §3 · p. 15
• Participants were also asked whether they had worked with TrustZone, TDX, or SEV, encountered reproducibility challenges there, and how those platforms differed from SGX.Source: §3 · p. 15

4. Improving Reproducibility

• The interview asked developers how they would address the reproducibility challenges they had identified.Source: §4 · p. 15
• The researchers asked whether Docker containers could make reproducibility testing feasible for participants’ projects.Source: §4 · p. 15
• The interview asked whether Docker-based reproducibility testing might also help other projects.Source: §4 · p. 15
• Participants were asked which channels, such as Intel’s website, GitHub, or documentation, could share reproducibility-related information.Source: §4 · p. 15
• The researchers asked how the research community could better support developers in achieving reproducibility.Source: §4 · p. 15

5. Offboarding

• The session concluded by inviting final questions or topics and then stopping the recording.Source: §5 · p. 15

Emails

• Figure 3 shows an email to SGX project developers and maintainers requesting interviews about reproducibility challenges.Source: Figure 3 · p. 15

Codebook

• Table 7’s codebook maps Organizational Barriers (OB), Known Issues (KI), and Security is a Badge (SB) to Sections 5.2 and 5.3; (*) marks containers for sub-codes excluded from coding.Source: Table 7 · p. 16