PaperlayerView exact revision on arXiv

Source-linked AI summary

CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation

Anirudh Khatry, Robert Zhang, Jia Pan, Ziteng Wang, Qiaochu Chen, Greg Durrett, Isil Dillig

arXiv:2504.15254v3cs.SEcs.CLcs.LG

TL;DR

Automated C-to-safe-Rust transpilation lacks a benchmark that tests repository-scale correctness under Rust’s safety constraints. CRUST-Bench fills this gap with 100 repositories, safe Rust interfaces, and tests, finding that frontier models remain substantially unreliable even with repair.

  • Problem

    Existing evaluation does not adequately test whether systems can translate realistic C repositories into safe, idiomatic Rust that satisfies correctness tests.

    Source: §1 · p. 1

  • Method

    CRUST-Bench pairs 100 C repositories with manually authored safe Rust interfaces and test cases, validating interface adherence, compilation, and functional correctness.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    OpenAI o3 with iterative compiler- and test-failure repair succeeds on 48% of benchmark tasks, while frontier models achieve 13-22% one-shot and 32-48% with repair loops.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    CRUST-Bench provides a repository-scale testbed for developing transpilation systems that must reason about complex dependencies while preserving Rust safety.

    Source: §1 · pp. 2–3

    Syntax-directed transpilers and C2SAFERRUST are excluded because they commonly generate unsafe code incompatible with the benchmark’s safe-interface constraints.

    Source: §4 · p. 7

Abstract

from arXiv · show

C-to-Rust transpilation is essential for modernizing legacy C code while enhancing safety and interoperability with modern Rust ecosystems. However, no dataset currently exists for evaluating whether a system can transpile C into safe Rust that passes a set of test cases. We introduce CRUST-Bench, a dataset of 100 C repositories, each paired with manually-written interfaces in safe Rust as well as test cases that can be used to validate correctness of the transpilation. By considering entire repositories rather than isolated functions, CRUST-Bench captures the challenges of translating complex projects with dependencies across multiple files. The provided Rust interfaces provide explicit specifications that ensure adherence to idiomatic, memory-safe Rust patterns, while the accompanying test cases enforce functional correctness. We evaluate state-of-the-art large language models (LLMs) on this task and find that safe and idiomatic Rust generation is still a challenging problem for various state-of-the-art methods and techniques. We also provide insights into the errors LLMs usually make in transpiling code from C to safe Rust. The best performing model, OpenAI o1, is able to solve only 15 tasks in a single-shot setting. Improvements on CRUST-Bench would lead to improved transpilation systems that can reason about complex scenarios and help in migrating legacy codebases from C into languages like Rust that ensure memory safety. You can find the dataset and code at https://github.com/anirudhkhatry/CRUST-bench.

1 Introduction

CRUST-Bench addresses the challenge of evaluating C-to-safe-Rust migration at repository scale, where correctness must include memory safety, idiomatic interfaces, compilation, and tests. It combines manually authored Rust specifications and validation cases with an empirical study of frontier models and their repair strategies.

Motivation

  • Safe migration requires more than functional equivalence: generated code must compile and execute within Rust’s memory-safe, idiomatic subset.The benchmark therefore targets migration from non-memory-safe C semantics to safe Rust rather than merely equivalent output.Source: §1 · p. 1

Benchmark contribution

  • Existing benchmarks often emphasize isolated or narrow programs, whereas CRUST-Bench evaluates realistic multi-file projects with structured interfaces and integration-level correctness.This scope is designed to capture dependencies and safety constraints that function-level evaluations can miss.Source: §1 · p. 2
  • CRUST-Bench evaluates C-to-Rust transpilation using 100 repositories paired with safe Rust interfaces and correctness-enforcing tests.The interfaces specify signatures, types, and ownership constraints, while tests validate the generated implementation's behavior.Source: §1 · pp. 2–3

Benchmark construction

  • CRUST-Bench uses a hybrid annotation process in which humans author interfaces and tests, then validate interfaces with the Rust compiler.This combination of human expertise and type checking is intended to provide a rigorous assessment framework.Source: §1 · p. 2

2 Motivation and Related Benchmarks

Prior C-to-Rust benchmarks commonly have narrow task scope, limited structural guidance, or weak correctness and safety validation. CRUST-Bench expands evaluation to 100 moderately sized real-world repositories with safe interfaces and test files.

Prior benchmarks

  • Earlier datasets range from 20 C programs in CROWN to 520 mostly self-contained functions in C-to-Rust translation studies, limiting project-scale evaluation.FLOURINE includes real-world projects but retains narrow individual tasks, while C2SaferRust covers seven large programs.Source: §2 · p. 3

CRUST-Bench scope

  • CRUST-Bench contains 100 real-world repositories averaging 958 lines of code, balancing project-level complexity with feasibility for LLM-guided transpilation.It targets migration challenges without requiring models to process excessively large codebases.Source: §2 · p. 4

CRUST-Bench design

  • CRUST-Bench adds safe Rust interfaces and corresponding tests to address gaps in structural guidance, memory-safety evaluation, and robust correctness validation.These artifacts make interface adherence and functional behavior explicit evaluation targets.Source: §2 · pp. 3–4

3 Benchmark Structure

CRUST-Bench evaluates repository-scale C-to-Rust transpilation through safe Rust interfaces, compilation, and functional tests across diverse real-world projects.

Benchmark Structure

  • The benchmark represents repositories as parallel C and Rust file structures, requiring each Rust file to directly correspond to a C source file.This formulation preserves multi-file translation structure and repository-level dependencies.Source: §3 · p. 4
  • CRUST-Bench defines success as conforming to a safe Rust interface, compiling without errors, and passing every supplied test.Interfaces specify types, function signatures, and ownership constraints; compilation checks Rust’s type and borrow systems.Source: §3 · p. 4
  • Projects are sourced from open-source C repositories spanning compilers, algorithmic libraries, system utilities, and other application domains.Eligible repositories compile with GCC 11.4.0 and Clang 14.0.0 and satisfy complexity and testing requirements.Source: §3 · p. 4
  • Each benchmark pairs manually authored safe Rust interfaces with adapted Rust tests that exercise the specified functions and data structures.Four annotators define idiomatic Rust types and signatures, compile-check the annotations, and manually review the benchmarks.Source: §3 · p. 5

4 Experimental Setup: Models and Systems Evaluated

The evaluation compares closed-source and open-weight language models using deterministic generation, iterative repair, and agent-assisted workflows under CRUST-Bench’s safety constraints.

Models and Systems Evaluated

  • The study evaluates closed-source and open-weight LLMs, including OpenAI, Anthropic, Google, and open-weight coding models.The evaluated systems include o3, o1, o1-mini, GPT-4o, Claude models, Gemini-1.5-Pro, Virtuoso, QwQ-32B, and Qwen-Coder variants.Source: §4 · p. 5
  • Models use greedy decoding where temperature control is available, enabling deterministic pass@1 evaluation under long-context and strict correctness requirements.Each model generates approximately 5,165 tokens per task on average.Source: §4 · p. 6
  • The repair experiments run three bounded rounds using compiler feedback alone or compiler feedback plus failing tests.A pilot found one initial generation followed by greedy repair outperformed sampling-based alternatives.Source: §4 · p. 6
  • Pipelined SWE-agent first receives an LLM-generated Rust implementation, then edits it iteratively in a Docker environment to repair compiler errors.This evaluates SWE-agent as a post-processing mechanism rather than a standalone transpiler.Source: §4 · p. 6
  • The evaluation excludes syntax-directed transpilers because they commonly generate unsafe Rust or rely on libc, conflicting with CRUST-Bench’s safe interface constraints.Adapting these systems to the interface-first evaluation would require substantial reengineering beyond the study’s scope.Source: §4 · p. 7

5 Results and Analysis

Single-shot transpilation is weak, while iterative repair substantially improves test success but can reduce build stability. Errors concentrate in Rust type and ownership reasoning, incomplete implementations, and interface-level mismatches.

Single-Shot Transpilation Results

  • 15%: OpenAI o1 passes CRUST-Bench tests in the single-shot setting, where models receive no compiler or test feedback.LLaMA-3-8B produces no correct solutions.Source: §5 · p. 7

Self-Repair Improves Success Rate

  • Up to 13%: three compiler-repair rounds improve test pass rates, while Test repair adds 0–9% over Compiler repair.Compiler repair also improves build success by as much as 37% for o1.Source: §5 · p. 7
  • 5–20%: moving from Compiler to Test repair lowers build success despite improving test correctness.Test feedback encourages more aggressive semantic changes that can introduce borrowing violations or type mismatches.Source: §5 · p. 7

Agent-Based Debugging

  • 32%: pipelined SWE-agent improves Claude 3.7 Sonnet from a 13% pass@1 baseline, matching rather than exceeding Claude’s Test-repair result.The agent navigates files and invokes build and test commands, but many steps are spent reading or navigating.Source: §5 · pp. 7–8

Error Analysis

  • Unsafe-code errors are rare under prompts that explicitly instruct models to avoid unsafe Rust features, whereas unimplemented errors remain widespread.The error breakdown is presented across models and configurations in Table 5.Source: Table 5 · p. 8, §5 · p. 9
  • Type and ownership errors are common, including mismatches and borrowing violations involving lifetimes, mutability, and compatibility.Other frequent failures include missing variables, incorrect argument counts, missing traits, unsafe features, and incomplete implementations caused by truncation or placeholders.Source: §5 · pp. 8–9

6 Related Work

Prior code-generation and repository benchmarks address isolated generation, editing, completion, or debugging, but C-to-Rust benchmarks often lack repository-scale scope, safe interfaces, and robust validation. CRUST-Bench targets these gaps while relating to both syntactic and learning-based transpilation approaches.

Code Generation Benchmarks

  • CRUST-Bench differs from code-generation benchmarks by evaluating repository-scale C-to-Rust translation rather than short, competition-style program generation.HumanEval and related benchmarks commonly use short tasks with test cases, while repository benchmarks focus on issue resolution, completion, or lifecycle stages.Source: §6 · p. 9

C-to-Rust Transpilation

  • Existing C-to-Rust benchmarks often use small or synthetic examples and provide limited correctness validation or memory-safety evaluation.CROWN includes 20 multi-file C programs but lacks structured interfaces and annotations; other work commonly uses isolated C functions.Source: §6 · pp. 9–10
  • Syntactic tools such as C2RUST often preserve compatibility through libc or #[repr(C)] but do not guarantee memory safety and frequently emit unsafe blocks.C2SAFERRUST can also fall back to unsafe code when target constraints cannot be satisfied.Source: §6 · pp. 9–10
  • Learning-based systems use approaches including LLM post-processing, property-based testing, and equivalence checking to improve or verify C-to-Rust generation.These methods complement CRUST-Bench’s emphasis on safe interfaces and test-based repository evaluation.Source: §6 · pp. 9–10

7 Conclusion

CRUST-Bench evaluates whether LLM-generated Rust follows target interfaces, compiles, and passes tests across 100 C projects. Even the strongest reported configuration succeeds on only 48% of tasks, leaving substantial room for improvement.

7 Conclusion

  • 48%: OpenAI o3 with iterative repair from compiler errors and test failures is the strongest reported approach, succeeding on fewer than half of benchmark tasks.The benchmark checks interface adherence, compilation, and provided Rust test cases.Source: §7 · p. 10

A Code Coverage Analysis

Coverage analysis on 25 buildable benchmarks finds that low coverage often reflects untested drivers and illustrative files rather than absent core functionality. The suites are not exhaustive, and substantially higher coverage was outside scope.

A Code Coverage Analysis

  • Coverage was measured with gcov on 25 benchmarks selected from projects that built with make or cmake.Some cmake projects linked external files that caused gcov failures, which were not resolved across the full benchmark.Source: A Code Coverage Analysis · p. 14
  • Low coverage commonly comes from untested driver files and illustrative examples that are not executed by the test suite.These files may invoke library functions while remaining uncovered in gcov analysis.Source: A Code Coverage Analysis · p. 14
  • The test suites are not exhaustive, but models’ tendency to succeed or fail across related functions suggests they are sufficiently representative for correctness evaluation.Localized errors were rare when several functions shared an abstract data type.Source: A Code Coverage Analysis · p. 14
  • Higher coverage would require targeted tests for every function in every benchmark, requiring detailed understanding of program intent and edge cases.This effort was outside the scope of the work.Source: A Code Coverage Analysis · p. 14

B Greedy self-repair vs sample-and-repair

The benchmark compares greedy compiler-guided self-repair with sample-and-repair under matched repair budgets, using compiler and test feedback while requiring safe, compiling Rust. Full experiments found three greedy repair rounds sufficient, with no substantial gains beyond them.

B Greedy self-repair vs sample-and-repair

  • The comparison balances greedy self-repair against sampling multiple outputs and repairing each, with both strategies receiving compiler-message feedback.The repair setup evaluates alternative ways to spend a fixed repair budget rather than comparing unequal amounts of feedback or computation.Source: B Greedy self-repair vs… · p. 14
  • Three rounds of greedy self-repair were sufficient in the full experiments, with no substantial improvements beyond the third round.The pilot compared one sampled candidate repaired repeatedly against three sampled candidates repaired once each under a six-call budget.Source: B Greedy self-repair vs… · p. 14
  • The transpilation and repair prompts require preserving interfaces and return types while producing complete, compiling Rust without unsafe code or foreign-function calls.The workflow also requires corresponding Rust files, project-local imports, and safe handling of compiler or test failures.Source: C Prompts · pp. 15–16

D Analysis of the Self-Repair Pipeline

Self-repair reduces several compiler-error categories but can introduce new failures and remains weak on missing logic, domain-specific constructs, and deeper ownership semantics. The observed failures show that incremental compiler feedback does not guarantee semantic correctness.

D Analysis of the Self-Repair Pipeline

  • Self-repair generally fails to recover unresolved functions and imports, and it increases unimplemented functions in 2% of benchmarks.The analysis attributes this weakness to editing existing code being easier for the model than synthesizing entirely new logic.Source: D Analysis of the Self-Repair Pipeline · p. 19
  • Repair can cascade across error categories, as successive fixes may resolve one issue while introducing missing methods, borrow violations, or other failures.The impcheck example progresses through omitted files, a missing Clone implementation, a missing capacity method, and ownership errors.Source: D Analysis of the Self-Repair Pipeline · pp. 19–20
  • Graph-recogniser failures persist because the model cannot satisfy custom Graph ownership constraints or incorporate a domain-specific hashing macro from compiler messages alone.This illustrates a boundary where iterative repair does not provide enough information to reconstruct project-specific semantics.Source: D Analysis of the Self-Repair Pipeline · p. 20

E Results using SWE-Agent

Pipelined SWE-agent combines initial C-to-Rust generation with an autonomous repair agent that edits repositories using compiler and test feedback. Its performance improves over single-shot execution with modest interaction but plateaus as budgets increase, because successful repairs usually occur early.

Characterization of SWE-Agent Behavior

  • SWE-agent explores source and test files, averages its first cargo build at step 12, then iteratively edits files and reruns builds until success or budget exhaustion.After a successful build, it runs cargo test and continues editing when tests fail before submitting the patch.Source: E Results using SWE-Agent · p. 20
  • At a $2 budget, the agent frequently navigates directories, rarely scrolls beyond the first 100 lines, and repeatedly invokes cargo build and cargo test regardless of success.These behaviors indicate reliance on interactive build and test feedback during repository repair.Source: E Results using SWE-Agent · p. 20

Comparison Across Cost Budgets

  • Approximately 80% of resolved test failures are addressed within SWE-agent’s first 50 steps under a $2 budget, showing early convergence when failures are recoverable.Under $4, most tasks are addressed within 40 steps, and only one task requires more than $3.5.Source: Figure 3 · p. 21, Figure 4 · p. 22
  • Even a $1 budget outperforms baseline pass@1, but performance plateaus at $2 with no observable gains between $2 and $4.The reported plateau suggests that additional budget does not improve the agent once its effective repair strategies are exhausted.Source: E Results using SWE-Agent · p. 21, Table 6 · p. 21
  • Increasing the budget from $1 to $2 substantially increases build and test invocations, whereas increasing it from $2 to $4 produces no additional successful builds.The command-frequency analysis supports the conclusion that effective repair strategies are usually exhausted within the lower budget range.Source: E Results using SWE-Agent · p. 21, Figure 5 · p. 22

Related papers

Loading 2504.15254v3…