Source-linked AI summary
OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration
Qirui Jin, Kenan Li, Liao Zhu, Lingching Tung, Yikai Zhang, Yijia Wu, Zijian Jin, Pengfei Gao, Yu Kang, Qingwei Lin, Wenke Lee
TL;DR
Repository-level SWE benchmarks provide limited evidence about agents’ ability to migrate existing implementations across operating systems while preserving behavior on both. OS-Bench constructs Linux–Windows migration tasks using differential execution coverage and evaluates agents on both platforms. The strongest configurations solve only 26.4% of Linux-target tasks and 20.4% of Windows-target tasks.
Problem
Repository-level SWE benchmarks usually evaluate tasks in one environment, leaving cross-OS migration with source-platform behavior preservation insufficiently tested.
Source: §1 · p. 1Method
Results
The strongest configurations solve only 26.4% of Linux-target tasks and 20.4% of Windows-target tasks when success requires correctness on both systems.
Source: §1 · p. 3Takeaways & Limitations
OS-Bench provides a testbed for measuring progress on agents’ ability to track migration scope, restore platform-specific behavior across files, and validate changes on both operating systems.
Source: §6 · p. 10The coverage-based construction pipeline cannot detect migration features implemented through cross-platform abstractions that follow the same code paths on both systems.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Language-model (LM) agents are increasingly evaluated on repository-level software engineering (SWE) benchmarks derived from GitHub issues and pull requests. Although these benchmarks cover a broad range of development tasks, they provide limited insight into a common yet technically distinct challenge: migrating an existing implementation from one operating system (OS) to another while preserving behavior on both. We introduce OS-Bench, an executable benchmark for cross-OS code migration between Linux and Windows. Motivated by case studies showing that cross-platform features are frequently implemented on one OS before support is added for another, OS-Bench identifies migration opportunities by comparing execution coverage collected from the same repository on both systems. Platform-specific regions are then verified, organized into five migration scenarios, and transformed into repository-level tasks with target-platform tests and source-platform regression tests. The resulting benchmark contains 140 validated tasks from 111 repositories across C, C++, C#, Go, and Rust, comprising 91 Windows-to-Linux and 49 Linux-to-Windows migrations. We evaluate six models with SWE-agent, Claude Code, OpenHands, and Win-agent. The best configurations achieve only 26.4% success on Linux-target tasks and 20.4% on Windows-target tasks while preserving correctness across both systems. These results indicate that cross-OS migration is a challenging and underexplored capability, and OS-Bench is a testbed for improving platform-aware SWE agents.
1 Introduction
Repository-level SWE benchmarks generally evaluate tasks in one environment, leaving agents untested on transferring behavior between OS-specific implementations while preserving source-platform functionality. OS-Bench constructs cross-OS migration tasks from differential execution coverage and finds that agents solve only a minority of tasks under two-platform correctness requirements.
1 Introduction
- Cross-platform support can require adapting system APIs, paths, process behavior, networking, encodings, toolchains, and command-line environments.The migration patch must implement target behavior while preserving source-platform functionality.Source: §1 · p. 1
- Migration histories are difficult to turn directly into coherent tasks because changes may span commits or mix migration work with unrelated edits.Tracing the relevant changes across commits requires substantial manual effort.Source: §1 · p. 2
- The strongest configurations solve only 26.4% of Linux-target tasks and 20.4% of Windows-target tasks when patches must work correctly on both systems.Performance also varies across languages, migration directions, and harnesses.Source: §1 · p. 3
- OS-Bench formulates migration as restoring missing target-platform behavior while preserving source-platform functionality.Tasks include target fail-to-pass tests and source-platform regression tests.Source: §1 · p. 3
- Differential execution coverage identifies platform-specific implementations, which are verified and removed to create tasks without relying on historical commit boundaries.The pipeline compares coverage traces from native Linux and Windows environments and retains target and regression tests.Source: §1 · pp. 2–3
- OS-Bench contains 140 validated tasks from 111 repositories across five languages and both migration directions.The benchmark includes 91 Windows-to-Linux and 49 Linux-to-Windows tasks.Source: §1 · pp. 2–3
2 Related Work
Prior repository-level benchmarks largely assess work in a single operating-system environment, while specialized benchmarks address other forms of software maintenance and evolution. OS-Bench extends this area to Linux–Windows migration and evaluates the native Windows interaction challenges faced by SWE agents.
2.1 SWE Benchmarks
- General repository-level benchmarks evaluate issue resolution or repository reconstruction, but typically run tasks in one OS environment, leaving cross-platform transfer largely untested.The SWE-bench family addresses repository-level issue resolution [2] [8], while ProgramBench evaluates repository reconstruction.Source: §2.1 · p. 3
- Specialized benchmarks cover Java-version migration [11], Linux-library compatibility across Linux versions [12], and sequences of interdependent tasks [2]; OS-Bench targets Linux–Windows migration.The paper describes OS-Bench as the first executable repository-level benchmark specifically designed for Linux–Windows code migration.Source: §2.1 · p. 3
2.2 SWE Agents
- SWE agents use tools to navigate repositories, edit code, execute commands, and iterate on test feedback.Examples include SWE-agent [3] [13], OpenHands [14], Codex, and Claude Code.Source: §2.2 · p. 4
- Most SWE agent benchmarks and interfaces remain Linux-centric, despite native Windows challenges in shell semantics, paths, filesystems, process management, compilation, and containers.OS-Bench addresses this through native environments for both platforms, a Windows-oriented harness, and counter-platform regression testing.Source: §2.2 · p. 4
3 OS-Bench
OS-Bench constructs repository-level migration tasks by identifying and masking verified OS-specific code, then checks repairs on both operating systems. Its benchmark composition spans multiple languages, migration directions, and task sizes.
3 OS-Bench
- An OS-Bench task restores removed support for a target operating system, and the submitted patch is evaluated on both target and source platforms.The paired evaluation checks target functionality and whether source-platform behavior is preserved.Source: §3 · p. 4
3.1 Task Formulation
- A migration succeeds only when target-platform repair and regression checks pass and source-platform tests remain passing; dual-platform tests cover behavior a target-only suite cannot exercise.Passing the checks establishes correctness only for behaviors covered by those tests.Source: §3.1 · p. 5
3.2 Repository and Environment Selection
- Repositories are retained when a fixed revision can be built and tested in native Linux and Windows environments, preserving platform-native shells and toolchains.The selected languages are C, C++, C#, Go, and Rust.Source: §3.2 · p. 5
3.3 Differential Coverage and Verification
- Differential coverage identifies lines executed on only one OS, and an LM verifier filters candidate regions for platform-specific behavior before retained regions form migration tasks.The verifier distinguishes platform conditionals, APIs, paths, encodings, other migration features, and false positives.Source: §3.3 · pp. 5–6
3.4 Constructing the Incomplete State
- The incomplete state masks verified target-specific regions, retains source-platform code and tests, and requires at least one target test to fail, skip, or disappear after masking.Difficulty is labeled by verified-region size: hard tasks exceed 100 estimated lines or five files; medium tasks exceed 30 lines or two files; others are easy.Source: §3.4 · p. 6
3.5 Benchmark Composition
- OS-Bench contains 140 tasks from 111 repositories; tasks average seven files and 271 estimated platform-specific lines, and difficulty counts are 44 easy, 38 medium, and 58 hard.Table 1 reports counts and per-task averages; files and estimated lines are measured over verified regions.Source: §3.5 · p. 6, Table 1 · p. 7
4 Experiments
The experiments evaluate six models across agent harnesses, using separate harness choices for Linux- and Windows-target tasks. Success requires passing target checks and preserving source-platform tests.
Models and harnesses
- For Linux-target tasks, six models are evaluated with Claude Code, OpenHands, and SWE-agent; Windows-target tasks use Win-agent because the Linux-based harnesses do not support Windows Docker containers.Win-agent runs natively on Windows with file tools, PowerShell execution, and patch submission.Source: §4 · p. 7
- Each run is limited to 100 agent steps and five hours, and instructions ask agents to fix compatibility issues and pass target regression tests.Each harness supplies its own workflow instructions.Source: §4 · p. 7
Evaluation
- The primary metric is the percentage of tasks passing both target-platform checks and source-platform preservation checks, using all tasks in the corresponding split as the denominator.Runs without a validated success count as failures.Source: §4 · p. 7
5 Results
Even the strongest configurations solve only about one quarter of Linux-target tasks and one fifth of Windows-target tasks under the two-platform success criterion. Results also vary by migration direction, language, harness, and failure pattern.
5.1 Overall Performance and Source Preservation
- 26.4% of Linux-target tasks are solved by Claude-Opus-4.8 with OpenHands, while GPT-5.6-Sol with Win-agent solves 20.4% of Windows-target tasks.Both figures require target repair and source-platform preservation; Table 2 summarizes cross-platform success by model.Source: §5.1 · p. 7, §5 · p. 7, Table 2 · p. 8
- 54.9% of GPT-5.6-Sol with Claude Code’s Linux-target patches pass Stage I, but only 24.2% pass both stages.Of the 50 target-accepted patches, 28 (56.0%) lack a validated source-platform pass.Source: §5.1 · pp. 7–8
5.2 Migration Direction and Programming Language
- The best Windows score trails the best Linux score by 6.0 percentage points, but task composition, model capability, and harness differ across platforms.Rust accounts for 18 of 24 Linux successes for Claude-Opus-4.8 with OpenHands, while Windows success is higher on C# for leading models; small C/C++ groups limit broader conclusions.Source: §5.2 · p. 8
5.3 Sensitivity to the Agent Harness
- Claude-Sonnet-5 ranges from 11.0% with SWE-agent to 23.1% with OpenHands on Linux, and no harness leads for every model.For GPT-5.6-Luna, OpenHands has higher target-only success than Claude Code, but Claude Code has higher both-platform success.Source: §5.3 · p. 9
5.4 Failure Patterns
- 76.1–82.2% of Linux tasks in each failure category remain unsolved under the combined criterion, and Windows encoding and path tasks also remain difficult.Table 3 labels overlap; for Linux path tasks, unsolved cases rise from 25 of 56 under target-only evaluation to 44 of 56 with source preservation.Source: §5.4 · p. 9, Table 3 · p. 9
- Among 35 recorded Linux Stage-I failures for GPT-5.6-Sol with Claude Code, 19 runs (54.3%) had verification that excluded tests, suites, or build modes required by the benchmark.Audit labels can overlap, and the benchmark marks runs unsuccessful when required checks do not pass.Source: Table 4 · p. 9, §5.4 · p. 10
6 Conclusion
OS-Bench provides 140 Linux–Windows migration tasks that require restoring functionality on one platform while preserving behavior on the other. The best configurations solve 26.4% of Linux-target tasks and 20.4% of Windows-target tasks, highlighting the difficulty of reliable cross-platform migration.
6 Conclusion
- OS-Bench contains 140 code-migration tasks between Linux and Windows.The benchmark is presented as a testbed for measuring progress on platform-aware software engineering agents.Source: §6 · p. 10
- 26.4% of Windows-to-Linux tasks and 20.4% of Linux-to-Windows tasks are solved by the best evaluated configurations while preserving behavior on both systems.Each task requires restoring missing functionality on one platform and preserving behavior on the other.Source: §6 · p. 10
- Reliable migration requires tracking the full migration scope, restoring platform-specific behavior across files, and validating changes on both operating systems.The paper identifies OS-Bench as a testbed for measuring progress toward these capabilities.Source: §6 · p. 10
Limitations
OS-Bench’s coverage-based construction misses migration features implemented through identical cross-platform code paths. Its evaluation and use also have boundaries around test coverage and security.
Limitations
- The pipeline cannot detect migration features implemented through cross-platform abstractions that follow the same code paths on both systems.Recovering such cases through commit history is difficult because commits often mix unrelated changes and are rarely labeled as platform migrations.Source: Limitations · p. 10
Security Considerations
- Evaluation checks target-platform repair and source-platform preservation only for behaviors covered by repository tests.Migration patches may still introduce vulnerabilities despite passing tests on both platforms.Source: Security Considerations · p. 11
Responsible AI
- Migration capabilities could be misused to adapt malicious software to additional platforms.For production use, the authors recommend human review, security checks, regression testing on both platforms, execution logs, and rollback procedures.Source: Responsible AI · p. 11
A Detailed Experiments
The experiment uses bounded agent runs in native platform environments, with migration objectives in prompts and independent source-platform preservation checks.
A.1 Budgets and Execution Environments
- Each run is capped at 100 agent steps or model calls and five hours, with SWE-agent stopping at its 100-call limit.The harness determines whether the applicable limit is steps or model calls.Source: §A.1 · p. 14
- Linux agents run in native Linux environments, while Win-agent runs in native Windows with PowerShell.Source: §A.1 · p. 14
- Prompts specify the migration direction and target-platform test objective, while the evaluator independently checks source-platform preservation.Source: §A.1 · p. 14
- The wrappers constrain test-file edits and Git use.Source: §A.1 · p. 14
A.2 Evaluation Stages
Evaluation has two stages: patches must pass required target-platform checks, then preserve the complete repository’s passing tests on the source platform.
A.2 Evaluation Stages
- Stage I applies the patch to the incomplete repository, runs native build and test commands, and records individual test results.Every required fail-to-pass and pass-to-pass check must be recorded as passing.Source: §A.2 · p. 14
- A run is unsuccessful if compilation fails, times out, omits a required test result, or fails any required test.Source: §A.2 · p. 14
- Stage II tests patches that passed Stage I on the source platform against tests passing in the complete source repository.Source: §A.2 · p. 14
- Source-platform preservation passes only when the reference test set is nonempty and every test in it still passes after applying the agent patch unchanged.Tests already failing in the complete source repository are excluded; a migration is solved only when both stages pass.Source: §A.2 · p. 14
B Agent Harness Tool Set Comparison
Table 5 compares the tool sets of the four agent harnesses under evaluation.
B Agent Harness Tool Set Comparison
- Table 5 compares the tool sets of the four agent harnesses under evaluation.Source: Table 5 · p. 14, B Agent Harness Tool Set Comparison · p. 14
C Single-Platform Success Rates
This section presents instructions and prompt materials for repository-fixing agents and verifying platform-specific migrations. It notes that success assessments may differ when source-platform preservation is required, while the repository-fixing instructions report no success-rate results.
C Single-Platform Success Rates
- Linux target-only evaluation accepted 50 of 91 GPT-5.6-Sol tasks with Claude Code (54.9%), versus 22 (24.2%) on both platforms; Windows success fell from 13 of 49 (26.5%) to ten (20.4%) with Win-agent.The Linux difference is 28 tasks (30.8 percentage points), and 56.0% of target-accepted tasks lack a validated source-platform pass; these gaps do not identify why each source evaluation failed.Source: Table 6 · p. 15, C Single-Platform Success Rates · p. 15
- For GPT-5.6-Luna on Linux, OpenHands leads Claude Code on target-only success (46.2% versus 41.8%), but trails on both-platform success (16.5% versus 17.6%).Across configurations, Linux rates range from 22.0% to 54.9% target-only and 11.0–26.4% on both platforms; Windows ranges are 8.2–26.5% and 0.0–20.4%, respectively.Source: C Single-Platform Success Rates · pp. 15–16
D Prompts
- The appendices reproduce benchmark-specific prompt templates used in the construction pipeline and evaluation harnesses.Source: D Prompts · p. 16
D.1 Platform-Specific Region Verification
- The verifier uses annotated cross-platform coverage and full-file context to classify platform-only code, including genuine dependencies and false positives.Its taxonomy distinguishes conditional code, OS-specific APIs, paths, encodings, and other migration features; it asks agents to analyze every block and identify its function and feature group.Source: §D.1 · pp. 16–17
D.3 Migration Task Instruction
- The migration-task prompt frames repository repair as moving code from its original OS to a target OS and passing target regression tests.Source: §D.3 · p. 21
D.4 SWE-agent
- The SWE-agent prompt introduces the model as a helpful assistant able to interact with a computer.Source: §D.4 · p. 22
- SWE-agent is instructed to make minimal non-test changes, run regression and reproduction checks, restore any changed tests, and avoid Git except where explicitly permitted.The prompt also says not to commit and to submit after review.Source: §D.4 · p. 22
D.5 Claude Code
- Claude Code is asked to explore source files, run regression tests, fix Linux-compatibility issues, and restore modified tests while avoiding other Git commands.The prompt also specifies that responses should contain exactly one text content and one tool call until submission.Source: §D.5 · p. 23
D.6 OpenHands
- OpenHands follows staged reading, testing, exploration, fix analysis, implementation, verification, and final review, with focused source changes and test-file restoration.Its verification instructions include reproduction scripts, edge cases, and related existing tests.Source: §D.6 · pp. 23–24
D.7 Win-agent
- Win-agent is prompted to inspect the repository, run regression tests, edit and retest the fix, remove created tests, and use Git only to restore test files.The prompt specifies a Windows PowerShell environment and prohibits using `exit $LASTEXITCODE`.Source: §D.7 · p. 24
D.2 Masking the Target Implementation
The masking procedure labels verified platform-specific lines by migration direction, then removes those lines to create incomplete repositories. Deletions are shaped to hide their locations while still causing downstream errors or regression-test failures.
D.2 Masking the Target Implementation
- Verified target-specific lines receive <linux_specific> labels for Windows-to-Linux tasks or <windows_specific> labels for Linux-to-Windows tasks.Source: §D.2 · p. 21
- Marked platform-specific lines are removed in continuous blocks and replaced with placeholders that keep files natural while obscuring the deleted locations.The procedure repeats until no marked lines remain.Source: §D.2 · p. 21
- A deletion is valid only if it causes later code to fail or makes regression tests fail, and unrelated bugs must remain untouched.The masking loop continues until no marked lines remain.Source: §D.2 · p. 21