Source-linked AI summary
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke
TL;DR
SWE-Lancer asks whether frontier language models can perform and economically match real-world freelance software engineering, where existing benchmarks provide limited evidence. It constructs and evaluates 1,488 Upwork tasks spanning implementation and management, using engineer-verified end-to-end testing and real payouts. Frontier models remain unable to solve most tasks: Claude 3.5 Sonnet resolves 26.2% of IC SWE issues and earns $208,050 on the Diamond set, while the benchmark also releases a public evaluation split.
Problem
Existing benchmarks focus largely on isolated, self-contained tasks, providing limited evidence about full-stack software engineering, management decisions, and economic value in real-world work.
Method
The paper evaluates 1,488 real Upwork jobs using IC code-patch and SWE Management proposal-selection tasks, engineer-verified end-to-end tests, and actual freelance payouts.
Results
Frontier models remain unable to solve most tasks; Claude 3.5 Sonnet resolves 26.2% of IC SWE issues and earns $208,050 on the SWE-Lancer Diamond set.
Takeaways & Limitations
SWE-Lancer maps model performance to real monetary value and supports research on the economic impact and agentic safety of automated coding models.
Takeaways & Limitations
Tasks come exclusively from Upwork and the Expensify repository, underrepresenting infrastructure engineering and potentially limiting evaluation scope.
Abstract
from arXiv · showhide
We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at \$1 million USD total in real-world payouts. SWE-Lancer encompasses both independent engineering tasks--ranging from \$50 bug fixes to \$32,000 feature implementations--and managerial tasks, where models choose between technical implementation proposals. Independent tasks are graded with end-to-end tests triple-verified by experienced software engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model performance and find that frontier models are still unable to solve the majority of tasks. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond (https://github.com/openai/SWELancer-Benchmark). By mapping model performance to monetary value, we hope SWE-Lancer enables greater research into the economic impact of AI model development.
1. Introduction
SWE-Lancer addresses the limits of isolated coding benchmarks with real-world freelance software engineering and management tasks. It evaluates frontier models using realistic workflows, end-to-end tests, and monetary payouts, finding substantial remaining difficulty.
- Benchmark and motivation: Prior benchmarks emphasize self-contained tasks, whereas real software engineering requires full-stack reasoning about codebase interactions and tradeoffs.SWE-Lancer targets user-facing product work involving broader repository context and cross-component dependencies.
- Benchmark and motivation: SWE-Lancer evaluates 1,488 Upwork jobs worth $1,000,000, combining code-patch tasks with technical-lead proposal selection.IC tasks require resolving real-world issues; SWE Manager tasks require choosing the best implementation proposal.
- Evaluation design: IC tasks use browser-automated end-to-end tests triple-verified by professional engineers to assess application behavior and resist isolated-function loopholes.The tests can exercise complete workflows, such as logging in, uploading a profile picture, and interacting with another account.
- Evaluation design: SWE-Lancer includes real market-derived payouts and management assessment based on actual freelancer proposals, extending evaluation beyond code generation.Task values reflect amounts paid to freelancers, while managers must weigh competing proposals in repository context.
- Results and release: Claude 3.5 Sonnet scores 26.2% on IC SWE tasks and 44.9% on SWE Management tasks, earning $208,050 of $500,800 on Diamond.On the full dataset, it earns over $400,000 of the possible $1,000,000.
- Results and release: SWE-Lancer Diamond releases a public $500,800 evaluation split while retaining a private holdout to reduce contamination from training or search.The full dataset is available upon request.
2. SWE-Lancer
SWE-Lancer is built from real Expensify freelance work posted on Upwork, covering both implementation and management decisions. Its construction combines market-based task values with multi-engineer review and realistic repository snapshots.
- Dataset composition: 1,488 Expensify tasks posted on Upwork are collectively valued at $1 million USD and divided into IC SWE and SWE Management categories.The benchmark includes 764 IC tasks worth $414,775 and 724 management tasks worth $585,225.
- Dataset composition: IC SWE tasks provide an issue description, pre-fix codebase checkpoint, and repair objective for implementing features or resolving bugs.The model’s solution is evaluated within the benchmark environment.
- Dataset composition: SWE Management tasks provide competing proposals, a pre-fix codebase snapshot, and the objective of selecting the best solution.The model acts as a software engineering manager using proposals from the original issue discussion.
- Pricing and evaluation: IC tasks are graded with end-to-end tests that mirror how experienced engineers would evaluate freelance work.The task prompts preserve the original Upwork issue framing.
- Pricing and evaluation: The construction pipeline selects previously solved Expensify tasks, reviews them for executability, and validates high-value tasks and test coverage with multiple engineers.One hundred professional engineers reviewed tasks, proposals, and codebases; high-value IC tasks received validation from ten experienced engineers.
3. Experiments and Results
SWE-Lancer evaluates frontier models on realistic software-engineering tasks using isolated execution and pass@1, earnings, and earn rates. Models perform substantially better on management decisions than implementation tasks, while increased attempts and reasoning effort improve performance but do not close the gap to human-valued payouts.
- Main results: All models earn well below the full $1 million USD possible payout on the full SWE-Lancer dataset.The benchmark reports total payout as an economic measure of model performance.
- Main results: All models perform better on SWE Manager than IC SWE tasks, with IC SWE pass@1 and earnings rates below 30%.SWE Manager performance is often more than double IC SWE performance.
- Increasing test-time compute: Allowing more attempts consistently increases pass rates, and o1’s six additional attempts nearly triple its percentage of solved tasks.GPT-4o with pass@6 reaches 16.5%, matching o1 with pass@1.
- Increasing test-time compute: Higher reasoning effort raises o1 pass@1 from 9.3% to 16.5%, earnings from $16K to $29K, and earn rates from 6.8% to 12.1%.The improvement is particularly evident on harder, more expensive problems.
- Evaluation setup: 1,488 tasks worth $1,000,000 comprise the full SWE-Lancer benchmark, including 764 IC SWE tasks and 724 SWE Manager tasks.The Diamond set contains 502 tasks worth $500,800.
4. Limitations
The evaluation is limited by its task sources, scope, modalities, environment, and potential contamination. These constraints narrow how broadly SWE-Lancer results should be interpreted.
- Diversity of repositories and tasks: Tasks come exclusively from Upwork and the Expensify repository, leaving infrastructure engineering and other sources underrepresented.The authors suggest incorporating other freelance sites and open-source repositories to broaden coverage.
- Scope: Freelance tasks are more self-contained than full-time engineering, and all tasks build on an established codebase.The authors therefore caution against extrapolating results beyond freelance software engineering.
- Modalities: The text-only evaluation does not measure how screenshots or videos might improve model performance.Many issues include such media, but the harness does not support multimodal inputs.
- Environments: Models cannot ask clarifying or follow-up questions, unlike real engineers.This creates an environmental difference between the benchmark and real engineering work.
- Contamination: Public GitHub issues from 2023–2024 may permit training-data contamination or direct solution lookups.The authors recommend disabling browsing or restricting accessed sites and post-hoc filtering for cheating.
5. Future Work
Future work can use SWE-Lancer to study broader societal and economic effects while extending evaluation beyond text-only inputs. The current harness prioritizes text because multimodal support is not widespread.
- Economic analysis: SWE-Lancer can support research into autonomous agents’ effects on labor markets, productivity, and AI R&D acceleration.Its economic analysis links model performance to actual payouts sent to freelancers through Upwork.
- Multimodality: Future evaluations should extend beyond text-only inputs because many Expensify issues contain videos and occasional screenshots.The current harness cannot let models view these materials or completion screenshots.
- Multimodality: The current evaluation exposes prompt and environment information through text-based trajectories because widespread multimodal support is unavailable.This was a deliberate prioritization in the present harness.
6. Conclusion
SWE-Lancer evaluates real-world freelance software engineering through complex full-stack and management tasks while mapping performance to monetary value. The benchmark is intended to support research on AI’s economic impact.
- Conclusion: SWE-Lancer evaluates model performance on real-world freelance software engineering work.The benchmark is designed around freelance engineering tasks rather than only self-contained coding problems.
- Conclusion: The benchmark maps model performance to real monetary value and evaluates complex full-stack software engineering and management skills.These design choices are presented as improvements in benchmark realism.
- Conclusion: The authors hope SWE-Lancer enables greater research into the economic impact of AI model development.The stated intended use is research into economic impact.
Impact Statement
Strong real-world software engineering models could improve productivity and access to engineering capabilities, but may also shift labor demand and create broader risks. The impact statement highlights both potential benefits and risks of advanced systems.
- Potential benefits: Strong real-world software engineering models could enhance productivity and expand access to high-quality engineering capabilities.The passage also links these benefits to reducing barriers to technological progress.
- Labor impacts: Such models could shift labor demand, especially in the short term for entry-level and freelance software engineers.The passage identifies this as a potential implication for the software industry.
- Risks: Advanced systems could carry autonomy risks in self-improvement, while automatically generated code may contain security flaws.These risks are presented as reasons that improving AI software engineering is not without risk.
A.1. Pass@1 Rate Analysis by Category
SWE-Lancer Diamond reports pass@1 rates by task nature, with categories determined through repeated review of complete GitHub issue threads and manual resolution of disagreements.
- Table 3 presents SWE-Lancer Diamond pass@1 rates categorized by the tasks’ nature of work.
- Each task category was assigned after o1 reviewed the entire GitHub issue thread three separate times.The review included all descriptions and comments for each IC SWE and SWE Management task.
- Disagreements among the three categorizations were resolved through manual assignment of the final category.
A.2. Contamination Analysis
Table 4 analyzes Diamond pass@1 rates according to when tasks were created.
- Table 4 reports Diamond pass@1 rate analysis based on task creation date.
- The analysis uses task creation date as the organizing dimension for examining Diamond pass@1 rates.
- This contamination analysis is presented through a table of Diamond pass@1 rates grouped by task creation date.
A.3. Further exploration of SWE Management Tasks
SWE Management proposals may be selected for nuanced implementation considerations, while Figure 11 illustrates the iterative process of submitting and choosing solutions to GitHub issues.
- Proposal selection criteria: SWE Management winners are often selected for nuanced reasons, including edge-case coverage or fewer external dependencies.
- Proposal selection criteria: An image-pasting proposal for iOS handled multiple clipboard formats, minimized permission prompts, and matched native iOS behavior.The proposal enabled direct pasting into the compose bar.
- Iteration process: Figure 11 illustrates the real-world iteration process for submitting and choosing solutions to GitHub issues in SWE-Lancer.
A.4. Coding and Software Engineering Evaluations Comparison Table
SWE-Lancer differs from existing coding and software-engineering benchmarks through its monetary mapping, task coverage, and engineer-authored execution-based evaluation, while related benchmark results provide comparison points.
- Benchmark comparison: SWE-Lancer maps model performance to monetary metrics, includes full-stack and management tasks, and uses engineer-handcrafted end-to-end tests.
- Benchmark comparison: Other benchmarks rely on pre-existing repository tests or tests generated by language models rather than SWE-Lancer’s engineer-handcrafted tests.
- Benchmark comparison: GPT-4o solved 12.2% of visual programming tasks on SWE-Bench Multimodal with the best scaffold and multimodal capabilities.
- Benchmark comparison: GPT-4o scored 8.0% on IC SWE Diamond tasks when it could ingest image screenshots through the user tool.
- Benchmark comparison: On SWE-Bench Verified, GPT-4o currently scores 38.8% using the best available approach.
A.5. Additional Scaffold Details
The benchmark combines curated real-world tasks, end-to-end evaluation, and agent execution constraints to assess software engineering performance. Examples show that models often identify relevant code but fail to implement comprehensive, validated fixes.
- Execution setup: Agents run in Azure virtual machines and Docker containers with 2 vCPUs, 8GB RAM, up to 100 tool calls, and a 3-hour limit.The environment uses a pre-built open-sourced image and excludes multimodal inputs; experiments generally use a single rollout except for pass@k studies.
- Task curation: 1,488 tasks were screened by experienced software engineers, with accepted issues required to be clear and previously resolved.Management tasks additionally required at least two well-defined proposals containing sufficient information about the correct fix.
- Evaluation: SWE-Lancer uses human-validated end-to-end tests rather than narrow unit tests, making evaluations more resistant to cheating.The tests emulate human-engineer workflows and are intended to reflect genuine solutions to real-world engineering challenges.
- Qualitative evaluation: Claude 3.5 Sonnet fixed the specified validation bug but failed end-to-end testing because it did not handle valid international postal codes and countries without ZIP codes.The example illustrates that resolving the immediate symptom is insufficient when broader application behavior is tested.
- Economic analysis: GPT-4o and o1 incurred API costs equal to 0.08% and 0.7% of freelancer costs, while solved-task earnings per API dollar were $109 and $20.These ratios use total API cost for cost ratio, but condition earnings on correctly solved tasks.
- Open-source model evaluation: Open-source models often failed to use or parse the user-tool, and their results may improve with a customized scaffold rather than the shared default scaffold.The preliminary analysis used one seed and the same scaffold as OpenAI and Anthropic models; observed failures included poor instruction following, shallow repository inspection, and early termination.