PaperlayerView exact revision on arXiv

Source-linked AI summary

Remote Labor Index: Measuring AI Automation of Remote Work

Mantas Mazeika, Alice Gatti, Cristina Menghini, Udari Madhushani Sehwag, Shivam Singhal, Yury Orlovskiy, Steven Basart, Manasi Sharma, Denis Peskoff, Elaine Lau, Jaehyuk Lim, Lachlan Carroll, Alice Blair, Vinaya Sivakumar, Sumana Basu, Brad Kenstler, Yuntao Ma, Julian Michael, Xiaoke Li, Oliver Ingebretsen, Aditya Mehta, Jean Mottola, John Teichmann, Kevin Yu, Zaina Shaik, Adam Khoja, Richard Ren, Jason Hausenloy, Long Phan, Ye Htet, Ankit Aich, Tahseen Rabbani, Vivswan Shah, Andriy Novykov, Felix Binder, Kirill Chugunov, Luis Ramirez, Matias Geralnik, Hernán Mesura, Dean Lee, Ed-Yeremai Hernandez Cardona, Annette Diamond, Summer Yue, Alexandr Wang, Bing Liu, Ernesto Hernandez, Dan Hendrycks

arXiv:2510.26787v1cs.LGcs.AIcs.CL

TL;DR

It remains unclear how AI benchmark gains translate into economically valuable remote work. The Remote Labor Index evaluates agents on real-world freelance projects, where current agents perform near the floor and the highest Automation Rate is 2.5%.

  • Problem

    It remains unclear how benchmark gains translate into economically valuable remote work, while existing benchmarks cover specialized or simple tasks rather than the diversity of remote work.

    Source: §1 · p. 1

  • Method

    The Remote Labor Index evaluates agents on economically valuable freelance projects spanning diverse remote-work categories.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across evaluated models, performance on RLI is near the floor; the highest Automation Rate is 2.5% (Manus).

    Source: §1 · p. 2, §4.2 · p. 10

  • Takeaways & Limitations

    RLI establishes an economically grounded measure for monitoring AI automation capacity in remote work.

    Source: §6 · p. 12

    RLI excludes client-interaction, teamwork, and other work types, so it does not represent several types of remote work.

    Source: §5 · p. 12

Abstract

from arXiv · show

AIs have made rapid progress on research-oriented benchmarks of knowledge and reasoning, but it remains unclear how these gains translate into economic value and automation. To measure this, we introduce the Remote Labor Index (RLI), a broadly multi-sector benchmark comprising real-world, economically valuable projects designed to evaluate end-to-end agent performance in practical settings. AI agents perform near the floor on RLI, with the highest-performing agent achieving an automation rate of 2.5%. These results help ground discussions of AI automation in empirical evidence, setting a common basis for tracking AI impacts and enabling stakeholders to proactively navigate AI-driven labor automation.

1 Introduction

The Remote Labor Index (RLI) provides a standardized empirical measure of AI’s ability to automate economically valuable remote work, intended to ground discussions of automation and help stakeholders navigate its effects.

1 Introduction

  • Existing agent benchmarks measure specialized skills or simple cross-profession tasks but capture little of remote work’s diversity and complexity, limiting insight into labor automation.Examples include software engineering [13] [18] [26], basic computer use [34] [7] [14] [17] [32], and simple tasks shared across professions [23].Source: §1 · p. 1
  • RLI evaluates agents on entire, complex projects sourced directly from online freelance platforms, spanning diverse computer-based work and grounded in real economic transactions.Projects include game development, product design, architecture, and data analysis; each includes the original work brief and a human freelancer’s gold-standard deliverable.Source: §1 · p. 2, Figure 1 · p. 2
  • 2.5% was the highest automation rate achieved by evaluated agents on RLI, and all models fell well short of the aggregate human baseline.A rigorous manual evaluation compared AI outputs against human gold-standard deliverables; Elo-based pairwise comparisons showed models steadily approaching higher automation rates across projects.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • Elo-based pairwise comparisons detect granular performance shifts across projects, even though all evaluated models remain well below the aggregate human baseline.Models are steadily approaching higher automation rates across projects.Source: §1 · p. 3

2 Related Work

Prior benchmarks have broadened from academic knowledge to interactive agent tasks, but often cover simplified or domain-specific work rather than end-to-end remote projects. RLI addresses this gap by evaluating automation on real-world projects from remote labor markets, relative to the economy’s diverse specializations and skills.

Evaluating AI agents

  • Benchmarks have expanded from closed-ended academic knowledge to agentic tasks involving computer use, web browsing, and realistic API calls.Source: §2 · p. 3

Benchmarking real-world value

  • Knowledge benchmarks at the limits of human skill are becoming saturated, while current agent benchmarks often use simplified environments covering only a small fraction of remote work.Source: §2 · p. 3
  • Patwardhan et al. [23] find near-human parity on tasks such as writing, web search, and administration, indicating augmentation potential but not measuring end-to-end project automation.Source: §2 · p. 3
  • RLI measures agents’ automation ability on end-to-end projects sourced from real-world remote labor markets, grounding evaluation in actual economic transactions.Source: §2 · p. 3
  • Unlike benchmarks of general human-level cognition representing well-educated individuals, RLI targets automation capacity across the remote labor economy’s diverse specializations and skills.Source: §2 · pp. 3–4

3 Remote Labor Index

The Remote Labor Index (RLI) is a benchmark of 240 end-to-end remote freelance projects sourced from professionals to evaluate AI agents on practical, economically valuable work.

3.1 Dataset Description

  • RLI projects combine a brief, input files, and a professional-produced gold-standard deliverable, spanning 23 Upwork work categories and a wide range of file types.The benchmark includes substantial coverage of design, operations, marketing, administration, data/BI, and audio–video production; human completion times average 28.9 hours (median 11.5), and project costs average $632.6 (median $200).Source: §3.1 · pp. 4–5, Figure 3 · p. 4

3.2 Dataset Collection

  • RLI was collected from freelancers and online examples, then reviewed, cleaned, and standardized to retain self-contained, reproducible benchmark projects.The collection included 207 projects from freelance-platform sourcing, 7 from long-tail sourcing, and 33 permitted online examples; multiple review rounds removed unsuitable projects and sought clarifications when needed.Source: §3.2 · pp. 5–6

3.3 Metrics

  • RLI reports automation rate against human deliverables, Elo for relative agent performance, and dollars earned from successfully completed projects.Automation rate is the percentage of projects where an AI deliverable is judged at least as good as the human deliverable; Elo uses pairwise preferences, while dollars earned sum the human-deliverable costs of projects the agent completes.Source: §3.3 · pp. 6–7

3.4 Evaluation

  • Evaluators manually compare AI deliverables with project requirements and human references using holistic judgments from a reasonable-client perspective.Automation-rate judgments use a three-point scale, with ratings of 2 or 3 counting as completion; agreement was 94.4% for automation rate and 56.9% for ternary Elo judgments, versus 33.0% random chance.Source: §3.4 · pp. 8–9
  • Current AI agents solve less than 3% of tasks in the RLI benchmark.The benchmark evaluates deliverables against project requirements and human gold standards.Source: §3.4 · p. 8, Table 1 · p. 9

4 Experiments

Across RLI projects, the best agent, Manus, achieved only a 2.5% Automation Rate, while Elo scores showed measurable relative progress; failures commonly involved technical, completeness, quality, and consistency problems.

4.1 Experimental Setup

  • The evaluation tested six frontier agents using computer-use or CLI scaffolds, reporting GPT-5’s better-performing CLI setup in the main tables.For models without computer-use support, the researchers used the OpenHands CLI scaffold; computer-use models used a Scale AI scaffold.Source: §4.1 · p. 9

4.2 Quantitative Results

  • Elo rankings show measurable progress between models, generally placing newer frontier models above older ones, while all remain below the human baseline of 1,000.Pairwise comparisons capture relative progress in project completion and deliverable quality even when agents do not fully complete most projects.Source: Figure 8 · p. 10, §4.2 · p. 10
  • Manus achieved the highest Automation Rate at only 2.5%, and Dollars Earned and Autoflation were also close to the floor.Agents failed to complete most projects at a level acceptable as commissioned work in a realistic freelancing environment.Source: §4.2 · p. 10

4.3 Qualitative Findings

  • Across roughly 400 evaluations, rejections clustered around technical or file-integrity, incomplete or malformed, poor-quality, and inconsistent deliverables.Examples included corrupt or empty files, truncated videos, poor professional quality, and mismatches between related files.Source: §4.3 · p. 11
  • A small subset of AI deliverables matched or exceeded human output, predominantly in creative audio and image work, writing, and data retrieval or web scraping.Successes also included report writing and code for interactive data visualization.Source: §4.3 · p. 11
  • Many failures reflected difficulty verifying work and fixing mistakes, while successes tended to occur in text-processing and image-creation tasks.Verification challenges were especially evident in complex interactive audiovisual projects such as architecture, game development, and web development; the passage relates this analysis to Hendrycks et al..Source: §4.3 · p. 11

5 Discussion

AI’s general cognitive skills may distinguish it from task-specific automation and enable automation of new jobs, while RLI is a substantial but incomplete measure of remote-economy automation. Its exclusions mean even 100% automation on RLI would not establish performance across all remote work, and reported project costs likely understate current economic value because they are not inflation-adjusted.

Discussion

  • Simple web visualizations requiring only code are within Sonnet 4.5’s capabilities, but such work makes up only a small slice of remote labor.Figure 9 shows a successful project completion; additional successes and failures appear in Figures 16 and 17.Source: Figure 9 · p. 12

Generalization to automating new jobs

  • AI is being developed to automate human intelligence and already captures substantial human-level cognitive generality, potentially enabling new-job automation [15].That potential assumes an AI can automate all current remote work without overfitting; RLI is a step toward measuring remote-economy automation broadly, rather than only current tasks.Source: §5 · p. 12

Limitations

  • RLI excludes client-interaction, teamwork, and other work types, so even a 100% automation rate may leave AI underperforming humans on unevaluated work.Although RLI is the broadest benchmark of its kind, it does not represent several types of remote work because of its requirements.Source: §5 · p. 12
  • Reported human-professional project costs are not inflation-adjusted and likely underestimate current economic value, since most known completion dates are within the past five years.The reported costs reflect prices at project completion.Source: §5 · p. 12

6 Conclusion

RLI measures AI automation capacity using economically valuable digital freelance projects; frontier agents achieve less than 3% automation, highlighting a gap between computer-use evaluations and real, economically valuable work.

6 Conclusion

  • Less than 3%: frontier AI agents’ automation rate on RLI, revealing a gap between computer-use evaluations and real, economically valuable work.The benchmark is intended to help stakeholders monitor AI capabilities, forecast labor-market impacts, and navigate AI-driven automation.Source: §6 · p. 12
  • RLI covers 240 projects across 23 domains of digital freelance work, each anchored in demonstrated market value.Source: §6 · p. 12
  • RLI aims to provide an empirical foundation for monitoring AI capabilities, forecasting labor-market impacts, and proactively navigating AI-driven automation.Source: §6 · p. 12

A Additional Results · A.1 Full Results · A.2 Autoflation

The appendix reports model-level Elo, automation, and earnings results, then defines autoflation as a way to track changes in the effective cost of RLI projects over time.

A.1 Full Results

  • Table 3 reports precise Elo scores and automation rates for all models, including GPT-5 with CLI and CUA scaffolds.The supplied passage does not provide the table’s values, so no model comparison is stated here.Source: §A.1 · p. 16
  • Table 4 reports dollars earned for all evaluated models, and current AI agents earn only a small fraction of total dataset project costs.The supplied passage does not give the table’s dollar values.Source: §A.1 · p. 16

A.2 Autoflation

  • Autoflation tracks changes over time in the cost reduction for RLI’s fixed bundle of projects, analogously to price indices for bundles of goods.Figure 10 plots the measure as new models are released.Source: §A.2 · p. 16
  • For each project, autoflation uses the cost difference from the human-produced deliverable, based on the lowest-cost AI method that achieves an acceptable deliverable.The measure is calculated project by project.Source: §A.2 · p. 16
  • A project’s cost reduction is zero when no AI method completes it at a lower effective cost than the human baseline.Source: §A.2 · p. 16
  • Because autoflation is sensitive to false-positive annotations, all AI deliverables marked successful are audited to minimize the false-positive rate.Source: §A.2 · p. 16

A.3 Effect of Agent Scaffolds

GPT-5 performed better with a CLI-based agent than with the Computer-Use Agent setup on both Elo and automation rate, suggesting current models cannot fully take advantage of computer-use environments.

A.3 Effect of Agent Scaffolds

  • GPT-5 scored 436.7 Elo and 1.7% automation with CLI, versus 431.6 Elo and 0.8% with CUA, suggesting current models cannot fully exploit computer-use environments.The authors expect more vertical integration of model scaffolds to yield stronger performance.Source: §A.3 · p. 16

B Evaluation Details · B.1 Model Details · B.2 Elo Computation

The evaluation appendix describes how model preferences were sampled, aggregated, converted into normalized Elo scores, and how the evaluated models and evaluator time limits were chosen.

B.1 Model Details

  • Evaluations used Gemini 2.5 Pro rather than Gemini 2.5 Computer Use, which struggled in the computer-use environment because it was tuned for browser-only environments.Source: §B.1 · p. 17

B.2 Elo Computation

  • Model pairs were compared on at least 10 projects each (median 25), using stratified project sampling and random pair ordering to reduce order effects.The preference data were combined with model-versus-human automation-rate evaluations.Source: §B.2 · p. 17
  • Project-level preferences used majority voting across two independent evaluations, with a third evaluation to break ties, then were averaged across projects.A split among preferences for AD-1, indifference, and AD-2 was coded as indifference; a majority for indifference was coded as 50/50.Source: §B.2 · p. 17
  • Global Bradley-Terry fitting on sampled preference edges produced utility scores called Elo scores, with 95% confidence intervals estimated from 100 project-level bootstrap samples.Bootstrap preferences were re-averaged over sampled projects; the method follows Chatbot Arena [6].Source: §B.2 · p. 17
  • Elo utilities were scaled so the human baseline scored 1,000 and a 400-point difference represented 10:1 odds of winning.Source: §B.2 · p. 17
  • Evaluators had soft time limits of 20 minutes for model-versus-human and 30 minutes for model-versus-model evaluations; majority voting required two or three evaluations per project.The stated limits were adequate for nearly all projects in preliminary testing, and total time per project was higher because multiple evaluations were performed.Source: §B.2 · p. 17, Figure 11 · p. 18

B.3 Evaluation and Generation Budgets · B.4 Evaluation Instructions

Evaluators followed time-limited workflows and judged AI deliverables against project briefs and human references using client-oriented criteria. The final instructions achieved 94.4% inter-annotator agreement.

B.3 Evaluation and Generation Budgets

  • Automation evaluations allowed at most 20 minutes per project and Elo evaluations 30 minutes, and most evaluations finished within those limits.The longer Elo limit reflected the need to inspect two AI deliverables; the limits were selected based on preliminary testing.Source: §B.3 · p. 18
  • The authors expect evaluation time per project to increase as AI deliverables become more complex and closer to solving RLI projects.Source: §B.3 · p. 18

B.4 Evaluation Instructions

  • Annotators judged deliverables holistically from a reasonable client’s perspective and treated the human reference’s minor flaws or non-critical omissions as acceptable error.The reference established the baseline quality accepted by the original client.Source: §B.4 · p. 18
  • Training highlighted common AI errors such as raster images when vector graphics were required, nonsensical image text, and visual inconsistency across files.Source: §B.4 · p. 18
  • Evaluators reviewed the brief and human reference before assessing AI deliverables, and were told to stop early and fail a project for a critical flaw making it unusable.The workflow allotted 20 minutes for Human vs. Model and 30 minutes for Model vs. Model evaluations.Source: §B.4 · pp. 18–19
  • The final evaluation instructions achieved 94.4% inter-annotator agreement after instruction iteration and annotator-quality audits.Audits targeted at least 85% agreement on a random subset of projects.Source: §B.4 · p. 19
  • Automation evaluations rated AI deliverables on a three-point scale, and projects rated 2 or 3 counted toward the automation rate.Rating 2 meant matching the reference at a quality acceptable to a reasonable client; rating 3 meant exceeding it, a distinction retained for Elo computations.Source: §B.4 · p. 19
  • Elo evaluations compared two AI deliverables on separate three-point scales for closeness to satisfying the brief and overall project quality.Both deliverables were shown alongside the human reference.Source: §B.4 · p. 19

B.5 Evaluation verification … C.3 Analysis Details

This section documents RLI evaluation checks and methods, including agent setups, platform, dataset taxonomy and filtering, and comparison analyses. It also illustrates category 3 work and describes how HCAST and Upwork task distributions are estimated, noting that the Upwork estimate is not job-level.

B.5 Evaluation verification

  • No false negatives appeared in a manual evaluation of Human vs Model pairs from 50 random projects, giving a false-negative rate of ≤5.8% with 95% confidence.Two co-authors conducted the manual evaluation.Source: §B.5 · p. 19

B.6 Agent Setup

  • Agents used integrated, computer-use, or OpenHands scaffolds, with model-specific scaffold choices and specialized tools for image, speech, and video generation.GPT-5 (CLI) was reported in the main results because it outperformed GPT-5 (CUA); computer-use-capable models defaulted to the computer-use scaffold, while Grok 4 and Gemini 2.5 Pro used OpenHands.Source: §B.6 · pp. 19–20

B.7 Evaluation Platform Details

  • The evaluation platform supports a broad range of file formats, and fewer than 20 projects had evaluator notes with project-specific guidance.The notes could, for example, tell evaluators to ignore features in a human deliverable that were outside the project brief.Source: §B.7 · pp. 21–22

C.1 Categorization

  • RLI uses the Upwork taxonomy for freelance-work coverage, and its final dataset spans 9 major categories and 23 subcategories.The authors say O*NET’s broad activity and occupational structure does not align with self-contained freelance projects.Source: §C.1 · pp. 23–24

C.2 Filtering & Cleaning Criteria

  • After cleaning and filtering, the dataset contains 240 projects from 23 Upwork subcategories.Sourcing criteria focused on remote, self-contained work that one freelancer could complete independently and that could be evaluated on the spot; the cleaning process also checked completeness, anonymization, human-deliverable quality, and renderability.Source: §C.2 · pp. 25–26

C.3 Analysis Details

  • The comparison analyses used reported completion times for GDPval and HCAST, sampled 275 Upwork jobs from 60 freelancers, and classified RLI and GDPval project briefs into three types with a judge LLM.The types were software engineering/coding, research and writing, and other.Source: §C.3 · p. 26
  • Category 3 examples include research with simulations and a report, as-built building drawings, educational videos, and video-game QA with bug reports.The QA example involves playing the game.Source: §C.3 · p. 26
  • HCAST's task distribution is manually classified from Table 1 of the HCAST paper [26].Source: §C.3 · p. 26
  • The Upwork estimate applies the prompt to taxonomy category names, yielding a work-type distribution rather than a job-level distribution, which is more skewed toward software tasks.Source: §C.3 · p. 26

C.4 Data Collection Details … C.7 Detailed Project Examples

These appendices describe project sourcing and anonymization, how project cost and completion time were measured, and examples of AI deliverables and extended project briefs.

C.4 Data Collection Details

  • Freelancer-sourced projects required authors to verify rights to sell the work, sensitive or copyrighted details were redacted or replaced with synthetic details, and long-tail works were purchased or linked with author permission.Source: §C.4 · pp. 26–27

C.5 Project cost and completion time.

  • 0.785 was the Pearson correlation between project cost and completion time on a log-log scale.The correlation is reported for projects with both values available.Source: Figure 15 · p. 27, §C.5 · p. 27
  • Project costs and completion times were generally self-reported by the professionals who created the deliverables.Cost was freelancer earnings or a professional-estimated fair price in USD; completion time was hours spent on the project.Source: §C.5 · p. 27
  • Costs were available for 95% of projects and completion times for 84%; both measures were roughly log-normal, reaching $22,500 and 450 hours, respectively.Ranges were recorded at their midpoint; 5% of projects lacked both measures but were retained for quality, and projects missing values required for an experiment or metric were dropped.Source: §C.5 · p. 27

C.6 AI Deliverable Examples

  • Manus used image-generation tools to complete some marketing projects, while Gemini 2.5 Pro and GPT-5 failed to complete the vast majority of projects in the examples.Source: Figure 16 · p. 28, Figure 17 · p. 29

C.7 Detailed Project Examples

Related papers

Loading 2510.26787v1…