Source-linked AI summary
CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks
Pattaraphon Kenny Wongchamcharoen, Kris Gulati, Min Min Fong, Abhishek Nagaraj
TL;DR
Existing benchmarks emphasize automation, although models often assist other agents. CentaurBench evaluates both roles across seven work tasks and finds that their rankings differ, with assistance sometimes reducing performance.
Problem
Existing benchmarks emphasize autonomous task performance, leaving models’ ability to improve another worker’s performance undermeasured.
Method
The study compares models as autonomous solvers and assistants guiding a standardized lower-capacity worker across seven professional tasks.
Results
The top-ranked model differs between automation and augmentation on five of seven tasks, while unaided workers outrank every assisted condition on three tasks.
Takeaways & Limitations
Automation rankings are an incomplete proxy for assistance quality, supporting role-specific evaluation that considers the worker and task.
Takeaways & Limitations
The evaluation relies on LLM-judge evidence rather than definitive human or domain-expert judgments, so evaluator bias cannot be eliminated.
Abstract
from arXiv · showhide
Most LLM benchmarks rank models on their ability to automate work tasks. In practice, however, models are often used to assist other (human or LLM) agents. The question that drives model selection is therefore not only which model produces the best output, but which model most improves the work of another (weaker) agent. We introduce a unified framework that evaluates the capability of models to automate and augment another agent's performance. Across seven economically grounded real-world tasks, an assistant model writes assistance text for a standardized lower-capacity worker model, which produces the deliverable. In automation mode, the assistant produces the output directly. Outputs are scored through blind pairwise comparisons by an LLM judge panel with task-specific rubrics, replicated across ten runs. Rankings across the two regimes are only modestly correlated, and the automation winner loses augmentation on five of seven tasks. Assistance is not reliably positive. The unaided worker outranks every assisted condition on three tasks, and only one model's guidance beats no guidance on average. These results suggest that automation ability is an incomplete proxy for assistance quality, motivating benchmarks that evaluate models according to the roles they play in human-AI and multi-agent systems.
1 Introduction
CentaurBench evaluates LLMs as both autonomous task solvers and assistants across seven professional tasks, holding the downstream worker constant to isolate guidance quality. It shows that automation and augmentation rankings diverge, assistance can harm performance, and model suitability depends on the role and task.
- Motivation: Real-world LLM use often involves assisting human or weaker-LLM workers rather than producing deliverables independently.Assistants may help plan, decompose tasks, identify errors, and revise work while workers retain responsibility for the final output.
- Research gap: Existing benchmarks do not compare automation and assistance on identical work-related tasks across multiple professional domains.Prior work typically evaluates autonomous task solving or assistance within a single domain and scores the interaction itself.
- Framework: CentaurBench evaluates frontier LLMs across seven heterogeneous professional tasks in automation and augmentation modes.In augmentation, each focal model writes assistance text for a fixed worker model; task-specific LLM judges then produce task- and model-level rankings.
- Key results: 0.48 is the approximate model-level rank correlation between augmentation and automation, ranging from −0.04 for travel planning to 0.85 for tax preparation.Rankings diverge meaningfully across usage modes and task domains, and no single model dominates augmentation tasks.
- Key results: GPT-5-mini ranks first in both regimes, yet the unaided GPT-3.5-Turbo worker achieves the second-best average augmentation rank.Overly complex, poorly matched, or task-miscalibrated guidance can leave workers worse off than working alone.
- Contributions: Automation performance does not determine augmentation performance, so organizations should select models for specific roles and tasks rather than rely on one general leaderboard.The study introduces a common framework, measures guidance’s marginal value by fixing the worker, and motivates role-specific, workflow-aware benchmarks.
2 Related Literature
Prior benchmarks increasingly evaluate realistic, economically relevant work but still measure models as independent end-to-end task solvers, leaving augmentation undermeasured. Related research motivates evaluating which models provide assistance to a fixed worker across tasks and comparing their automation and augmentation roles.
- Benchmarking LLMs: Traditional and newer benchmarks standardize model comparisons and increasingly assess realistic professional work, but their unit of evaluation remains independent end-to-end task completion.Examples include broad knowledge and reasoning benchmarks, industrial workflows, occupational tasks, and economically valuable work products.
- Benchmarking LLMs: These benchmarks provide realistic measures of automation but do not directly measure the value a model creates by augmenting another party’s work.In workplace applications, models may instead help another party plan, execute, review, or revise a deliverable.
- AI-assisted work: Field experiments show that generative AI can improve worker productivity, but they usually study one tool, population, and domain rather than compare assistant models for the same worker.Their purpose is generally to estimate whether access to a particular AI system improves performance in a particular setting.
- Human–AI complementarity: Human–AI collaboration is not automatically beneficial: combinations frequently underperform the better standalone condition, with assistance depending on capability distribution, work allocation, and calibration.These findings motivate comparing which model provides assistance and how its value varies across tasks.
- Study positioning: This study addresses these gaps by placing models in an assistance role for a fixed worker and comparing the same models across automation and augmentation modes, professional domains, and tasks.The design measures the marginal value of guidance and tests whether rankings remain stable across roles.
3 Methodology
CentaurBench evaluates models both as direct task solvers and as assistants guiding a fixed lower-capability worker across seven economically meaningful tasks. Its process-oriented assistance constraint, observable rubrics, blind pairwise judging, and repeated runs separate guidance quality from direct task completion.
- Framework and usage modes: The framework compares automation, where focal models complete tasks end-to-end, with augmentation, where assistants guide a fixed worker model without writing the deliverable.In augmentation, assistance consists of plans, checklists, constraints, and self-review steps; GPT-3.5-Turbo serves as the worker.
- Judging and aggregation: Outputs undergo blind pairwise comparisons by four LLM judges using task-specific and general rubrics, with leave-one-family-out masking and ten independent pipeline runs.The procedure produces cross-mode, cross-task rankings for ten models and a profile for each model rather than a single score.
- Benchmark tasks: Seven benchmark tasks span counseling, market analysis, meal planning, operations research, tax preparation, travel planning, and tutoring, varying in structure, knowledge, risk, and human-facing judgment.Each task defines an observable deliverable with concrete requirements for objective constraint checking.
- Evaluation design: Task prompts are paired with micro-rubrics that score required and general dimensions on a 1–10 scale, including task-specific criteria such as dietary safety and mathematical correctness.Pairwise grading supports contrastive comparisons while rubrics ground judgments in observable dimensions.
- Assistance protocol: Augmentation assistants follow a universal three-phase prompt—requirements check, execution plan, and final checklist—and are explicitly prohibited from producing the final deliverable.This isolates planning and decomposition quality from whether the assistant solved the task itself.
- Reliability: 71.0% of cases had unanimous winner selection among judges across 6,265 comparisons, with higher agreement in automation (74.5%) than augmentation (67.8%).Across non-tied comparisons, the selected response received the higher mean rubric score in 99.7% of cases.
4 Results
Model performance is task-specific and mode-specific: automation rankings differ from augmentation rankings, and assistance quality is an incomplete proxy for direct-solving ability. Assistance is task-contingent, sometimes worsening the fixed worker’s output, while effective guidance adds analytical specificity and explicit structure.
- Task-specific assistance: Different tasks reward different assistant behaviors: GPT-4.1 leads counseling, GPT-O4-Mini leads market-trends analysis, GPT-5-Mini leads menu planning and tutoring, and GPT-OSS-120B leads operations research among assistants.These patterns correspond to professionally framed guidance, concise actionable synthesis, constraint tracking and structured explanation, and optimization framing with trade-off reasoning.
- Cross-mode rankings: ρ = 0.48; two-sided p = 0.187 for overall automation–augmentation rankings, indicating automation performance is an incomplete proxy for augmentation performance.Tax preparation has the strongest cross-mode correlation (ρtax = 0.85), while travel planning is essentially unrelated (ρ = −0.04).
- Cross-mode rankings: In five of seven tasks, the augmentation winner differs from the automation winner, showing that model rankings depend on the mode and task.The automation winner loses augmentation in counseling, market-trends analysis, and operations research, among other tasks.
- Augmentation outcomes: GPT-3.5-Turbo (plain) ranks first in operations research, tax preparation, and travel planning as the unaided worker baseline.Across all seven tasks, the plain baseline has a mean rank of 3.79, while GPT-5-Mini is the only assisted condition with a better overall mean rank of 3.66.
- Guidance mechanisms: Effective guidance adds analytical specificity beyond the prompt and embeds explicit structural requirements into the workflow.Examples include pairing each trend with its driver and market implication, and requiring assumptions, clarifying questions, and a named itinerary section sequence.
5 Discussion
The discussion argues that automation and augmentation are distinct, task-contingent capabilities requiring role-specific model selection and evaluation. It presents practical applications while emphasizing that the pilot provides preliminary, simulation-based evidence requiring broader human and workplace validation.
- Automation versus augmentation: The top-ranked model differs between automation and assistant-only augmentation in five of seven tasks, showing that the capabilities produce systematically different rankings.Augmentation is task-contingent: no single model dominates, and several strong direct solvers perform poorly as assistants.
- Automation versus augmentation: On operations research, tax preparation, and travel planning, unaided GPT-3.5-Turbo outranks every assisted condition.The finding indicates that assistance can be conditional and negative when guidance distracts the worker, constrains execution, or moves it away from task requirements.
- Practical applications: Teams can prioritize automation for reliable, repeatable solvers and augmentation for workflows where AI improves human output without replacing human judgment.The framework also supports assigning models to subtasks according to comparative advantage and keeping humans in the loop where assistance yields the largest gains.
- Evaluation implications: Evaluations that test only automation or augmentation are incomplete by design, so model profiles should report both axes for human-in-the-loop and multi-agent deployment.The authors propose adopting the automation-versus-augmentation distinction as a structural requirement for capability evaluation.
- Limitations and validation: The framework is an initial pilot rather than a definitive capability map or comprehensive benchmark, and LLM-judge results are scalable simulation evidence rather than definitive human judgments.Human validation remains important because evaluator bias cannot be eliminated and professional expertise is consequential for several tasks.
- Limitations and validation: Future research should test multi-turn workflows, vary downstream worker capability and model family, expand task diversity, and compare LLM-judge preferences with domain-expert assessments.These extensions address the single-assistance design, fixed GPT-3.5-Turbo worker, modest seven-task set, and uncertain correspondence between simulated and workplace value.
A Models and Evaluators
The evaluation assigns focal models dual roles: direct solvers in automation and planning assistants for a fixed worker in augmentation. Judges use a leave-family-out constraint when evaluating model outputs.
- Model roles: All focal models serve as direct solvers in automation and planning assistants in augmentation.Their roles differ by evaluation regime.
- Worker and judges: GPT-3.5-Turbo is the fixed augmentation worker, while judges are excluded from evaluating outputs from their own provider family.Model families are grouped to implement the leave-one-family-out judging rule.
B Task Prompts and Evaluation Rubrics
The section defines a standardized augmentation workflow, universal judging procedure, and task-specific prompts and rubrics spanning counseling, travel, meal, and tax planning. Assistance is process-focused, while evaluation prioritizes usefulness, accuracy, completeness, and reliability.
- B.2 Universal Judge Prompt: The judge scores both responses from 1–10, applies task-specific dimensions primarily, uses general criteria as tie-breakers, and selects the higher average overall.General criteria include instruction following, accuracy and specificity, practical usefulness, organization and readability, and tone and audience fit.
- B.3 Counseling: Counseling requires empathic, non-diagnostic support with evidence-informed framing, actionable coping and goal-setting steps, and ethical safety or escalation guidance.Its six rubric dimensions cover empathy, pattern recognition, psychological framing, actionable recommendations, ethical boundaries, and single-session usefulness.
- B.4 Travel Planning: Travel planning requires a budget-aware 5-day Tokyo itinerary for one traveler departing San Francisco, with assumptions, clarifying questions, estimates, and cost confirmation.The total budget is $2,500 USD, excluding passport costs, and the rubric assesses completeness, arithmetic realism, practicality, and itinerary quality.
- B.5 Meal Planning: Meal planning requires a cheap, simple 7-day plan tailored to specified allergies, intolerances, preferences, flavor sensitivities, snacks, groceries, and preparation needs.The rubric emphasizes dietary safety, preference fit, affordability, nutritional adequacy and variety, and simplicity and usability.
- B.6 Tax Preparation: Tax preparation asks for 2025 California return review, discrepancy identification, rule explanations, recalculations, correction forms, and an estimate or framework when information is insufficient.The scenario includes $68,500 W-2 wages, $4,200 freelance income, unavailable business expenses, $9,800 mortgage interest, and dependent-related filing issues.
- B.6 Tax Preparation: The tax case contains a $1,800 discrepancy between reported freelance income and the 1099-NEC, with no Schedule C, Schedule SE, or self-employment tax recorded.Form 1040 reports $2,400 instead of $4,200, while the prefilled return also labels a $9,800 deduction as the standard deduction.
B.9 Market Trends Analysis
The Market Trends Analysis task asks for a balanced 2026 U.S. natural-gas-market briefing with exactly three bullish and three bearish trends, explaining interacting market drivers and concluding with an investor-oriented outlook. Evaluation emphasizes completeness, economic accuracy, and causal reasoning.
- Task requirements: The task requires exactly three positive and three negative trends likely to influence U.S. natural gas prices in 2026.The trends should be presented as a client briefing.
- Task requirements: Analyses must connect supply, demand, weather, storage, LNG exports, infrastructure, and policy to explain market trends.The prompt specifically asks how these factors interact rather than treating them as isolated drivers.
- Task requirements: The response must conclude with 2–3 sentences summarizing the overall outlook for energy-sector investors or companies.This conclusion follows the bullish and bearish trend analysis.
- Evaluation rubric: Scores range from 1 to 10 across rubric dimensions including trend completeness and balance, economic and market accuracy, and causal reasoning.The rubric penalizes missing, extra, or poorly categorized trends, fabricated statistics, implausible causal logic, and isolated trend lists.
C Rank Tables With Standard Error
The rank tables report mean ranks and standard errors across ten independent runs, both by task and for models’ seven-task averages. Separate figures present these uncertainty estimates for automation and augmentation.
- Uncertainty estimation: Each task–model cell reports the mean rank across ten independent runs and its standard error across those observations.The standard error quantifies variation across the ten runs.
- Uncertainty estimation: For each model, seven-task average ranks are computed within each run before calculating standard error across the ten run-level values.This produces ten observations per model for the cross-task uncertainty estimate.
- Automation: Automation mean ranks by task are shown with standard error across ten independent runs.Figure 10 presents the automation rank tables with uncertainty estimates.
- Augmentation: Augmentation mean ranks by task are shown with standard error across ten independent runs.Figure 11 presents the augmentation rank tables with uncertainty estimates.
D Best Model by Task and Usage Mode
Table 4 identifies the best-performing model for each task and usage regime using mean rank across ten independent runs.
- Best Model by Task and Usage Mode: Table 4 reports the best-performing model by task and regime, selected by lowest mean rank across ten independent runs.The table compares models separately across tasks and usage modes.
E General Rubric Profiles
In augmentation mode, models show relatively flat average general-rubric profiles across tasks and judges. Ranking differences therefore are not driven by specialization on a single general-rubric dimension.
- General rubric profiles: Augmentation scores were averaged across tasks and judges for five general-rubric dimensions.The dimensions were instruction following, accuracy and specificity, practical usefulness, organization and readability, and tone/audience fit.
- General rubric profiles: Most models display relatively flat profiles across the general-rubric dimensions.No single listed dimension dominates the aggregated profiles for most models.
- General rubric profiles: Ranking differences are not driven by specialization on a single general-rubric axis.This conclusion follows from the relatively flat profiles observed for most models.
F Validity of LLM-as-Judge Evaluation
The LLM-as-judge procedure shows internal and directional validity, but only moderate inter-judge reliability and no substitute for human expert validation. Accordingly, the benchmark should be interpreted as a scalable simulation-based tool rather than a final measure of workplace value.
- Limitations: LLM judgments are not treated as ground truth because robustness checks do not substitute for human expert validation.The authors interpret the benchmark as a scalable simulation-based benchmark and hypothesis-generation tool, not a final human-validated measure of workplace value.
- Internal coherence: 99.7% of 15,120 non-tied comparisons selected the option with the higher average rubric score, indicating strong choice–score consistency.Agreement ranged from 98.9% to 100.0% across the four individual judges, and mean rubric scores correlated with pairwise win rates at r = 0.53 across 4,200 cells.
- Inter-judge reliability: 71.0% agreement and Krippendorff’s nominal α = 0.414 across 11,257 judge-pair decisions indicate meaningful but moderate common signal.Reliability was higher in automation (74.5%; α = 0.448) than augmentation (67.8%; α = 0.354).
- Inter-judge reliability: Krippendorff’s alpha ranged from 0.231 for operations research augmentation to 0.615 for market-trends automation, showing substantive task-mode variation.Automation produced higher agreement than augmentation overall and in every task shown.
- Criterion validity: 89% of ordered GPT-family comparisons recovered the prespecified release-order gradient, while GPT-4.1 preferred Claude-Opus-4.8 in 62.9% of Claude-family task–run comparisons.These checks provide internal directional validation but do not imply that complete automation rankings reproduce external benchmark rankings.
G Outputs for GPT-4.1 and DeepSeek-V3.1
GPT-4.1’s Rank 1 augmentation output offers empathetic validation, possible explanatory patterns, evidence-based approaches, and concrete coping steps. DeepSeek-V3.1’s Rank 10 output follows a similar structure, emphasizing emotional acknowledgment, possible causes, collaborative reframing, and professional support.
- GPT-4.1: GPT-4.1 begins by validating emotional exhaustion, work anxiety, and uncertainty, emphasizing that these feelings are understandable and shared by others.The response acknowledges the difficulty of feeling emotionally drained and says the reader is not alone.
- GPT-4.1: It presents burnout, perfectionism, avoidance, low self-efficacy, and disconnection from values as possibilities rather than diagnoses.The response explicitly frames these patterns as possibilities to consider, not labels or diagnoses.
- GPT-4.1: GPT-4.1 recommends CBT, motivational interviewing, and positive psychology, then suggests mindfulness, challenging negative self-talk, clarifying values, and setting manageable goals.The response links these approaches to reframing thoughts, exploring values, building resilience, and developing coping strategies.
- DeepSeek-V3.1: DeepSeek-V3.1 acknowledges emotional strain and uncertainty, then identifies burnout, perfectionism, avoidance, low self-efficacy, and value mismatch as possible patterns.It states that it is not diagnosing while presenting these patterns as potentially relevant to the reader’s challenges.
- DeepSeek-V3.1: DeepSeek-V3.1 combines CBT, motivational interviewing, and positive psychology with collaborative thought reframing, stress regulation, value clarification, achievable goals, and professional support.The response describes breaking overwhelming feelings into manageable steps and seeking additional guidance when distress is severe or persistent.