Source-linked AI summary
MLOps-Bench: Benchmarking AI Agents on Production ML Engineering in Real Repositories
Quoc Co Tran, Hitesh Laxmichand Patel, Diwakar Mahajan, Kshitij Bakliwal, Avi Sil, Katrin Kirchhoff
TL;DR
Existing benchmarks emphasize repository issue resolution or ML experimentation, leaving safe, executable MLOps changes inside real repositories under-specified. MLOPS-BENCH evaluates repository-grounded lifecycle tasks with behavioral tests; the best system completes 31.2% of tasks, while results indicate many patches remain behaviorally incomplete.
Problem
Existing benchmarks only partially measure whether agents can make safe, executable MLOps changes inside existing ML repositories.
Source: §1 · p. 1Method
MLOPS-BENCH grounds tasks in repository lifecycle evidence and evaluates executable changes across MLOps stages using behavioral tests, oracle patches, and buggy baselines.
Source: §1 · p. 1Results
The best system completes 31.2% of evaluation tasks, and many patches make partial progress but fail at least one scoring test.
Source: §4 · pp. 8–9Takeaways & Limitations
The benchmark evaluates both functional correctness and preservation of existing behavior in production-oriented repository changes.
Source: §5 · p. 9The benchmark uses public open-source repositories and synthesized, test-oriented tasks, so it is not fully representative of industrial MLOps or every production interaction and failure mode.
Source: §6 · p. 9
Abstract
from arXiv · showhide
AI agents are increasingly expected to modify and maintain machine-learning systems in production, yet existing benchmarks primarily evaluate either competition-style ML experimentation or generic repository issue resolution. We introduce MLOps-Bench, a repository-grounded benchmark for production ML engineering. Unlike prior ML-agent benchmarks, MLOps-Bench evaluates executable lifecycle changes inside real ML repositories, spanning data pipelines, feature engineering, model training, model serving, monitoring, CI/CD and governance, and end-to-end integration. Each task is grounded in repository evidence and includes a natural-language requirement, oracle patch, plausible buggy baseline, fail-to-pass tests, pass-to-pass tests, and provenance metadata. The benchmark contains 212 validated tasks from 31 public ML repositories, with 636 fail-to-pass tests and 856 pass-to-pass tests. We evaluate six agents and find that the best system solves 31.2% of tasks while passing 68.7% of tests, showing that current agents often make partial progress but still fail production-relevant behavioral requirements. MLOps-Bench shifts agent evaluation beyond model building and issue fixing toward repository-native MLOps work where correctness depends on lifecycle reasoning, behavioral robustness, and preservation of existing system behavior.
1 Introduction
Production ML engineering requires agents to make changes that respect repository-specific lifecycle contracts, a capability existing repository-repair and ML-experimentation benchmarks only partially measure. MLOPS-BENCH grounds executable tasks in real repository evidence and tests both lifecycle requirements and preservation of existing behavior.
1 Introduction
- Existing benchmarks leave under-specified whether agents can make safe, executable MLOps changes inside existing ML repositories.Repository-repair benchmarks and data-science or ML-agent benchmarks emphasize issue resolution, analysis, experimentation, or model improvement instead.Source: §1 · p. 1
- MLOPS-BENCH defines task validity through lifecycle evidence in native repository surfaces, rather than issue availability, leaderboard performance, or generic programming difficulty.Examples include loaders, schemas, feature transforms, training configurations, serving paths, monitoring outputs, and governance metadata.Source: §1 · p. 1
- The benchmark covers six MLOps stages plus end-to-end integration, pairing repository-native requests with behavioral tests, oracle implementations, buggy baselines, and provenance metadata.These tests target production-relevant contracts and realistic failures while checking preservation of existing behavior.Source: §1 · p. 1
- The evaluation pipeline freezes a repository snapshot, extracts lifecycle evidence, creates a task and patches, generates behavioral tests, then scores agents’ patches against that snapshot.The paper organizes its evaluation around repository-native capability, stage-level difficulty, and behavioral robustness.Source: Figure 1 · p. 2, §1 · p. 2
1. A repository-grounded MLOps bench-
MLOPS-BENCH constructs repository-native tasks from public ML repositories and lifecycle evidence. Its validated task set spans MLOps stages and includes oracle and buggy patches alongside behavioral tests.
1. A repository-grounded MLOps bench-
- The generation pipeline derives executable agent tasks from 31 public ML repositories spanning a combined 20 programming languages, frameworks, and tooling stacks.The passage also begins to describe the repositories’ ML application coverage, but does not provide the complete phrase.Source: §1 · p. 2
- The benchmark contains 212 validated tasks covering six core MLOps stages and one cross-stage integration category.The stages are data pipelines, feature engineering, model training, model serving, monitoring, CI/CD and governance, and end-to-end lifecycle integration.Source: §1 · p. 3
- Each task is grounded in repository evidence, including native code paths, configuration files, tests, workflow files, artifacts, monitoring utilities, and metadata.The context-to-requirement protocol is designed to ensure each task is native to its source repository rather than a generic MLOps prompt.Source: §1 · p. 3
- For each task, the benchmark supplies an oracle patch, a plausible buggy baseline, fail-to-pass tests, and pass-to-pass tests.Across the benchmark, it includes 212 oracle patches, 212 buggy baselines, 636 fail-to-pass tests, and 856 pass-to-pass tests.Source: §1 · p. 3
5. Auditable validation and human ground-
The benchmark attaches provenance and validation metadata to each instance, and all released tasks pass validation checks. A human audit compares generated tasks with MLOps-candidate SWE-bench pull requests.
5. Auditable validation and human ground-
- All released tasks pass validation checks, and each instance carries provenance, licensing, snapshot, dependency, stage-label, requirement, oracle-rationale, and validation metadata.A human audit compared generated tasks with MLOps-candidate SWE-bench PRs.Source: §5 · p. 3
- The human audit supports constructing MLOps tasks directly from lifecycle evidence.The audit compares generated tasks with MLOps-candidate SWE-bench PRs.Source: §5 · p. 3
- The study evaluates 6 agents/models and reports task completion, test pass rate, stage-level performance, and categorized error analysis.These reports characterize agent performance on repository-native MLOps tasks.Source: §5 · p. 3
2 Related Work
Prior work supplies repository-level evaluation methods, realistic ML development settings, and accounts of lifecycle reliability, but the paper identifies no benchmark that combines these with behavioral tests for production patches. MLOPS-BENCH positions itself at that intersection.
Repository-level software engineering benchmarks
- Repository-level software benchmarks evaluate issue resolution through code changes and tests, while their semantics remain centered on issue resolution.The paper identifies SWE-bench as the closest methodological precedent because it uses the repository as the evaluation environment.Source: §2 · p. 3
Data-science and machine-learning agent benchmarks
- Data-science and ML-agent benchmarks study analysis, experimentation, or model development, whereas MLOPS-BENCH targets production-oriented lifecycle changes inside existing repositories.The cited examples include DS-1000, MLAgentBench, MLE-bench, ML-Bench, and ML-Dev-Bench.Source: §2 · p. 3
MLOps, ML reliability, and data-centric evaluation
- MLOps and reliability research describes lifecycle dependencies and recurring failures, but typically studies isolated bugs, data-quality tasks, or analysis settings rather than executable repository-native agent tasks.The cited work discusses dependencies across schemas, features, training, serving, monitoring, and governance.Source: §2 · pp. 3–4
Positioning of MLOPS-BENCH
- The paper identifies a gap in benchmarks combining repository-grounded executable evaluation, MLOps lifecycle semantics, and behavioral tests of production patches.MLOPS-BENCH addresses this gap with tasks spanning six lifecycle stages and end-to-end integration.Source: §2 · p. 4
3 Method
MLOPS-BENCH builds executable MLOps tasks from repository evidence, then validates them with oracle and buggy patches, behavioral tests, and human audits. Its 212 tasks cover six lifecycle stages and end-to-end integration across 31 selected repositories.
3 Method
- The generation pipeline turns evidence from real ML repositories into traceable tasks, then constructs oracle patches, plausible buggy baselines, and executable validation.Each accepted task is linked to repository evidence and evaluated by tests.Source: §3 · p. 4
3.1 Crawling Repositories
- The 31 selected repositories were chosen for lifecycle coverage, domain diversity, buildability, and manageable audit size, with tasks spanning several programming and configuration languages.Repositories with weak structure, limited ML-pipeline evidence, or poor fit for realistic MLOps tasks were excluded.Source: §3.1 · p. 4
3.2 Repository Grounding and Context
- Repository context bundles capture business, technical, MLOps-lifecycle, and requirement views so generated tasks reflect native components and operational behavior.Snapshots also preserve repository provenance, dependencies, and evaluation commands.Source: §3.2 · p. 4
3.3 Task Generation
- For each lifecycle stage, task generation verifies repository evidence, defines a testable stage contract, and identifies a minimal native edit surface; end-to-end tasks target cross-boundary behavior.Problem statements specify business motivation, technical context, requirements, constraints, and evaluation intent, with evidence stored for audit.Source: §3.3 · p. 5, §3.4 · p. 5
3.5 Oracle, Buggy Baseline, and Tests
- Oracle patches provide one valid implementation, while realistic buggy baselines and fail-to-pass tests target observable contract failures and pass-to-pass tests protect existing behavior.Acceptance requires the oracle to pass both test sets and the buggy baseline to fail an intended fail-to-pass test.Source: §3.5 · pp. 5–6
3.6 Validation and Evaluation
- Tasks are accepted only after grounding, stage-contract, behavioral-test, and preservation checks; evaluation uses a separate split of 29 repositories and 187 tasks.Two repositories and 25 tasks were reserved for development, and agents were scored on completion, test pass rate, runtime, patch size, changed files, and error categories.Source: §3.6 · p. 6
3.7 Dataset Statistics
- The benchmark contains 212 tasks, 636 fail-to-pass tests, and 856 pass-to-pass tests across six MLOps stages and end-to-end integration.It includes 25 development tasks and 187 evaluation tasks; five candidate tasks were rejected during quality checks.Source: §3.7 · p. 6
3.8.1 Repository Issue Resue
- In an audit, only 48% of sampled SWE-bench PRs were unanimously judged valid MLOps tasks, while all 25 generated tasks received unanimous confirmation.Separate audits found that 9 of 10 fail-to-pass suites fully captured their stated contracts, and all 10 reviewed agent patches were accepted as correct alternatives to the oracle after adjudication.Source: §3.8.1 · p. 6, §3.8.2 · p. 7, §3.8.3 · p. 7
4 Results
Across the evaluation, the best system completes 31.2% of tasks, and stage performance varies. The gap between completion and test pass rates and the failure analysis show that many patches make partial progress but miss behavioral requirements.
Results by MLOps Stage
- Monitoring/observability and end-to-end tasks show the strongest coding-agent performance, while feature engineering and serving remain difficult for several agents.Figure 2 reports task completion rates by MLOps stage.Source: Figure 2 · p. 8, §4 · p. 8
Overall Results
- The gap between task completion and test pass rate indicates that many runs make substantial partial progress but fail at least one scoring test.Task completion requires a fully correct repository state, whereas test pass rate captures partial progress.Source: §4 · pp. 7–8
Error Analysis
- Failures are dominated by behaviorally incomplete implementations; coding agents also show integration and dependency errors, while one-shot base agents fail without browsing, test execution, or repair.Base Agent Sonnet often breaks package structure and imports, whereas Base Agent GPT 5.2 often produces plausible but semantically insufficient patches.Source: §4 · pp. 8–9
Summary with respect to the research questions
- 31.2% of evaluation tasks are completed by the best system, while aggregate scores conceal variation across lifecycle stages and many patches remain behaviorally incomplete.The study reports that monitoring/observability and end-to-end tasks show the strongest coding-agent performance, while feature engineering and serving remain difficult for several agents.Source: §4 · pp. 8–9
5 Conclusion
MLOps-Bench evaluates production-oriented ML engineering through repository-grounded lifecycle tasks and executable behavioral tests. The evaluation shows that agents make progress but remain unreliable on production-relevant requirements.
5 Conclusion
- 31.2% task completion and 68.7% test pass rate were achieved by the best-performing system, with agents often missing important behavioral requirements.Stage-level and error analyses also identify incorrect functionality, missing imports, dependency issues, and incomplete handling of repository-specific contracts.Source: §5 · p. 9
- Tasks cover data pipelines, feature engineering, training, serving, monitoring, CI/CD and governance, and end-to-end MLOps integration.Each task is grounded in repository evidence and paired with executable tests, an oracle patch, a buggy baseline, and provenance metadata.Source: §5 · p. 9
- MLOps-Bench targets lifecycle work unlike Kaggle-style ML or generic repository-repair benchmarks, measuring functional correctness and preservation of existing behavior.Source: §5 · p. 9
6 Limitations
The benchmark uses public open-source repositories and synthesized tasks, so its results do not fully represent commercial deployments or naturally occurring production work. Finite tests and the task-generation process also leave limits on what the evaluation can establish.
6 Limitations
- 31 public repositories are open-source codebases, so the benchmark should not be treated as fully representative of all industrial MLOps environments.The repositories were selected for domain diversity, lifecycle evidence, and reliable freezing, building, auditing, and testing.Source: §6 · p. 9
- Tasks synthesized from repository evidence may be more localized, explicit, and test-oriented than real MLOps requests.Finite test suites approximate production correctness and cannot cover every downstream interaction, workload, data distribution, or operational failure mode.Source: §6 · p. 9
- Generation relies on one model family, creating a possible same-family advantage that subtler stylistic alignment between tests and agents cannot fully rule out.The highest-scoring system was not from the generation family, but the authors flag the remaining concern for further study.Source: §6 · p. 9
7 Ethical Considerations
The benchmark is built from public, licensed repositories and excludes private organizational information. Its intended use is controlled evaluation, not autonomous production modification.
7 Ethical Considerations
- Repository selection records provenance, source snapshots, dependencies, and license evidence, and includes only projects whose licenses permit research and benchmark use.Tasks come from lifecycle evidence in source snapshots, not private user data or confidential production incidents.Source: §7 · p. 9
- The authors exclude private information, credentials, secrets, and non-public organizational data, along with repositories raising obvious privacy, credential, or reproducibility concerns.Source: §7 · pp. 9–10
- Benchmark success is not evidence of readiness for autonomous deployment; generated patches may be incomplete or unsafe outside the harness and require human review.The benchmark is intended for controlled evaluation on frozen repositories.Source: §7 · p. 10
- Generative AI tools assisted with language editing and exposition, while human authors reviewed and approved the scientific content and final text.Source: §7 · p. 10
A Benchmark Example
The appendix illustrates an end-to-end task requiring coordinated work across repository surfaces and release stages. It also describes the stage handoffs and the repository profile and workload counts.
A Benchmark Example
- A representative end-to-end task requires coordinated changes across data, feature, model, serving, monitoring, and release-governance surfaces, rather than isolated code completion.The structured JSON-style representation mirrors the metadata-rich task records while keeping the example human-readable.Source: A Benchmark Example · pp. 11–12, Table 8 · p. 12
A.1 Example E2E MLOps Stage Flow
- Release approval requires the dataset, features, model artifact, serving manifest, monitoring contract, and rollback metadata to agree on one release contract.The end-to-end task implements an integration layer that verifies each handoff before deployment.Source: §A.1 · p. 12
B Repository Counts by Profile and Workload
- Python appears in 29 repositories, while JavaScript appears in 3 and TypeScript, C++, and C/C++ appear in 1 repository each.The workload table classifies model objectives from repository-analysis summaries and generated problem statements; model-agnostic repositories are counted as platform/template/mixed.Source: B Repository Counts by Profile and… · p. 12
B.1 Benchmark sample with Codex solutions
The appendix describes repository-grounded oracle generation and a human audit protocol for judging MLOps-task validity. A stage rubric distinguishes lifecycle work from generic ML or software issues.
C Oracle Generation Prompt
- Oracle generation uses repository-analysis metadata to produce code and an auditable grounding map, while evaluated agents receive only the task prompt and editable repository snapshot.The prompt includes problem context, repository analysis, a native repository profile, the target stage contract, and the source repository URL.Source: C Oracle Generation Prompt · p. 12
D Human Audit Annotation Protocol
- The human audit uses stage-specific accept/reject criteria, but reports one binary judgment: whether a problem statement is a valid MLOps task.Reviewers apply their data-science and MLOps judgment when a statement is ambiguous.Source: D Human Audit Annotation Protocol · p. 12, §D.2 · p. 13
D.1 What Annotators Reviewed
- Annotators reviewed 25 repurposed SWE-bench items and 25 generated items under the same validity question; generated prompts omitted pipeline metadata, and Figure 3 shows Codex pass/fail scenarios.For SWE-bench items, annotators judged the scraped statement and consulted linked issues or PRs only to clarify context, not to infer unstated implementation details.Source: §D.1 · pp. 12–13, Figure 3 · p. 13
D.2 MLOps stage Rubric
- The rubric accepts data-pipeline work involving data correctness or traceability and reusable feature work involving consistency or freshness, while excluding generic parsing and ordinary preprocessing.Source: §D.2 · p. 14
- Training qualifies when it concerns pipelines, tracking, artifacts, validation, or quality gates; training used only to reproduce a generic framework bug does not.Source: §D.2 · p. 14
- The rubric treats deployment-context inference, post-signal monitoring, and ML release controls as serving, monitoring, and CI/CD-governance work, respectively.It excludes local-example inference, training-only metrics, and ordinary CI failures unless they directly concern ML release safety.Source: §D.2 · pp. 14–15