Source-linked AI summary
Forward-Deployed Full-Stack Engineering for Autonomous Cloud MLOps
Sagar Srinivas Sakhinana, Venkataramana Runkana
TL;DR
Operationalizing machine-learning applications requires coordinating a broad MLOps lifecycle, while prior work has not fully addressed converting natural-language cloud engineering tasks into verified deployments. This paper presents an evidence-gated multi-agent framework with graph orchestration, bounded correction and recovery, and controlled execution environments. Its evaluation reports enforcement of supported transitions and termination in either verified deployment or auditable terminal failure.
Problem
MLOps requires coordinating application, pipeline, infrastructure, security, deployment, monitoring, retraining, recovery, and rollback activities, while the paper asks whether natural-language tasks can be transformed into verified operational deployments with evidence for consequential transitions.
Method
The framework uses a stateful Graph Orchestrator, specialized lifecycle agents, bounded reflection and repair, runtime adaptation and recovery, and controlled execution environments on Google Cloud.
Results
The evaluation covers repository completeness, controlled execution, evidence-gated transitions, cloud promotion, and bounded recovery, with results showing unsupported transitions are prevented and runs reach verified deployment or auditable terminal failure.
Takeaways & Limitations
Machine-checkable evidence governs lifecycle progression, while bounded recovery handles recoverable failures and auditable termination handles unresolved failures.
Takeaways & Limitations
The current evaluation focuses on a representative frontend–backend cloud application rather than a complete production-grade cloud stack.
Abstract
from arXiv · showhide
Across industries, machine-learning systems support applications ranging from prediction and anomaly detection to forecasting, optimization, and scheduling, yet operationalizing these systems requires coordinating application development, model pipelines, cloud infrastructure, security, deployment, monitoring, retraining, recovery, and rollback. We present an evidence-gated multi-agent framework for transforming a natural-language MLOps cloud engineering task into a verified repository and operational cloud deployment. The framework combines graph engineering, loop engineering, and agent harness engineering. A stateful Graph Orchestrator coordinates specialized agents for repository generation, review, execution, verification, release, and monitoring while governing workflow dependencies, evidence gates, retry bounds, recovery paths, and termination. Consequential lifecycle transitions proceed only when their required predicates are supported by verifiable execution or runtime evidence. Verification failures activate bounded reflection, repair, and re-verification, while runtime evidence of failure, drift, degradation, or policy violation can trigger bounded adaptation, recovery, or rollback. Agent harness engineering constrains repository generation, review, and repair, artifact execution, and cloud operations through controlled capabilities and isolated execution environments. We realize the framework on Google Cloud Platform and evaluate repository completeness, controlled execution, evidence-gated transitions, cloud promotion, and bounded recovery. Our experimental results show that the framework prevents unsupported lifecycle transitions and drives each run toward either a verified operational deployment or an auditable terminal failure.
1 Introduction
Autonomous cloud MLOps must coordinate a broad operational lifecycle, but prior work has not addressed transforming natural-language engineering tasks into verified deployments. The proposed framework combines graph orchestration, bounded correction and adaptation, and controlled execution to enforce evidence-gated transitions and evaluate verified outcomes.
- MLOps spans reproducible pipelines, infrastructure, testing, security, deployment, observability, retraining, recovery, auditability, and provenance across diverse applications.
- Prior work largely targets reasoning, code synthesis, repository modification, workflow optimization, or bounded task completion rather than end-to-end verified cloud deployment.
- The framework transforms a natural-language task into a deployable repository and operational deployment through evidence-gated lifecycle transitions.
- A stateful Graph Orchestrator coordinates agents for generation, review, execution, verification, release, and monitoring while managing dependencies, retries, recovery, and termination.
- Controlled VS Code and Chrome sandboxes, a gVisor-isolated GKE Sandbox, and cloud infrastructure constrain repository work, artifact execution, verification, release, monitoring, recovery, and rollback.
- The evaluation examines repository completeness, isolated execution, evidence-gated transitions, cloud promotion, and bounded recovery, assessing verified deployment or auditable terminal failure.
2 Problem Formulation
The framework models an MLOps execution as stateful lifecycle transitions that require machine-checkable evidence. Verification failures follow bounded corrective or terminal paths, while runtime problems can activate bounded adaptation, recovery, or rollback.
- Objective: The framework transforms a natural-language MLOps requirement into a verified repository and operational cloud deployment.Consequential transitions are permitted only when supported by machine-checkable evidence.
- State representation: The execution state combines lifecycle, repository, deployment, evidence, and retry-budget components.The deployment space includes ⊥ to represent that no deployment has yet been established.
- Evidence gates: Verification predicates govern repository, release, and runtime acceptability, so progression depends on verified system state rather than agent-reported completion.Evidence can include validation, tests, security and policy checks, provenance, controlled-execution traces, deployment verification, and telemetry.
- Correction and termination: Recoverable verification failures decrement transition-specific retry budgets and route execution through reflection, repair, review, execution, and fresh verification evidence.When retries are exhausted or transitions are non-retryable, the Graph Orchestrator follows the configured failure path to terminal failure.
- Runtime adaptation: The same bounded mechanism applies to runtime failure, drift, degradation, or policy violation and may trigger reflection, repair, adaptation, recovery, or rollback.Correction continues only while the relevant predicate remains unsatisfied and retry budget remains.
3 Overall Framework
The framework realizes its lifecycle as a Google Cloud multi-agent system coordinated by a Graph Orchestrator. Specialized agents progress from repository generation through review, execution, verification, staged release, live-cloud verification, promotion, and monitoring using policy-scoped capabilities.
- Agent realization: Google ADK and GKE host the Graph Orchestrator and specialized agents for generation, review, execution, verification, reflection, repair, release, and monitoring.The orchestrator coordinates them according to execution state and evidence-gated transition rules.
- Lifecycle path: The primary invocation path is generation, review, sandbox execution, repository verification, staging release, live-cloud verification, production promotion, and monitoring.The first verification and release invocations concern repository artifacts and build/staging, while the second concern live-cloud verification and production promotion.
- Agent harness: MCP mediates policy-scoped access across repository and browser sandboxes, gVisor-isolated GKE execution, and cloud deployment and release operations.These paths constrain how agents access tools and operational capabilities.
4 Experiments
The evaluation measures repository completeness, controlled execution, evidence-gated progression, cloud promotion, and bounded recovery across a 100-task cloud MLOps benchmark. Results indicate that the framework achieves strong verified deployment outcomes while supporting bounded recovery and auditable termination.
- Evaluation design: The evaluation covers five research questions spanning repository completeness, controlled artifact execution, lifecycle progression, cloud release, and bounded recovery and termination.Metrics are defined for each research question and reported on a common [0, 1] scale, with higher values indicating better performance.
- Experimental scenarios: The experimental suite includes one nominal scenario, three recoverable verification-perturbation scenarios, and one retry-budget-exhaustion scenario.All scenarios use the same execution graph, verification predicates, transition rules, retry semantics, recovery policies, and cloud configuration.
- Evaluation design: The benchmark contains 100 distinct natural-language cloud MLOps engineering tasks evaluated using a Google Cloud Platform configuration.The benchmark spans diverse datasets and applications and records verification outcomes, lifecycle transitions, recovery attempts, tool calls, token usage, execution time, retained evidence, and terminal state.
- Results and analysis: GPT-5.6 Sol achieves the highest VODR in every reported scenario, followed consistently by Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite.Each model–scenario pair comprises 100 executions; VODR is the fraction of executions whose terminal state satisfies repository, release, and runtime verification predicates.
- Results and analysis: The framework enables recovery from repository, cloud, and runtime failures while enforcing termination when recovery is exhausted.This result supports bounded autonomy by separating model-independent verification enforcement from model-dependent recovery capability.
5 Conclusion
The paper concludes that evidence-gated orchestration, bounded correction and recovery, and controlled agent execution can govern long-horizon cloud MLOps workflows. Its evaluation supports verified progression, blocked unsupported transitions, bounded re-verification, and auditable termination, while broader production coverage and generalizability remain open.
- Conclusion: The framework combines graph-controlled lifecycle orchestration, bounded correction and recovery, and controlled agent execution for autonomous cloud MLOps.The evaluation supports machine-checkable evidence as the basis for governing multi-step workflows.
- Conclusion: Verified transitions proceed, unsupported transitions are blocked, and recoverable or unresolved failures receive bounded re-verification or auditable termination.These outcomes support the framework’s central hypothesis for long-horizon MLOps workflows.
- Limitations and future work: The current evaluation focuses on a representative frontend–backend cloud application rather than a complete production-grade cloud stack.The paper identifies DNS, custom domains, external HTTPS load balancing, TLS, WAF, CDN delivery, distributed rate limiting, and broader audit analytics as future extensions.
- Limitations and future work: Further evaluation across application architectures, workloads, cloud platforms, and failure modes is needed to assess framework generalizability.This scope boundary accompanies the paper’s conclusion about the current evaluation.
6 Technical Appendix
The evaluation examines repository quality, controlled execution, evidence-gated progression, cloud promotion, recovery, termination, and benchmark scope across a 100-task cloud MLOps suite. Results indicate that the Graph Orchestrator enforces verification gates and bounded retry behavior while supporting verified deployment or auditable failure.
- 6.1 Additional Results: GPT-5.6 Sol achieves the highest RCS and RAR in all reported scenarios, while repository completeness and acceptance remain distinct properties.The larger RCS–RAR gaps for Gemini models show that artifact-category coverage does not by itself imply repository acceptance.
- 6.1 Additional Results: CER is 1.00 in all reported cases that reach controlled artifact execution through the GKE Sandbox with gVisor and retained machine-checkable evidence.
- 6.1 Additional Results: VPR and BFR are both 1.00 in all reported cases, showing that forward progression requires a satisfied verification predicate and unsatisfied predicates block progression.These outcomes support consistent enforcement of the evidence-gated transition rule by the Graph Orchestrator.
- 6.1 Additional Results: GPT-5.6 Sol achieves the highest RPSR in every reported scenario, while cloud-verification perturbations produce the largest reduction for Gemini models.RPSR is computed over executions satisfying repository verification; runtime perturbation occurs only after production promotion.
- 6.1 Additional Results: ETR is 1.00 for every model reaching the assigned retry-exhaustion target, while model capability primarily affects successful recovery within the retry budget.GPT-5.6 Sol has the highest RSR at each recoverable target, whereas Gemini models have their lowest RSR for repository-verification failures.
- 6.2 Benchmark and Experimental Setup: The benchmark contains 100 natural-language cloud MLOps tasks spanning multiple application types, each with a frontend and authenticated backend inference APIs.Tasks are evaluated unchanged across five experimental scenarios, with perturbations introduced only by the experimental procedure.
- 6.3 Ablation Studies: Evidence-Gate Bypass yields BFR, RSR, and VODR of 0.00, whereas Zero Retry Budget preserves BFR at 1.00 but yields RSR and VODR of 0.00.Without a corrective path, induced failures remain uncorrected; without retries, failures terminate at the failed gate.
- 6.3 Ablation Studies: Runtime RSR increases from 0.74 with one permitted attempt to 0.92 with five and 0.97 with twenty, with limited additional gains beyond ten attempts.VODR and RSR are non-decreasing with retry budget, and successful re-verification ends recovery before budget exhaustion.
SPIFFE / SPIRE + mTLS
The supplied passages identify service-to-service authentication and encrypted authenticated communication as security-control elements.
- SPIFFE / SPIRE + mTLS: Service-to-service authentication is identified as a security-control layer.
- SPIFFE / SPIRE + mTLS: Encrypted authenticated communication is identified as a complementary security-control layer.
RBAC + ABAC
The supplied passages identify network isolation and runtime observability with evidence derivation as components of the framework’s control architecture.
- RBAC + ABAC: Network isolation is implemented through Cilium and Kubernetes NetworkPolicy.
- RBAC + ABAC: The framework includes multi-layer zero-trust controls spanning cloud identity, workload identity, authorization, network isolation, and secret management.
- RBAC + ABAC: Runtime observability and evidence derivation support evidence-gated verification for continued operation or bounded recovery.