PaperlayerView exact revision on arXiv

Source-linked AI summary

Event-Driven ML Pipeline Orchestration for Manufacturing: An AWS Industry Experience

Zhengyang, Gu, Thomas Cook, Fredaljohn Rohrbaugh, Joseph E. Hernandez, Chris Couch

arXiv:2610.06890v1cs.LG

TL;DR

Manufacturing retraining creates bursts of long-running GPU jobs, and the paper reports operating an event-driven AWS pipeline for that workload. Its admission-controlled dispatcher buffers work and gates GPU launches; production telemetry and simulation report lower cost than always-on infrastructure and zero job loss with admission control, while testing remains limited in scale.

  • Problem

    Manufacturing ML requires continuous retraining of product-specialized models as production conditions change, creating bursts of long-running GPU workloads.

    Source: §I · p. 1

  • Method

    The system uses an admission-controlled Lambda dispatcher and durable SQS queues to coordinate ECS GPU workloads and downstream training stages.

    Source: D. Admission-Controlled Dispatcher · p. 4, E. Handler Wrapper and Message Draining · p. 5

  • Results

    The system reports 72–78% lower cost than always-on alternatives; simulation found naive dispatch lost 65% of jobs, while queue-draining median latency was 11.8 h versus 12.5 h for Step Functions.

    Source: 2) Results · p. 7, §VI · p. 9

  • Takeaways & Limitations

    For batch-oriented, GPU-intensive workloads, event-driven composition of cloud primitives can match or outperform managed ML platforms.

    Source: §VI · pp. 9–10

    The system’s MAX_WORKERS=2 setting is small, and behavior at 100–1,000 concurrent GPU tasks remains unverified.

    Source: D. Threats to Validity · p. 9

Abstract

from arXiv · show

We present an industry experience report on three years of operating an event-driven cloud infrastructure for continuous machine learning training in automotive manufacturing. Our system orchestrates GPU-accelerated training of product-specialized model pairs, a physics prediction model and a reinforcement-learning control policy, across multiple plants, coordinating long-running GPU workloads triggered by manufacturing events. The architecture combines Amazon ECS with EC2 GPU capacity providers, SQS-based messaging with dead-letter queues, and an admission-controlled Lambda dispatcher that enforces cluster concurrency limits. A Conductor orchestrator on ECS Fargate initiates dependency-aware retraining chains on a weekly schedule. The entire infrastructure is codified in modular Terraform with multi-account separation. From 40000+ production training jobs we report a 72-78% cost reduction versus always-on GPU infrastructure. A discrete-event simulation confirms that admission control is necessary (naive dispatch loses 65% of jobs) and that queue-draining matches AWS Step Functions latency while eliminating per-job startup overhead. We provide lessons learned and release the simulator and Terraform module skeletons as open-source artifacts.

I. INTRODUCTION

Manufacturing ML creates bursty demand for long-running, product-specialized GPU training, so the paper treats retraining as an event-driven batch workload rather than an always-on service. It presents an AWS pipeline for dependent model training and evaluation, built from cloud services and modular Terraform.

I. INTRODUCTION

  • Event-driven retraining addresses manufacturing workloads where product-specialized models are triggered by equipment changes, performance degradation, or data drift.A plant may require dozens of independently trained models.Source: §I · p. 1

A. Problem and Domain

  • LPP physics-model training and backtesting must pass before LCP control-policy training and backtesting can proceed to edge deployment.Durable SQS queues decouple transitions, and the LCP depends on a validated LPP.Source: A. Problem and Domain · pp. 1–2, Figure 1 · p. 2

B. AWS Building Blocks

  • The architecture combines SQS queues, Lambda dispatch, ECS Fargate and GPU-enabled EC2, EventBridge scheduling, and Auto Scaling Groups.Terraform provisions the infrastructure declaratively in modular, reusable configurations.Source: B. AWS Building Blocks · p. 2

C. Contributions

  • The paper contributes an admission-controlled dispatcher, a discrete-event simulation, an architecture comparison, and operational evidence from 40,000+ GPU training jobs.The simulation and Terraform module skeleton are released as open-source artifacts.Source: C. Contributions · p. 2

II. RELATED WORK

The paper positions its work among ML platforms, GPU scheduling, workflow orchestration, event-driven architectures, and infrastructure-as-code. It argues that bursty, GPU-intensive training calls for infrastructure that coordinates GPU capacity and avoids paying for idle operation.

A. ML Training Infrastructure

  • Existing ML platforms range from hyperscale systems and operationally demanding open-source tools to managed SageMaker Pipelines, which the paper says add 3–5 minutes of instance warm-up and pricing premiums.The paper identifies limited academic treatment of Terraform-based ML infrastructure and presents a production-derived reference architecture.Source: A. ML Training Infrastructure · p. 2, E. Infrastructure as Code for ML Systems · p. 3

B. GPU Cluster Scheduling

  • Event-driven, queue-buffered GPU provisioning is presented as a complement to workload-aware scheduling, with GPU instances present only when work is queued.The cited scheduling work reports average shared-cluster GPU utilization of 30–50%.Source: B. GPU Cluster Scheduling · pp. 2–3

C. Workflow Orchestration Platforms

  • For weekly bursts of 50–100 requests per plant separated by days of inactivity, the paper favors a serverless dispatcher over Airflow’s always-on scheduler.Airflow’s concurrency controls do not integrate with the underlying GPU infrastructure.Source: C. Workflow Orchestration Platforms · p. 3

D. Event-Driven Cloud Architectures

  • The architecture uses Lambda for admission control and dispatch while ECS supplies GPU compute for training jobs that exceed Lambda’s 15-minute limit.This is presented as a hybrid of serverless orchestration and container-based compute.Source: D. Event-Driven Cloud Architectures · p. 3

E. Infrastructure as Code for ML Systems

  • The work provides a production-derived, multi-account Terraform reference architecture for ECS clusters, SQS queues, Lambda dispatchers, and cross-account CI/CD.Terraform-based ML infrastructure has received limited academic treatment, according to the paper.Source: E. Infrastructure as Code for ML Systems · p. 3

A. Design Requirements

  • The requirements include LPP+LCP training chains lasting 1.5–3.5 hours, jobs initiated by discrete events, and facilities operating independently on their own schedules with facility-specific configurations.Source: A. Design Requirements · p. 3

B. Multi-Account AWS Topology

The system separates AWS accounts and assigns pipeline stages to compute tiers suited to their resource needs. GPU training dominates monthly cost, while non-GPU stages contribute less than five percent.

B. Multi-Account AWS Topology

  • Three AWS accounts separate tooling, customer compute, and shared services to enforce least-privilege access and contain the blast radius of misconfiguration.CodeBuild assumes a restricted deployment role in the customer compute account; encrypted Terraform state resides in shared services.Source: B. Multi-Account AWS Topology · p. 3

C. Heterogeneous Compute Tiers

  • GPU training accounts for 96% of monthly cost, while all non-GPU stages collectively contribute less than 5%.The system assigns different pipeline stages to compute tiers matched to their resource profiles, with SQS connecting the tiers.Source: C. Heterogeneous Compute Tiers · pp. 3–4

D. Admission-Controlled Dispatcher

An admission-controlled Lambda dispatcher connects SQS messages to ECS workloads and gates GPU task launches against configured capacity. It routes requests by model type and defers overflow to a task queue when the concurrency limit is reached.

D. Admission-Controlled Dispatcher

  • The dispatcher bridges SQS queues and ECS GPU or Fargate tasks, while the handler wrapper processes messages during long-running GPU tasks.The dispatcher runs in Lambda for under one second; the handler runs on GPU for hours.Source: D. Admission-Controlled Dispatcher · p. 4
  • The dispatcher routes each typed training request to the appropriate cluster, task definition, and task message queue.Requests specify plant, facility ID, request ID, and model type.Source: D. Admission-Controlled Dispatcher · p. 4
  • When active tasks reach MAX_WORKERS, the dispatcher forwards the request to a task queue; otherwise, it launches an ECS task.The source SQS message is automatically deleted after a successful Lambda return in either path.Source: D. Admission-Controlled Dispatcher · p. 4
  • The dispatcher concurrency limit and Auto Scaling Group maximum derive from one Terraform variable, preventing capacity mismatches between admission and GPU placement.The Lambda reserves concurrency of 1 to serialize admission decisions and prevent race conditions.Source: D. Admission-Controlled Dispatcher · p. 4

E. Handler Wrapper and Message Draining

The handler wrapper processes an initial work item, then drains overflow messages from a task queue while GPU tasks are running. This lets a bounded number of ECS tasks process bursts without launching a new task for every request.

E. Handler Wrapper and Message Draining

  • The wrapper first executes its launch-time work item, then polls the task queue and processes each received request like the initial item.It exits after three consecutive empty polls, or 60 seconds of silence.Source: E. Handler Wrapper and Message Draining · p. 5
  • During bursts of 30–50 requests, the dispatcher launches tasks for the first available slots and forwards subsequent messages for queue draining.The passage specifies two initial tasks in the example; later messages go to the task message queue.Source: E. Handler Wrapper and Message Draining · p. 5
  • A bounded number of GPU instances can process an unbounded stream of training requests by continuously draining the queue until it is empty.Overflow messages are processed sequentially by already-running tasks, without additional ECS task launches.Source: E. Handler Wrapper and Message Draining · p. 5
  • The handler wrapper’s queue-draining phase uses long-polling until three consecutive empty polls, corresponding to 60 seconds idle.The initial work item is supplied through CLI arguments; the dispatcher sends overflow messages to the task queue for running tasks to consume.Source: Figure 4 · p. 5, Figure 3 · p. 5, E. Handler Wrapper and Message Draining · p. 5

F. SQS Configuration and Fault Handling

The system pairs each of its eight SQS queues with a dead-letter queue and configures timeouts and retries to handle long-running jobs. Scheduled cleanup tasks classify failures, redrive transient ones, and alert on persistent ones.

F. SQS Configuration and Fault Handling

  • Each of the eight SQS queues has a paired DLQ; messages are routed there after five retries, with visibility timeouts calibrated to at least 1.5× expected maximum job duration.Training queues use 12-hour visibility timeouts; backtest and completion queues use 30-minute to 1-hour timeouts.Source: F. SQS Configuration and Fault Handling · p. 6
  • CloudWatch alarms on visible DLQ messages trigger cleanup tasks that log message and failure metadata, classify failures, redrive transient failures, or alert on persistent ones.All queues use server-side encryption and seven-day retention.Source: F. SQS Configuration and Fault Handling · p. 6
  • Active DLQ management reduces mean-time-to-resolution by 60% compared to passive monitoring alone.Cleanup tasks log failure details, classify failures, redrive transient messages, and alert human reviewers about persistent failures.Source: F. SQS Configuration and Fault Handling · p. 6

G. Conductor Orchestration

The Conductor scans each facility’s model inventory on a per-facility schedule, evaluates retraining needs, and submits requests while preserving downstream dependencies. The infrastructure uses reusable Terraform modules and a cross-account CI/CD flow; a discrete-event simulator provides controlled comparisons of dispatch strategies.

G. Conductor Orchestration

  • A Conductor run scans a facility’s model inventory, evaluates retraining needs per product and line, and submits requests; a plant run typically creates 30–50 requests within minutes.It submits LPP requests, while downstream LCP requests follow successful LPP backtests through SQS.Source: G. Conductor Orchestration · p. 6

H. CI/CD and Infrastructure as Code

  • The infrastructure is provisioned with reusable Terraform modules, and adding a facility requires only an entry in the scheduler_facilities map.Terraform state is stored in encrypted S3 with DynamoDB locking.Source: H. CI/CD and Infrastructure as Code · p. 6, Table III · p. 6
  • CodeBuild builds and pushes a CUDA-based image, then a second build assumes a cross-account role to update the Coordinator Lambda; ECS tasks pull the latest image on later launches.The role uses an external ID and is restricted to Lambda code updates and ECR reads.Source: H. CI/CD and Infrastructure as Code · p. 6, Figure 5 · p. 6

A. Controlled Micro-Benchmark

  • A discrete-event simulator compares three GPU dispatch strategies on identical workloads under controlled conditions, and the simulator and baseline implementations are open-sourced.Source: A. Controlled Micro-Benchmark · p. 6

1) Methodology:

The study simulates a 52-week bursty training workload to compare naive dispatch, Step Functions, and admission-controlled queue dispatch. Admission control avoids job loss, while queue-draining achieves similar latency without repeated per-job startup overhead.

2) Results:

  • 65% of jobs were lost by naive dispatch, while Step Functions and admission-controlled dispatch lost zero on the 2,077-request workload.The simulation attributes naive losses to ECS placement failures exhausting retries during burst arrivals; the two controlled approaches use different concurrency-limiting mechanisms.Source: Figure 6 · p. 7, 2) Results · p. 7

1) Methodology:

  • The comparison modeled naive dispatch with retries, a Step Functions Map state limited to two concurrent jobs, and an SQS-to-Lambda-to-ECS dispatcher with active-task inspection.The simulation constrained concurrent GPUs to two and included each strategy’s stated startup assumptions.Source: 1) Methodology · p. 7
  • The simulation replayed 2,077 jobs over 52 weeks, using the same fixed-seed workload for all three strategies.The workload modeled weekly arrivals of approximately 40 requests, each requiring 0.5–1.5 hours of GPU time.Source: 1) Methodology · pp. 6–7, 2) Results · p. 7

2) Results:

  • Median latencies were within 0.7 hours (11.8 h vs 12.5 h), and queue-draining eliminated repeated per-job startup costs.Step Functions incurred a 5-minute cold start per job; the admission-controlled handler processed overflow messages from within a GPU task.Source: 2) Results · p. 7, Figure 6 · p. 7

B. Architecture Comparison

The operational results compare event-driven GPU spending and utilization with an always-on baseline, alongside qualitative architecture alternatives. The production system processed over 40,000 training jobs across multiple facilities.

B. Architecture Comparison

  • SageMaker Pipelines was rejected for 3–5-minute fresh-instance startup, management fees, and container-customization constraints; Step Functions lacks cross-execution GPU concurrency limiting.The Step Functions comparison also notes a 5-minute cold start per job and pricing that scales with state transitions.Source: B. Architecture Comparison · p. 7
  • MWAA requires an always-on scheduler, while its pool-based concurrency does not bind directly to GPU capacity; EKS maintenance was considered disproportionate for a 3–4-engineer team.The cited workload has weekly bursts with days of inactivity between them.Source: B. Architecture Comparison · p. 7

V. OPERATIONAL RESULTS AND DISCUSSION

  • The system had run since 2023 across multiple automotive facilities, executing over 40,000 training jobs.Operational metrics came from AWS CloudWatch Container Insights, Cost Explorer, and SQS CloudWatch metrics.Source: §V · p. 7

A. Cost Analysis

  • $760/month with Spot pricing was the event-driven cost per facility, a 76% reduction from the approximately $3,200/month always-on baseline.Using on-demand pricing for all GPU jobs cost approximately $910/month, a 72% reduction; Spot applied only to checkpoint-enabled LPP training, while LCP remained on-demand.Source: A. Cost Analysis · pp. 7–8, Figure 7 · p. 8

B. Throughput and GPU Utilization

  • 94% burst-period GPU utilization was measured while processing bursts of 30–50 requests on a two-instance cap.Overflow messages were buffered and drained sequentially; a typical burst took approximately 15–25 hours to process.Source: B. Throughput and GPU Utilization · p. 8

C. Failure Modes and Recovery

The reported dead-letter queue rate was approximately 2%, with data quality issues the leading classified cause. The discussion also identifies boundaries in portability, scale, cost reporting, and simulator realism.

C. Failure Modes and Recovery

  • Approximately 2% of messages reached the DLQ over three years, and those messages persist for seven days rather than being lost.The retention period provides an audit trail, and cleanup tasks surface DLQ depth, message age, and failure reasons on a Grafana dashboard.Source: C. Failure Modes and Recovery · pp. 8–9
  • Data quality issues dominated DLQ messages, followed by GPU OOM and transient EC2 capacity failures.The figure classifies causes as 65% data quality, 18% GPU OOM, 12% EC2 capacity, and 5% visibility-timeout cases; OOM and capacity cases were addressed by configuration or SQS retry.Source: C. Failure Modes and Recovery · p. 8, Figure 9 · p. 9

D. Threats to Validity

  • Portability remains unvalidated: production telemetry comes from one AWS-deploying organization, despite analogous logical components in Azure and GCP.The pattern comprises a durable queue, admission-controlled dispatcher, and container-based compute.Source: D. Threats to Validity · p. 9
  • Behavior at 100–1,000 concurrent GPU tasks is unverified; the reported production limit is MAX_WORKERS=2.The micro-benchmark suggests linear scaling, but that behavior remains unvalidated at extreme scale.Source: D. Threats to Validity · p. 9
  • Cost figures come from the authors’ Cost Explorer dashboards rather than independent third-party reporting.The authors provide replication methodology but cannot share account-level financial data.Source: D. Threats to Validity · p. 9
  • The simulator omits network jitter, API rate limits, and availability-zone failures, so its numbers are relative comparisons rather than absolute predictions.Source: D. Threats to Validity · p. 9

E. Lessons Learned

Three years of operation yielded practical lessons for managing event-driven GPU training, and the reported architecture reduced costs while supporting production workloads. The authors conclude that simple cloud primitives can match or outperform managed ML platforms for batch-oriented, GPU-intensive workloads.

E. Lessons Learned

  • SQS visibility timeouts set to 1.5× the maximum expected duration balanced duplicate execution against prompt retries after failures.A too-short timeout caused mid-execution redelivery; duplicate execution wasted about $2,000 in GPU compute before detection.Source: E. Lessons Learned · p. 9
  • Lambda reserved concurrency of 1 prevents task-count races that can overprovision GPU capacity and cause placement failures.A race once launched four GPU tasks from a two-instance cluster, causing two placement failures.Source: E. Lessons Learned · p. 9
  • SQS-triggered Lambda responded in under a second while costing nothing during idle periods, reducing orchestration cost by 95% versus five-minute polling.Source: E. Lessons Learned · p. 9
  • Variable-driven Terraform facility maps reduced facility onboarding from two weeks of manual console work to four hours.Adding a facility required one map entry and terraform apply.Source: E. Lessons Learned · p. 9
  • The 2% of messages reaching DLQs exposed root causes effectively; active cleanup and visualization reduced mean time to resolution by 60%.Source: E. Lessons Learned · p. 9
  • Shared IaC variables aligned orchestration limits with infrastructure capacity, while multi-account separation contained IAM misconfigurations and prevented two potential incidents.A single Terraform variable governing worker limits and ASG maximum size eliminated a class of configuration drift; account separation limited blast radius.Source: E. Lessons Learned · p. 9

VI. CONCLUSION

  • 72–78% lower cost than always-on alternatives; simulation found 65% job loss without admission control, equivalent latency to Step Functions with queue-draining, and 94% GPU utilization during bursts.The architecture was used across automotive manufacturing plants for 40,000+ GPU training jobs.Source: §VI · p. 9
  • Event-driven composition of simple cloud primitives can match or outperform managed ML platforms when workloads are batch-oriented and GPU-intensive.The authors report this as a practitioner takeaway from the operating experience.Source: §VI · pp. 9–10
Loading 2610.06890v1…