Source-linked AI summary
Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointing
Angshuman Chakravertty, MD Rayyan
TL;DR
Spot’s short reclaim notice can be inadequate for expensive, interruption-prone multi-node training, motivating a test of when prediction beats simple checkpointing. Argus is a Kubernetes operator evaluated on CIFAR-10 and real EKS, with controlled policy and lead-time comparisons. The results show that prediction can eliminate wasted compute with a suitable lead, but excess lead incurs overhead, while periodic checkpointing remains a strong ML-free baseline.
Problem
The paper asks when predictive checkpointing can beat the free two-minute Spot notice, whose usefulness depends on interruption frequency and whose losses can be severe for multi-node training.
Source: §1 · p. 1Method
Argus is a Kubernetes operator that combines risk-triggered checkpointing with reactive Spot-warning handling, evaluated through a real-EKS drain and controlled checkpoint-policy and lead-time experiments.
Source: §2 · p. 2, §3.2 · p. 3, §3.1 · p. 3Results
At MTBF 120 s, wasted compute was 202.3 s for no-protection, 169.4 s for reactive, 4.3 s for periodic, and 0.0 s for predictive; the lead sweep found zero waste at every tested lead, while excess lead increased overhead.
Source: §3.2 · p. 3, §3.3 · p. 3Takeaways & Limitations
A small lead can suffice for zero wasted compute, but it should stay below the interruption interval; periodic checkpointing is a strong ML-free baseline.
Source: §3.3 · p. 3, §5 · p. 5The risk model is advisory, trained on a proxy label rather than real interruptions; large-model and multi-node validation remain future work.
Source: §3.1 · p. 3, §4 · p. 4
Abstract
from arXiv · showhide
Elastic Compute Cloud (EC2) Spot is 60% to 90% cheaper than On-Demand but can be reclaimed on just a 2-minute notice; for expensive multi-node training this loss can be severe, with one reclaim costing hours of synchronous progress. We build Argus, a Kubernetes operator, and ask empirically, on a CIFAR-10 testbed, when predicting interruptions beats simple checkpointing. Argus on real EKS survives a real Spot drain with a graceful SIGTERM checkpoint, resuming from epoch 8 and losing only the in-progress epoch. Alongside, we further find that in an 80-trial benchmark, the reactive-on-notice degrades toward no protection once interruption outpaces the fixed 2-minute notice, and predictive wasted compute is driven to zero, but with an oversized fixed lead it over-migrates so severely that at the fastest rate only one of five runs completes, while periodic is a strong ML-free baseline. A lead-time sweep turns the lead prediction into a guideline where a small lead suffices for zero waste, but excess lead is wasteful. The predictor built is advisory (a proxy label); real interruption labels and large-model-scale validation are future work.
1 Introduction
Argus asks when predictive checkpointing can beat a free two-minute Spot warning, whose usefulness depends on interruption frequency and compounds with job scale. It evaluates the question with a Kubernetes operator and a controlled CIFAR-10 testbed.
1 Introduction
- A single reclaim can discard hours of synchronous multi-GPU progress, while an illustrative 16-node A100 job loses roughly $190 of compute against a checkpoint cost of cents.The study uses CIFAR-10 as a controlled testbed; it claims no large-model experiments, and multi-node validation remains future work.Source: §1 · p. 1
- At p = 0.05, a 16-node job has about a 56% per-interval chance of at least one reclaim, and fixed two-minute notice may offer no useful lead at high interruption rates.The paper asks when predictive checkpointing beats the free reactive notice and validates Argus on real EKS.Source: §1 · p. 1
- The paper contributes real-EKS interruption survival, a controlled comparison of predictive, periodic, and reactive checkpointing, and a lead-time guideline.Argus is a Kubernetes operator evaluated empirically on a controlled testbed.Source: §1 · p. 2
2 System Design
Argus combines risk prediction, Kubernetes orchestration, and checkpoint-resuming training. Predictive risk and the real Spot warning use the same interruption-handling path.
2 System Design
- Argus separates prediction, orchestration, and training: a Transformer risk model serves predictions, a Kubernetes operator polls them, and training resumes from checkpoints persisted to S3.The operator writes a flush marker and reschedules; the training pod performs the actual checkpoint flush, keeping checkpointing decoupled from training internals.Source: §2 · p. 2
- The operator leaves the checkpoint mechanism decoupled: it signals a flush rather than accessing training internals.The training pod polls the marker and writes model.pt.Source: §2 · p. 2
- A predictive risk-threshold trigger and AWS’s real two-minute Spot warning both initiate SIGTERM, checkpoint, reschedule, and resume.The reactive warning reaches the pod through the node termination handler; the pod’s handler checkpoints.Source: §2 · p. 2, Figure 1 · p. 2
3 Evaluation
Evaluation exercises graceful recovery on real EKS and compares checkpoint policies under controlled interruptions, then sweeps prediction lead time and reports the advisory risk model and operator observability.
3.1 Surviving a real Spot drain on real EKS
- On real EKS, a schema-conformant Spot warning triggered graceful SIGTERM checkpointing; the replacement resumed from epoch 8, losing only the in-progress epoch.The interruption was not an AWS-issued FIS reclaim, although the drain-to-resume path was exercised.Source: §3.1 · pp. 2–3
3.2 When does prediction pay?
- 202.3 s wasted compute for no-protection, 169.4 s for reactive, 4.3 s for periodic, and 0.0 s for predictive at MTBF 120 s; periodic wins on makespan and checkpoint count.Predictive’s zero-waste result assumes a configured lead; only one of five predictive runs completed cleanly at this rate because its 600 s lead exceeded the interruption interval.Source: §3.2 · p. 3, Table 1 · p. 3
3.3 How much lead time does prediction need? (sensitivity)
- 0.0 s wasted compute at every tested lead; at 60 s, predictive migrated about 93 times and makespan reached 218 s vs. 73 s at 2 s.The sweep used a fixed 20 s mean interruption interval; makespan crosses the roughly 78 s periodic reference around 10–15 s, supporting a lead just above checkpoint-write time and below the interval.Source: §3.3 · p. 3, Figure 2 · p. 4
3.4 The risk model (brief)
- 14.68× base-rate lift on proxy price-spike labels; the risk model is advisory and is not claimed to predict real interruptions.The proxy label is Spot price spikes greater than 1%, not actual reclaims.Source: §3.4 · p. 4
3.5 Observability
- Predicted risk crossing its threshold triggers a checkpoint, and the proactive firing was captured on the real operator against real S3/SQS.The operator emits three Prometheus metrics, with a Grafana dashboard showing the checkpoint firing.Source: §3.5 · p. 4
4 Limitations
Argus’s evidence is bounded by its injected-warning and synthetic benchmark settings, advisory proxy-label model, and lack of large-model-scale validation.
4 Limitations
- The benchmark uses a synthetic job with locally injected interruptions, and Argus has no large-model-scale validation.Both are stated as scope boundaries and future work.Source: §4 · p. 4
- The predictor is advisory and trained on a proxy label, making label quality its ceiling.Source: §4 · p. 4
- The real-EKS interruption was an injected schema-conformant warning, not an AWS-issued FIS reclaim.The account lacked the subscription needed for FIS; NTH could not distinguish the injected warning, so the drain-to-resume path was exercised.Source: §4 · p. 4
5 Related Work and Conclusion
Argus studies when predictive checkpointing beats reactive notice and periodic checkpointing, rather than proposing a new resilient-training mechanism. Its benchmark and lead-time results form a guideline, while its advisory proxy-label predictor and lack of large-scale validation bound the evidence.
5 Related Work and Conclusion
- Argus asks when predictive checkpointing beats free reactive notice, an orthogonal policy question to existing resilient-training mechanisms.Those mechanisms use elastic rescaling, pipeline templates, or redundant computation.Source: §5 · p. 4
- The 80-trial benchmark finds periodic is a strong ML-free baseline, while predictive’s zero wasted compute is clean only when its lead matches interruption rate.Source: §5 · p. 5
- A lead-time sweep indicates that a small lead can suffice for zero waste, while excess lead over-migrates.Future work includes real interruption-label capture and multi-node large-model validation; the risk model remains advisory and uses a proxy label.Source: §5 · p. 5
A Technical appendices and supplementary material
The appendix contains supporting artifacts and full benchmark data referenced in the paper’s body.
A Technical appendices and supplementary material
- The appendix holds supporting artifacts and full benchmark data referenced in the body.Source: A Technical appendices and… · p. 6
A.1 Real-EKS interruption survival
Table 2 presents the real-EKS Spot interruption timeline, while Figure 3 shows the checkpoint persisted to S3 during the drain.
A.1 Real-EKS interruption survival
- Figure 3 shows the checkpoint object persisted to the real S3 bucket during the Spot drain.Table 2 provides the full drain-to-resume timeline.Source: Figure 3 · p. 6, §A.1 · p. 6, Table 2 · p. 6
A.2 Full benchmark results
Figure 4 pools wasted compute by arm across four interruption rates, while Table 3 reports lead-time sensitivity and its periodic baseline.
A.2 Full benchmark results
- Table 3 provides the complete per-lead data behind Figure 2.Source: §A.2 · p. 6
- Table 3 reports lead-time sensitivity for the predictive arm at a 20 s mean interruption interval across five repetitions.The passage gives the periodic baseline as 4.3 s wasted, 78.1 s makespan, and 29.0 checkpoints.Source: Table 3 · p. 7
- Figure 4 shows wasted compute by arm pooled across all four tested interruption rates.The passage identifies the figure's scope but does not provide its plotted values.Source: Figure 4 · p. 7
A.3 Observability
Figure 5 presents Argus observability through a live Grafana dashboard showing a predicted-risk trigger and proactive checkpoint.
A.3 Observability
- The observability event is shown on the live Grafana dashboard.Source: §A.3 · p. 7
- Figure 5 shows predicted risk crossing the trigger threshold as a proactive checkpoint fires.Source: Figure 5 · p. 8
- The figure depicts this event on the real operator with real S3/SQS.Source: Figure 5 · p. 8