Source-linked AI summary
ContinuumBench: Benchmarking Joint Autoscaling and Placement Across Evaluation Regimes in the Cloud-Edge Continuum
Lanpei Li, Antonino Vaccarella, Vincenzo Lomonaco, Massimo Coppola
TL;DR
Cloud-edge evaluations can obscure whether gains come from control choices or from workload, connectivity, and scoring assumptions, while completed-task metrics omit unfinished work. ContinuumBench controls these factors and finds that elastic capacity drives completion in its studied regimes, while scoring choices can change controller rankings.
Problem
Cloud-edge evaluations often leave workload, connectivity, and calibration assumptions implicit, and completion-only metrics omit unfinished tasks, complicating comparison of controller gains.
Source: §1 · p. 2Method
ContinuumBench extends ECLYPSE with runtime dynamics and compares placement-only and scale-capable controllers under declared regimes using completion-aware accounting.
Source: §1 · p. 2Results
Elastic capacity drives completion in the evaluated regimes; HPA, the joint heuristic, and KEDA reach 99–100%, 97–100%, and 89–100% of the over-provisioned reference, respectively.
Source: §7 · p. 9, §8 · p. 11Takeaways & Limitations
Once capacity suffices, autoscaling policy determines how much work arrives on time, and completion-only versus completion-aware scoring can rank controllers differently.
Source: §8 · p. 11The evaluated regimes are capacity-bound, and because heuristics never relocate, only the migrating MILP optimizer tests re-planning.
Source: §7 · p. 11
Abstract
from arXiv · showhide
Cloud-edge controllers coordinate service placement, replica scaling, and resource pre-warming to keep end-to-end latency within application deadlines. But evaluations often obscure the source of a reported gain: placement and scaling are studied separately; workload, connectivity, and calibration assumptions remain implicit; and metrics over completed tasks hide unfinished work. We present ContinuumBench, a benchmark that controls these factors. Its completion-aware accounting treats late, unfinished, and discarded tasks as deadline misses. A common protocol compares placement-only and scale-capable controllers under declared regimes and stressors. Built on the ECLYPSE simulator, ContinuumBench adds arrivals, worker elasticity, intermittent transport, buffering, and failures to close the control loop. We evaluate nine controllers across four scenarios and two regimes. The studied regimes are capacity-bound: elastic capacity, not placement sophistication, drives completion, and once capacity suffices, the choice of autoscaling policy decides how much of that work arrives on time. Placement re-planning has no measurable effect without relocation, while cost-free migration defines the observed exception. Consequently, scale-capable controllers approach an over-provisioned reference while placement-only controllers degrade with load; and placement quality separates controllers only once capacity is exhausted. Finally, the accounting choice itself changes the reported result: completion-only and completion-aware scoring can rank controllers differently.
1 Introduction
ContinuumBench studies when control sophistication improves cloud-edge resource management and how delivered work should be measured. It controls evaluation assumptions while comparing placement and scaling decisions.
1 Introduction
- Placement and scaling interact: insufficient burst capacity can defeat good placement, while scaling without placement awareness can waste scarce edge resources.Intermittent links can also make a structurally feasible path unavailable, so evaluations must track buffered and unfinished tasks.Source: §1 · p. 2
- ContinuumBench compares placement-only, scaling, and joint controllers under a common protocol with declared regimes and stressors.Its ECLYPSE-based control layer adds runtime dynamics while separating them from substrate feasibility.Source: §1 · pp. 2–3
- Completion-aware accounting counts late, unfinished, and discarded eligible tasks as deadline misses, keeping unfinished work in the score.The benchmark makes explicit how the accounting choice alone changes reported rankings.Source: §1 · p. 2
- Controlled ablations isolate re-planning schedule, elastic capacity under rising load, and placement quality under a placement bottleneck.Each lever is tested while holding the evaluation fixed.Source: §1 · p. 3
2 Related Work
Prior work addresses continuum control, simulators, and systems benchmarking, but does not combine a controlled placement-and-autoscaling protocol with completion-aware scoring. ContinuumBench adds that evaluation layer to ECLYPSE.
2 Related Work
- Joint autoscaling-and-placement methods and ProScale report improvements under study-specific graphs, traces, networks, and metrics, without isolating control-lever gains from evaluation setup.ContinuumBench holds workload, regime, stressors, and completion-aware metric fixed while comparing placement-only, fixed-placement scaling, and joint control.Source: §2 · p. 3
- ECLYPSE models feasibility, residual resources, and routing, while ContinuumBench adds the controller and evaluation protocol left to simulator users.iFogSim and faas-sim are also described as substrate simulators.Source: §2 · p. 3
- TailBench and DeathStarBench benchmark latency or microservice graphs, but report latency over completed requests rather than unfinished and discarded work.ContinuumBench varies controllers under declared regimes and stressors and counts unfinished and discarded tasks as deadline misses.Source: §2 · p. 3
3 ContinuumBench Architecture
ContinuumBench adds an epoch-based control and evaluation loop to ECLYPSE, combining controller actions with workload, transport, and worker dynamics. Its design distinguishes placement feasibility from runtime reachability.
3 ContinuumBench Architecture
- Each ContinuumBench run specifies six components: application and infrastructure graphs, workload process, controller, execution regime, and analysis configuration.The application is a DAG of stages with placement constraints and resource demands; the infrastructure supplies tiered nodes and links.Source: §3 · pp. 3–4
- ECLYPSE provides placement-feasibility checks, residual-resource state, and routing; ContinuumBench adds arrivals, time-varying contacts, and worker scaling.The epoch loop updates links and failures, applies controller actions, then flushes buffers, admits and processes tasks, and records outcomes.Source: Figure 1 · p. 4, §3 · p. 5
- At each epoch, controllers choose stage placement, active worker counts, or both, subject to node capacity, tier anchoring, and ECLYPSE feasibility checks.Routing is fixed across controllers, and the objective is completion-aware on-time delivery.Source: §3 · p. 4
- Scaling changes worker activation within bounded pools, with configurable startup delay; the heuristics place new workers but do not relocate existing ones.Only the exact optimizer migrates, and migration downtime is set to zero to test relocation under favorable conditions.Source: §3 · p. 5
- Structural feasibility and transport availability are separate: a link outage can delay, buffer, or drop work without invalidating the active mapping.The mapping stays fixed while buffered work awaits store-and-forward delivery.Source: §3 · p. 5
4 Scenarios and Workloads
ContinuumBench defines four scenario families around source–transform–sink workflows, each stressing a distinct cloud-edge resource-management mechanism. Seeded arrivals and a configurable tiered infrastructure support repeatable evaluation.
4 Scenarios and Workloads
- The Earth-observation scenario combines anchored space/IoT sources, a fast path, and an elastic bulk path to stress contact windows and store-and-forward buffering [7].Its workflow distinguishes latency-sensitive paths, elastic paths, and intermittent-contact boundaries.Source: §4 · p. 5, Figure 2 · p. 6
- The V2X scenario uses an anchored vehicle source, critical planner path, and elastic offload pool to stress source-to-processing reachability [9].Its workflow includes latency-sensitive and elastic paths.Source: §4 · p. 5, Figure 2 · p. 6
- Split inference pairs an anchored camera source and edge-hosted early-exit path with an optional elastic refinement pool, stressing placement versus worker activation.Its workflow includes stages whose placement is fixed and paths controlled by placement and scaling.Source: §4 · p. 5, Figure 2 · p. 6
- Industrial IoT combines an anchored sensor, deadline-sensitive anomaly path, and elastic archival path to stress mixed-criticality SLO separation.The four scenario families use source–transform–sink structures to isolate different mechanisms.Source: §4 · pp. 5–6
- Seeded Poisson arrivals create repeatable low-load, burst, and recovery phases, while contact-trace variants affect intermittent boundaries.Each run has 120 workload epochs and 60 arrival-free drain epochs; the default tiered infrastructure defines R1.Source: §4 · p. 6
5 Evaluation Regimes, Stressors and Metrics
ContinuumBench declares regimes and stressors, then reports both completed-work metrics and completion-aware deadline misses. R1 and R2 vary calibrated inputs while keeping the scenario DAG, topology, capacities, and stage demands fixed.
5 Evaluation Regimes, Stressors and Metrics
- R2 changes up to four calibrated inputs from R1, while the DAG, topology, capacities, and stage demands remain identical across regimes.Table 1 defines scenario scale and per-regime workload, including stage count, node split, worker-pool cap, and expected arrivals.Source: §5 · p. 6, Table 1 · p. 7
- Stress tests vary arrival load, contact severity, or one-epoch node unavailability, with failures independently redrawn at probabilities 0.03 or 0.08 per epoch.Representative space–edge and edge–cloud profiles are used rather than reproducing measurements; IoT–edge remains synthetic.Source: §5 · p. 7
- The pipeline reports delivered, pending, and failed work, latency, and SLO violations at epoch and run granularity.Pending work is max(0, G − C − F), and failed work combines buffer drops, buffer expirations, and node-failure losses.Source: §5 · p. 7
- Completion-aware SLO scoring counts eligible late, unfinished, and discarded tasks as misses, while completed-task p95 latency excludes unfinished work.Tasks whose deadlines have not elapsed at run end remain outside the score; the 60-epoch drain reduces this boundary set.Source: §5 · p. 7
- For each scenario–regime pair, the benchmark selects the best placement-only and scale-capable representatives using a completion-aware score with secondary latency and migration penalties.Perturbing the four score weights across an 81-point grid changes the selected representative in one of eight contexts, but not the family-level comparison.Source: §5 · pp. 7–8
6 Experimental Setting
The main experiment compares nine controllers across four scenarios and two regimes. Separate sensitivity experiments vary one stressor at a time, and results are reported across seeds with multiple performance measures.
6 Experimental Setting
- The evaluation includes four placement-only and five scale-capable controllers, including Kubernetes HPA and KEDA and a MILP controller.Scale-capable controllers place newly activated workers with min-latency, except Meta-rule, which selects its own placement; startup and migration delays are zero.Source: §6 · p. 8
- The main experiment evaluates all nine controllers on four scenarios under R1 and R2 with ten seeds per configuration.Sensitivity experiments vary one stressor at a time and evaluate six controllers over the same horizon with three seeds each.Source: §6 · p. 8
- Results report means and standard deviations across seeds, compare family representatives seed by seed, and interpret ranks alongside completion, SLO violation, and p95 latency.Source: §6 · p. 8
7 Results
Across the evaluated scenario–regime pairs, scale-capable controllers match or outperform placement-only controllers, while the results show placement quality matters chiefly when capacity or placement becomes a bottleneck. Re-planning without relocation has no measurable effect, and scoring unfinished work as deadline misses can change controller rankings.
7 Results
- Without relocation, heuristic re-planning changes completion by less than 10^-3; a zero-delay-migration MILP increases completion by 0.045.The MILP performs approximately 55 worker-relocation events per run, while worker capacity binds in every non-migrating schedule.Source: §7 · pp. 9, 11
- In a 480-run R1 capacity sweep (arrival rates 0.6×–2.0×), HPA reaches 99–100% of an over-provisioned reference; placement-only controllers reach 44–100% and degrade with load except EO.Across all 14 non-saturated scenario–load combinations, every scale-capable controller outperforms placement-only control. The reference activates all workers at t = 0 and is not a proven completion optimum.Source: §7 · p. 9
- In all eight scenario–regime pairs, scale-capable control achieves equal or higher completion, lower SLO violation, and lower p95 latency than placement-only control.The SLO ordering holds in all 80 seed-level comparisons; completion ordering holds in all but one saturated Industrial IoT R1 seed, where mean completion ties at 1.00.Source: §7 · p. 8, Table 3 · p. 10
- Under a tight placement bottleneck, Cost-greedy completes 102% as many tasks as the best on-time strategy but delivers only 6% on time versus 47%.Completion-only scoring reports a near-tie despite an approximately 8× on-time-delivery gap; the six strategies’ on-time spread is 0.42 in this setting, 0.12 with settings relaxed, and 0.00 in unmodified EO.Source: §7 · p. 10
- Scoring violations over completed tasks alone makes them decrease as failures rise in 21 of 24 configurations; counting unfinished tasks removes the inversion in all but one.Under R2 in Industrial IoT, HPA and KEDA both complete all tasks, yet their completion-aware violation rates are 0.01 and 0.57.Source: §7 · p. 10
- Contact stress changes 3 of 6 controller ranks in EO and 4 of 6 in Driving V2X, while load stress changes 3 of 6 in Industrial IoT.Split inference remains stable under the targeted stressors.Source: §7 · p. 10
- At 4–6 nodes, p95 controller decision time stays below 3 ms per epoch, except for the MILP reference at 8–25 ms.On the 48-node re-planning variant, per-epoch routing raises even HPA’s p95 to 181 ms.Source: §7 · pp. 10–11
- For the capacity-bound regimes studied, the results suggest provisioning elastic capacity first; the simulation-only findings remain conditional on ECLYPSE and representative, externally unvalidated scenarios and calibration profiles.The simulator omits transport contention, container scheduling, and handover dynamics, and the infrastructures are hand-built graphs.Source: §7 · p. 11
8 Conclusions and Future Work
ContinuumBench finds that elastic capacity drives completion in the evaluated regimes, while autoscaling policy affects how much work arrives on time once capacity suffices. Future work will test costs, learning-based controllers, larger networks, and synthetic topologies.
8 Conclusions and Future Work
- In the evaluated regimes, elastic capacity drives completion, autoscaling policy affects on-time delivery once capacity suffices, and scoring can rank controllers differently.Completion-only and completion-aware scoring can produce different rankings.Source: §8 · p. 11
- An explicit cost model will test whether penalizing over-provisioning makes anticipatory control advantageous.Source: §8 · p. 11
- Adding learning-based methods would extend a controller set that currently includes production policies and an exact optimizer.The benchmark logs per-epoch observations, actions, and seeds for these methods.Source: §8 · p. 11
- Scaling to thousand-node networks requires reducing per-epoch routing overhead through path caching, candidate pruning, and replacing the exact MILP reference.Source: §8 · p. 11
- Synthetic topologies, including sparser connectivity, will test whether the capacity-bound findings generalize beyond the evaluated topologies.Source: §8 · p. 11