Live Adaptation of Dataflow Stream Processing for Hybrid Edge-Cloud Analytics

1 Introduction

• Sluice extends datacenter-centric engines to hybrid deployments while preserving applications, familiar APIs, fault tolerance, out-of-order processing, and ecosystem integrations.Prior hybrid and edge systems often lack mature-engine features, making reimplementation and application migration substantial engineering work.Source: §1 · p. 1
• Prior work explored pure-edge and hybrid analytics, but the paper identifies resource assumptions, static scheduling, and disruptive recovery as challenges for dataflow engines.Static placements become obsolete as WAN bandwidth, edge capacity, and event rates change; job-wide recovery can cause downtime, backlog, or source-side data loss.Source: §1 · pp. 1–2
• Sluice dynamically reconfigures stateless preprocessing between edge and cloud while keeping stateful operators in the cloud to avoid state migration.Its automatic rewriter creates independently managed, fault-tolerant segments; routing controllers shift work without downtime and coordinate watermarks to preserve event-time correctness and no data loss.Source: §1 · p. 2
• Across five real-world applications, Sluice shifts computation between edge and cloud and quickly converges to a configuration meeting target throughput under variable bandwidth, rates, and scarce compute.Evaluation used Raspberry Pi devices and a 100-VM cloud testbed.Source: §1 · p. 2

2.1 Streaming dataflow concepts

• Dataflow pipelines are operator DAGs with stateless and stateful stages; watermarks track event-time progress, and consistent checkpoints restore state by replaying input from source offsets.If an operator cannot keep pace, backpressure stalls upstream stages and can lead to backlog and data loss when buffers fill.Source: §2.1 · pp. 2–3

2.2 Application scenarios

• The five scenarios span air-quality monitoring, taxi routes, smart-grid anomalies, sentiment analysis, and video-assisted traffic monitoring.Their pipelines commonly place lightweight stateless transformations before stateful operations or complex analytics, making early stages candidates for edge offloading to reduce WAN traffic.Source: §2.2 · p. 3
• Neither static Q5 deployment sustains target throughput throughout the experiment: cloud-only processing struggles during low bandwidth, while edge offloading struggles when edge compute is limited.The figure compares throughput under two static deployments alongside changing operating conditions.Source: §2.2 · p. 3, Figure 1 · p. 3

3 Sluice system design

• Sluice’s scheduler deploys query segments on edge and cloud resources, while metric collection feeds a routing controller that triggers reconfiguration as needed.The system design describes a monitoring-and-control loop for deployed jobs.Source: §3 · pp. 3–4

3.1 Adaptive routing mechanism

• Sluice adapts stateless operators before the first stateful operator by switching event channels between edge and cloud replicas, leaving stateful operators in the cloud.Routing operators direct each event to one replica; replicas are organized so each event is processed only once, and segments can follow different execution paths.Source: §3.1 · pp. 4–5

3.2 Query re-writer

• The query re-writer partitions the original graph at the first stateful operator, then adds stateless-operator replicas and routing operators to the upstream segment.Cloud-replica outputs bypass the next routing operator, so once an event is routed to the cloud, later operators remain in the cloud.Source: §3.2 · p. 5

3.3 Adaptive routing policy

• Sluice uses a metrics-based heuristic to adjust routing until the target throughput is sustained, distinguishing WAN bandwidth bottlenecks from constrained edge compute.The policy assumes a feasible routing plan exists and that cloud resources can scale sufficiently to prevent the stateful segment from becoming a bottleneck.Source: §3.3 · p. 6

4.1 Assumptions

• Sluice’s correctness guarantees assume broker event order, reliable delivery, high availability, persistence before acknowledgment, and checkpoint-based engine fault tolerance.The paper cites Kafka, Pulsar, Kinesis Data Streams, Flink, and Spark Structured Streaming as systems supporting these features.Source: §4.1 · p. 7

4.2 Semantic equivalence

• Sluice preserves execution-path equivalence: each event traverses the same ordered logical operators in the original and transformed graphs, although stateless operators may run in different locations.Each stateless operator has identical edge and cloud replicas, and routing prevents an event from switching back to the edge after moving to the cloud.Source: §4.2 · p. 7
• Sluice coordinates watermarks across edge and cloud paths so reordering during rerouting respects event-time boundaries and does not advance downstream progress ahead of in-flight events.A watermark union waits for upstream watermarks; the cloud segment tracks partition watermarks and computes the global watermark from their minimum.Source: §4.2 · pp. 7–8, Figure 5 · p. 7

4.3 Correctness under failures

• Sluice provides at-least-once processing under transient failures when source devices can buffer events during recovery, using checkpoint offsets and broker persistence to replay events without data loss.When edge and cloud jobs fail simultaneously, independent checkpoint offsets can cause duplicate events to be replayed and reprocessed.Source: §4.3 · p. 8
• Sluice avoids aligning checkpoint barriers across query segments, preventing a slow or disconnected edge device from holding up global checkpoint progress and allowing separate checkpoint intervals.Edge checkpoints can be less frequent to reduce resource use, while cloud checkpoints can be more frequent for faster recovery.Source: §4.3 · p. 8

4.4 Availability under failures

• When a broker partition is inactive beyond a user-specified period, Sluice excludes it from the global watermark calculation to prevent that partition from stalling the pipeline.The global watermark is the minimum of the maximum watermarks from active partitions.Source: §4.4 · pp. 8–9

5 Evaluation

• Sluice’s evaluation covered reconfiguration overhead, adaptation to changing conditions, individual parameter impacts, and scalability.The experiments used Apache Flink 1.20 and Apache Kafka 3.9.0.Source: §5 · p. 9

5.1 Experimental setting

• The evaluation used Testbed-RPi with five Raspberry Pi edge nodes and Testbed-100VMs with 100 edge-emulating VMs; five hybrid streaming queries were benchmarked against All-Edge, All-Cloud, and CAPSys.Runs generally lasted two hours, with five minutes of warm-up; throughput below the target indicated backpressure.Source: §5.1 · p. 9

5.2 Zero-downtime reconfiguration

• Sluice showed no visible throughput dips during operator shifts, while each vanilla Flink shift caused approximately 3 minutes of downtime.The comparison used Q2 on Testbed-RPi, with shifts triggered every 10 minutes.Source: Figure 7 · p. 9, §5.2 · p. 9

5.3 Sluice under variable operating conditions

• Under simultaneous workload, bandwidth, and edge-compute changes, Sluice alone sustained the target throughout the three-hour experiment on Q1 and Q5.It shifted operators to the edge when bandwidth was insufficient, then back to the cloud when edge compute contention became the bottleneck, and shifted again after bandwidth dropped.Source: Figure 8 · p. 10, §5.3 · p. 10

5.4 Impact of individual parameters

• Across all queries and gradual or abrupt input-rate patterns, Sluice alone met the target rate; All-Edge, All-Cloud, and CAPSys each fell short in specified cases.All-Edge was limited by edge compute, All-Cloud performed poorly for Q2, Q3, and Q5, and CAPSys missed high-rate periods for Q1, Q2, and Q3.Source: Figure 9 · p. 10, §5.4 · p. 11
• With variable bandwidth, Sluice alone consistently sustained the target; with edge compute contention, it alone always sustained the target across applications.Sluice shifted operators to the edge for network-intensive Q1, Q2, and Q5, and shifted toward the cloud when contention required it; Q5 needed no shift during contention.Source: Figure 10 · p. 11, §5.4 · p. 11

5.5 Sluice scalability

• After routing reconfiguration, all 100 devices sustained the target, averaging 99.8% throughput-to-target; repeated policy adjustments converged to backpressure-free configurations.Initially, 39% of devices missed the target with all operators in the cloud; after subsequent bandwidth changes, Sluice reconfigured again to restore target rates.Source: §5.5 · pp. 11–12, Figure 11 · p. 12

Edge stream processing engines

• Unlike edge-centric systems such as DART, EdgeWise, and Frontier, Sluice targets hybrid pipelines that produce data at the edge but need cloud analysis, reusing mature dataflow engines.DART supports dynamic placement and scaling, EdgeWise schedules resources on a single device, and Frontier replicates dataflow graphs for fault tolerance.Source: §6 · p. 12

Adaptive filtering and routing

• Compared with adaptive routing systems such as Shepherd, Falcon, and Frontier, Sluice makes its adaptive routing transparent to applications.Those systems respectively reroute tuples, support live key migration, and forward events through per-batch routing.Source: §6 · p. 12

Adaptive placement and scheduling

• Sluice rewrites queries into independent jobs and routes between edge and cloud replicas of stateless preprocessing without pausing execution, while preserving watermark coordination and event-time semantics.Prior systems including CAPSys, Trisk, WASP, and APEROL require at least partial stopping during reconfiguration; CAPSys also assumes homogeneous resources.Source: §6 · p. 12

• Sluice extends datacenter-centric dataflow engines to hybrid edge–cloud deployments without requiring application changes.Source: §7 · p. 12
• Sluice dynamically offloads computations to the edge and adjusts to operational and workload dynamics.Source: §7 · p. 12
• Users retain the underlying engine’s event-time and fault-tolerance semantics, ensuring correctness and no data loss.Source: §7 · p. 12

• Users submit Flink applications without manually splitting queries, managing resources, or placing tasks; the rewriter generates query segments and adaptive routing paths.Programs must include a valid source, define mobile operators as DataStream variables without chaining methods, and expose user-defined object fields through public JSON-serializable getters and setters.Source: A Implementation details · p. 17
• Heartbeat records carrying watermark timestamps and partition tags let the stateful segment compute a global watermark; idle partitions are excluded until they recover.Duplicate heartbeats are harmless because watermarks advance through a monotonic max-min reduction.Source: A Implementation details · p. 17
• The routing controller periodically gathers Flink metrics and evaluates routing policies in real time, independently for each segment.It sends decisions over HTTP; CP operators use a routing flag to direct events to the cloud or edge output.Source: A Implementation details · p. 17
• A custom Flink scheduler pins tasks to edge or cloud workers using a static placement map keyed by Task Manager IP address.ResourceProfile and SlotMatchingStrategy extensions match each task’s specified location to an available slot.Source: A Implementation details · p. 17
• Figure 12 is labeled as a comparison of Sluice with and without the watermark inactivity tolerance mechanism.Source: Figure 12 · p. 17
• The metrics collector records useful processing time and observed and true input and output rates per task for routing-plan recomputation.Source: A Implementation details · p. 17
• Each query uses a dedicated Kafka topic, with each edge device writing to a distinct partition that the cloud segment reads.Source: A Implementation details · p. 17

• The experiment runs Q1 on Testbed-RPi at a fixed input rate for one hour, with a five-minute warm-up, and cancels one of five edge segments at 900s.Source: §B.1 · p. 17
• Figure 12’s sliding-average output throughput uses the last 10 samples to smooth query-logic spikes and isolate the mechanism’s effect.Source: §B.1 · p. 17
• After an edge device is canceled at 900s, Sluice throughput stabilizes after the tolerance period; without idleness, it remains zero through the experiment.The global watermark advances using the four remaining live broker partitions, preventing system-wide downtime under partial edge-device failures.Source: §B.1 · pp. 17–18

• The remaining four Raspberry Pi devices show results similar to the single devices reported per query in the main text.Source: C Additional results · p. 18
• For Q3, mobile operators filter out all input on RPi2–4, where All-Cloud and All-Edge therefore perform better than on the other RPis.Source: C Additional results · p. 18
• Figures 13–16 cover Sluice under variable operating conditions for RPi1–4.Source: Figure 13 · p. 18, Figure 14 · p. 18, Figure 15 · p. 19
• Figures 17–20 cover the impact of input rate for RPi1–4.Source: Figure 17 · p. 20, Figure 18 · p. 20, Figure 19 · p. 21
• Figures 21–24 cover the impact of network bandwidth and compute contention for RPi1–4.Source: Figure 21 · p. 22, Figure 22 · p. 22, Figure 23 · p. 23