PaperlayerView exact revision on arXiv

Source-linked AI summary

IoT Stream Processing and Analytics in The Fog

Shusen Yang

arXiv:1705.05988v1cs.NI

TL;DR

Cloud-centered IoT streams raise latency and network-capacity concerns, while Fog streaming has not been comprehensively investigated in the supplied paper context. The article analyzes typical applications, streaming models, architecture, and design dimensions, synthesizing common properties and design opportunities and challenges.

  • Problem

    Cloud-centered IoT and mobile applications face latency and network-capacity concerns as large streams travel between end devices and distant Cloud servers.

    Source: §I · p. 1

  • Method

    The article studies typical Fog-streaming applications and develops general models, functional architecture, and a design-space analysis across system, data, human, and optimization dimensions.

    Source: §I · p. 1, C. Fog Data Streaming Architecture · p. 4, §IV · p. 5

  • Results

    The four application examples illustrate common properties and motivate general Fog-streaming models, architecture, and design opportunities and challenges.

    Source: §V · p. 7

  • Takeaways & Limitations

    The authors expect the growing roles of the network edge and stream processing to promote Fog data-streaming development in academia and industry.

    Source: §V · p. 7

Abstract

from arXiv · show

The emerging Fog paradigm has been attracting increasing interests from both academia and industry, due to the low-latency, resilient, and cost-effective services it can provide. Many Fog applications such as video mining and event monitoring, rely on data stream processing and analytics, which are very popular in the Cloud, but have not been comprehensively investigated in the context of Fog architecture. In this article, we present the general models and architecture of Fog data streaming, by analyzing the common properties of several typical applications. We also analyze the design space of Fog streaming with the consideration of four essential dimensions (system, data, human, and optimization), where both new design challenges and the issues arise from leveraging existing techniques are investigated, such as Cloud stream processing, computer networks, and mobile computing.

I. INTRODUCTION

The introduction motivates Fog streaming by latency and network-capacity concerns in cloud-centered IoT, then presents a study of typical applications, streaming models, architecture, and design issues. The applications section illustrates those ideas through examples including IoT query and analytics.

I. INTRODUCTION

  • Cloud-centered mobile and IoT applications face latency and network-capacity concerns as billions of devices exchange data with distant cloud servers.Augmented reality may require about 10 ms, while the passage describes typical cloud end-to-end latency as hundreds of milliseconds.Source: §I · p. 1
  • The article studies typical Fog applications, presents streaming models and functional architecture, and analyzes design issues across system, data, human, and optimization dimensions.Source: §I · p. 1

II. FOG STREAMING APPLICATIONS

  • The applications overview uses examples to illustrate typical Fog-streaming features and the architecture in distinct real-world contexts.Source: §II · p. 1, Figure 1 · p. 2

A. IoT Stream Query and Analytics

  • Gigasight processes crowdsourced video at a Cloudlet within a metropolitan area and sends metadata to the Cloud for Internet-wide SQL search.Its stated aim is to avoid massive video-stream transmissions over the backbone Internet.Source: A. IoT Stream Query and Analytics · p. 1
  • WSN databases such as TinyDB jointly use sensors and edge gateways for stream processing, while MongoDB can run on Cloud or Fog servers.The WSN systems focus mainly on low-power sensor energy minimization and provide basic stream-management and SQL-like query support.Source: A. IoT Stream Query and Analytics · pp. 1–2

B. Real-time Event Monitoring

Event monitoring distributes classification tasks across Fog servers according to application logic, stream features, and available resources. Industrial control uses real-time sensor and control streams, with Fog networks and servers supporting communication and computation requirements.

B. Real-time Event Monitoring

  • Event-detection jobs divide into low-level classifiers arranged as a reversed binary tree, with classifiers distributed across Fog servers according to their available resources.The listed resources include CPU, memory, storage, and network bandwidth.Source: B. Real-time Event Monitoring · p. 2

C. Networked Control Systems for Industrial Automation

  • An NCS control loop connects controllers, sensors, and control plants through a network carrying continuous sensor flows and control signals.Source: C. Networked Control Systems for… · p. 2
  • NCS applications may require a 10 ms delay, 5 Mbps data rate, and 10^-8 bit error; local Fog networks minimize component distances while Cloud supports remote administration.Source: C. Networked Control Systems for… · p. 2
  • Fog servers can provide resources for computation-intensive NCS tasks that embedded controllers on resource-limited end devices cannot support.Examples include solving high-order differential equations, learning system dynamics, and addressing disturbances and faults related to communication uncertainty.Source: C. Networked Control Systems for… · p. 2

D. Real-time Mobile Crowdsensing

Real-time mobile crowdsensing combines sensor-data processing with streams about human participation. The paper frames these applications alongside general models and architecture for typical Fog-streaming systems.

D. Real-time Mobile Crowdsensing

  • Mobile crowdsensing processes both sensor-data flows, such as queries and mining, and human-related streams about incentives, worker selection, and quality control.Source: D. Real-time Mobile Crowdsensing · pp. 2–3
  • Data-stream processing generally needs lower latency and higher bandwidth than human-related information processing, such as worker payments.Source: D. Real-time Mobile Crowdsensing · p. 3
  • A hierarchical Fog architecture partitions mobile participants geographically and processing tasks functionally, with the passage reporting better scalability, interactive responsiveness, and bandwidth savings than Cloud-based MCS.Source: D. Real-time Mobile Crowdsensing · p. 3

III. MODELS AND ARCHITECTURE

  • The models-and-architecture section characterizes common features across typical Fog-streaming systems and applications, including the four examples discussed earlier.Source: §III · p. 3

A. Life Cycle of Fog Data Streams

Fog data streams move through creation, collection, processing, and application, while processing workflows can be modeled as DAGs. The lifecycle may loop when applications produce streams, and DAGs represent processing dependencies and support parallel computation.

A. Life Cycle of Fog Data Streams

  • The Fog-stream lifecycle comprises stream creation, collection at Fog servers, application-specific processing, and consumption or storage of results.Processing may use one or multiple Fog servers, a Fog cluster, or a combination of Fog and Cloud servers.Source: A. Life Cycle of Fog Data Streams · p. 3
  • The lifecycle can loop when applications generate streams, as when an NCS produces control signals.Source: A. Life Cycle of Fog Data Streams · p. 3

B. Work flow and Operations of Fog Stream Processing

  • A stream-processing workflow is modeled as a DAG: vertices are processing elements, and directed edges carry streams from producers to consumers.Source: B. Work flow and Operations of Fog… · p. 3
  • DAGs describe application workflows such as Gigasight's multistage pipeline and event detection's binary-tree topology, and facilitate parallel processing of elements.Apache Storm is given as an example of an engine using spouts and bolts in a DAG topology.Source: B. Work flow and Operations of Fog… · p. 3
  • Fig. 3 presents the functional architectures of Cloud and Fog data-streaming systems.Source: Figure 3 · p. 4

C. Fog Data Streaming Architecture

The Fog streaming architecture adapts Cloud streaming components across six functional layers, while its physical network is more heterogeneous and dynamic than a Cloud data-center network.

  • The six-layer architecture includes application, processing, data management, resource management, virtualization, and physical network layers.The layers define job logic, application-specific processing, data storage and organization, resource scheduling, hardware virtualization, and network infrastructure.Source: C. Fog Data Streaming Architecture · p. 4
  • Virtualization configures and virtualizes hardware resources, while Fog APIs can also support sensing, caching, mobility, and control services.The passage names OpenStack, SDN, and NFV as virtualization techniques supported in both Cloud and Fog architectures.Source: C. Fog Data Streaming Architecture · p. 4
  • Fog's physical network is more heterogeneous and dynamic than Cloud data-center networks, so Cloud streaming approaches may not transfer directly.Fog also features delay-sensitive applications, mobility-driven network changes, additional resource types, and potentially unreliable services.Source: C. Fog Data Streaming Architecture · pp. 4–5

IV. THE DESIGN SPACE OF FOG DATA STREAMING

The design space examines system, data, optimization, and human dimensions, which are interrelated; the system dimension covers Fog streaming components, including stream-processing engines.

IV. THE DESIGN SPACE OF FOG DATA STREAMING

  • The four design dimensions—system, data, optimization, and human—are not orthogonal, so technical issues in one can relate to others.Fig. 4 presents the four dimensions of Fog data streaming.Source: §IV · p. 5, Figure 4 · p. 5

A. System

  • The system dimension comprises the algorithms, protocols, and software that form the Fog streaming architecture.The passage identifies three critical system issues, including stream-processing engines.Source: A. System · p. 5

1) Stream Processing Engine:

  • Existing Cloud engines can run on Fog servers, but latency-sensitive applications may require tuning for sub-10-ms end-to-end delay.The passage contrasts typical engine latency around 100 ms with Fog requirements below 10 ms, and identifies Storm topology size and Spark micro-batch size as configuration choices.Source: 1) Stream Processing Engine · p. 5
  • Fog applications also lack APIs for some tasks, including differential-equation solvers and control-error estimators for real-time networked control.Source: 1) Stream Processing Engine · p. 5

2) Streaming Task Partitioning:

Fog streaming tasks can be partitioned across end devices, Fog servers, and Cloud servers, while Fog also creates new opportunities for streaming algorithms.

2) Streaming Task Partitioning:

  • Application partitioning assigns tasks across end devices, Fog servers, and Cloud servers according to resources, privacy, latency, and fault-tolerance needs.Partitioning can occur across applications, streams, or functional components within one application.Source: 2) Streaming Task Partitioning · p. 5
  • Geographic partitioning distributes tasks among Fog servers, making load balancing important when server capacities and end-device densities vary.Source: 2) Streaming Task Partitioning · p. 5
  • Mobility-driven service migration needs real-time and fault-tolerance support; Cloud live migration performs poorly in Fog, and existing Fog schemes assume unrealistic mobility patterns.Source: 3) Streaming Service Migration · p. 5

B. Data

1) Data Stream Acquisition.:

Fog's hierarchy creates opportunities to distribute sensing and anomaly detection, while streaming analytics must balance processing loads without compromising mining performance.

1) Data Stream Acquisition.:

  • Fog servers could reconstruct local sensing sub-matrices for Cloud recovery of a global matrix, addressing computation and communication burdens in large-scale compressive sensing.The cited challenges include citywide-scale and real-time multimedia applications.Source: 1) Data Stream Acquisition · p. 6
  • Fog servers can detect anomalies locally in correlated streams from nearby sources, a configuration described as more fault-tolerant than detection on end devices.Source: 1) Data Stream Acquisition · p. 6

2) Stream Mining and Analyltics:

  • Fog mining must balance real-time computation loads between Fog servers and end devices while ensuring mining performance.Existing distributed stream-mining theory and software support provide a basis for Fog analytics, but the load-balancing challenge remains.Source: 2) Stream Mining and Analyltics · p. 6

C. Human

Because Fog systems are closer to users and device owners, human behavior matters in their design. The section discusses provider-consumer incentives, privacy, and crowdsourced-worker quality control.

1) Pricing and Icentivization:

  • Fog providers supply data or resources, while consumers discover, subscribe to, and consume streaming services.Their self-interest and strategic behavior make appropriate pricing and incentives important for efficient, trustworthy exchange.Source: 1) Pricing and Icentivization · p. 6
  • Pricing and incentivization mechanisms should address Fog systems’ heterogeneity and dynamics, building on related pricing and auction research.The article identifies cloud data pricing, smart-grid pricing, and crowdsensing auctions as potentially relevant prior work.Source: 1) Pricing and Icentivization · p. 6
  • Fog’s hierarchical tiers can support privacy preservation tailored to application contexts, while balancing data value against exposure risk.Gigasight denatures video streams at an edge Cloudlet to abstract features while preserving providers’ privacy.Source: 1) Pricing and Icentivization · p. 6
  • For real-time mobile crowdsensing, worker-specific audio allocation and Fog’s geographic and functional task partitioning can support online quality control.In speech captioning, audio streams with different speaking rates are assigned to workers according to their abilities.Source: 1) Pricing and Icentivization · p. 6

D. Optimization

Fog streaming optimization must account for dynamic, uncertain conditions and interactions across design dimensions. The section points to analytical models and methods that address these challenges.

D. Optimization

  • The optimization dimension develops theoretical models and methods to address challenges in system, data, and human dimensions.Source: D. Optimization · p. 6

2) Privacy:

  • Fog streaming models should account for dynamics and uncertainty arising from mobility, wireless communication, physical events, unreliable providers, faulty sensors, and server failures.One cited approach uses a Markov decision process for mobility-driven edge-cloud service migration.Source: 1) Dynamic Optimization · p. 6

1) Dynamic Optimization.:

  • Fig. 6 presents in-network data processing [8] as an example for demonstrating gains from jointly optimizing data processing and network operations.Source: Figure 6 · p. 7
  • Dynamic service configurations can be supported under arbitrary stochastic service-arrival processes.Source: 1) Dynamic Optimization · p. 7

2) Complex Resource Allocation:

Fog streaming resource allocation faces complex jobs and heterogeneous resources, while joint optimization across system, data, and human dimensions is presented as a promising direction. The article synthesizes applications, models, and design challenges for this setting.

2) Complex Resource Allocation:

  • DAG-like jobs and heterogeneous Fog resources make resource allocation challenging; a data-center approach for Apache Storm jobs may be extendable to Fog.The jobs arrive dynamically, and the data-center network topology resembles Fog network infrastructure.Source: 2) Complex Resource Allocation · p. 7

3) Optimization over System, Data, and Human Dimensions:

  • Joint optimization across system, data, and human dimensions outperforms optimizing each dimension individually; the cited in-network algorithm [8] achieves better energy-resource utilization and network throughput than pure network optimization.The article points to queuing theory, signal processing, and game theory as methods for cross-dimension optimization.Source: 3) Optimization over System, Data, and… · p. 7

V. CONCLUSION

  • The article studies four typical Fog streaming applications and uses their common properties to discuss general models, architecture, and design opportunities and challenges.The design discussion spans networked systems, data processing and management, human factors, and optimization methods.Source: §V · p. 7
Loading 1705.05988v1…