Source-linked AI summary
Sensing as a Service and Big Data
Arkady Zaslavsky, Charith Perera, Dimitrios Georgakopoulos
TL;DR
IoT will generate massive, heterogeneous, high-velocity data that challenges traditional data management. The paper surveys architectures, applications, sensing-as-a-service models, cloud technologies, and management challenges, concluding that advanced systems must extract high-value information from noisy data. It also identifies substantial potential value from big-data use across sectors.
Problem
IoT-generated data streams challenge traditional approaches to managing, storing, processing, and interpreting data at scale.
Method
The paper surveys IoT architecture, large-scale sensor networks, sensing-as-a-service, cloud technologies, and sensor-data management challenges.
Results
The paper identifies technologies and management priorities for processing massive sensor data, including scalable storage, real-time analytics, security, query optimization, and performance isolation.
Takeaways & Limitations
Advanced systems and innovative technologies can process massive amounts of dirty data and extract high-value information from it.
Abstract
from arXiv · showhide
Internet of Things (IoT) will comprise billions of devices that can sense, communicate, compute and potentially actuate. Data streams coming from these devices will challenge the traditional approaches to data management and contribute to the emerging paradigm of big data. This paper discusses emerging Internet of Things (IoT) architecture, large scale sensor network applications, federating sensor networks, sensor data and related context capturing techniques, challenges in cloud-based management, storing, archiving and processing of sensor data.
1. Introduction
IoT connects vast numbers of sensor-enabled devices that generate real-time data for diverse applications. Falling acquisition costs and rapidly increasing data volumes make scalable collection, analysis, and interpretation increasingly important.
- IoT emergence: IoT links billions of devices containing sensors, actuators, and processors for real-time data collection across many applications.Applications include environmental monitoring, industrial systems, business, and human-centric pervasive computing.
- Data growth: 1 ZB of data was generated worldwide in 2010, reaching 1.8 ZB by the end of 2011 and projected to reach 35 ZB in 2020.The paper notes that the 2020 estimate may be conservative.
- IoT emergence: 50 to 100 billion devices were predicted to connect to the Internet by 2020, shifting the focus toward an Internet of Things.The resulting data has value only if it can be collected, analyzed, and interpreted.
- Enabling technologies: Sensor data acquisition has become substantially cheaper, especially for continued acquisition, while sensor technologies such as Arduino also reduce initial costs.The paper describes declining acquisition costs as an enabler of broader sensing deployments.
2. Sensing Big Data
IoT sensing produces big data characterized by volume, variety, and velocity across applications ranging from infrastructure monitoring to disaster response. Managing, storing, processing, and sharing these heterogeneous streams creates major technical and organizational challenges.
- Big data characteristics: Big data is commonly characterized by volume, variety, and velocity rather than size alone.Volume concerns data size, variety concerns types and sources, and velocity concerns generation or processing frequency.
- Challenges: Big-data challenges include capture, storage, search, analysis, and virtualization.The paper also notes that data may be valuable in aggregate even when individual records appear uninformative.
- Applications: Very large sensor networks produce polymorphous, heterogeneous, large-scale, time-limited data requiring coordinated management of sensing and computational resources.Cloud computing is used to process such data, while storage and processing remain key challenges.
- Applications: IoT sensing applications include traffic management, greenhouse-gas monitoring, bridge monitoring, disaster response, logistics, and personal fitness.Examples range from California’s greenhouse-gas network to radiation sensing at Fukushima and sensor-equipped infrastructure.
- Sensing as a service: Sensing as a service lets organizations access government roadway-sensor data for different analyses and value-added services, potentially in exchange for a fee.The London roadway example illustrates how one organization’s sensing infrastructure can serve multiple data users.
3. Why Big Data?
Big data matters because IoT-generated streams can support timely decisions and more responsive operations across public and private sectors. Its value depends on analyzing high-volume, high-variety, high-velocity data for knowledge and prediction.
- Value of big data: Big data can support timely decisions, cost savings, and more optimized operations in public and private sectors.The paper connects these benefits to data-driven decision making.
- IoT and analytics: IoT sensors generate high-volume, high-variety data at high velocity, creating a need for analysis to extract knowledge.Supply-chain management is presented as a domain where these technologies are strongly interconnected.
- Applications: Big-data analytics is associated with applications including customer analysis, hospital intervention, weather prediction, and asset-placement optimization.These examples span business, healthcare, energy, and infrastructure decisions.
- Reported opportunities: $300 billion potential annual value to US health care and 60% potential increase in retailers’ operating margins are among the reported opportunities.The same summary reports potential public-sector, consumer-surplus, and workforce effects.
4. Technologies around Big Data
Big-data management combines scalable technologies with cloud-based processing and storage, but organizations still face a widening gap between available and processable data. Sensor-cloud integration is presented as important for sharing sensors and sensor data.
- Technologies: Machine learning, massively parallel processing databases, distributed file systems, and cloud computing are identified as technologies supporting big-data management.Existing techniques also need extension for big-data settings.
- Analytics: Big-data extraction retains conceptual steps such as define, search, transform, entity resolution, and answer the query, while implementation technology changes substantially.The paper gives SAP Hana as an example of faster real-time analysis.
- Storage: NoSQL systems address big-data and cloud requirements through key-value, document, wide-column, and graph database varieties.Graph databases are favored for complex relationships such as social-network, semantic, and linked data.
- Management gap: The gap between data available to organizations and data they can process is widening because producers and generation frequencies are increasing.Figure 5 presents the corresponding decline in the percentage of data organizations can analyze.
- Sensor-cloud integration: Cloud architectures can process high-dimensional sensor data efficiently, and connecting and managing sensors through the cloud supports sensing-as-a-service sharing.Examples include map-reduce processing for brain signals and an Arduino, Google App Engine, and Google Datastore setup.
5. Sensing as a service Model
Sensing as a service connects physical and virtual sensors to cloud-based services, enabling sensor owners to publish data and consumers to access it. The model supports real-time sensing and data processing while raising privacy, security, context-management, and scalability concerns.
- Cloud computing provides scalable storage and processing capabilities considered essential to realizing the IoT vision.
- Sensing as a service allows sensor owners to publish data for returns while mediating organizations connect owners with sensor-data consumers.
- IoT sensing-as-a-service deployments can reduce consumers’ need to deploy sensors themselves while providing real-time data, but privacy and security require appropriate handling.
- As IoT scales toward 50 billion connected things, sensor-generated data will reach unprecedented volumes and context annotations will increase storage requirements.
- Sensor-Cloud manages physical sensors through cloud connections and SensorML metadata, but does not focus on offering sensor data as a service.
- SenaaS encapsulates physical and virtual sensors as SOA services but emphasizes sensor management rather than sensor-data collection and dissemination.
- OpenIoT develops an open-source cloud middleware platform connecting Internet objects to support IoT applications.
6. Challenges in Big Data Management
Big data management challenges span engineering and semantic concerns, from efficient storage and querying to extracting meaning from massive, unstructured, dirty data. Cloud-based IoT systems additionally require energy-aware processing, real-time analytics, security, approximation policies, query optimization, and multi-tenant isolation.
- Big data challenges divide into engineering tasks such as query and storage management and semantic tasks such as extracting meaning from unstructured dirty data.
- High-volume processing must account jointly for processing power and cooling power in low-power digital architectures.
- Real-time big-data systems need data-adaptive machine-learning techniques capable of analyzing incoming data streams.
- Cloud-based sensing services must manage data security and privacy, including access control for big data.
- Approximate query results can be much faster than traditional execution, requiring decisions about where approximation preserves sufficient accuracy.
- Query optimization must consider energy, memory, processing time, and storage, with parallel processing identified as key in cloud environments.
- Multi-tenant cloud systems require performance isolation so one tenant’s workload does not undermine another’s service.
7. Conclusions
The conclusion characterizes big data as valuable information mixed with noise, errors, and raw data. Advanced systems and technologies are needed to process this mixture efficiently and extract high-value information.
- Big data mixes hidden high-value information with dirty noise, erroneous data, and raw data.
- Advanced systems and innovative technologies are needed to process massive dirty datasets efficiently and extract valuable information.