PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-supervised Deep Reinforcement Learning in Support of IoT and Smart City Services

Mehdi Mohammadi, Ala Al-Fuqaha, Mohsen Guizani, Jun-Seok Oh

arXiv:1810.04118v1cs.NIcs.AIcs.LG

TL;DR

IoT and smart-city services face limited labeled training data, motivating learning methods that use sensor-generated unlabeled data. The paper proposes a VAE-based semi-supervised DRL model and evaluates it for BLE indoor localization in smart buildings, reporting improved localization and rewards over supervised DRL.

  • Problem

    IoT sensor data are abundant but often difficult to label, limiting training data for smart services.

  • Method

    The paper proposes a semi-supervised deep reinforcement learning framework using a deep variational autoencoder and both labeled and unlabeled data.

  • Results

    23% improvement in localization accuracy and at least 67% more rewards were reported than with the supervised model.

  • Takeaways & Limitations

    The study supports using unlabeled data with semi-supervised DRL for IoT and smart-city learning agents requiring little supervision.

Abstract

from arXiv · show

Smart services are an important element of the smart cities and the Internet of Things (IoT) ecosystems where the intelligence behind the services is obtained and improved through the sensory data. Providing a large amount of training data is not always feasible; therefore, we need to consider alternative ways that incorporate unlabeled data as well. In recent years, Deep reinforcement learning (DRL) has gained great success in several application domains. It is an applicable method for IoT and smart city scenarios where auto-generated data can be partially labeled by users' feedback for training purposes. In this paper, we propose a semi-supervised deep reinforcement learning model that fits smart city applications as it consumes both labeled and unlabeled data to improve the performance and accuracy of the learning agent. The model utilizes Variational Autoencoders (VAE) as the inference engine for generalizing optimal policies. To the best of our knowledge, the proposed model is the first investigation that extends deep reinforcement learning to the semi-supervised paradigm. As a case study of smart city applications, we focus on smart buildings and apply the proposed model to the problem of indoor localization based on BLE signal strength. Indoor localization is the main component of smart city services since people spend significant time in indoor environments. Our model learns the best action policies that lead to a close estimation of the target locations with an improvement of 23% in terms of distance to the target and at least 67% more received rewards compared to the supervised DRL model.

I. INTRODUCTION

IoT and smart-city services need learning from abundant sensor data despite scarce labels. The paper proposes semi-supervised deep reinforcement learning using labeled and unlabeled data, evaluated for BLE indoor localization in smart buildings.

  • Motivation: IoT sensors generate large volumes of data without easily obtainable labels, creating a central challenge for smart-service development.
  • Motivation: Smart-building localization supports location-aware services and links indoor users to other IoT services.
  • Approach: The proposed framework combines deep generative models with reinforcement learning to address semi-supervised learning.
  • Approach: The model leverages both labeled and unlabeled data, targeting IoT settings where unlabeled data are more prevalent.
  • Evaluation: The study applies the model to indoor localization in smart buildings using BLE or iBeacon signals.

II. RELATED WORK

Prior work applied deep reinforcement learning across healthcare, vision, traffic control, resource management, language understanding, and energy management.

  • Applications: DRL has been used to learn medication-dosing policies that maximize therapeutic outcomes.
  • Applications: A CNN combined with reinforcement learning to guide visual attention for vehicle image classification.
  • Applications: Traffic-signal timing was formulated as DRL using lane queues as states and lane-control choices as actions.
  • Applications: DRL has been applied to job scheduling with rewards based on reciprocal job duration to minimize average slowdown.
  • Applications: LSTM representations and DQN Q-function approximation supported agents in text-based games, while DRL also addressed energy management.

B. Review of Indoor Localization

Indoor-localization research spans conventional, deep-learning, and semi-supervised approaches, but the paper identifies a gap in combining deep reinforcement learning with unlabeled data and BLE fingerprinting.

  • Prior indoor-localization studies use technologies and models including WiFi, BLE, CSI, SVM, KNN, Bayesian filtering, transfer learning, and neural networks.
  • Deep-learning and semi-supervised methods have reported improved fingerprinting accuracy over shallow or alternative methods.
  • Semi-supervised localization methods use unlabeled fingerprints with limited labeled data, including one approach reporting an average error of 2 meters.
  • The paper states that prior deep-learning localization solutions rely on WiFi fingerprinting and that BLE fingerprinting had not been studied with deep or reinforcement learning.

III. BACKGROUND AND PROPOSED APPROACH

The proposed approach uses variational autoencoders to extend deep reinforcement learning into a semi-supervised setting that learns from labeled and unlabeled data.

  • The model adopts a deep generative VAE framework for semi-supervised learning with labeled and unlabeled observations.
  • For unlabeled data, the label y is treated as a latent variable alongside z in the generative model.
  • M1 uses a Gaussian inference network for latent variable z, while M2 infers latent variables z and y with Gaussian and multinomial distributions.
  • The training objective combines generative and discriminative components, with α adjusting their contributions during optimization.

B. Semi-Supervised Deep Reinforcement Learning

The semi-supervised DRL design combines a reward-based Markov decision process and deep Q-learning with a VAE-based model that handles unlabeled observations.

  • The reward function reflects closeness of the resulting state to the true class label, assigning larger positive rewards to more compelling results.
  • The DRL agent approximates the optimal action-value function with a neural network and updates it through Q-learning and experience replay.
  • The proposed model replaces the supervised DRL network with a generative semi-supervised model to process unlabeled observations.
  • The VAE is extended with an additional hidden layer and an output that generates actions.
  • Training occurs offline, while policy prediction occurs online for high-dimensional and high-volume data.

IV. USE CASE: INDOOR LOCALIZATION

The case study applies the semi-supervised DRL model to BLE-based indoor localization in a campus environment, using labeled and unlabeled observations to estimate locations.

  • The approach is also described for smart-building energy management and intelligent transportation navigation use cases.
  • The algorithm accepts labeled and unlabeled data, infers labels for unlabeled samples, and computes approximate rewards from state closeness.
  • The localization experiment was conducted in a large, busy academic library rather than a simulated environment, small area, or isolated testbed.
  • The case study uses iBeacon RSSI measurements as input to identify indoor locations in a campus setting.
  • RSSI categories are used as preprocessing features because received signal strength fluctuates with environmental conditions.

A. Description of the Environment

The indoor-localization environment is a grid in which an agent uses recent RSSI observations and movement actions to approach a target location and receives distance-based rewards.

  • The agent observes RSSI values and uses its three most recent observations to select actions.
  • The agent can move among neighboring cells in eight directions, including cardinal and diagonal directions.
  • The environment is a grid of equal-size floor cells, with each position associated with iBeacon RSSI values.
  • The state includes RSSI values, the current row-and-column location, and target distance when labeled data are available.
  • The reward is positive within distance threshold δ and negative otherwise, increasing as the agent approaches the target.

V. EXPERIMENTAL RESULTS

The evaluation uses a real-world iBeacon deployment and RSSI-based smartphone sensing to assess indoor localization with the semi-supervised deep reinforcement learning model.

  • The evaluation was conducted in a real-world deployment on the first floor of Western Michigan University’s Waldo Library.
  • The experimental setup uses iBeacons and smartphone-collected RSSI values as input for identifying indoor locations.
  • The model learns from historical RSSI patterns and estimated positions to improve policy decisions for previously unseen RSSI values.

A. Dataset

The dataset comes from a real-world grid of 13 ceiling-mounted iBeacons deployed across a 200 ft. ×180 ft. library area with signal-obstructing pillars.

  • The dataset covers a 200 ft. ×180 ft. campus library area containing a grid of iBeacons.
  • Thirteen iBeacons were mounted on the first-floor ceiling of Waldo Library at Western Michigan University.
  • Pillars could deteriorate iBeacon signals, so the deployment arranged beacons to provide coverage from several devices.
  • Adjacent iBeacons were separated by 30-40 ft.

B. Preprocessing

Preprocessing augments raw RSSI readings with difference and categorical-membership features; the selected representation combines raw and S2 features into 169 inputs.

  • The preprocessing compares raw RSSI features with mutual-difference S1 and Boolean categorical-membership S2 features.
  • S1 represents pairwise RSSI differences between distinct iBeacons.
  • S2 assigns each RSSI value to a category defined by a specific interval, such as 10.
  • Adding S2 features improves average accuracy more than adding S1 features, especially with a categorical interval of 5.
  • The selected raw-plus-S2 representation contains 13 RSSI values and 156 range-membership features, totaling 169 features.
  • Each label identifies a specific location using row and column coordinates.

C. Evaluation

The evaluation compares supervised and semi-supervised DRL for indoor localization, showing that the semi-supervised model reaches higher rewards, smaller target distances, and faster convergence.

  • Experimental setup: The evaluation compares a fully connected supervised DRL model with a stacked variational autoencoder for semi-supervised learning.Both approaches are evaluated for indoor localization.
  • Episode comparison: In six episodes, the semi-supervised agent achieves higher rewards and smaller distances to the true target than the supervised agent.Figure 5 reports both received rewards and distance to the target.
  • Convergence: At least 4× faster convergence is reported for the semi-supervised localization policies.The comparison concerns average convergence speed to destination points under the same number of epochs.
  • Reward performance: At least 67% more rewards are obtained by the semi-supervised model, which reaches higher rewards faster while maintaining a stable reward trend.The model also achieves about twice the rewards of the supervised model.
  • Localization accuracy: The semi-supervised model improves localization by 6% to 23% across different epoch counts.Figure 7 compares average distance to target points for the two models.

VI. CONCLUSION

The paper proposes a semi-supervised DRL framework using a deep variational autoencoder to learn policies from mixed labeled and unlabeled data. In indoor localization, it reports improved accuracy and substantially more rewards than supervised DRL.

  • VI. CONCLUSION: The framework combines a small labeled dataset with a larger unlabeled dataset for semi-supervised deep reinforcement learning.It is presented as a learning mechanism for smart IoT services.
  • VI. CONCLUSION: A deep variational autoencoder network learns policies for the agent’s optimal actions.The VAE is the framework’s policy-learning component.
  • VI. CONCLUSION: The indoor-localization use case reports a 23% improvement in localization accuracy compared with using only labeled data in a supervised model.The conclusion characterizes the result as improved positioning performance for mixed labeled and unlabeled environment data.
  • VI. CONCLUSION: At least 67% more rewards are received by the semi-supervised model than by the supervised model.The comparison is reported for the indoor localization experiments.
  • VI. CONCLUSION: IoT and smart-city learning agents can benefit from unlabeled data while requiring little supervision through rewarding feedback.The conclusion frames this benefit within context-aware applications.
Loading 1810.04118v1…