Semi-supervised Deep Reinforcement Learning in Support of IoT and Smart City Services

Motivation

• IoT sensors generate large volumes of data without easily obtainable labels, creating a central challenge for smart-service development.Source: §I · p. 1
• Smart-building localization supports location-aware services and links indoor users to other IoT services.Source: §I · p. 1

Approach

• The proposed framework combines deep generative models with reinforcement learning to address semi-supervised learning.Source: §I · p. 2
• The model leverages both labeled and unlabeled data, targeting IoT settings where unlabeled data are more prevalent.Source: §I · p. 2

Evaluation

• The study applies the model to indoor localization in smart buildings using BLE or iBeacon signals.Source: §I · p. 2

Applications

• DRL has been used to learn medication-dosing policies that maximize therapeutic outcomes.Source: A. Deep Reinforcement Learning · pp. 2–3
• A CNN combined with reinforcement learning to guide visual attention for vehicle image classification.Source: A. Deep Reinforcement Learning · p. 3
• Traffic-signal timing was formulated as DRL using lane queues as states and lane-control choices as actions.Source: A. Deep Reinforcement Learning · p. 3
• DRL has been applied to job scheduling with rewards based on reciprocal job duration to minimize average slowdown.Source: A. Deep Reinforcement Learning · p. 3
• LSTM representations and DQN Q-function approximation supported agents in text-based games, while DRL also addressed energy management.Source: A. Deep Reinforcement Learning · p. 3

• Prior indoor-localization studies use technologies and models including WiFi, BLE, CSI, SVM, KNN, Bayesian filtering, transfer learning, and neural networks.Source: B. Review of Indoor Localization · p. 3
• Deep-learning and semi-supervised methods have reported improved fingerprinting accuracy over shallow or alternative methods.Source: B. Review of Indoor Localization · pp. 3–4
• Semi-supervised localization methods use unlabeled fingerprints with limited labeled data, including one approach reporting an average error of 2 meters.Source: B. Review of Indoor Localization · p. 4
• The paper states that prior deep-learning localization solutions rely on WiFi fingerprinting and that BLE fingerprinting had not been studied with deep or reinforcement learning.Source: B. Review of Indoor Localization · p. 4

• The model adopts a deep generative VAE framework for semi-supervised learning with labeled and unlabeled observations.Source: A. Semi-Supervised Learning Using VAE · p. 4, §III · p. 4
• For unlabeled data, the label y is treated as a latent variable alongside z in the generative model.Source: A. Semi-Supervised Learning Using VAE · p. 5
• M1 uses a Gaussian inference network for latent variable z, while M2 infers latent variables z and y with Gaussian and multinomial distributions.Source: A. Semi-Supervised Learning Using VAE · p. 5
• The training objective combines generative and discriminative components, with α adjusting their contributions during optimization.Source: A. Semi-Supervised Learning Using VAE · p. 5

• The reward function reflects closeness of the resulting state to the true class label, assigning larger positive rewards to more compelling results.Source: B. Semi-Supervised Deep Reinforcement… · p. 5
• The DRL agent approximates the optimal action-value function with a neural network and updates it through Q-learning and experience replay.Source: B. Semi-Supervised Deep Reinforcement… · p. 6
• The proposed model replaces the supervised DRL network with a generative semi-supervised model to process unlabeled observations.Source: B. Semi-Supervised Deep Reinforcement… · p. 6
• The VAE is extended with an additional hidden layer and an output that generates actions.Source: B. Semi-Supervised Deep Reinforcement… · p. 6
• Training occurs offline, while policy prediction occurs online for high-dimensional and high-volume data.Source: B. Semi-Supervised Deep Reinforcement… · p. 6

• The approach is also described for smart-building energy management and intelligent transportation navigation use cases.Source: §IV · p. 6
• The algorithm accepts labeled and unlabeled data, infers labels for unlabeled samples, and computes approximate rewards from state closeness.Source: §IV · p. 6
• The localization experiment was conducted in a large, busy academic library rather than a simulated environment, small area, or isolated testbed.Source: §IV · p. 7
• The case study uses iBeacon RSSI measurements as input to identify indoor locations in a campus setting.Source: §IV · p. 7
• RSSI categories are used as preprocessing features because received signal strength fluctuates with environmental conditions.Source: §IV · p. 7

• The agent observes RSSI values and uses its three most recent observations to select actions.Source: A. Description of the Environment · p. 7
• The agent can move among neighboring cells in eight directions, including cardinal and diagonal directions.Source: A. Description of the Environment · pp. 7–8
• The environment is a grid of equal-size floor cells, with each position associated with iBeacon RSSI values.Source: A. Description of the Environment · pp. 7–8
• The state includes RSSI values, the current row-and-column location, and target distance when labeled data are available.Source: A. Description of the Environment · p. 8
• The reward is positive within distance threshold δ and negative otherwise, increasing as the agent approaches the target.Source: A. Description of the Environment · p. 8

• The evaluation was conducted in a real-world deployment on the first floor of Western Michigan University’s Waldo Library.Source: §V · p. 8
• The experimental setup uses iBeacons and smartphone-collected RSSI values as input for identifying indoor locations.Source: §V · p. 8, Figure 4 · p. 8
• The model learns from historical RSSI patterns and estimated positions to improve policy decisions for previously unseen RSSI values.Source: §V · p. 8

• The dataset covers a 200 ft. ×180 ft. campus library area containing a grid of iBeacons.Source: A. Dataset · p. 8
• Thirteen iBeacons were mounted on the first-floor ceiling of Waldo Library at Western Michigan University.Source: A. Dataset · p. 8
• Pillars could deteriorate iBeacon signals, so the deployment arranged beacons to provide coverage from several devices.Source: A. Dataset · p. 8
• Adjacent iBeacons were separated by 30-40 ft.Source: A. Dataset · p. 8

• The preprocessing compares raw RSSI features with mutual-difference S1 and Boolean categorical-membership S2 features.Source: B. Preprocessing · p. 8
• S1 represents pairwise RSSI differences between distinct iBeacons.Source: B. Preprocessing · p. 8
• S2 assigns each RSSI value to a category defined by a specific interval, such as 10.Source: B. Preprocessing · pp. 8–9
• Adding S2 features improves average accuracy more than adding S1 features, especially with a categorical interval of 5.Source: B. Preprocessing · p. 9, Table II · p. 9
• The selected raw-plus-S2 representation contains 13 RSSI values and 156 range-membership features, totaling 169 features.Source: B. Preprocessing · p. 9
• Each label identifies a specific location using row and column coordinates.Source: B. Preprocessing · p. 9

Experimental setup

• The evaluation compares a fully connected supervised DRL model with a stacked variational autoencoder for semi-supervised learning.Both approaches are evaluated for indoor localization.Source: C. Evaluation · p. 9

Episode comparison

• In six episodes, the semi-supervised agent achieves higher rewards and smaller distances to the true target than the supervised agent.Figure 5 reports both received rewards and distance to the target.Source: C. Evaluation · p. 9

Convergence

• At least 4× faster convergence is reported for the semi-supervised localization policies.The comparison concerns average convergence speed to destination points under the same number of epochs.Source: C. Evaluation · p. 9, Table III · p. 9

Reward performance

• At least 67% more rewards are obtained by the semi-supervised model, which reaches higher rewards faster while maintaining a stable reward trend.The model also achieves about twice the rewards of the supervised model.Source: C. Evaluation · p. 9

Localization accuracy

• The semi-supervised model improves localization by 6% to 23% across different epoch counts.Figure 7 compares average distance to target points for the two models.Source: C. Evaluation · p. 9

VI. CONCLUSION

• The framework combines a small labeled dataset with a larger unlabeled dataset for semi-supervised deep reinforcement learning.It is presented as a learning mechanism for smart IoT services.Source: §VI · p. 9
• A deep variational autoencoder network learns policies for the agent’s optimal actions.The VAE is the framework’s policy-learning component.Source: §VI · p. 9
• The indoor-localization use case reports a 23% improvement in localization accuracy compared with using only labeled data in a supervised model.The conclusion characterizes the result as improved positioning performance for mixed labeled and unlabeled environment data.Source: §VI · p. 9
• At least 67% more rewards are received by the semi-supervised model than by the supervised model.The comparison is reported for the indoor localization experiments.Source: §VI · pp. 9–10
• IoT and smart-city learning agents can benefit from unlabeled data while requiring little supervision through rewarding feedback.The conclusion frames this benefit within context-aware applications.Source: §VI · p. 10