Source-linked AI summary
From Thermal Preference Prediction to Adaptive Thermal Intervention: A Reinforcement Learning Approach Using Physiological and Environmental Sensing
Isibor Kennedy Ihianle, Emmanuel Manu, Ehsan Asnaashari, Mojgan Jadidi, Pedro Machado, Amrit Sagoo, Ahmad Lotfi
TL;DR
Conventional HVAC comfort models and static controls do not capture substantial occupant-specific thermal variability, motivating more personalised approaches. The paper combines multimodal sensing with participant-specific probabilistic Comfort Oracles and reinforcement learning for adaptive intervention policies. Evaluation on a wearable thermal-comfort dataset supports Comfort Oracles as robust representations despite inter-participant variability, while controller choice depends on operational priorities.
Problem
Conventional HVAC systems rely on static setpoints and population-level comfort models that do not capture individual physiological variability.
Method
The paper combines multimodal physiological and environmental sensing with participant-specific probabilistic Comfort Oracles embedded in Contextual Bandit, Q-learning and DQN controllers.
Results
Personalised Comfort Oracles robustly represent individual thermal preference despite substantial inter-participant variability, while controllers exhibit differing comfort, intervention-efficiency and policy-stability characteristics.
Takeaways & Limitations
Controller selection should reflect whether the operational priority is maximising comfort, reducing intervention magnitude or balancing both objectives.
Takeaways & Limitations
The evaluation preserves participant-specific chronological train-validation-test partitions without random shuffling or stratified sampling.
Abstract
from arXiv · showhide
Personalised thermal comfort is essential for occupant wellbeing and for the development of more responsive building-control strategies, yet conventional Heating, Ventilation, and Air Conditioning (HVAC) systems rely on static setpoints and population-level comfort models that fail to capture individual physiological variability. This paper presents a two-stage personalised thermal comfort approach integrating multimodal physiological and environmental sensing with reinforcement learning-based decision-making.
1. Introduction
Conventional HVAC systems often use static, population-level comfort models despite substantial individual differences in thermal responses. The paper addresses this gap by combining personalised comfort prediction with reinforcement-learning intervention policies.
- Thermal comfort affects wellbeing, productivity and satisfaction, while suboptimal conditions are linked to reduced cognitive performance, impaired sleep and occupant complaints.
- Static setpoints, rule-based schedules and population-average models such as PMV/PPD fail to represent physiological, behavioural and environmental differences between occupants.
- Multimodal wearable and environmental sensing enables personalised comfort modelling from physiological signals and weather-derived variables.
- Prediction alone does not determine how an estimated cooler, warmer or unchanged preference should be translated into a thermal intervention.
- The proposed architecture develops probabilistic participant-specific Comfort Oracles and embeds them in reinforcement-learning controllers to recommend adaptive interventions.
- The study compares Contextual Bandit, Q-learning and DQN controllers across comfort, reward, intervention and preference-agreement measures.
2. Related Work
Prior work has advanced personalised comfort prediction from multimodal sensing and intelligent HVAC control through reinforcement learning, but these strands remain largely disconnected. The paper targets the missing integration of probabilistic personalised comfort outputs with sequential intervention decisions.
- Traditional PMV/PPD models assume steady-state conditions, fixed metabolic rates and uniform environments, limiting their ability to capture adaptation and individual variability.
- Wearable sensing measures physiological and environmental signals that support increasingly personalised thermal comfort inference.
- Ensemble, deep, hybrid and transformer models have been applied to multimodal data, while probability-based comfort outputs offer richer feedback for downstream control.
- RL-HVAC research includes DQN, SAC, PPO and multi-agent methods, with reported improvements in energy efficiency and comfort maintenance over rule-based controllers.
- Existing RL-HVAC studies often rely on indoor temperatures, setpoints or building thermal models unavailable in wearable-sensor datasets.
- The unresolved gap is the lack of integration between personalised comfort inference and sequential decision-making for real-time HVAC control.
- The paper embeds probabilistic Comfort Oracle predictions in RL states and compares multiple RL paradigms under identical personalised conditions.
3. The Proposed Two-Stage Architecture
The proposed architecture uses two stages: a participant-specific probabilistic Comfort Oracle followed by reinforcement-learning controllers that select equivalent environmental-temperature interventions. This links physiological and environmental sensing to adaptive thermal decision-making.
- Stage 1: Personalised thermal preference prediction: The architecture first predicts each occupant’s thermal preference from personalised physiological and environmental observations.
- Stage 1: Personalised thermal preference prediction: Stage 1 combines heart rate, wrist and ankle skin temperatures, body-proximity temperature, outdoor temperature, humidity, wind speed and solar radiation.
- Stage 2: Reinforcement-learning intervention: Stage 2 supplies physiological state, weather context, body-proximity state and oracle probabilities to a controller that selects a discrete equivalent environmental-temperature action.
- Stage 2: Reinforcement-learning intervention: Contextual Bandit, Q-learning and DQN are evaluated, contrasting one-step optimisation with controllers that consider future comfort rewards.
4. Experimental Methodology
The methodology develops participant-specific Comfort Oracles from multimodal sensing, then embeds their probabilistic outputs in reinforcement-learning controllers for adaptive proxy intervention policies. The evaluation keeps personalised data partitions and control objectives consistent while comparing controller strategies.
- Stage 1: Stage 1 predicts each participant’s preference for cooler, no-change, or warmer conditions from physiological and environmental feature representations.Models are trained independently per participant to account for individual differences.
- Stage 1: The Comfort Oracle outputs a probability distribution whose neutral-class probability estimates whether no environmental adjustment is required.This probabilistic output forms the interface with Stage 2 controllers.
- Stage 1: RF, GB, and ET are trained per participant and feature representation, ranked on validation data, then retrained before held-out testing.One selected Comfort Oracle is produced for every participant–feature representation combination.
- Stage 2: Stage 2 embeds participant-specific Comfort Oracles within CB, QL, and DQN controllers to generate adaptive intervention recommendations.The controllers operate on a unified personalised state representation rather than discrete preference labels.
- Stage 2: The controller state includes personalised comfort probabilities, physiological and environmental features, and the previous control action.Including the previous action provides short-term context and supports penalising rapid successive changes.
- Stage 2: Actions represent relative decreases, no change, or increases in equivalent environmental temperature, not direct thermostat setpoints or realised temperature changes.After each action, the proxy state is updated and re-evaluated by the participant-specific Comfort Oracle.
- Stage 2: The reward balances predicted comfort with intervention magnitude, action switching, and an intervention-based energy proxy.Identical state, action, and reward definitions support comparison of controller decision-making strategies.
5. Experiments and Results
The experiments evaluate the two-stage architecture sequentially: first assessing personalised Comfort Oracle prediction, then assessing reinforcement-learning control performance. Prediction and control use distinct metric sets covering predictive quality, comfort, intervention, efficiency, and behavioural stability.
- Evaluation: Stage 1 prediction performance is evaluated using accuracy, Macro F1, and Cohen’s κ.These metrics assess personalised Comfort Oracle predictive performance.
- Evaluation: Stage 2 control performance is assessed using comfort probability, reward, intervention magnitude, reward efficiency, and action agreement.The metrics capture both intervention outcomes and controller behaviour.
5.1. Stage 1 Personalised Thermal Preference Prediction
Stage 1 evaluates participant-specific Comfort Oracles using three ensemble classifiers and four physiological/environmental feature representations. Results show modest aggregate differences but substantial participant variability, supporting personalised model selection.
- Candidate models and representations: Three ensemble classifiers were evaluated across physiological-only, 60-minute, 480-minute, and combined environmental feature representations.The candidate models were Random Forest, Gradient Boosting, and Extra Trees.
- Aggregate predictive performance: Mean accuracies ranged from approximately 0.69 to 0.73, with Phys+480 consistently achieving the highest mean accuracy across classifiers.Environmental features provided only modest improvements over physiological features alone.
- Metric-specific model performance: Gradient Boosting achieved the highest Macro F1 and Cohen’s 𝜅, while Random Forest generally achieved the highest overall accuracy.Extra Trees remained competitive but showed greater variability across feature representations.
- Participant variability: Participant accuracies ranged from approximately 0.39 to 0.93, with corresponding variation in Macro F1 and Cohen’s 𝜅.Participant-level differences included consistently high performance for Participant 4 and lower, more variable performance for Participants 6 and 12.
- Personalised model selection: Selected Comfort Oracle models varied across participants and feature representations, with no single learning algorithm consistently providing the highest predictive performance.Random Forest was selected most frequently for more stable preference patterns, whereas Gradient Boosting and Extra Trees were selected for participants with greater class imbalance or variability.
- Representation comparison: Phys+480 achieved a mean prediction accuracy of 0.715, but its Macro F1 and Cohen’s 𝜅 improvements were modest and remained within one standard deviation of other representations.Friedman tests found no significant differences among feature representations for Accuracy (𝑝= 0.969), Macro F1 (𝑝= 0.180), or Cohen’s 𝜅 (𝑝= 0.430).
5.2. Stage 2 Thermal Control Performance
Stage 2 compares reinforcement-learning controllers that translate personalised Comfort Oracle outputs into closed-loop proxy interventions. Controller choice creates a comfort–intervention-efficiency trade-off, with statistically significant behavioural differences across participants and feature representations.
- Controller performance: CB achieved the highest comfort probability (0.684 - 0.700) and cumulative reward (87.4 - 89.9) across oracle representations.Its average temperature adjustments exceeded 3.3◦C, reducing reward and comfort efficiency relative to QL.
- Comfort–energy trade-off: QL produced the smallest HVAC interventions at approximately 2.6◦C and the highest reward efficiency (31.3 - 32.1) and comfort efficiency (0.249 - 0.254).Approximately 98% of its decisions maintained the current temperature.
- Closed-loop behaviour: CB generated more responsive, frequent adjustments, QL smoother trajectories with dominant “No Change” actions, and DQN more variable switching with larger action changes.The plotted room-temperature terms denote observed and controller-generated proxies rather than measured indoor temperatures or physical HVAC actuation.
- Action agreement: QL reached the highest overall action accuracy, 0.688 - 0.691, but low balanced accuracy, 0.327 - 0.332, Macro F1, 0.278 - 0.285, and negative Cohen’s 𝜅.Its accuracy was largely driven by frequent selection of the dominant “No Change” action.
- Action agreement: DQN achieved the strongest minority-class sensitivity under Phys+60, with balanced accuracy 0.458, Macro F1 0.395 and Cohen’s 𝜅 0.129, despite overall accuracy 0.442.CB had lower overall accuracy than QL but more consistent behavioural agreement across feature representations.
- Statistical comparison: Controller differences were statistically significant across all oracle representations (all 𝑝< 0.001), indicating policy selection had greater influence than feature representation on regulation performance.The controllers learned distinct comfort, intervention, and efficiency policies from the same personalised Comfort Oracle outputs.
5.3. Discussion
The discussion frames personalisation as the foundation for intelligent thermal control and reinforcement learning as the mechanism that closes the loop from preference prediction to intervention. Results indicate that controller selection, rather than feature representation, should follow operational priorities.
- Personalisation: Stage 1 found substantial inter-participant variability, with no single learning algorithm or feature representation consistently outperforming others across participants.Phys+480 had the highest average predictive accuracy, but differences were not statistically significant.
- Personalisation: Participant-specific Comfort Oracles therefore represent occupant thermal preference more appropriately than a single population-wide prediction model.The discussion attributes the dominant variation to individual physiological and behavioural characteristics rather than model architecture.
- From prediction to decision-making: Reinforcement learning extends personalised prediction into sequential intervention policies that balance comfort and intervention efficiency.This closes the decision loop beyond prediction-only studies and deterministic consensus-based adjustment.
- From prediction to decision-making: Controller design became the dominant factor governing adaptive regulation: CB maximised comfort and reward, whereas QL minimised interventions and improved efficiency.Controller selection should reflect whether the operational priority is comfort, reduced intervention magnitude, or a balance of both.
- Practical implications: The approach provides a pathway toward occupant-centric HVAC systems that adapt environmental-intervention policies to individual physiological responses.The study’s evaluation uses proxy interventions rather than measured HVAC operation, with real-time energy data identified as a future practical context.
6. Conclusion
The paper presents a two-stage system combining personalised Comfort Oracles with reinforcement learning for adaptive thermal intervention-policy generation. Stage 2 shows that controller policy has a greater influence on regulation than the evaluated feature representations.
- Conclusion: The approach integrates wearable physiological sensing, participant-specific Comfort Oracles and reinforcement learning for adaptive thermal intervention-policy generation.It combines personalised thermal preference modelling with sequential decision-making in a human-in-the-loop environmental-regulation framework.
- Conclusion: Personalised Comfort Oracles robustly represent individual thermal preference despite substantial inter-participant variability.Stage 1 found only modest differences among the evaluated feature representations and ensemble learning models.
- Conclusion: Stage 2 found that RL policy had a much greater influence on adaptive thermal regulation than the evaluated feature representations.CB maximised comfort and cumulative reward, QL achieved the highest reward and comfort efficiency conservatively, and DQN provided a more flexible but less stable strategy.
Appendix A. Additional Tables
The appendix provides supplementary tables covering participant-level Stage 1 performance and Stage 2 performance across oracle representations and controller variants.
- Table A.1: Table A.1 reports participant-level Stage 1 performance across all feature representations using Accuracy, Macro F1 and Cohen’s 𝜅.Values are presented as Accuracy/Macro F1/Cohen’s 𝜅.
- Table A.2: Table A.2 reports Stage 2 controller performance across all oracle feature representations and controller variants.It complements the selected-controller comparison reported in the main text.