Source-linked AI summary
The ASHRAE Great Energy Predictor III competition: Overview and results
Clayton Miller, Pandarasamy Arjunan, Anjukan Kathirgamanathan, Chun Fu, Jonathan Roth, June Young Park, Chris Balbach, Krishnan Gowri, Zoltan Nagy, Anthony Fontanini, Jeff Haberl
TL;DR
Building-energy prediction methods are difficult to compare across bespoke studies and restricted datasets. This paper describes GEPIII, a crowdsourced competition and benchmarking effort. The top five winners used ensembles, and the authors identify limited building-sector diversity as a boundary on applying the results elsewhere.
Problem
Comparing building-energy prediction methods is difficult because studies often use bespoke datasets, and prior benchmarking data were not fully open to the broader research community.
Source: §1 · p. 2Method
The paper describes GEPIII, a crowdsourced machine-learning competition for benchmarking commercial-building energy prediction workflows.
Source: §1 · p. 2Results
The top five winning teams consistently used ensembles, and the authors report that model accuracy was verified against tens of thousands of alternative models.
Source: §5 · p. 9, §8 · p. 17Takeaways & Limitations
The competition provides building-industry practitioners with models whose accuracy was checked against numerous alternatives.
Source: §8 · p. 17Most buildings in the dataset were from higher education, so applying the resulting models to other building sectors remains untested.
Source: §7.5 · p. 17
Abstract
from arXiv · showhide
In late 2019, ASHRAE hosted the Great Energy Predictor III (GEPIII) machine learning competition on the Kaggle platform. This launch marked the third energy prediction competition from ASHRAE and the first since the mid-1990s. In this updated version, the competitors were provided with over 20 million points of training data from 2,380 energy meters collected for 1,448 buildings from 16 sources. This competition's overall objective was to find the most accurate modeling solutions for the prediction of over 41 million private and public test data points. The competition had 4,370 participants, split across 3,614 teams from 94 countries who submitted 39,403 predictions. In addition to the top five winning workflows, the competitors publicly shared 415 reproducible online machine learning workflow examples (notebooks), including over 40 additional, full solutions. This paper gives a high-level overview of the competition preparation and dataset, competitors and their discussions, machine learning workflows and models generated, winners and their submissions, discussion of lessons learned, and competition outputs and next steps. The most popular and accurate machine learning workflows used large ensembles of mostly gradient boosting tree models, such as LightGBM. Similar to the first predictor competition, preprocessing of the data sets emerged as a key differentiator.
1. Introduction
Building-energy prediction research has grown, but comparing methods remains difficult because many studies use bespoke building datasets and restricted or inaccessible data. GEPIII uses a machine-learning competition to crowdsource and benchmark prediction workflows, extending a competition series begun in the 1990s.
1. Introduction
- Earlier reviews covered multiple prediction approaches, while more recent work focused on ensembles that usually outperformed other algorithms [4].The reviewed approaches included statistical and machine-learning methods, with ensembles combining models to reduce errors through diversity.Source: §1 · p. 1
- Bespoke studies and restricted data make it difficult to compare the growing range of building-energy prediction methods.Earlier benchmarking studies tested a limited set of techniques, and their data were not fully open to the broader research community.Source: §1 · p. 2
- GEPIII aims to crowdsource methods for hourly commercial-building energy prediction and provide tutorials for building scientists.The paper presents the competition as a source of machine-learning modeling knowledge and learning materials.Source: §1 · p. 2
1.1. Machine learning competitions as a means of crowdsourcing and benchmarking prediction models
- Machine-learning competitions compare workflows by giving contestants shared data, prediction objectives, constraints, and performance metrics.The paper presents competitions as a way to investigate solution generalizability and scalability.Source: §1.1 · p. 2, §1 · p. 2
- The 1993 Predictor Shootout I supplied hourly data from one institutional building, while the 1994 Shootout II used retrofit data from two buildings.The second contest recorded 47 submissions, of which four met all submission requirements.Source: §1.1 · pp. 2–3
2. Pre-Competition process
ASHRAE and Kaggle organized GEPIII around a multi-site dataset and a three-year prediction task. Contestants trained on 2016 data, used 2017 public validation data, and were judged on a withheld 2018 private test set.
2. Pre-Competition process
- The organizing committee planned GEPIII from 2018, secured funding and Kaggle’s hosting, and launched the competition in October 2019.The dataset and rules were prepared with Kaggle before the launch.Source: §2 · p. 3
2.1. Dataset development
- The dataset comprised 61,910,200 hourly energy measurements from 16 sites, with 73% from university campuses and 27% from municipal and healthcare repositories.Most sites were universities, so education was the most common building type; Figure 1 summarizes dataset characteristics and site contributions.Source: §2.1 · p. 3, Figure 1 · p. 4
2.2. Competition objective
- Contestants were tasked with counterfactual hourly energy prediction across four energy types using historical consumption and coincident weather data.The task framed retrofit savings as the difference between post-retrofit consumption and modeled pre-retrofit use.Source: §2.2 · pp. 3–4
2.2.1. Training data set
- Contestants received 2016 meter and weather data for training and were tasked with predicting the 2017 and 2018 data.They were expected to preprocess the data, create input features, and train machine-learning models.Source: §2.2.1 · p. 4
2.2.2. Public test /validation data set - the public leaderboard
- The withheld 2017 data served as public validation, and contestants could use discovered public-source data shared on the forum to refine strategies.Contestants were not told the exact division between public validation and private test data.Source: §2.2.2 · pp. 4–5
2.2.3. Private test data set - the private leaderboard
- The withheld 2018 data formed the private test set, whose score determined winners and was revealed on the private leaderboard after the competition.A site found to have an API was removed from the private test set to prevent leakage from affecting the final score.Source: §2.2.3 · p. 5, Figure 2 · p. 5
3. Competition procedures and rules
GEPIII used Kaggle’s submission process and rules to structure participation, scoring, and awards. The paper describes the RMSLE evaluation metric, the data supplied to contestants, and the competition timeline.
3. Competition procedures and rules
- Participants joined the Kaggle competition, accessed training, metadata, and weather data, then submitted predictions for an automatically scored public leaderboard.Figure 3 outlines this submission workflow.Source: §3 · p. 5, Figure 3 · p. 6
3.1. Rules overview
- Rules permitted teams of up to five and two daily entries, while restricting multiple accounts and private code sharing; automated-tool users could not win prizes.External data was allowed only if made freely available to all competitors and disclosed on the forum.Source: §3.1 · p. 5
3.2. Evaluation metric
- RMSLE scored submissions and reduced the influence of meters with much larger consumption values on the competition score.The nonprofit competition structure prevented the technical team from creating a custom scoring metric.Source: §3.2 · p. 5
3.3. Prizes and winners’ obligations
- The five prizes ranged from $10,000 for first place to $1,000 for fifth; winners had to deliver the final model’s code and documentation, capable of generating the winning submission, with resources needed to build and run it.Source: §3.3 · p. 6
3.4. Data files
- Contestants received training data files, building and meter metadata, and coincident weather data from the individual sites.Table 1 outlines the supplied files and associated data columns.Source: §3.4 · p. 6
3.5. Timeline
- The competition opened on October 15, 2019, and closed with the final submission deadline on December 19, 2019.Registration and team merger both closed on December 12, 2019.Source: §3.5 · p. 6
4. Overview of the competitors
The competition drew thousands of participants, whose experience and leaderboard scores varied by Kaggle tier; discussion-board analysis examined participant topics and sentiment.
4. Overview of the competitors
- Across 3,614 teams, 2,522 participants (58.1%) listed a country of origin, representing 80 countries; Japan, the USA, India, China, and Russia were the top five countries of origin.Source: §4 · p. 7, Figure 4 · p. 8
4.1. Discussion board overview
- The discussion analysis covered 293 posts and 2,508 comments from 713 competitors; manually tagged posts most often concerned questions, data sources, features, preprocessing, and modeling.Source: §4.1 · p. 7
4. Overview of the competitors
- Novices were the largest tier at 2,413 participants, versus 29 Grandmasters; higher ranks aligned with more prior competitions and lower (better) scores.Nearly all Novices had competed in one or fewer previous competitions, and they averaged the lowest scores.Source: §4 · p. 7, Figure 5 · p. 8
4.1. Discussion board overview
- Cooperation-focused posts tended positive, while posts about using public data for the public test or validation set were more negative.Source: §4.1 · p. 8
- TextBlob sentiment polarity was right-shifted, with a mean above zero; the authors present it as a view of this competition, not a comparison with other competitions.A polarity of +1.0 represents maximum positive sentiment and -1.0 maximum negative sentiment.Source: §4.1 · p. 8
5. Overview of the top five and medal-winning teams or individuals
The top five workflows were compared by preprocessing, feature, modeling, and post-processing strategies, and winners shared their workflows as a condition of receiving prizes. The solutions varied in their data handling and model combinations, while medal outcomes also differed across Kaggle ranking tiers.
5.1. First-place solution - Group-based ensembles using CatBoost, LightGBM, and MLP
- 1.231 private and 0.938 public; first place used grouped CatBoost, LightGBM, and MLP models with 28 engineered features.The team combined individual predictions with a generalized weighted mean.Source: §5.1 · pp. 9–10
5.2. Second-place solution - Intensive pre-processing and a huge XGBoost, LightGBM, CatBoost, and FFNN ensemble
- 1.232 private and 0.937 public; second place used XGBoost, LightGBM, CatBoost, and FFNN models, with manual outlier filtering.The team ensembled predictions using empirically determined weighted means; its FFNN models were limited to electrical meter id 0.Source: §5.2 · pp. 10–11
5.3. Third-place solution - Catboost and LightGBM with weighted post-processing
- 1.234 private and 0.946 public; third place combined CatBoost, neural networks, and LightGBM using weights informed by public datasets.The solution used 23 features, including derived and lagged weather features.Source: §5.3 · p. 11
5.4. Fourth-place solution - XGBoost and LightGBM
- Jun's final private leaderboard score was 1.235 and public score was 0.936 (rank 48); predictions ensembled two XGBoost and one LightGBM model, with weights determined using leaked data.The XGBoost models used 2-fold and 5-fold cross-validation, while LightGBM used 3-fold cross-validation. No systematic approach was followed to select optimal hyperparameters.Source: §5.4 · p. 11
5.5. Fifth-place solution - LightGBM ensemble with postprocessing weighting
- 1.237 private and 0.940 public; fifth place used only LightGBM in a two-step, building-and-meter-specific modeling process.The ensemble weights were determined from the public dataset.Source: §5.5 · p. 11
5.6. Other medal winners
- About 90% of Novice and Contributor participants earned no medals, while about a third of 29 Grandmasters earned one.Medal tiers were awarded by final private leaderboard ranking: Gold to the top 0.2%, Silver to the top 5%, and Bronze to the top 10%.Source: §5.6 · p. 11
6. Shared machine learning workflows using notebooks
Participants shared notebooks during the competition, creating examples of analysis and complete workflows. The paper examines notebook topics and presents the shared solutions as a directory for future users.
6. Shared machine learning workflows using notebooks
- Participants shared workflows as notebooks, which combine code, instructions, explanations, and other content on a shareable page.The paper says sharing helped level the playing field for mundane tasks such as detecting erroneous data.Source: §6 · p. 12
6.1. Overview of the notebook topics
- More than 96% of the 415 notebooks used Python, almost 84% used gradient boosting trees, and nearly half focused on preprocessing.Prediction models accounted for a third of notebooks; exploratory data analysis made up 16.1% and data scraping 3.86%. Figure 13 groups notebooks by language, model, general category, and user-assigned tag.Source: §6.1 · pp. 12–13, Figure 13 · p. 14
6.2. Complete reproducible machine learning solution examples
- The openly shared notebooks and discussion-board material yielded full machine-learning workflows, organized as a directory for users seeking particular methods or data-analysis approaches.Table 3 lists full solutions.Source: §6.2 · pp. 13–14
7. Discussion
The competition compared modeling approaches for its supplied data, and the authors identify preprocessing, gradient boosting trees, and ensembles as key findings. They also note that complex ensembles can be slower to train and predict.
7.1. Objective judgment of model types and steps for building energy prediction
- The authors frame their findings as generalizable to the type of data provided in the competition, based on winning solutions and shared full-solution examples.Their objective was to compare modeling configurations developed by thousands of practitioners.Source: §7.1 · p. 14, §7 · p. 14
7.1.1. Pre and postprocessing steps are essential and can’t always be automated
- Top teams spent substantial time filtering anomalous data and applying post-prediction weightings, while manual inspection illustrated that filtering cannot always be automated.The authors identify domain understanding as important for removing data that adds false signals to training.Source: §7.1.1 · p. 14
7.1.2. Gradient boosting tree models dominate
- Gradient boosting trees appeared in all winning solutions and most shared notebooks, with LightGBM the most commonly used framework.Other cited models include CatBoost, XGBoost, and LiteMORT; winners also used neural networks and random forests in large ensembles.Source: §7.1.2 · pp. 14–15
7.1.3. Ensembles of models create more accurate (but slower) solutions
- The top five used complex ensembles trained on varied data subsets; the authors describe them as more accurate but computationally intensive, requiring hours or days for training and prediction.The paper identifies complexity versus practical use as an area for further analysis and notes ensemble methods in prior building-science research.Source: §7.1.3 · p. 16
7.2. Convergence of skills and talent from the engineering and data science domains
The competition brought together data-science expertise and engineering knowledge relevant to building energy prediction.
7.2. Convergence of skills and talent from the engineering and data science domains
- Most contestants were outside building science or research, but engineering backgrounds were relevant to the context for a top-team member and the third-place winner.Source: §7.2 · p. 16
7.3. Dataset creation and preparation
Preparing the competition dataset required more data networking and schema work than expected, and cleaning errors remained in the data after launch.
7.3. Dataset creation and preparation
- Unit-conversion mistakes and weather timestamps misaligned with local meter time persisted after launch, but competitors identified the issues and shared notebook-based fixes.The team also found it harder than expected to collect enough datasets and to organize them in a consistent schema.Source: §7.3 · p. 16
7.4. Inclusion of publicly available data in the competition
The organizers used public data for the public leaderboard while reserving non-public data for winner determination. Competitors found publicly available test data online, prompting frustration but also shared methods and additional validation uses.
7.4. Inclusion of publicly available data in the competition
- Organizers used publicly available data only for public validation while reserving non-public data for the private leaderboard to evaluate winning solutions.The split was intended to allow a larger dataset while protecting winner evaluation integrity.Source: §7.4 · p. 16
- Competitors quickly found online public test data and called it leakage, but shared scraping methods openly and used discovered datasets to generate additional validation data.The winning solutions did not use all publicly available data that participants uncovered.Source: §7.4 · p. 16
- The private leaderboard, based on non-public test data, determined the winners; one unexpectedly exposed site was moved to the public leaderboard and removed from final scoring.The committee was unaware that the single site's data were online.Source: §7.4 · p. 16
- The organizers concluded future competitions should avoid public data even in public leaderboards, or disclose its presence to contestants from the start.They judged the resulting frustration more disruptive than the data's benefit.Source: §7.4 · p. 16
7.5. Generalizability of the competition results
The competition's results have limited application beyond the dataset's building mix, which was dominated by higher-education buildings.
7.5. Generalizability of the competition results
- Most buildings in the dataset were from education, especially higher education, so applying the resulting models to other building sectors remains untested.The authors identify dataset diversity as the primary constraint on applying the results to practice and research.Source: §7.5 · p. 17
- The authors propose community-driven dataset growth to test whether techniques need tuning or modification for broader generalizability.They also identify more representative building types and climates, plus objectives such as peak-load prediction, as future competition directions.Source: §7.5 · p. 17
8. Conclusion
The competition produced a crowdsourced benchmarking exercise intended to help practitioners improve building-performance prediction. Its outputs include reproducible winning solutions, open data, curated workflow examples, and future competition and dissemination plans.
8. Conclusion
- The competition created a crowdsourced benchmarking exercise, and practitioners can compare model accuracy against tens of thousands of alternatives developed by experts.The stated objective was to create value for practitioners improving prediction models for building performance analysis.Source: §8 · p. 17
8.1. A reproducible repository of the winning solutions
- The winning teams' solutions were published with reproduction instructions, execution resources, and simpler variants for users balancing complexity against accuracy.The repository includes the original submitted code, solution summaries, and team explanation videos.Source: §8.1 · p. 17
8.2. Open-source building energy benchmarking data
- The training and public test data remain available, and an expanded collection of buildings was released as Building Data Genome Project 2 (BDG2) [28].Researchers can also submit predictions to compare them against the public and private leaderboard data.Source: §8.2 · p. 17
8.3. Curated directory of machine learning workflow examples
- A continuously updated wiki curates competition notebooks by category, including tutorials for building-science professionals learning data science.The associated repository also contains data and code used to create several paper figures.Source: §8.3 · p. 17
8.4. Future workshops and seminars
- ASHRAE seminars and workshops were planned to disseminate results, including an online 2020 conference track on the top three winning solutions.The planned meetings were intended to connect winners with the ASHRAE community and transfer knowledge.Source: §8.4 · p. 17
8.5.1. CityLearn challenge
- The CityLearn Challenge explored reinforcement-learning agents for urban demand response, with centralized single-agent and decentralized multi-agent options.CityLearn is described as an OpenAI Gym environment; scaling direct energy management and control across buildings remains a major challenge.Source: §8.5.1 · pp. 17–18
8.5.2. Great Predictor IV competition
- A possible Great Predictor IV would use millions of building-management data streams to infer metadata and make energy-savings calculations more scalable.The proposed focus is time-series classification of building performance or operations data from thousands of buildings.Source: §8.5.2 · p. 18