Source-linked AI summary
Constrained Hyperparameter Optimization for Streaming Data
Bruno Veloso, João Gama
TL;DR
Streaming hyperparameter optimization must adapt to concept drift while keeping generated configurations within bounded search spaces. The paper integrates four boundary-constraint strategies into two online optimizers and evaluates them empirically. Reflection and Random do not improve optimizer performance, whereas other strategies provide improvements that depend on the optimizer and machine-learning task.
Problem
Online learning requires hyperparameter self-tuning for evolving data streams, while optimization operators can generate invalid configurations outside bounded search spaces.
Method
The study integrates Centroid, Random, Reflection, and Wrapping boundary strategies into heuristic and evolutionary online hyperparameter optimization algorithms.
Results
Reflection and Random do not improve optimizer performance, while the other strategies can improve performance depending on the optimizer and machine-learning task.
Takeaways & Limitations
Boundary-constraint strategy choice affects online optimizer efficacy, and its benefit is dependent on the optimizer and task.
Takeaways & Limitations
The findings motivate further exploration of more sophisticated boundary-constraint strategies for online optimization algorithms.
Abstract
from arXiv · showhide
Optimization of hyperparameters is a critical factor to obtain optimal model performance. While existing research has predominantly concentrated on batch-learning scenarios, addressing the complexities inherent in data streams presents a challenge. The deployment of sophisticated methodologies to manage data streams becomes highly important. Consequently, the capacity for self-adjusting hyperparameters during on-line learning phases emerges as a goal. Many hyperparameters exhibit constraints and are confined within bounded search spaces, rendering specific solutions unacceptable upon applying optimization operators. To solve this issue, employing boundary constraint- handling techniques becomes imperative to rectify invalid solutions. This paper presents strategies for effectively managing boundary constraints within constrained numerical optimization problems. Recent methodologies, including heuristic and evolutionary-based optimization, employ a "boundary" strategy, wherein values that surpass boundary thresholds for a given hyperparameter are realigned to the respective limits. Our study introduces four strategies to navigate boundary constraints in online optimization algorithms. Through empirical investigations conducted on established datasets, we demonstrate that adopting boundary strategies outperforms the "boundary" strategy.
1 Introduction
Online hyperparameter optimization must handle concept drift and bounded search spaces while adapting models during streaming data. This study adapts online optimizers and integrates boundary-constraint strategies to rectify invalid configurations.
- Motivation: Static-data models often struggle with concept drift, requiring tuning to restart when the current data distribution changes.The paper identifies online hyperparameter self-tuning as an important response to evolving data streams.
- Motivation: Optimization operators can generate solution vectors outside specified hyperparameter bounds, producing invalid results.Boundary-constraint methods are therefore needed to restore solutions to the admissible region.
- Approach: The study adapts Nelder-Mead and micro-evolutionary online optimizers for boundary-constrained optimization.The objectives include a single pass over data, concept-drift response, and correction of invalid configurations.
- Approach: Centroid, Random, Reflection, and Wrapping strategies are integrated to restore invalid vectors to the admissible region.These methods were not originally designed for online constrained problems requiring solutions to remain feasible throughout optimization.
- Evaluation: The paper evaluates boundary-constraint strategies within two online hyperparameter optimization algorithms.The contribution is framed as an evaluation of how these strategies affect online optimization methods.
2 Related Work
Prior work covers online hyperparameter optimization, evolutionary adaptation, and boundary handling, but these strands have rarely been developed together for constrained online optimization. Existing online strategies also face difficulties during exploration on dynamic data streams.
- Online hyperparameter optimization: Online hyperparameter optimization research is relatively recent compared with broader AutoML research, and the literature search focuses on online tuning and boundary constraints.This frames the related work around streaming adaptation rather than general algorithm selection.
- Online hyperparameter optimization: Hyper-gradient methods have been integrated into online learning, but they do not explicitly accommodate concept drift according to cited work.Other systems apply AutoML to real-time streams and emphasize adaptation to drift events.
- Online hyperparameter optimization: Evolutionary approaches have addressed dynamic online optimization, including genetic tuning with and without explicit concept-drift handling.The cited studies include deep recurrent models and an online CASH formulation.
- Boundary constraint strategies: Boundary handling includes projection to violated limits, reflection from the violated bound, wrapping from the opposite bound, and random perturbation.These strategies are described as general approaches for optimization problems with bounded variables.
- Boundary constraint strategies: The Centroid method corrects an invalid vector using a centroid formed from nearby solution vectors and randomly corrected vectors.The supplied passage describes the method but truncates the precise number of vectors involved.
- Research gap: Prior boundary approaches were not explicitly developed for constrained online optimization, where admissible solutions must remain inside a defined feasible region.Online strategies also face exploration challenges because streaming data are inherently dynamic.
3 Proposed Method
The proposed method combines online hyper-parameter tuning with boundary-constraint strategies that restore invalid configurations to the admissible search region. It uses heuristic and micro-evolutionary optimizers alongside five computationally inexpensive constraint-handling methods.
- Online hyper-parameter tuning: SPT alternates exploration and exploitation, restarting exploration when ADWIN or DDM detects concept drift.Exploration uses expansion, contraction, shrinkage, and reflection operators; exploitation uses the best configuration identified during exploration.
- Online hyper-parameter tuning: MESSPT uses mutation and crossover during exploration in a micro-evolutionary online tuning procedure.The supplied passage identifies its exploration operators but does not complete the description of its second operational mode.
- Boundary constraint strategies: The integrated boundary strategies are Boundary, Centroid, Random, Reflection, and Wrapping, selected for their relatively low computational cost.These methods restore invalid vectors to the admissible region, although they were not originally designed for online constrained problems.
- Boundary constraint strategies: Boundary projection clips out-of-range values to Min_hp or Max_hp, whereas Reflection returns them across the violated bound according to the violation amount.Both methods preserve values already inside the interval [Min_hp, Max_hp].
- Boundary constraint strategies: Centroid computes a rectification value from k previous configurations, while Random replaces an invalid variable with a uniformly generated value inside its bounds.Wrapping treats each dimension as periodic and repositions values beyond a bound within the search space.
4 Results
The experiments evaluate five boundary constraint methods integrated with SPT and MESSPT using sequential online learning. Results vary by optimizer and task, with distinct strategies performing best under classification, regression, and drift conditions.
- Evaluation Protocol: The experimental framework integrates SPT and MESSPT into RiverML and evaluates them using the Prequential protocol.New instances are tested sequentially and then used for training.
- Evaluation Protocol: The study evaluates five boundary constraint methods across classification and regression tasks using average accuracy or RMSE.The methods are assessed across datasets and optimizer types.
- Optimizer Results: For SPT, centroid performs best for classification, whereas reflection performs best for regression.These task-specific comparisons are reported from the average dataset results.
- Optimizer Results: For MESSPT, Boundary performs best for classification, while Wrapper performs best for regression.The reported evolutionary-optimizer results differ from the heuristic-optimizer pattern.
- Concept Drift: MESSPT behaves better under drift conditions, while ADWIN restarts optimization after detected drift.The restart design may introduce latency in detecting concept drift.
- Concept Drift: Under concept drift, SPT improves with Random or Centroid for classification, while evolutionary optimization favors Wrapper and Boundary.The comparison uses datasets containing abrupt concept drifts.
5 Conclusions
The study integrates boundary constraint strategies into two online optimization algorithms to repair invalid configurations. Its results show that strategy effectiveness depends on the optimizer and machine-learning task, motivating more sophisticated methods.
- Contribution: The study integrates randomization, centroid adjustment, reflection, and wrapper strategies into two online optimization algorithms.These methods rectify invalid configurations produced during optimization.
- Findings: Boundary constraint strategies affect optimizer efficacy, but Reflection and Random do not improve performance consistently.Other strategies can improve results depending on the optimizer and machine-learning task.
- Future Work: The findings motivate further exploration of more sophisticated boundary constraint strategies for online optimization algorithms.This is presented as a direction for future research.