Source-linked AI summary
Multi-Target Regression via Input Space Expansion: Treating Targets as Inputs
Eleftherios Spyromitros-Xioufis, Grigorios Tsoumakas, William Groves, Ioannis Vlahavas
TL;DR
Multi-target regression must exploit dependencies among continuous targets, but the applicability of target-as-input methods from multi-label classification had not been established. The paper introduces SST and ERC, identifies a training–prediction discrepancy, and develops out-of-sample-estimate extensions. When the discrepancy is mitigated, the extensions consistently improve over independent regressions, while extended ERC versions significantly outperform four state-of-the-art methods.
Problem
Multi-target regression and multi-label classification share the challenge of exploiting dependencies among multiple targets, but target-as-input methods successful in multi-label classification had not been studied in multi-target regression.
Method
The paper adapts two multi-label classification methods into Stacked Single-Target and Ensemble of Regressor Chains, using other targets as additional inputs and out-of-sample estimates during training.
Results
When the discrepancy is mitigated, the proposed methods consistently improve over independent regressions, and extended ERC versions significantly outperform four state-of-the-art methods.
Takeaways & Limitations
Treating targets as additional inputs is an effective way to exploit target dependencies in multi-target regression when the training–prediction discrepancy is appropriately handled.
Takeaways & Limitations
The ideal amount of discrepancy mitigation differs by target, so some targets perform better with in-sample estimates or actual target values than with out-of-sample estimates.
Abstract
from arXiv · showhide
In many practical applications of supervised learning the task involves the prediction of multiple target variables from a common set of input variables. When the prediction targets are binary the task is called multi-label classification, while when the targets are continuous the task is called multi-target regression. In both tasks, target variables often exhibit statistical dependencies and exploiting them in order to improve predictive accuracy is a core challenge. A family of multi-label classification methods address this challenge by building a separate model for each target on an expanded input space where other targets are treated as additional input variables. Despite the success of these methods in the multi-label classification domain, their applicability and effectiveness in multi-target regression has not been studied until now. In this paper, we introduce two new methods for multi-target regression, called Stacked Single-Target and Ensemble of Regressor Chains, by adapting two popular multi-label classification methods of this family. Furthermore, we highlight an inherent problem of these methods - a discrepancy of the values of the additional input variables between training and prediction - and develop extensions that use out-of-sample estimates of the target variables during training in order to tackle this problem. The results of an extensive experimental evaluation carried out on a large and diverse collection of datasets show that, when the discrepancy is appropriately mitigated, the proposed methods attain consistent improvements over the independent regressions baseline. Moreover, two versions of Ensemble of Regression Chains perform significantly better than four state-of-the-art methods including regularization-based multi-task learning methods and a multi-objective random forest approach.
1 Introduction
The paper adapts dependency-modeling methods from multi-label classification to multi-target regression, introducing SST and ERC while addressing a training–prediction discrepancy in target-derived inputs. Experiments show that mitigating this discrepancy yields improvements over independent regressions and strong performance against state-of-the-art methods.
- Motivation: Multi-target regression predicts multiple continuous variables from shared inputs, and its central challenge is exploiting dependencies among targets.Applications include ecological modeling, economics, and energy.
- Contribution: The paper adapts two successful multi-label classification methods to multi-target regression, introducing Stacked Single-Target and Ensemble of Regressor Chains.Both methods treat other targets as additional input variables.
- Discrepancy and solution: The additional target-derived inputs differ between training and prediction, and this discrepancy can drastically reduce performance.The proposed extensions use out-of-sample target estimates during training, obtained through internal cross-validation.
- Evaluation: 18 diverse real-world datasets, including 14 newly used and publicly released datasets, support the experimental evaluation.The study compares the proposed methods with independent regressions and existing state-of-the-art approaches.
- Results: Direct adaptations fail to outperform independent regressions, whereas discrepancy-mitigating extensions produce consistent improvements; extended ERC versions significantly outperform four state-of-the-art methods.The compared methods include regularization-based multi-task learning and a multi-objective random forest approach.
2 Methods
The methods extend independent single-target regression by using other target estimates as additional inputs, either through stacking or ordered chains. Cross-validation-based out-of-sample estimates address the mismatch between meta-inputs during training and prediction, while ensembles reduce sensitivity to chain ordering.
- Stacked Single-Target: Stacked Single-Target (SST) trains independent first-stage regressors, then learns one meta-model per target on inputs augmented with estimated target values.At prediction time, first-stage outputs form the expanded inputs for the second-stage models.
- Regressor Chains: Regressor Chains (RC) select an ordered target permutation and sequentially predict each target using previous targets as additional inputs.Because previous true targets are unavailable for unknown instances, prediction uses estimates generated by preceding chain models.
- Ensemble of Regressor Chains: Ensemble of Regressor Chains (ERC) averages predictions from multiple RC models built with different random chains and bootstrap samples.The ensemble adapts the classifier-chain ensemble idea to regression and is intended to reduce sensitivity to a single chain ordering.
- Discussion: Both stacking and chaining add features that can reduce bias but increase variance, so gains depend on whether the bias reduction outweighs the variance increase.The methods may therefore benefit from base regressors or feature-selection procedures that reduce variance.
- Generation of Meta-inputs: The original training procedures create a train–prediction discrepancy because meta-inputs are generated differently across the two phases.SST uses in-sample estimates during training, whereas ERC uses actual target values; both rely on estimated values at prediction time.
- Generation of Meta-inputs: The proposed extensions use internal f-fold cross-validation to generate out-of-sample target estimates for all training examples, making training meta-inputs more compatible with prediction values.This avoids losing most second-stage training examples under a simple held-out approach.
3 Related Work
Prior MTR work includes statistical, tree-based, neural, Gaussian-process, and regularization-based multi-task methods, while this paper adapts dependency-aware MLC methods and investigates their meta-input discrepancy in MTR.
- Early MTR methods include Reduced Rank Regression, FICYREG, and two-block PLS, followed later by Curds and Whey.
- Statistical methods such as Curds and Whey, RRR, and FICYREG exploit correlated targets through transformed or shrunk output estimates.
- Other MTR approaches use predictive clustering trees, neural networks, Gaussian processes, or regularization-based multi-task learning to model multiple prediction tasks.
- Multi-task learning generalizes MTR by allowing tasks to differ in training examples, descriptive features, and data types while exploiting shared representations.
- Prior MLC work found discrepancy-related deterioration and reported benefits from rectified or estimate-based chain methods, motivating the present MTR analysis.
- This paper is the first to study the meta-input discrepancy issue in MTR and evaluates out-of-sample cross-validation estimates alongside prior in-sample comparisons.
4 Experimental Setup
The experiments compare proposed and competing multi-target methods across diverse datasets using cross-validated RRMSE and statistical tests, with separate analyses across datasets and target variables.
- Methods, Parameters and Implementation: The evaluation includes SST and ERC variants, the independent-regression ST baseline, and four state-of-the-art methods: MORF, TNR, DIRTY, and RLC.
- Methods, Parameters and Implementation: Proposed transformation methods decompose MTR into single-target regressions, typically using bagged regression trees; ERC ensembles use k = 10 random-chain models.
- Methods, Parameters and Implementation: SSTcv and ERCcv use f = 10 internal cross-validation folds to obtain out-of-sample target estimates during training.
- Evaluation: Performance is measured per target with RRMSE, comparing prediction RMSE against the RMSE of predicting the training-set target mean.
- Evaluation: RRMSE is estimated using k = 10-fold cross-validation on most datasets, with a reduced-fold procedure for datasets containing more than 9000 examples.
- Evaluation: Method differences are assessed with the Friedman test and, when p < 0.01, the Nemenyi post-hoc test.
- Evaluation: Because multi-target evaluation yields multiple measurements per dataset, significance is analyzed using both average RRMSE per dataset and RRMSE per target.
- Datasets: The study compiles a large, diverse benchmark collection combining five previously used public datasets with additional datasets made available for future benchmarking.
5 Experimental Analysis
Experiments show that handling meta-input discrepancy is crucial: estimate-based SST and ERC variants improve over independent regression, while direct adaptations generally do not. ERC variants perform especially strongly against state-of-the-art methods, although gains over a strong base regressor involve additional computation.
- 5.1 Base Regressor Exploration: ST-BAG obtained the lowest average error in nine datasets and was selected as the base regressor for subsequent experiments.ST-SGB won five datasets, while ST-RIDGE and ST-SVR each won two.
- 5.2 Evaluation of Direct Adaptations: SSTtrain ranked best among the direct adaptations, but neither SSTtrain nor ERCtrue significantly outperformed ST, and both sometimes performed worse.SSTtrain was significantly better than several state-of-the-art methods, while ERCtrue remained competitive but generally ranked lower.
- 5.3 Evaluation of Meta-input Generation Variants: Using target estimates instead of actual target values significantly improved both SST and ERC, demonstrating that the training–prediction meta-input discrepancy strongly affects performance.The true variants had the worst average ranks, while train and cross-validation variants performed better.
- 5.3 Evaluation of Meta-input Generation Variants: ERCtrain and ERCcv were significantly better than the remaining methods in per-target analysis, and every ERC variant ranked above its corresponding SST variant.The four estimate-based variants ranked below ST, although only ERCtrain and ERCcv were significantly better in the per-target analysis.
- 5.4 Comparison with the State-of-the-art: The best proposed variants significantly outperformed TNR, DIRTY, and MORF, while ERCcv and ERCtrain also significantly outperformed RLC in per-target analysis.They were also more cautious than TNR, DIRTY, MORF, and RLC, with fewer extreme degradations relative to ST.
- 5.6 Discussion: ERCcv and ERCtrain improved over ST at increased computational cost, which was considered manageable especially for ERCtrain.Parallelization, smaller ensemble size, or fewer cross-validation folds can reduce the cost.
6 Conclusion
The paper introduces SST and ERC for multi-target regression by adapting multi-label methods that use other targets as inputs. Extended variants mitigate training–prediction discrepancy with out-of-sample target estimates, yielding consistent and significant improvements over independent regression.
- 6 Conclusion: SST and ERC adapt established multi-label classification methods for multi-target regression.Both treat other prediction targets as additional input variables to exploit target dependencies.
- 6 Conclusion: Directly adapted SST and ERC variants can fail to improve, or can degrade, independent regression performance.The degradation is attributed to discrepancies between additional-input values during training and prediction.
- 6 Conclusion: Extended variants use out-of-sample target estimates during training to mitigate the discrepancy between training and prediction inputs.The paper reports consistent and significant improvements over the independent-regression baseline.
- 6 Conclusion: The study concludes that exploiting relationships between similar machine-learning tasks can be effective even through relatively simple method adaptations.This conclusion follows from the reported improvements obtained by adapting multi-label classification methods.
- 6 Conclusion: The authors identify target-specific mitigation as a future research direction because different targets may favor different estimate types.They also propose studying which problem characteristics favor each variant and adapting the methods to settings where variables are available only during training.
A Datasets
The evaluation uses a diverse collection of real-world multi-target regression datasets from domains including machining, energy, advertising, public services, employment, transportation, river flow, and supply chains.
- A Datasets: The Electrical Discharge Machining dataset predicts two numeric machining-control targets from 16 continuous inputs.Each target takes one of three values: {−1,0,1}.
- A Datasets: The Energy Building dataset predicts heating and cooling load requirements from building parameters such as glazing area, roof area, and height.It represents an energy-efficiency prediction problem.
- A Datasets: The advertising dataset contains 12 targets representing monthly sales during the first 12 months after product launch.Examples with missing target values and attributes with one distinct value were removed.
- A Datasets: The SCPF dataset predicts views, clicks, and comments for civic issues using a random 1% sample spanning four US cities and 12 months.The data come from the See Click Predict Fix competition.
- A Datasets: Additional datasets cover employment counts, airline ticket prices, river flows, and supply-chain prices across multiple targets and forecasting horizons.The passages describe OES, ATP, RF, and SCM datasets, including 16 SCM regression targets.
B Detailed Experimental Results
The detailed-results section reports base-regressor comparisons, multi-target regression results, and running-time analyses, including average and per-target RRMSE reporting for ST.
- B.1 Base Regressor Exploration Results: The base-regressor analysis compares ST using RIDGE, TREE, SVR, BAG, and SGB.Table 5 reports average RRMSE over all targets and RRMSE for each target.
- B.1 Base Regressor Exploration Results: Table 5 identifies the lowest error in each row using bold typesetting.This supports comparison across the reported average and per-target RRMSE values.
- B.2 Multi-target Regression Results: The multi-target regression results are presented in Table 6 and its continuation pages.The supplied continuation contains numeric result rows for datasets including mtlp13a through mtlp16a.