PaperlayerView exact revision on arXiv

Source-linked AI summary

NPBoost: Neural Processes with Gradient-Boosted Fixed Effects

Andrea Nava, Ken Rölli, Armin Begic, Fabio Sigrist

arXiv:2609.28122v1stat.MLcs.LG

TL;DR

Most Neural Process extensions focus on network architecture, while shared structure and task-specific variation can instead be modeled separately. NPBoost assigns shared patterns to boosted trees and task-specific effects to a Neural Process, improving RMSE across most real-world comparisons and generally matching or improving on standard NPs in synthetic settings.

  • Problem

    Most Neural Process extensions focus on improving network architecture rather than explicitly separating structure shared across tasks from stochastic task-specific variation.

    Source: §1 · p. 2

  • Method

    NPBoost models shared fixed effects with a gradient-boosted tree ensemble and task-specific random effects with a Neural Process, jointly training them through a shared predictive objective and boosting algorithm.

    Source: §1 · p. 2

  • Results

    Across synthetic settings, NPBoost achieved lower RMSE and CRPS than standard NP in most settings; on real-world datasets it reduced RMSE in seven of eight comparisons, but CRPS improvement was not consistent.

    Source: §6 · p. 8

  • Takeaways & Limitations

    For tabular meta-learning, NPBoost offers an extension that decomposes predictions into shared tree-ensemble structure and task-specific residual effects without modifying the NP architecture.

    Source: §6 · p. 8

    The formulation assumes a mixed-effects model for each task.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Neural Processes (NPs) are model-based meta-learners that implicitly learn a stochastic process and adapt to a new task from a small context set. Most extensions of NPs focus on improving the neural network architecture. We instead develop an extension motivated by the shared hierarchical interpretation of meta-learning and mixed-effects models. Specifically, we introduce Neural Process Boosting (NPBoost), which decomposes structured response variability into tree-boosted fixed effects shared across tasks and NP random effects that capture stochastic task-to-task variation. We propose to train the two components jointly using a boosting algorithm in which an NP learns residual task-specific structure and a tree ensemble estimates common patterns across tasks. Across synthetic and real-world tabular meta-learning problems, this decomposition improves over a standard NP when the shared structure contains discontinuities or other irregular patterns that boosted trees can represent effectively.

1 INTRODUCTION

The paper reframes meta-learning through a hierarchical, mixed-effects perspective and proposes separating shared response structure from stochastic task-specific variation. NPBoost combines a tree ensemble and a Neural Process under a shared predictive objective, with joint training and evaluation on tabular meta-learning problems.

1 INTRODUCTION

  • The authors motivate the decomposition by the hierarchical view that information learned across tasks acts as a prior when adapting to a new task.This perspective connects meta-learning with mixed-effects models, which distinguish shared fixed effects from task-specific random effects.Source: §1 · p. 1
  • The approach differs from architecture-focused Neural Process extensions by explicitly separating shared structure from stochastic task-specific variation.The authors identify boosted trees as suited to irregular, nonlinear, and discontinuous shared patterns, while the NP supports uncertainty-aware adaptation from a small context set.Source: §1 · pp. 1–2
  • NPBoost decomposes response variation into a shared fixed-effects component modeled by gradient-boosted trees and task-specific random effects modeled by a Neural Process.The components are coupled through a shared marginal predictive objective and trained jointly with a boosting algorithm.Source: §1 · p. 2
  • The contribution includes context–target cross-fitting, which enables second-order boosting updates despite the context-dependent nature of Neural Process predictions.The paper also formulates the components under a shared marginal predictive objective.Source: §1 · p. 2
  • The paper evaluates NPBoost on synthetic and real-world tabular meta-learning problems.The evaluation is one of the three stated contributions.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The background establishes task-based prediction and relates meta-learning to hierarchical models, Neural Processes, and mixed-effects models. It also situates NPBoost as a change to the response decomposition rather than an architectural modification.

2.1 Notation and Setup

  • The setup considers related tasks with potentially varying observation counts and predicts target outputs from labeled context observations, either within training tasks or on unseen tasks.The two settings are within-task prediction and few-shot prediction on unseen tasks.Source: §2.1 · p. 2

2.2 Meta-Learning as Hierarchical Learning

  • Hierarchical meta-learning learns shared global parameters across task realizations, then uses a new task’s context as a prior update for target prediction.The background relates this view to prior-data fitted networks and to MAML as approximate hierarchical Bayesian inference.Source: §2.2 · p. 2

2.3 Neural Processes

  • Latent Neural Processes use a context-conditioned latent representation to produce predictive distributions, with marginalization capable of yielding non-Gaussian predictions and within-task dependence.The figure illustrates predictions for three unseen tasks, with predictive means, 95% intervals, true functions, and labeled context observations.Source: §2.3 · p. 3, Figure 1 · p. 3
  • Neural Process variants modify context representation and processing, whereas this work changes the probabilistic response decomposition and is complementary to those architectural developments.Examples include attention, convolution, recurrence, dimension-agnostic, and streaming-context variants.Source: §2.3 · p. 4

2.4 Mixed-Effects Models

  • Mixed-effects models represent grouped tasks with shared fixed effects and task-specific random effects, commonly estimated with likelihood-based procedures across tasks.The paper’s connection concerns independent task realizations from a common population distribution.Source: §2.4 · p. 4

3 NEURAL PROCESS BOOSTING

NPBoost combines a tree ensemble for shared fixed effects with an NP for stochastic task-specific random effects. Its training procedure couples residual-based NP updates with boosting updates and uses context–target cross-fitting to construct derivatives across observations.

3.1 Model Formulation

  • NPBoost represents shared fixed effects with regression trees and task-specific random effects with a latent NP, without separately identifying the two additive components.The decomposition is used for prediction and does not impose a centering constraint on the random effects.Source: §3.1 · p. 4

3.2 Monte Carlo Predictive Objective

  • The training objective approximates each task’s context-conditioned marginal predictive likelihood by sampling latent variables and averages risk over the training context-sampling scheme.Stochastic optimization draws task minibatches, context–target splits, and latent samples.Source: §3.2 · p. 5

3.3 Training using a Boosting Algorithm

  • Each boosting round updates the NP on responses residualized by the current fixed-effects function, then updates the tree ensemble while holding the NP parameters fixed.For tree updates, the gradient is taken through the explicit fixed-effects term while sampled latents and NP outputs are held fixed.Source: §3.3 · p. 5

3.4 Context–Target Cross-Fitting

  • Context–target cross-fitting uses each fold as context in turn and averages derivatives from the splits where observations are targets, yielding an unbiased gradient estimate over context sets of the chosen size.When the context size matches NP training, the averaging uses the same context-sampling distribution as the predictive risk.Source: §3.4 · p. 6

3.5 Prediction

  • Prediction approximates the full predictive distribution with an equally weighted Gaussian mixture and uses its mean for point predictions.Sampling from the mixture captures uncertainty over both the task-level latent variable and conditional Gaussian predictions.Source: §3.5 · p. 6

3.6 Attention-Based Extension

  • ANPBoost replaces the standard NP with an Attentive Neural Process using a deterministic multi-head cross-attention path, while retaining the NPBoost formulation and training procedures.The attentive path uses eight attention heads.Source: §3.6 · p. 6

4 EXPERIMENTS

The experiments examine NPBoost and ANPBoost on synthetic and real-world tabular tasks, comparing them with NP, tree, mixed-model, and tabular foundation-model baselines. They evaluate both predictions within training tasks and few-shot adaptation to unseen tasks.

4 EXPERIMENTS

  • The experiments ask whether mixed-effects decomposition helps over standard NPs with shared task structure, when cross-attention helps, and how NPBoost compares with mixed-model and boosted-tree baselines.Evaluation uses synthetic and real-world tabular datasets.Source: §4 · p. 6

4.1 Prediction and Evaluation Protocol

  • Evaluation separates within-task prediction on held-out observations from few-shot prediction on unseen tasks given small labeled context sets.Within-task data use 50%/25%/25% training, validation, and test splits; few-shot tasks are partitioned into disjoint training, validation, and test sets.Source: §4.1 · p. 7

4.2 Synthetic Datasets

  • Synthetic data use piecewise-constant shared effects, Gaussian-process task effects, and Gaussian noise across 600 tasks, with five independently repeated seeds.Each task has 100 observations, and few-shot validation and test tasks provide 20 context points.Source: §4.2 · p. 7

4.3 Real-World Datasets

  • The real-world evaluation covers four datasets from different domains: cars, Spotify, cows, and bikes.Feature definitions, preprocessing, and data-splitting procedures are deferred to Appendix H.Source: §4.3 · p. 7

5 RESULTS AND DISCUSSION

NPBoost improves on standard Neural Processes across synthetic and real-world tabular benchmarks, while attention’s benefits vary by setting. Results also distinguish NPBoost from baselines and show complementary strengths between NPBoost and pooled TabICL.

5.1 Results for Synthetic Datasets

  • Across synthetic settings, NPBoost matches or improves on NP in RMSE and CRPS for within-task and few-shot prediction.The mixed-effects decomposition consistently improves the standard NP.Source: §5.1 · p. 7, Table 1 · p. 8
  • On synthetic datasets, ANP consistently improves over NP and often performs similarly to or slightly better than NPBoost.Adding the boosted fixed effect to ANP can still yield gains, but they are smaller.Source: §5.1 · p. 7
  • On synthetic datasets, GPLinear, LME, and both TabICL variants are less competitive than NP-based models; Task-ID GBT drops sharply in few-shot prediction.Task-ID GBT performs reasonably on some within-task problems.Source: §5.1 · p. 7

5.2 Results for Real-World Datasets

  • NPBoost achieves lower RMSE than NP in seven of eight real-world dataset–scenario comparisons.It also outperforms both tree baselines and LME throughout, and GPLinear in seven of eight comparisons.Source: Table 2 · p. 8, §5.2 · p. 8
  • On real-world datasets, ANPBoost and ANP do not consistently improve on their non-attentive counterparts.The authors suggest that relatively small context sets may reduce the benefit of cross-attention.Source: §5.2 · p. 8
  • NPBoost or ANPBoost performs best in four of eight real-world RMSE comparisons, while pooled TabICL performs best in the other four.NPBoost models perform best on the largest dataset, cars, in both prediction settings; TabICL P is especially strong on smaller datasets and several few-shot settings. TabICL T has higher RMSE than NPBoost in all settings.Source: §5.2 · p. 8

6 CONCLUSION

The conclusion presents NPBoost as a Neural Process extension that separates shared tree-ensemble structure from task-specific latent-NP residuals. It reports gains over standard NP in most synthetic settings and seven of eight real-world RMSE comparisons, and describes the authors’ use of generative AI tools.

6 CONCLUSION

  • NPBoost separates predictions into a shared tree-ensemble component and a task-specific residual modeled by a latent NP.The components share a predictive Monte Carlo objective and are trained with alternating NP and boosting updates.Source: §6 · p. 8
  • NPBoost achieves lower RMSE and CRPS than standard NP in most synthetic settings and reduces RMSE in seven of eight real-world comparisons.The improvement did not consistently extend to CRPS; model selection used validation RMSE rather than CRPS.Source: §6 · p. 8

AI use statement

  • The authors used generative AI for methodology feedback, literature work, and drafting, and say they reviewed and verified its suggestions and edits.They state that the experiments and reported results were produced and checked by the authors.Source: AI use statement · p. 9

A PROOF OF PROPOSITION 1

The proof fixes a context set to average gradients over latent draws, then averages across random context partitions. It uses the uniformity of the folds to relate these averages to gradients conditional on context size and exclusion of the target index.

A PROOF OF PROPOSITION 1

  • For a fixed context set, the latent-averaged gradient is the expected split-specific gradient over repeated latent draws from the encoder distribution.The context is held fixed while the latent variables are resampled.Source: A PROOF OF PROPOSITION 1 · p. 11
  • Under a uniform random partition, each fold is marginally uniform over subsets of the specified context size.For a target index excluded from context, the corresponding subsets are uniform among those that exclude that index.Source: A PROOF OF PROPOSITION 1 · p. 11
  • The proof equates the averaged latent gradient with the gradient expectation conditional on context size and exclusion of the target index.It obtains this equality by iterated expectation after averaging over the random partition.Source: A PROOF OF PROPOSITION 1 · p. 11
  • The diagonal-Hessian argument follows the same reasoning as the gradient argument.Source: A PROOF OF PROPOSITION 1 · p. 11

B DERIVATION OF BOOSTING GRADIENT AND HESSIAN

The appendix derives boosting derivatives for NPBoost’s tree updates, gives the training procedure and prediction-cost analysis, and specifies the synthetic settings used for evaluation.

B DERIVATION OF BOOSTING GRADIENT AND HESSIAN

  • The diagonal Hessian entries are not guaranteed nonnegative and are not clipped, although negative values were rare and caused no numerical issues in the experiments.The authors note that negative entries could be clipped to a small positive value if required.Source: B DERIVATION OF BOOSTING GRADIENT AND… · p. 13

C ALGORITHM

  • For each boosting round, NP parameters are updated on residualized responses before cross-fitted derivatives are aggregated to update the fixed-effects predictor.During the tree update, NP parameters and representations are held fixed, so derivatives do not propagate through the encoder or decoder.Source: B DERIVATION OF BOOSTING GRADIENT AND… · p. 11, C ALGORITHM · p. 13

D TEST-TIME COMPUTATIONAL COMPLEXITY

  • For fixed latent-sample count, tree count, and average depth, NPBoost retains linear prediction complexity in the combined context and target size.The tree ensemble adds O(MD(nc + nt)) work; cross-attention adds O(ncnt), while fixed context size keeps that term linear in target size.Source: D TEST-TIME COMPUTATIONAL COMPLEXITY · p. 14

E ROBUSTNESS EXPERIMENTS

  • The robustness suite varies task count and observations per task, random-effect and noise distributions, and the presence of irrelevant input features.The more-tasks setting uses 1,200 tasks with 50 observations each and 10 context points; Brownian-LN uses centered lognormal noise.Source: E ROBUSTNESS EXPERIMENTS · p. 14

F SYNTHETIC DATASETS

  • Synthetic datasets use shared fixed effects, task-specific stochastic functions, and observation noise, with 600 tasks and 100 observations per task by default.Inputs are sampled independently from a uniform range unless otherwise stated.Source: F SYNTHETIC DATASETS · p. 14

F.1 Fixed-Effects Function

  • The shared fixed effect is piecewise constant in one or two dimensions and is scaled to unit variance.The one-dimensional function has four regions; the two-dimensional function is a piecewise-constant surface.Source: §F.1 · p. 14

F.2 Task-Specific Processes

  • Task-specific effects use independent zero-mean Gaussian processes with an RBF kernel by default, while Brownian settings use a shifted Brownian-sheet kernel.In one dimension, the Brownian process is non-stationary and produces rough sample paths.Source: §F.2 · p. 15

F.3 Observation Noise

  • The Brownian-LN setting uses centered lognormal noise with mean zero and variance 0.25, which remains asymmetric.The appendix also evaluates a GP-regression setting with zero fixed effects, RBF covariance, and Gaussian noise as a benchmark-aligned sanity check.Source: §F.3 · p. 17, §F.4 · p. 17

F.5 Data Splits and Context-Set Construction

The appendix distinguishes prediction within training tasks from few-shot adaptation to held-out tasks and defines the accuracy and probabilistic scoring criteria.

F.5 Data Splits and Context-Set Construction

  • Within-task splits use 50% of each task’s observations for training and 25% each for validation and testing.The raw observations are identical across within-task and few-shot settings; only the train-validation-test partition changes.Source: §F.5 · p. 17
  • Within-task prediction holds out observations from tasks used in training, whereas few-shot prediction evaluates on unseen tasks given a small labeled context set.In the reference few-shot setting, each validation or test task supplies 20 context points and predictions are evaluated on the remaining 80.Source: §F.5 · p. 17

G EVALUATION CRITERIA

  • RMSE measures point accuracy of the predictive mean, while CRPS evaluates probabilistic predictions using the predictive cumulative distribution function.Because NP-based predictive CDFs lack a closed form, their CRPS is approximated by Monte Carlo.Source: G EVALUATION CRITERIA · p. 18

H REAL-WORLD DATASETS AND PREPROCESSING

The appendix documents preprocessing and task construction for four real-world tabular datasets, alongside their task and observation counts in Table 3.

H REAL-WORLD DATASETS AND PREPROCESSING

  • Table 3 reports the post-preprocessing task and observation counts for the real-world datasets.Its caption defines K as the number of tasks and N as the number of observations.Source: Table 3 · p. 18

H.1 Preprocessing and Splits

  • Preprocessing removes tasks with fewer than 10 observations, shares the same partitions across methods, standardizes continuous predictors, and one-hot encodes categorical predictors.For fixed-plus-task-specific models, all predictors enter the fixed component, but only continuous predictors enter the task-specific component.Source: §H.1 · p. 19

H.2 Cars

  • Cars predicts log used-car prices by car model, retaining 1,255 tasks and 72,518 observations after filtering.Few-shot validation and test splits each hold out 200 tasks; RMSE and CRPS are reported on the log scale.Source: §H.2 · p. 19

H.3 Spotify

  • Spotify predicts track danceability by artist, with 418 tasks and 7,868 observations after filtering.Few-shot validation and test sets each hold out 100 artists.Source: §H.3 · p. 19

H.4 Cows

  • Cows predicts milk protein content from repeated measurements, defining tasks by cow across 1,337 observations from 79 cows.The dataset is described as a small but standard mixed-effects benchmark.Source: §H.4 · p. 19

H.5 Bikes

  • Bikes predicts log hourly rental counts across date-defined tasks, with cyclic encodings for time features and temporally ordered few-shot task partitions.The processed data contain 352 tasks with 24 observations each and one task with 17 observations.Source: §H.5 · p. 19, Table 4 · p. 20

I IMPLEMENTATION DETAILS

The appendix specifies the model implementations, training and model-selection procedures, and the hyperparameter search strategy.

I IMPLEMENTATION DETAILS

  • NP components follow the Dubois et al. implementation, while boosted-tree components use LightGBM and mixed-effects baselines use GPBoost.The implementation uses PyTorch for NP models and reports the package versions used.Source: I IMPLEMENTATION DETAILS · p. 20
  • Models are selected by validation RMSE, with test RMSE and CRPS averaged across five seeds.NP-based CRPS uses 400 predictive samples; early stopping is used for NP, boosted-tree, and NPBoost models.Source: I IMPLEMENTATION DETAILS · p. 20
  • The NP architecture uses 128-dimensional representations and latent variables with 20 latent samples; attentive variants add an eight-head cross-attention path.Attentive models use target inputs as queries and context inputs as keys and values.Source: I IMPLEMENTATION DETAILS · p. 20

J PARAMETER TUNING

  • Hyperparameters are tuned by 32-trial random search, with more focused search ranges for NPBoost and ANPBoost because they have more hyperparameters.LME and GPLinear have no tuning parameters, and TabICL is evaluated without fine-tuning or hyperparameter optimization.Source: J PARAMETER TUNING · p. 21

K COMPLETE RESULTS

The appendix supplies complete predictive-performance tables for the synthetic and real-world datasets, alongside model plots and experimental hardware details. The plots report very good calibration for NP-based models, with the lowest MACE differing by prediction setting.

K COMPLETE RESULTS

  • Tables 6–11 report complete synthetic-dataset results, including GP regression and the GBT baseline without task ID, while Tables 12–15 cover real-world datasets, including that baseline.Tables 6 and 12 report entries as mean ± standard error over 5 seeds.Source: K COMPLETE RESULTS · p. 22, Table 6 · p. 22, Table 12 · p. 23

L MODEL PLOTS

  • NP-based models exhibit very good calibration; ANPBoost has the lowest MACE within-task, while NP has the lowest MACE few-shot.Figure 7 presents predictive-interval coverage calibration for the one-dimensional reference setting.Source: L MODEL PLOTS · p. 24, Figure 7 · p. 24
  • Dense-prediction intervals use empirical quantiles from 20 latent and 20 decoder samples for NP-based models, while TabICL uses its provided prediction quantiles.For the reference experiment and Steps 1D fixed effect, plots show the best-performing seed-0 model for each class under both prediction settings.Source: L MODEL PLOTS · pp. 23–24
  • The dense-prediction figures compare NPBoost and NP in the one-dimensional reference setting across within-task and few-shot scenarios.The appendix also includes plots showing NPBoost predictions alone in both scenarios.Source: Figure 8 · p. 24, Figure 10 · p. 25, Figure 9 · p. 25

M HARDWARE DESCRIPTION

  • Models with NP components and TabICL ran on GPU; the other models ran on CPU.The hardware setup is summarized in Table 16.Source: Table 16 · p. 28
Loading 2609.28122v1…