Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations

1 Introduction

• Related tasks can share structured, low-dimensional dynamics variation, so experience with prior instances may help an agent model and adapt to a new one.The paper illustrates this with bats whose differing lengths smoothly affect dynamics.Source: §1 · p. 1
• HiP-MDP assumes a bounded set of latent parameters fully specifies dynamics and remains fixed during each task instance.The agent is assumed to know when a new instance begins.Source: §1 · p. 1
• The paper presents a data-driven method for learning HiP-MDP structure using Indian Buffet Processes and Gaussian processes.The passage introduces these models as components of the generative model.Source: §1 · p. 1
• On acrobot, the learned HiP-MDP rapidly identifies new-instance dynamics without specifying the number of latent parameters in advance.The introduction reports efficient control in this domain.Source: §1 · pp. 1–2

2 Background

• Bayesian reinforcement learning places a prior over unknown transition dynamics and turns learning an MDP into planning in a POMDP with hidden dynamics.The hidden state combines the fully observed world state with the unknown transition function.Source: §2 · p. 2
• The HiP-MDP is presented as a way to simplify Bayesian RL by using related tasks to find a low-dimensional representation of transition dynamics.The passage identifies high-dimensional, continuous-state POMDP planning as a challenge.Source: §2 · p. 2
• An Indian Buffet Process prior models which latent parameters are relevant to predicting a transition output.The IBP permits an unbounded number of columns and gives each row a Poisson-distributed number of nonzero entries.Source: §2 · p. 2
• Gaussian processes place priors over continuous functions, and the paper uses additive mixtures of them to model the transition function.The GP prior specifies joint Gaussian distributions over function outputs at given inputs.Source: §2 · p. 2

3 Hidden Parameter Markov Decision Processes

• The model focuses on learning dynamics while assuming the reward function is shared across instances.The passage gives the example of always wanting to swing the bat.Source: §3 · p. 2
• A HiP-MDP instance is obtained by fixing a parameter vector drawn at the task’s start, which remains unchanged until the next instance.The transition dynamics depend on that instance’s hidden parameter vector, while the reward function is assumed fixed across instances.Source: §3 · p. 2
• The IBP-GP prior models dynamics semiparametrically with few assumptions on transition-function form while maintaining computational tractability.The passage contrasts this with potentially brittle analytic derivations of dynamics.Source: §3 · p. 3
• Unlike a general POMDP, a HiP-MDP has Markov dynamics conditioned on its parameters, which remain fixed throughout a task and simplify inference.Each instance can also be learned as its own distinct MDP.Source: §3 · p. 3

• Additive basis-function regression learns the factor count, instance weights, relevance filters, and task-specific basis functions.The filters identify which factors matter for each action and state dimension.Source: §4 · p. 3
• Fixing the first filter and weight to one sets the latent-parameter scale and makes the first basis function the domain's mean dynamics.Source: §4 · p. 3
• The IBP prior expects few latent factors to matter for most predictions while allowing new actions or state dimensions to be incorporated consistently.It treats the possible instance-specific scalar latent parameters as countably infinite.Source: §4 · p. 3
• Batch observational data from several instances fits shared filters and basis functions and computes instance-specific weight posteriors for efficient online inference.Source: §4 · pp. 3–4
• The preprocessing step represents each transition with state-difference targets at support points, which Gaussian processes predict by action and state dimension.Support points are selected iteratively to minimize maximum reconstruction error within each batch.Source: §4 · p. 4
• A blocked Gibbs sampler infers filters, weights, and basis values; conditional posteriors for basis values and weights are Gaussian.Source: §4 · p. 4
• Online filtering updates a Gaussian belief over instance weights from each transition, using basis values evaluated at the current state.Basis values at new states are interpolated from support points with Gaussian-process prediction.Source: §4 · p. 5
• Using only the mean of each basis function avoids extra online computation, and the authors report that it still gave significant learning gains in practice.Source: §4 · p. 5

5.1 Cartpole

• On cartpole, IBP-GP performed similarly to a single GP for predicting x and θ, while using only current-instance training points had the highest error across all four outputs.The evaluation compared relative test log-likelihoods and mean-squared errors; the table reports cartpole MSE with 95% confidence intervals.Source: §5.1 · p. 6, Table 1 · p. 6
• Across five cartpole MCMC runs, four latent parameters were inferred, with features linked to different output dimensions and to pole mass or length.The feature used only for predicting ˙x that varied most was suggested as a possible correction for nonlinear effects at more extreme pole lengths.Source: §5.1 · p. 6

5.2 Acrobot

• For acrobot, IBP-GP had lower mean-squared error on angular velocities but slightly higher error on angle predictions.The paper identifies accurate angular-velocity prediction as critical for planning the swing-up task.Source: §5.2 · p. 7, Table 2 · p. 8
• By 5 episodes, IBP-GP reached near-optimal acrobot performance, while the average model took 15 episodes to reach that level.The HiP-MDP model was already near true-model performance by episode 2.Source: §5.2 · pp. 7–8, Figure 2 · p. 8

6 Discussion and Related Work

• HiP-MDP models dimensions of variation across related tasks, unlike transfer methods that identify a task from discrete MDPs or transfer representations across value functions or task representations.The related methods include Hidden-Goal MDPs, hierarchical model-based transfer, and representation transfer approaches.Source: §6 · p. 8
• Unlike multi-output Gaussian processes focused on related outputs within one task, HiP-MDP parameterizes related tasks with stationary, low-dimensional vectors.GP-LVMs instead find time-varying low-dimensional representations for a single system, and these efforts emphasize modeling rather than control.Source: §6 · p. 8
• Future work could relax inference approximations, examine which uncertainties to model, and use clustering or hierarchical methods to share strength among basis functions.A fully Bayesian treatment of filter parameters and basis functions may help when no initial batch from several instances is available.Source: §6 · p. 8

7 Conclusion

• Traditional model-based planning may fail under large inter-instance variation, while reinforcement learning may fail to leverage information from related instances.The conclusion contrasts both approaches with repeated experiences of domains that vary in limited and specific ways.Source: §7 · pp. 8–9
• HiP-MDP models inter-instance variation as a compromise between standard control paradigms, for domain families with small parametrizations relative to their models and similar objectives.The conclusion cites similar objects, cars, and network flows as examples where related-task batch data may be available in advance.Source: §7 · p. 9
• With batch data from related tasks, HiP-MDP is aimed at generalizing dynamics from only a few interactions in a new operating regime.The stated application scope is robust, reliable decision-making while adapting to new situations.Source: §7 · p. 9