PaperlayerView exact revision on arXiv

Source-linked AI summary

Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes

Andrew Bennett, Nathan Kallus

arXiv:2110.15332v2cs.LGmath.OCmath.STstat.MLstat.TH

TL;DR

Off-policy evaluation from observational trajectories is challenging when hidden factors affect actions and outcomes. This paper extends proximal causal inference to partially observed decision processes and develops efficient estimators. In confounded simulations, its method is consistent while the MDP benchmark is generally not, though it has greater variance.

  • Problem

    The paper asks how to identify and estimate a target policy’s value from partially observed trajectories when the logging policy may depend on hidden states.

    Source: §1 · p. 1

  • Method

    The paper extends proximal causal inference to sequential partially observed settings, using bridge functions and nuisance-function estimation to construct policy-value estimators.

    Source: §1 · pp. 1–2

  • Results

    In the confounded POMDP simulation, PRL is consistent while the MDP method is not, except for πeasy, and PRL has greater variance than the MDP benchmark.

    Source: §6.1.2 · p. 16

  • Takeaways & Limitations

    The framework provides identification and efficient estimation results for policy evaluation under partial observation, with experiments validating the theory in synthetic and sepsis-simulator settings.

    Source: §1 · p. 2

    The authors describe the work as largely theoretical and identify applying the theory and estimators to real-world sequential decisions with unmeasured confounding as future work.

    Source: §7 · p. 18

Abstract

from arXiv · show

In applications of offline reinforcement learning to observational data, such as in healthcare or education, a general concern is that observed actions might be affected by unobserved factors, inducing confounding and biasing estimates derived under the assumption of a perfect Markov decision process (MDP) model. Here we tackle this by considering off-policy evaluation in a partially observed MDP (POMDP). Specifically, we consider estimating the value of a given target policy in a POMDP given trajectories with only partial state observations generated by a different and unknown policy that may depend on the unobserved state. We tackle two questions: what conditions allow us to identify the target policy value from the observed data and, given identification, how to best estimate it. To answer these, we extend the framework of proximal causal inference to our POMDP setting, providing a variety of settings where identification is made possible by the existence of so-called bridge functions. We then show how to construct semiparametrically efficient estimators in these settings. We term the resulting framework proximal reinforcement learning (PRL). We demonstrate the benefits of PRL in an extensive simulation study and on the problem of sepsis management.

1 Introduction

Off-policy evaluation estimates a target policy’s average reward from data generated by a different behavior policy, but unobserved confounding can make this difficult. The paper extends identification and efficient estimation to partially observed settings using proximal causal inference.

1 Introduction

  • Off-policy evaluation matters when experiments are expensive, impractical, or unethical, yet unobserved factors can confound observational data.The paper frames the problem around healthcare, education, and robotics applications.Source: §1 · p. 1
  • The paper asks when policy value remains identifiable under partial observation and how to estimate it efficiently.A POMDP can represent settings where unobserved factors jointly affect observed covariates, actions, and outcomes.Source: §1 · p. 1
  • The paper extends prior tabular identification to continuous settings, then develops a broader identification result using proximal causal inference.The new result applies in more general settings than the prior approach.Source: §1 · p. 1
  • Proximal reinforcement learning expresses policy value as a logging-policy expectation and yields consistent, asymptotically normal, semiparametrically efficient estimators under stated conditions.The paper also gives a tractable nuisance-function algorithm and reports experiments on a synthetic scenario and a sepsis simulator.Source: §1 · p. 2

2 Related Work

Related work spans off-policy evaluation under varied forms of unmeasured confounding, proximal causal inference, and online POMDP policy learning. The paper distinguishes its contribution by combining general POMDP identification with estimators having desirable statistical properties.

2 Related Work

  • Prior work addresses diverse confounding structures, including POMDP confounding, but often targets partial identification, counterfactual sampling, warm starts, or instrumental-variable methods.The paper identifies Tennenholtz et al. [2020] and Nair and Jiang [2021] as the closest POMDP studies.Source: §2 · p. 2
  • Tennenholtz et al. [2020] and Nair and Jiang [2021] provide POMDP identification results limited to tabular settings, without developing estimators with consistency or asymptotic-normality guarantees.This work proposes a more general identification result and a class of OPE estimators with such properties.Source: §2 · p. 2
  • Proximal causal inference uses conditionally independent confounder proxies to construct outcome or action bridge functions, and prior work applies it to contextual-bandit policy values.This paper extends PCI to POMDP off-policy evaluation and proposes adversarial machine-learning estimation of bridge functions.Source: §2 · p. 2
  • Ying et al. [2021] study multi-step PCI for fixed treatment sequences, whereas this paper evaluates policies that adapt to available context.Their setting may optionally condition on observable context at the first time step.Source: §2 · p. 3
  • Unlike online POMDP policy-learning work, this paper studies offline policy evaluation where identification is an issue.The authors note that their approach could potentially augment or warm-start online methods when offline data are available.Source: §2 · p. 3

3 Problem Setting

The problem setting is a partially observed decision process in which logged trajectories omit hidden states even though the logging policy may depend on them. The goal is to estimate the value of a target policy using those trajectories.

3 Problem Setting

  • The POMDP specifies states, actions, observations, a horizon, and observation, reward, and transition kernels that may vary over time.Rewards are bounded, and the observed trajectory omits the states included in the full trajectory.Source: §3 · p. 3
  • Logged trajectories are sampled under a logging policy that uses hidden states, inducing confounding even though those states are absent from the data.The model is Markovian with respect to hidden states, but those states cannot be conditioned on in the observed data.Source: §3 · p. 3
  • The deterministic evaluation policy may use the current observation and the full past observable trajectory because observations alone need not satisfy the Markov property.The task is to estimate its discounted value under the evaluation-policy trajectory distribution from data drawn under the logging policy.Source: §3 · pp. 3–4

4 Identification Theory

This section develops identification strategies for target-policy value in partially observed MDPs, extending a discrete-state approach and introducing a proximal causal inference framework based on bridge functions. It describes the proxy conditions required for the framework’s identifying assumption.

4.1 Identification by Time-Independent Sampling and Its Limitations

  • The time-independent-sampling extension generalizes Tennenholtz et al. [2020] beyond discrete settings, but its Pind expectation requires averaging n^s terms and lacks properties such as Neyman orthogonality.Under Tennenholtz et al. [2020, Theorem 1] conditions, the identification quantity matches their result.Source: §4.1 · p. 5

4.2 Identification by Proximal Causal Inference

  • Under negative-control independence and existing action and outcome bridge functions, Theorem 2 identifies policy value as E_Pb[ψDR], enabling standard semiparametric efficiency analysis.Assumption validity depends on the proxy and confounder choices and on what inputs the evaluation policy uses.Source: §4.2 · p. 7

4.3 Specific Proximal Causal Inference Reductions and Resulting Identification

  • Practical PCI reductions require justifying bridge-function existence, and their usefulness depends partly on how strongly the action-side and outcome-side proxies correlate.The section summarizes several reduction choices and the policy classes they allow.Source: §4.3 · p. 8

4.3.1 Current and previous observation

  • Using the current hidden state as confounder and current and previous observations as proxies supports policies depending on the current observation alone.This reduction suits noisy observations or settings where decisions should be based on an interpretable current observation.Source: §4.3.1 · p. 8

4.3.2 Current and k-prior observation

  • Using observations k steps apart as proxies allows policies based on recent history, but adds a second confounder state and may weaken proxy correlation as k grows.Weaker correlation can make bridge functions harder to fit, while the extra confounder can make bridge existence harder to satisfy.Source: §4.3.2 · p. 9

4.3.3 Two views of current observation

  • When observations split into private and public views, the private view can serve as an outcome-side proxy and the public view can support policies using observation history.This can accommodate covariates unavailable at deployment or excluded from policies for ethical reasons.Source: §4.3.3 · p. 9

4.4 Example: Tabular POMDPs Using Previous and Current Observation as Proxies

  • In the tabular previous/current-observation reduction, bridge equations are solvable when observation and evaluation-induced transition matrices are full rank; observation rank requires |O|≥|S| and distinguishes states.The transition-matrix condition depends on the evaluation policy, though invertible state-transition kernels for every action can justify it for all evaluation policies.Source: §4.4 · p. 10
  • Assumption 2 requires proxies sufficiently correlated with confounders, such as full-rank proxy distributions, and at least as informative, such as |O| ≥ |S|.Source: §4.4 · p. 10

5 Policy Value Estimators

The section develops cross-fitted policy-value estimators and gives conditions for their consistency, asymptotic normality, and semiparametric efficiency. It also describes sequential nuisance-function estimation and discusses how estimator variance may grow with the horizon.

5 Policy Value Estimators

  • The estimator uses cross-fitting, fitting nuisance functions on other folds before evaluating scores on each held-out fold.The nuisance estimates are plugged into the doubly robust score, whose values are averaged.Source: §5 · p. 10

5.1 Consistency and Asymptotic Normality

  • Nuisance estimates must be consistent, with product-error terms converging faster than n^-1/2; individual op(n^-1/4) rates can suffice.The stated rate condition permits slower-than-parametric nuisance estimation, with trade-offs when some nuisances are estimated faster.Source: §5.1 · p. 11
  • With nuisance norms of comparable order as H grows, the efficient asymptotic variance grows roughly as O(H^2), and may grow faster if inverse problems worsen.The variance bound is expressed through norms of the bridge functions; increasing ill-conditioning can increase those norms.Source: §5.1 · p. 12

5.2 Semiparametric Efficiency

  • MPCI gives uniquely defined nuisances and a uniquely defined target V(P), which equals vγ(πe) for the logging distribution.The efficiency result identifies ψDR(τH) − vγ(πe) as the efficient influence function at the observational distribution.Source: §5.2 · p. 13
  • Under the stated conditions, the estimator is semiparametrically efficient relative to MPCI.The result combines the efficient influence function with appropriate nuisance estimation.Source: §5.2 · p. 13

5.3 Nuisance Estimation

  • Nuisance estimation reframes intervention-distribution moment equations as nested restrictions under the observed distribution, then estimates q(1) through q(H) and h(H) through h(1).The proposed variational method of moments solves these restrictions sequentially.Source: §5.3 · pp. 13–14

6 Experiments

The experiments use a three-state NoisyObs POMDP and a sepsis-management simulator to assess PRL against unadjusted and MDP baselines. They examine consistency under confounding and practical hyperparameter selection.

6 Experiments

  • The experiments include a toy proof-of-concept scenario and a benchmark based on a confounded sepsis simulator.The sepsis setting is intended as a closer reflection of real applications.Source: §6 · p. 14, §6.2.1 · p. 16

6.1.2 Results

  • With observation noise ϵnoise = 0.2, PRL was consistent while the MDP estimator was not, except for a policy where MDP happened to have small bias.PRL had more variance than MDP, while the unadjusted MeanR and TIS baselines were inconsistent.Source: Figure 4 · p. 15, §6 · p. 15, §6.1.2 · p. 16

6.1.1 Experimental Setup

  • The NoisyObs experiment varies observation noise and evaluation policies, using a three-state, two-action POMDP with horizon H = 3.The observation noise parameter controls whether observations match the state; the target policies depend only on current observations.Source: §6.1.1 · p. 15

6.2.1 Experimental Setup

  • The sepsis experiment censors diabetes status for 25% of patients and evaluates a policy learned from 10,000 observational trajectories over horizon H = 3.The learned policy treats observations as true states, although those observations are confounded.Source: §6.2.1 · pp. 16–17

6.2.2 Results

  • In sepsis management, automatic hyperparameter selection produced slightly higher bias but much lower variance and mean squared error than the single best hyperparameter setup.The best setup was approximately unbiased but highly variable; the heuristic avoids needing ground truth for hyperparameter selection.Source: Table 2 · p. 17, §6.2.2 · p. 17
  • The automatic hyperparameter heuristic always predicted that the proposed sepsis policy was worse than the baseline, while MDP always incorrectly predicted an improvement.The true values were −2.275 for the proposed policy and −1.799 for the baseline.Source: §6.2.2 · pp. 17–18, Table 2 · p. 17

7 Conclusion

The paper develops proximal identification and efficient estimation for off-policy evaluation in unknown POMDPs, then demonstrates the approach in synthetic and sepsis-management settings. It identifies practical algorithms, semiparametric efficiency, and real-world application as areas for further work.

7 Conclusion

  • The framework combines a new identification approach, double machine learning with cross-fitting, and an implemented estimator demonstrated in synthetic and sepsis-management settings.The applications involve confounding from noisy measurements and missing diabetes measurements, respectively.Source: §7 · p. 18
  • The authors show that, under appropriate conditions, their estimators are asymptotically normal and semiparametrically efficient.The framework reduces identification sequentially to proximal causal inference problems.Source: §7 · p. 18
  • Even in a toy POMDP with about 1,000 trajectories, the proposed estimators can have high variance.The experiments were intended as proof of concept; sequentially solving the q(t) and h(t) conditional moment problems may introduce cascading errors.Source: §7 · p. 18
  • Extending the semiparametric efficiency model to allow multiple nuisances while keeping the parameter well-defined remains an open challenge.Source: §7 · p. 18
  • Applying the theory and policy-value estimators to real-world sequential decisions involving unmeasured confounding remains future work.The authors describe the work as largely theoretical.Source: §7 · p. 18

A Identification by Time-Independent Sampling

The section identifies a target policy value through an expectation under a time-independent sampling distribution, subject to completeness and bridge-function conditions. It also relates the general result to an analytic solution in the tabular setting.

A Identification by Time-Independent Sampling

  • In the finite-state setting, the completeness condition requires at least as many categories in observations Ot as in states St.The section characterizes this as Ot capturing all degrees of variation in St.Source: A Identification by Time-Independent… · p. 21
  • Identification requires completeness and functions ρ(t) satisfying the stated bridge condition; the assumptions also implicitly require P(At = a | Wt) > 0.Completeness requires that a function of the latent state with zero conditional expectation given observed variables and action is itself zero almost surely.Source: A Identification by Time-Independent… · p. 21
  • The identification result expresses the target-policy value as an expectation under Pind, with data at each time point sampled independently.The result applies for any given γ when Pind is identified with respect to the behavior distribution Pb.Source: A Identification by Time-Independent… · p. 21
  • In the tabular case, an invertible Q(t,a) gives an analytic solution for ρ(t), and substituting it recovers Tennenholtz et al.'s result [2020].The section also notes that invertibility implies Assumption 5 when St has no more than k categories.Source: A Identification by Time-Independent… · p. 21

B Identification by Proximal Causal Inference

This section presents a proximal-causal-inference identification result built around recursively defined quantities and bridge-function equations. Under the stated assumptions, the resulting importance-sampling, regression, and doubly robust scores all represent the target policy value.

B Identification by Proximal Causal Inference

  • Under Assumptions 1 and 2, the target policy value equals the behavior-distribution expectation of each of ψIS, ψReg, and ψDR.The section connects this result to Theorem 2.Source: B Identification by Proximal Causal… · p. 26
  • The three score forms have structures resembling importance sampling, direct-method, and double-reinforcement-learning estimators in MDP settings.The q and h terms play roles analogous to importance weights and quality-function terms in the cited comparison.Source: B Identification by Proximal Causal… · p. 26
  • Assumption 2 ensures solutions to the bridge equations used in the identification construction, enabling the recursive argument.The section states that its main result follows by backward induction from the preliminary lemmas.Source: B Identification by Proximal Causal… · pp. 26, 29, 30

C Proof of Consistency and Asymptotic Normality

The appendix establishes Neyman orthogonality and asymptotic normality conditions for the doubly robust score, then reviews semiparametric efficiency concepts and a strategy for deriving an efficient influence function.

C Proof of Consistency and Asymptotic Normality

  • The proof establishes Neyman orthogonality of ψDR with respect to q(t) and h(t), under the theorem conditions and finite supremum norms for these nuisance functions.The proof then applies the conditions of Chernozhukov et al. [2016, Theorem 3.1] to obtain convergence to a normal distribution.Source: C Proof of Consistency and Asymptotic… · pp. 31, 35

D Background on Semiparametric Efficiency Theory

  • The semiparametric-efficiency review frames the problem as limits on estimating a parameter while requiring estimators to be consistent and well-behaved locally.Source: D Background on Semiparametric… · p. 36

D.1 Definitions

  • Regular estimators have limiting distributions robust to local changes in the data-generating process.Source: §D.1 · p. 36

D.2 Characterizations

  • For a pathwise differentiable parameter, an asymptotically linear estimator is regular exactly when its influence function is a gradient; the efficient influence function is the projection of a gradient onto the tangent space.Source: §D.2 · p. 37

D.3 Strategy to calculate the EIF

  • The stated strategy is to derive a gradient, characterize the tangent space, and either show the gradient lies in it or project it onto that space.The gradient can often be obtained by differentiating the parameter along a submodel and rewriting the derivative as an expectation involving the score function.Source: §D.3 · p. 37

D.4 Optimalities

The appendix characterizes the efficiency bound and gives conditions for attaining it, then establishes boundedness properties used in the semiparametric-efficiency proof.

D.4 Optimalities

  • The local asymptotic minimax and convolution results bound estimator risk using the variance of the efficient influence function (EIF).The bound applies to worst-case risk under local perturbations and to regular estimators.Source: §D.4 · pp. 37–38
  • An estimator is efficient if and only if it is asymptotically linear with influence function equal to the EIF.Efficiency here means regularity and a limiting normal distribution with variance equal to the efficiency bound.Source: §D.4 · p. 38

E Semiparametric Efficiency Theory for Proximal RL Estimator

  • The proof shows that Fbounded(u) is closed under addition and multiplication, enabling boundedness arguments for nuisance functions.The appendix also uses forward and backward induction to establish uniform bounds for q and h nuisance functions under Assumption 4.Source: E Semiparametric Efficiency Theory for… · p. 38
  • Under Theorem 4's conditions, the appendix establishes bounds on functions in the ranges of the conditional operators and their adjoints.These bounds support the subsequent control of nuisance functions used to derive the path derivative of the policy value.Source: E Semiparametric Efficiency Theory for… · pp. 39–40

E.1 Discussion of Issues with Tangent Spaces in Past Work

The authors question whether tangent-space restrictions proposed in earlier proximal causal inference work are sufficient or fully justified for the models considered. They define assumptions yielding a locally saturated model with a well-defined parameter.

E.1 Discussion of Issues with Tangent Spaces in Past Work

  • For the tangent-space choice discussed by Kallus et al. [2022], the authors say it is unclear whether necessary nuisance-differentiability conditions are sufficient to construct a matching parametric submodel.They also question whether nuisance differentiability is necessary, since they derive the policy-value gradient without assuming it.Source: §E.1 · pp. 43–44
  • When bridge functions are not uniquely determined, restrictions may need to hold for every valid h and q, not just one selected pair.The authors say sufficiency of restrictions for a single chosen pair is unclear.Source: §E.1 · p. 44
  • The authors note that latent-confounder PCI distributions impose additional necessary restrictions on score functions.They give a conditional decomposition of scores as an example of the restrictions implied by the latent-variable model.Source: §E.1 · p. 44
  • Their assumptions make the model locally saturated at the logging policy, with a tangent set of all square-integrable functions and a locally unique, well-defined parameter.The parameter agrees with the target policy value for distributions arising from a valid PCI model satisfying the assumptions.Source: §E.1 · p. 44

E.2 Proof of Semiparametric Efficiency Theorem

The appendix proves semiparametric efficiency by establishing the gradient property without requiring differentiable nuisance functions. It also describes nuisance-estimation procedures, including sequential kernel VMM estimation.

E.2 Proof of Semiparametric Efficiency Theorem

  • A pointwise-convergent function family that remains uniformly bounded for small perturbations permits the dominated-convergence argument used in the proof.Lemma 14 supplies this step for expectations along a parametric submodel.Source: §E.2 · pp. 44–45
  • The proof of Theorem 4 reduces to showing that ψDR is a gradient, since the tangent space is all square-integrable functions and ψDR lies in it.The authors do not assume that the nuisance functions q are differentiable.Source: §E.2 · p. 45

F.1 Proof of Nuisance Estimation Lemma

F.2 Meta-Algorithm for Nuisance Estimation with Kernel Critics

  • With RKHS critic classes and quadratic norm regularizers, kernel VMM has a closed-form inner supremum and is described as having consistency, asymptotic-normality, and efficiency properties.The cited theoretical properties are attributed to Bennett and Kallus [2023].Source: §F.2 · p. 48
  • The sequential procedure estimates q(t) in ascending time order and h(t) in descending order, plugging in previously estimated nuisances.Prior nuisance estimates may come from any method and need not be consistent; for discrete data, the described procedure has computational cost linear in n.Source: §F.2 · p. 48

F.3 Derivation of Meta-Algorithm

The appendix derives the kernel VMM nuisance-estimation procedures and describes the experimental environment and baselines. The authors report that their experiment is a proof of concept using hand-selected hyperparameters, leaving automatic data-driven selection for future work.

F.3 Derivation of Meta-Algorithm

  • The kernel VMM derivation verifies ComputeQ and ComputeH through representer-theorem reductions and closed-form optimization of their critic objectives.The main algorithm estimates q(t) in increasing order and h(t) in decreasing order, plugging in previously estimated nuisance functions.Source: §F.3 · p. 50

G.1 Environment Details

  • NoisyObs uses deterministic, time-independent transitions and rewards, with noisy observations drawn from the current state’s observation with probability 1 − ϵnoise.Figure 5 encodes actions with different edge styles and labels edges with rewards.Source: Figure 5 · p. 51, §G.1 · p. 51

G.2 Policy Details

  • The evaluation policies described for NoisyObs are deterministic and depend on the current observation, not the time index.The initial-state procedure also samples a prior state with probabilities 0.5, 0.3, and 0.2 for s1, s2, and s3, respectively.Source: §F.3 · p. 51, §G.1 · p. 51

G.3 Method Details

  • The experimental methods include the proposed estimator and MeanR, MDP, and TIS baselines.The provided method descriptions specify the MDP baseline as fitting a count-based tabular model treating observations as states, then evaluating the policy by dynamic programming.Source: §G.3 · p. 51, §G.3.1 · p. 51, §G.3.2 · p. 52

G.3.1 Ours

  • The proposed estimator uses five-fold cross-fitting, Algorithm 1 for nuisance estimation, and the PCI reduction Zt = Ot−1 and Wt = Ot.The experiments use tabular nuisance-function classes and hand-select hyperparameters after grid search; automatic data-driven selection is left for future work.Source: §G.3.1 · pp. 51–52

G.3.2 MeanR

  • MeanR is included as an extremely simple baseline.The supplied passage does not state its formula.Source: §G.3.2 · p. 52

G.3.3 MDP

  • The MDP baseline treats observations as states, fits a tabular model from observed transition counts, and evaluates the target policy by dynamic programming.Source: §G.3.3 · p. 52

G.3.4 TIS

  • TIS estimates a time-independent sampling identification quantity from observed counts and uses an empirical analogue of the independent-trajectory expectation.The estimator sums over nH combinations of separately sampled observed trajectories and normalizes by n−H.Source: §G.3.4 · p. 52

G.4 Results for the MDP Setting

Without confounding, the MDP baseline is consistent and highly accurate, while the PRL estimate remains consistent but has higher variance. The sepsis-simulator experiment modifies the initial-state distribution, diabetes observation, and reward function.

H.1 Environment and Policy Details

  • The sepsis simulator uses a uniform initial-state distribution over non-terminal states and gives patients a 20% chance of censored diabetes status at t = 1.The modified reward function scores safe indicators and treatments, penalizes unsafe indicators, and continues rewards after absorbing terminal states.Source: §H.1 · pp. 52, 54

G.4 Results for the MDP Setting

  • Without confounding, the MDP baseline is consistent and highly accurate; the PRL estimate is consistent but has greater variance than the baseline.The PRL estimate has lower variance than in the POMDP setting, while MeanR and TIS still perform badly.Source: §G.4 · p. 52, Figure 6 · p. 53

H.2.1 Benchmarks

  • The MeanR and MDP benchmarks were implemented identically to those in the prior experiment.Source: §H.2.1 · p. 54

H.2.2 Our Method

The sepsis experiment uses a proximal causal inference reduction with two diabetes-status proxies and applies the paper’s estimator with neural-network nuisance models. Hyperparameters are selected either by the best tested configuration or an automatic heuristic.

H.2.2 Our Method

  • The method sets Z_t = (G_{t−1}, X_t) and W_t = (G_t, X_t), with identical diabetes proxies because the censored observation does not change over time.The authors state this is consistent with their assumptions because G_t is deterministically determined by the full state.Source: §H.2.2 · p. 54
  • The estimator uses regularized nuisance functions, Gaussian-kernel combinations, and neural networks for H(t) and Q(t).The kernel inputs concatenate embeddings of diabetes status, observations, and treatment; the neural networks process treatment and observation embeddings through bilinear and linear layers with GeLU nonlinearities.Source: §H.2.2 · pp. 54–55

H.3 Hyperparameter Details

  • Among 81 tested configurations, the best overall-MSE setting used λ_h = 10^-4, λ_q = 10^-2, α_h = 10^-2, and α_q = 10^-4.The automatic heuristic selected across configurations by taking the median of predictions after discarding out-of-bound values on each of 50 replications.Source: §H.3 · p. 55
Loading 2110.15332v2…