Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition

• Reward misspecification can cause unintended behavior, while specifying rewards for visual tasks may require specialized sensing or motion-capture systems.Source: §1 · p. 1
• VICE generalizes inverse reinforcement learning to settings where users provide desired final outcomes rather than full demonstrations.This reduces the required supervision because the expert need not know how to perform the task.Source: §1 · p. 1, Figure 1 · p. 2
• The framework learns event probabilities from data and uses neural function approximators, supporting high-dimensional observations such as images.Source: §1 · p. 2
• VICE replaces conventional reward specification with event probabilities and inference queries that encode when and how often desired events should occur.Rewards correspond to log-probabilities of events, while queries can express intentions such as occurrence at least once or at a specific timestep.Source: §1 · p. 2
• VICE substantially outperforms sparse-reward RL baselines, suggesting that learned event probabilities provide an automated shaping effect for difficult tasks.Source: §1 · p. 2

• VICE learns event probabilities with neural networks, allowing the framework to operate on high-dimensional observations such as images.Source: Figure 2 · p. 2
• Unlike maximum-entropy RL methods that embed standard reward objectives in probabilistic inference, VICE aims to avoid specifying arbitrary scalar-valued reward functions.Source: §2 · p. 3
• VICE generalizes inverse reinforcement learning by using examples of desired outcomes rather than expert demonstrations of how to achieve them.Source: §2 · p. 3
• The framework combines event specification with inference-query selection, including decisions about when and how often an agent should cause an event.Source: §2 · p. 3

3 Preliminaries

• An MDP is defined by state and action spaces, transition dynamics, reward function, discount factor, and initial-state distribution.Source: §3 · p. 3

3.1 Control as inference

• The graphical model adds an event variable e_t conditioned on the state and possibly action, with reward-based event probabilities yielding maximum-entropy RL.Source: §3.1 · p. 3, Figure 3 · p. 3
• Conditioning on e_1:T = 1 produces backward messages whose logarithms satisfy soft backup equations for Q and V.Source: §3.1 · p. 3
• For stochastic dynamics, maximum-entropy control is optimistic and risk-seeking because posterior inference can alter the effective next-state distribution.Source: §3.1 · p. 4

4 Event-based control

• Event-based control represents task specification through event probabilities and inference queries rather than arbitrary scalar-valued rewards.The unified formulation changes the query while leaving the underlying problem parameters unchanged.Source: §4 · p. 4

4.1 ALL and AT queries

• ALL requires an event at every timestep, whereas AT requires it at one specified timestep, covering persistent and time-specific goals.Source: §4.1 · pp. 4–5

4.2 ANY query

• ANY maximizes the probability that an event occurs at least once before episode termination, regardless of its exact timing.This query is suited to reaching goals or completing tasks when repeated or precisely timed occurrence is unnecessary.Source: §4.2 · p. 5
• The ANY query applies when event occurrence is uncertain because it need not be directly observed, and it matches first-exit RL under deterministic event probabilities.Source: §4.2 · p. 5

4.3 Sample-based optimization using policy gradients

• Policy-gradient optimization extends the event-based backup equations to large problems with unknown dynamics and high-dimensional neural function approximators.Source: §4.3 · p. 5
• VICE alternates policy rollouts, discriminator training on expert states and actions, and policy updates using the appropriate inference objective.Source: §4.3 · p. 6

5 Learning event probabilities from data

• VICE generalizes inverse reinforcement learning by learning event probabilities from desired states or observations instead of full expert trajectories.Under ANY or AT queries, supervision can consist of successful states, including goal images in vision-based robotics.Source: §5 · p. 6
• The event model is trained by maximum likelihood on successful state-action data, with a gradient combining data samples and samples from the model.The second gradient term corresponds to the partition function and requires sampling from pθ(s, a|e = 1).Source: §5 · p. 6

5.1 Sample-based optimization with discriminators

• In high-dimensional observations, a discriminator distinguishes policy samples from data, while a variational generator policy samples from pθ(s, a|evidence).This provides a practical approximation for the gradient update in Eqn. 1.Source: §5.1 · p. 6
• The discriminator procedure alternates classifier training with policy optimization that minimizes KL divergence between the sampler and the learned event model.The corresponding reward is log Dθ(s, a) − log(1 − Dθ(s, a)) under the appropriate inference objective.Source: §5.1 · p. 6

6 Experimental evaluation

• The evaluation asks whether query choice changes behavior, whether VICE beats offline classification, and whether learned event probabilities shape rewards better than binary indicators.These questions span pre-specified event probabilities and image-based settings where manual rewards are difficult to specify.Source: §6 · p. 7, §6.2 · p. 8

6.1 Inference with pre-specified event probabilities

• The ANY query reaches the target more accurately than ALL despite similar average distance, because it permits max-seeking behavior rather than penalizing every timestep away from the goal.The ALL query generally achieves superior returns, whereas ANY achieves a closer minimum distance.Source: §6.1 · p. 7, Table 1 · p. 7

6.2 Learning event probabilities

• VICE outperforms the classifier-based setup on Maze, Ant, and Pusher, with substantially larger gains on Ant and Pusher.Table 2 reports final distance to the goal state, where lower is better.Source: Table 2 · p. 7, §6.2 · p. 8
• The learning experiments use successful states as supervision across pixel-based Maze and Pusher tasks and a state-based Ant task.Maze uses 1000 success states, Ant 500, and Pusher 10K.Source: §6.2 · pp. 7–8
• VICE significantly outperforms naïve classification and true binary event indicators on Pusher, while matching an oracle hand-engineered reward.The comparison is averaged across five random seeds and uses lower-is-better performance curves.Source: Figure 6 · p. 8, §6.2 · p. 8

7 Conclusion

• The event-based framework replaces rewards with events whose probabilities may be user-provided or learned from data.Different inference queries express different goals in the corresponding graphical model.Source: §7 · pp. 8–9
• Learning event probabilities generalizes inverse reinforcement learning by using states and actions where events occur instead of assuming full expert demonstrations.Standard inverse reinforcement learning corresponds to events occurring at every timestep, while the framework also supports single-timestep events.Source: §7 · p. 9

A Message Passing Updates for Reinforcement Learning

• Backward state-action and state messages represent the probabilities that future events occur, providing the basis for policy computation.The messages are β(s_t,a_t)=p(e_t:T=1|s_t,a_t) and β(s_t)=p(e_t:T=1|s_t).Source: A Message Passing Updates for… · p. 11
• Defining rewards as log event probabilities and value functions as log messages yields soft Bellman-style backup equations.With a uniform reference policy, Q(s,a)=log β(s,a) and V(s)=log β(s)−log C.Source: A Message Passing Updates for… · p. 11
• Under deterministic dynamics, the resulting recursion reduces to Q(s_t,a_t)=r(s_t,a_t)+V(s_{t+1}), aligning with maximum-causal-entropy and soft Q-learning formulations.A constant log C can be absorbed into the reward without changing the optimal policy for the fixed-horizon task.Source: A Message Passing Updates for… · p. 11
• Bayes’ rule converts the event-conditioned messages into the optimal policy p(a_t|s_t,e_1:T=1).Conditional independence reduces conditioning on all events to conditioning on future events from the current timestep.Source: A Message Passing Updates for… · p. 11

B Control as Variational Inference

• Direct graphical-model inference can produce optimistic, risk-seeking behavior because it allows inferred dynamics to conspire toward positive outcomes.The issue is that posterior inference need not preserve p(s_{t+1}|s_t,a_t)=p(s_{t+1}|s_t,a_t).Source: B Control as Variational Inference · p. 12
• Variational inference constrains the trajectory distribution’s initial state and dynamics to the true model while allowing only the policy q_θ(a_t|s_t) to vary.This restriction addresses the dynamics mismatch while retaining policy optimization.Source: B Control as Variational Inference · p. 12
• The resulting objective is equivalent to undiscounted entropy-regularized reinforcement learning, with q_θ(a|s) serving as the policy.A discounted objective is recovered by adding a (1−γ) transition probability to a zero-reward absorbing state.Source: B Control as Variational Inference · p. 12
• The derivation shows that maximum-entropy reinforcement learning is recovered by applying variational inference with a specific variational distribution.This provides a unified variational treatment of control as inference.Source: B Control as Variational Inference · p. 12

C.1 ALL query

• The ALL query requires an event at every timestep and is mathematically identical to maximum-entropy reinforcement learning.Its derivation therefore reuses the maximum-entropy RL formulation.Source: §C.1 · p. 12

C.2 ANY query

• The ANY query requires an event at least once, regardless of when it occurs, and therefore needs a new backup equation.Forward messages track the cumulative probability that the event has already happened.Source: §C.2 · pp. 12–13
• The ANY-query derivation represents the first event time with a stopping time and recursively combines current event probability with the probability of a future event.This produces backward messages and backup rules for the event-occurrence objective.Source: §C.2 · pp. 12–13
• The ANY-query policy may continue seeking the event after it has already occurred, although this does not affect behavior before triggering or episodes terminated at success.A forward pass can compute the non-seeking policy by tracking whether the event has happened.Source: §C.2 · p. 13

E Policy Gradients for Events

• Policy-gradient estimation for ANY events must use future returns and weight them by the cumulative probability that the event has not already occurred.After the event, the return estimator becomes zero.Source: E Policy Gradients for Events · p. 15

F Variational Inverse Control with Events (VICE)

• For inverse control, AT, ALL, and ANY queries define data distributions over event states and actions, trajectories, or first-event states, respectively.Trajectory-based objectives upper-bound the relevant sampler-to-model KL divergences, and samples can be obtained by executing trajectories and selecting the relevant timestep.Source: F Variational Inverse Control with… · p. 15, §F.1 · pp. 15–16, §F.2 · p. 16

F.3 ANY query VICE

• The ANY-query inverse formulation uses states and actions at the first timestep when an event occurs and derives an upper bound through convex-combination and KL-divergence arguments.Samples are collected by executing full trajectories and using the first state where the event is triggered.Source: §F.3 · pp. 16–17

F.4 Justification for using the discriminator

• The discriminator-based implementation represents the model through the event probability and approximates the sampler state-action marginal using the reference-policy state distribution.The model is parameterized by p_θ(e=1|s,a), while q(s,a)=q(a|s)\bar p(s) is used for discrimination.Source: §F.4 · p. 17
• Using q(a|s)\bar p(s) instead of q(a|s)q(s) creates an inconsistency in the policy objective and prevents unbiased negative examples for discriminator training.The discrepancy arises because the reference-policy state marginal is not the state distribution induced by the learned policy.Source: §F.4 · p. 17

G.1 Experimental details for prespecified events

• The prespecified-event experiments use diagonal Gaussian policies with 32x32 neural-network means and TRPO training for 1000 iterations.The TRPO batch size is 40000 on Lobber and 10000 on HalfCheetah.Source: §G.1 · p. 17

G.2 Experimental details for learning event probabilities

• Performance evaluation compares VICE with classifier-based methods across Ant, Maze, and Pusher using five random seeds and mean results.The binary indicator observes the event and is therefore not directly comparable to methods that do not.Source: §G.2 · p. 17
• VICE uses Gaussian policies with neural-network means, learned diagonal covariance, and neural-network event distributions; additional hyperparameters are listed in Table 3.The policy and event-distribution networks are configured separately for each task.Source: §G.2 · p. 17, Table 3 · p. 18
• On Ant, both the policy mean and event-distribution networks use two 200-unit hidden layers with ReLU activations.Source: §G.2 · p. 17
• On Maze, the policy uses two 5 × 5 convolutional layers followed by two 32-unit fully connected layers, while the event distribution uses two 5 × 5 convolutions and a 16-unit output layer.Source: §G.2 · pp. 17–18
• On Pusher, both policy and event-distribution networks use a three-layer convolutional architecture with 3 × 3 filters, autoencoder pretraining, and two 200-unit fully connected layers.The convolutional layers use strides of 2, 1, and 1 with 64, 32, and 16 filters, respectively.Source: §G.2 · p. 18

G.3 Detailed learning curves for learning event probabilities

• VICE-ALL and VICE-ANY consistently solve Ant, Maze, and Pusher, while the naive classifier often fails and the binary indicator fails in harder environments.The binary indicator works reasonably on Maze but does not solve the more challenging environments.Source: Figure 7 · p. 19
• Detailed learning curves show every one of the five random-seed runs, with the mean trajectory displayed in bold.Source: Figure 8 · p. 20