The Value Equivalence Principle for Model-Based Reinforcement Learning

1 Introduction

• Value equivalence means two models yield the same Bellman updates for selected policies and functions.The selected policies and functions encode information about the model’s intended use.Source: §1 · p. 2
• As the considered policy and function sets expand, the class of value-equivalent models shrinks and can collapse to the environment’s true model.The collapse corresponds to a model that perfectly describes the environment.Source: §1 · p. 2
• When planning needs only a subset of value functions, models can work under more restrictive conditions than traditional counterparts.With limited representational capacity, appropriately defined bases can support models requiring less memory or computation than approaches such as maximum likelihood estimation.Source: §1 · p. 2
• The paper compares value-equivalent model learning with more conventional counterparts and argues that it underlies several recent empirical successes in RL.The cited examples include Value Iteration Networks, the Predictron, Value Prediction Networks, TreeQN, and MuZero.Source: §1 · p. 2

2 Background

• An MDP is specified by state and action spaces, rewards, a transition kernel, and a discount factor.The reward may depend on the transition’s source state, action, and next state.Source: §2 · p. 2
• A policy maps states to action distributions, and the agent seeks a policy that maximizes state values.Values are defined by expected discounted rewards along policy-induced trajectories.Source: §2 · p. 2
• Policy evaluation can compute a candidate policy’s value function using its Bellman operator.Repeated application of the operator converges to the policy value function from any starting function.Source: §2 · p. 2
• Model-free RL estimates expectations through sampling, whereas model-based RL learns reward and transition approximations to compute Bellman updates.The paper focuses on the model-based setting.Source: §2 · p. 2

3 Value equivalence

• Models are value equivalent for selected policies and functions when their Bellman operators produce the same updates on those functions.Within those sets, the models are functionally identical for value-based planning.Source: §3 · p. 3

3.1 The topology of the space of value-equivalent models

• Reducing the candidate model space reduces its value-equivalent subset, while expanding policy or function sets shrinks that subset.If the true model is in the candidate space, it remains value equivalent for all policy and function sets.Source: §3.1 · pp. 3–4
• When all policies and functions are considered, the value-equivalent set either contains the true model or is empty.If the true model is included among candidates, enlarging the policy and function sets can pin it down.Source: §3.1 · pp. 3–4

3.2 A basis for the space of value-equivalent models

• For discrete policy and function sets, value equivalence is preserved by their pointwise policy span and function span.Adding a function already expressible as a linear combination of existing functions does not change the equivalent-model space; bases can cover spaces of interest.Source: §3.2 · p. 4
• With m independent policies and a new independent function, the dimension bound on P(Π, V) decreases by a factor of m.When m = |A| and the function span has dimension |S|, P(Π, V) reduces to {P*}; with m < |A|, expanding V alone cannot pin down effects for every action.Source: §3.2 · p. 5

4 Model learning based on the value-equivalence principle

• MLE may fail to produce a value-equivalent model even when one exists, because its objective ignores the policies and functions the model will serve.It can therefore settle for an approximate solution where a model with zero value-equivalence loss is feasible.Source: §4 · p. 5
• Minimize Bellman-update mismatch over chosen policies and functions; the empirical loss uses sampled transitions, with initial states from D′ and actions sampled under each policy.The paper notes that D′ can arise from direct environment interaction.Source: §4 · pp. 5–6

4.1 Restricting the sets of policies and functions

• Rather than fit a model for all policies and functions, select those needed for planning; if V′ is Bellman-closed under Π and spanned by V, equivalent models preserve the solution from V′.The proposition applies from any starting value function in V′.Source: §4.1 · p. 6
• When the hypothesis space intersects the value-equivalent model space, the resulting VE model has zero loss; the figure contrasts this with the corresponding MLE model.The caption identifies the intersection and zero-loss outcome but does not state the MLE model’s loss.Source: Figure 2 · p. 6
• Planning structure can guide V: value sequences may stay within a low-dimensional subspace, while the MDP value-function polytope offers actual policy values as candidate functions.The paper suggests spanning as much of the polytope as possible.Source: §4.1 · p. 6
• Under state aggregation, models with the same transition probabilities between state clusters are indistinguishable to the agent, making piecewise-constant functions a suitable choice for V.These functions belong to the value-function space representable by the agent.Source: §4.1 · pp. 6–7
• With linear value-function approximation, using the approximator’s features for value equivalence yields models that perform no worse than any other model; the approach also applies to nonlinear approximators.The representable value functions define the relevant set, though choosing a spanning set is less clear for nonlinear approximators.Source: §4.1 · p. 7

5 Experiments

• The shared protocol collects transitions under uniformly random actions, fits rewards and transitions with MLE or VE, plans using the learned model, and evaluates the resulting policy in the environment.Implementation details vary with the environment and function approximation.Source: §5 · p. 7
• The experiments vary model representational capacity and the size of the function set; they use policies that always select a particular action to cover the policy span.The analysis focuses on how the choice of functions affects VE.Source: §5 · p. 7
• VE models outperform MLE counterparts across the reported four-rooms and catch scenarios, especially under stricter model-capacity constraints.The comparison uses true value functions of randomly generated policies.Source: §5 · p. 7, Figure 3 · p. 8
• VE largely outperforms MLE on catch and cart-pole using functions from the agent’s representational space, with neural networks used for both models and value functions in cart-pole.The authors report some significant performance improvements.Source: §5 · p. 7, Figure 4 · p. 8
• Although the theoretical analysis assumes a value-equivalent model exists, the experiments report a considerable performance gap favoring VE even without that guarantee.The experiments are presented as insight into the principle rather than as a guarantee of existence.Source: §5 · p. 8

6 The value-equivalence principle in practice

• Predictron, Value Prediction Networks, TreeQN, and MuZero [33] are presented as successful empirical methods that can be understood through value equivalence.The authors frame this interpretation as a way to understand their model-learning components.Source: §6 · p. 8
• These methods can use uncontrolled or policy-conditioned operators, n-step or λ-weighted updates, open-loop action sequences, greedy search, and value functions that vary over time.The paper also describes cases where the true and approximate operators differ or use approximate values or policies.Source: §6 · pp. 8–9
• The paper’s characterization of prior methods is a conjecture about their essential model-learning idea and explicitly sets aside many important details.The table describes each method by its operator, policies, and functions.Source: §6 · p. 9
• All listed methods except VIN sample the Bellman operator rather than computing full expectations, and all jointly learn state representations with value-equivalent models.The authors suggest MuZero’s use of many policies and functions may help span what complex-environment planning requires, but present this as a possibility.Source: §6 · p. 9

7 Related work

• Value-aware model learning studies robust objectives, optimization guarantees, and variants including iterative learning, Wasserstein minimization, and policy-gradient applications; its theory complements this paper’s characterization of value-equivalent models.The cited work focuses on the induced optimization problem, whereas this paper characterizes the space of equivalent models.Source: §7 · p. 9
• Other approaches emphasize that prediction error need not match policy performance, or maintain models consistent with recent value estimates and derive regret bounds for value-targeted regression.Source: §7 · p. 9
• State-equivalence notions can be recast as state aggregation, but value equivalence also applies to other function approximations and can exploit structure without clear state abstractions [29].Source: §7 · p. 9
• The analysis assumes access to a well-defined state; extending value-equivalent learning to observations and learned state representations is left as a future research question [9] [44] [7].The paper asks whether the learned model could induce compatible state representations.Source: §7 · p. 9

8 Conclusion

• Value equivalence relates models by whether they produce the same updates for selected functions and policies.Source: §8 · p. 9
• The principle formalizes tailoring models to their future use and provides a way to incorporate that use into model learning.Source: §8 · p. 9
• The authors argue that value equivalence unifies important recent work and may support future theoretical and algorithmic innovations.Source: §8 · p. 9

• The paper’s results are largely foundational theoretical results about model learning for model-based reinforcement learning agents.Source: Broader impact · p. 10
• The results illuminate desirable model properties and facilitate subsequent training of agents using them.Source: Broader impact · p. 10
• The authors identify no foreseeable societal impact beyond general concerns about progress in artificial intelligence.Source: Broader impact · p. 10

• The supplement supplies proofs of the paper’s theoretical results and detailed descriptions of its experimental procedure.Source: Supplementary Material · p. 13
• A toy MDP illustrates VE outperforming MLE both when an exact value-equivalent model exists and when none exists in the model class.Source: Supplementary Material · p. 13
• Appendix A.1.1 gives derivations, A.1.2 adds toy-MDP examples and an approximate-VE result, and A.2 details the experimental pipeline and reports additional results.Source: Supplementary Material · p. 13

A.1.1 Proofs

• M(Π, V) either contains the true model or is empty; reducing Π or V enlarges the feasible set, and the true model belongs for every Π and V.Source: §A.1.1 · p. 13
• For discrete Π and V, M(Π, V) equals M(p-span(Π) ∩ Π, span(V)).Source: §A.1.1 · p. 14
• dim[P(Π, V)] ≤ |S|(|S||A| − mk) for m pointwise linearly independent policies and k linearly independent value vectors.The bound follows from the rank of the stacked policy matrix and the matrix constraints defining value equivalence.Source: §A.1.1 · p. 15
• MLE may leave the value-equivalent model set, while a model matching a single one-hot value vector can be value-equivalent even when MLE is not exact.In the stated transition-matrix parameterization, MLE sets θi = pii; exactness fails when off-diagonal probabilities in a row differ.Source: §A.1.1 · pp. 15–16
• If V′ is closed under Bellman updates for policies in Π, every model in M(Π, V) yields the true solution from any starting value in V′.The proposition assumes p-span(Π) ⊇ Π and span(V) = V′.Source: §A.1.1 · p. 16

A.1.2 Examples with a simple MDP

• In the three-state MDP, the two-dimensional value subspace R = {[x, y, y]⊤} is closed under arbitrary Bellman updates.Once an update sequence reaches R, it remains in R.Source: §A.1.2 · p. 17
• MLE can produce a value function not equivalent to the true value function for some policies, whereas value-equivalent models exactly match environment values for all policies and therefore the optimal value function.Source: §A.1.2 · pp. 17–18
• In the simple MDP, the true environment and VE model favor action a, while MLE favors the suboptimal action b, even though VE is only approximate.The example has no exactly value-equivalent model in the considered model class.Source: §A.1.2 · p. 18

A.2.1 Environment description

• Catch and Four Rooms use finite tabular states, whereas Cart-pole has a continuous five-dimensional state; Figure 6 describes their tasks and rewards.Catch has 250 states and Four Rooms has 68; Cart-pole states encode cart position and velocity, pole-angle sine and cosine, and angular velocity.Source: Figure 6 · p. 19, §A.2.1 · p. 19

A.2.2 Experimental pipeline

• Experiments collect data with uniformly random actions, train capacity-restricted models, construct policies using environment-appropriate planning or learning, and evaluate policy quality.Model capacity is restricted by rank, using factored stochastic matrices for tabular domains and factorized network layers for continuous-state domains.Source: §A.2.2 · pp. 19–21

A.2.3 Classes of experiments

• One experiment class approximates the environment’s value-function polytope with values of sampled deterministic policies and varies model rank and policy count.These experiments use tabular environments and exact value iteration.Source: §A.2.3 · p. 22
• For restricted value-function classes, experiments vary model rank and representation size, using k-means state aggregations for linear features or stochastic random-network bases for neural networks.The neural-network experiments resample basis networks during gradient descent; five random elements are used.Source: §A.2.3 · pp. 22–23

A.2.4 Additional results

• The appendix provides the complete bivariate experiment results, beyond the subset of slices shown in the main text with the fixed variable set to its median.The additional figures show Catch results for fixed V or fixed model and for both value-function settings.Source: §A.2.4 · p. 23, Figure 7 · p. 23, Figure 8 · p. 24
• Five additional figures show Four Rooms and Cart-pole results under specified fixed-model and value-function configurations.The Four Rooms captions specify fixed V or ˜m with V = ˜V or V = {vπ1, . . . , vπn}; the Cart-pole captions specify fixed V or ˜m with span(V) ≈˜V.Source: Figure 11 · p. 25, Figure 12 · p. 26, Figure 13 · p. 26

A.2.5 Hyperparameters

• The experiments’ hyperparameters are listed in Table 1.Source: §A.2.5 · p. 27, Table 1 · p. 28