Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning

1 INTRODUCTION

• Long-horizon task solving requires inferring scene contents and affordances, then sequencing simple skills such as picking, placing, and opening drawers.The agent must determine which skills are appropriate and plan their order.Source: §1 · p. 1
• Hierarchical reinforcement learning typically abstracts actions, but high-dimensional observations also make state abstraction important for simplifying perception and planning.The suitable state abstraction depends on the decisions required by the available skills.Source: §1 · p. 1
• VFS stacks the value functions of available low-level skills into a state representation for high-level planning.The skills are used as high-level actions without modification; skill discovery and test-time adaptation are outside the paper’s scope.Source: §1 · p. 2
• By mapping states with different object or arm positions, backgrounds, and distractors to the same embedding, VFS simplifies planning and supports generalization to novel environments.The states are functionally equivalent in terms of the skills available for planning.Source: §1 · p. 2
• VFS outperforms contrastive and information-theoretic representations on long-horizon tasks and generalizes zero-shot to novel environments.The paper evaluates maze-solving and robotic manipulation tasks.Source: §1 · p. 2

2 RELATED WORK

• Hierarchical reinforcement learning commonly treats hierarchy as temporal abstraction, with options providing temporally extended actions, but high-dimensional observations make option-model planning challenging.Prior work has studied sample-based planning in discrete and continuous state spaces.Source: §2 · p. 2
• Prior work improves lower-level policies through skill discovery, end-to-end training, and goal-conditioned behaviors; this paper instead assumes skills are given.Its focus is a representation suitable for model-based or model-free high-level control.Source: §2 · p. 3
• Prior skill-composition methods combine skills through value functions or pseudo-rewards, whereas this work uses value functions to build a state abstraction for composing skills.The paper also provides a high-level policy for temporally extended tasks.Source: §2 · p. 3
• Representation-learning approaches have used reconstruction, future prediction, contrastive learning, and transfer objectives, while this work grounds representations in low-level skill capabilities.The paper reports generalization to novel environments while remaining robust to functionally irrelevant distractors.Source: §2 · p. 3
• Model-based RL research includes learned dynamics, belief representations, bisimulation, and graph- or tree-based planning; this paper uses its state abstraction with model-based controllers.Developing a better model-based algorithm is not the paper’s primary objective.Source: §2 · p. 3

3 PRELIMINARIES

• The agent is assumed to have a finite set of temporally extended skills, whose production is not addressed in this work.The paper notes that prior work has produced skills through manual rewards, relabeling, or unsupervised discovery.Source: §3 · p. 3
• Each skill has a maximum rollout length and an associated critic estimating its expected cumulative skill reward from the current state.The paper notes that value functions may be obtained through policy evaluation for skills that do not already have them.Source: §3 · pp. 3–4
• The options framework describes skills through a policy, termination condition, and initiation set; this work does not assume an initiation set and uses value functions to provide that information.Skills are assumed to have a termination condition or fixed horizon.Source: §3 · p. 4
• The high-level decision problem is formulated as a fully observable SMDP with continuous states, finite temporally extended skills, transitions, rewards, and a discount factor.The high-level policy selects among the available skills.Source: §3 · p. 4
• The paper assumes full observability for image-based problems and leaves explicit treatment of partial observability outside its scope.In partially observed problems, the full observation-action history may be needed to describe state.Source: §3 · p. 4

4 SKILL VALUE FUNCTIONS AS STATE SPACES

• Because value predicts skill capabilities, a representation derived from skill values captures low-level skill affordances for high-level planning.The paper empirically evaluates the representation’s effectiveness for planning.Source: §4 · p. 4
• VFS maps each state to a k-dimensional embedding formed by stacking the value functions of k available skills.The paper presents this as a state abstraction capturing functional information about the agent’s interactions with the environment.Source: §4 · p. 4
• The skill-value representation can encode scene positions, skill preconditions, and the effects of executing feasible skills for high-level planning.In the desk example, high “Pick” values indicate liftable blocks, and the “Open Drawer” value increases after that skill is executed.Source: §4 · p. 4
• VFS is intended to ignore exogenous variations such as background distractors and task-irrelevant objects, enabling generalization to novel environments with the same skills.The paper illustrates functionally equivalent configurations that differ in arm pose, desk color, or additional objects.Source: §4 · p. 4

5.1 AN ALGORITHM FOR HIERARCHICAL RL

• The high-level controller learns a Q-function over VFS states and skills as temporally extended actions, using DDQN with a slower-updated target network and fixed low-level skills.The authors use a simple DDQN pipeline to evaluate VFS as a state representation, rather than focusing on improved model-free RL algorithms.Source: §5.1 · p. 5

5.2 EVALUATING LONG-HORIZON PERFORMANCE IN MAZE-SOLVING

• MiniGrid experiments compare representations on fully observable MultiRoom mazes of up to 10 rooms and KeyCorridor tasks of up to 7 rooms, with sparse goal rewards and tasks potentially exceeding 200 steps.The low-level skills include GoToObject, PickupObject, DropObject, and UnlockDoor; holdout MultiRoom layouts are used for zero-shot evaluation.Source: §5.2 · p. 5
• VFS scores 98% on the simplest maze task and up to 68% on the most challenging, outperforming all baselines and beating the next-best method by over 25%.Table 1 reports success rates across increasing room counts; the authors note VFS is largely orthogonal to other representation-learning algorithms and can be combined with them.Source: Table 1 · p. 6, §5.2 · p. 6
• Offline AE and CPC improve smaller-task performance but not larger tasks, while online VAE and CURL improve across tasks; VFS performs best across all tasks.The authors hypothesize that representations learned independently of the controller may miss information needed by the high-level policy; VFS instead captures low-level skill capabilities.Source: §5.2 · p. 6

5.3 ZERO-SHOT GENERALIZATION

• VFS generalizes zero-shot from KeyCorridor to MultiRoom, outperforming the next-best baseline by up to 180% and closely matching HRL trained from scratch in the target environment.The high-level agent is not trained in MultiRoom, and policies are not updated in target environments; the evaluation considers MultiRoom with 4 and 10 rooms.Source: Table 2 · p. 7, §5.3 · p. 7

6 MODEL-BASED PLANNING WITH VALUE FUNCTION SPACES

• VFS-MB uses VFS as the state representation for planning and is evaluated against alternative representations and oracle state.The section studies this model-based approach in robotic manipulation.Source: §6 · p. 7

6.1 A SIMPLE ALGORITHM FOR MODEL-BASED RL

• A one-step supervised model predicts the next VFS state; random shooting scores candidate skill sequences against the goal, and model-predictive control executes one skill before replanning.The model operates over high-level policy steps, each of which may span multiple low-level skill steps.Source: §6.1 · pp. 7–8
• The study tests VFS as a representation rather than optimizing the model-based algorithm; more sophisticated models and planning methods are outside scope and could improve performance.The authors specifically mention adjusted sampling methods and path-integral optimal control as possible alternatives.Source: §6.1 · p. 8

6.2 APPLICATION: ROBOTIC MANIPULATION

• Evaluation uses randomized O5/O10 layouts with horizons H=7/15, 90 MoveANearB skills, and raw-image, VAE, CPC, and oracle-state baselines.The simulated manipulation task uses an 8-DoF arm and visual observations.Source: §6.2 · p. 8
• 80% O10 task success with VFS closely matches the oracle's 85%; VFS also succeeds in all O5 tasks.Table 3 reports success rates for robotic manipulation.Source: Table 3 · p. 9, §6.2 · p. 9
• Unlabeled t-SNE clusters show VFS grouping observations across trajectories by objects and affordances while ignoring table-object and arm poses.The visualizations include different objects held by the arm and objects remaining on the table.Source: §6.2 · p. 9, Figure 6 · p. 9

7 DISCUSSION

• VFS encodes state using low-level skill value functions and is compatible with both model-free and model-based hierarchical policies, with reported gains on long-horizon tasks and zero-shot generalization.The authors connect the generalization benefit to invariances in the representation.Source: §7 · p. 9
• The work assumes a pre-specified skill set and does not address skill learning; jointly improving skills and the high-level policy is left for future work.The authors also suggest using unsupervised skill discovery as a route to unsupervised representation learning.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

• VFS implementation concatenates available skills' value functions into an embedding vector for hierarchical RL or planning.The paper points to Sections 5.1 and 6.1 and Appendix A for setup information.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 MODEL-FREE RL

• MiniGrid experiments use MultiRoom and KeyCorridor environments, with a 10-room modification made fully observable through a top-down grid view.The registered environments include MultiRoom variants with two, four, and six rooms and KeyCorridor variants.Source: §2 · p. 17, §A.1 · p. 17, §1 · p. 17

2. KeyCorridor

• The maze-solving agents use 13 temporally extended skills, including trained text-conditioned GoTo skills capped at 50 steps and oracular remaining skills.Invalid key and unlock actions in MultiRoom leave the world unchanged; GoTo skills use sparse success rewards without shaping or relabeling.Source: §2 · p. 17

A.1 MODEL-FREE RL

• Model-free baselines use 13-dimensional embeddings; AE and CPC train offline on 10^5 skill-collected observations, and HRL policies train to convergence.High-level control uses Double-DQN, with four fully connected layers for represented inputs and a convolutional network for raw observations.Source: §2 · p. 17

2. KeyCorridor

• MiniGrid evaluation averages success across five seeds and 20 layouts per task and environment size; generalization uses the same setting.The report states 100 experimental runs per baseline per task.Source: §2 · p. 18

A.2 MODEL-BASED RL

• For manipulation, a one-step skill-dynamics model is trained on one million open-loop transitions, and representations are evaluated on 20 random start-goal configurations.Success requires all semantic or proximity relations to hold, with objects considered close below 5 cm.Source: §A.2 · p. 18

• The section adds metrics, ablations, and extended analysis for the experiments presented in Section 5.2.Source: B ADDITIONAL EXPERIMENTS AND ABLATIONS · p. 18

B.1 COMBINING WITH REPRESENTATION LEARNING

• Adding VFS to learned embeddings improves SPL by up to 100% over the original baselines, while added baselines improve up to 9% over vanilla VFS, sometimes insignificantly.SPL weighs success by the optimality of actions; the reported results suggest most of the performance gain comes from VFS.Source: §B.1 · p. 18, Table 4 · p. 19

B.2 INCREASING EMBEDDING SIZE OF BASELINES

• VFS continues to outperform CURL, AE, and VAE on maze-solving when their embedding sizes increase to 2x and 5x.Some baselines improve marginally with larger embeddings.Source: §B.2 · p. 19, Table 5 · p. 19

• In Figure 7, the coffee-cup skill value is high before pickup and reaches 1 after the cup is picked up.The passage interprets these values as the skill precondition and success outcome, respectively.Source: §B.3 · p. 20, Figure 7 · p. 20
• In Figure 8, values for the cup, bottle, and bowl are high beforehand, while the cup’s value reaches 1 when it is picked up.Source: §B.3 · p. 20, Figure 8 · p. 21
• In Figure 9, values for picking the bowl rise while the apple is inside it, then drop sharply when the apple falls off.The passage describes this as a complex affordance captured by VFS.Source: §B.3 · p. 20, Figure 9 · p. 21
• The authors describe these few rollouts as examples of expected VFS behavior that qualitatively captures preconditions and outcomes.Source: §B.3 · p. 20