PaperlayerView exact revision on arXiv

Source-linked AI summary

World Models with Configurable Perception

Yijun Shen, Delong Chen, Xianming Hu, Kai Zhang, Pascale Fung

arXiv:2610.27914v1cs.CV

TL;DR

Task-agnostic world models use fixed representations even though different tasks may require different information from the same observation. ConfigWM uses a task query to construct task-relevant visual states and learns action-conditioned dynamics over them. Across increasing dynamic distraction, it remains robust and improves average planning performance by 22.3%.

  • Problem

    Task-agnostic world-model representations remain unchanged across queries, although task-relevant information can vary and irrelevant dynamics can make prediction and planning difficult.

    Source: §1 · p. 1

  • Method

    ConfigWM uses language as a configuration signal to steer visual state construction, then learns action-conditioned dynamics over the resulting task-specific states.

    Source: §1 · p. 2, §4 · p. 4, §4.1 · p. 4

  • Results

    22.3% average planning improvement: ConfigWM remains consistently robust as dynamic, target-like distraction increases across controlled comparisons.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ConfigWM’s states better preserve task-relevant information while suppressing noise, alongside stable planning and representation-analysis results under distraction.

    Source: §1 · p. 2

Abstract

from arXiv · show

Real-world observations are noisy and often contain excessive task-irrelevant information. Effective world modeling and planning need to abstract observations into world state representations while filtering out irrelevant information. However, encoders in existing world models are typically non-adaptive, leading to state representations that are easily distracted by task-irrelevant information. As a result, the model may not selectively filter observations to cater to different task demands. To address this problem, we introduce ConfigWM, which realizes configurable perception through task-conditioned state extraction. Given a language description of the task at hand, ConfigWM uses it as a configuration signal to steer how visual embeddings are mapped into a task-specific state space, selectively preserving information relevant to the current task while suppressing task-irrelevant variations. Action-conditioned dynamics are then learned in this task-specific state space for planning. We evaluate ConfigWM in environments with increasing levels of distraction. While existing methods exhibit substantial declines in planning success rate as the environments become increasingly noisy, ConfigWM remains comparatively stable and outperforms the strongest competing method by 22.3% on average across diverse environments. Consistent gains in physical-quantity probing (+0.365 Pearson r) and distracted-to-clean retrieval (+75.5%) further indicate that ConfigWM learns states that preserve task-relevant information while remaining robust to distractors, supporting reliable world modeling and planning in noisy worlds. Project Page: https://configwm.github.io

1 Introduction

Task-agnostic world models use fixed representations even though the information needed for a sufficient state depends on the task. ConfigWM conditions state construction on the task query and learns action-conditioned dynamics over the resulting states, remaining robust as dynamic distractors increase.

Motivation

  • Task-agnostic representations can retain irrelevant dynamics, making prediction and planning difficult when only some coexisting objects matter to the task.The same observation may require different representations under different task queries.Source: §1 · p. 1

Main results

  • ConfigWM remains robust as dynamic distraction increases, with an average planning improvement of 22.3% across controlled comparisons.Baseline methods degrade substantially under distraction, and training them with distractors does not solve the problem.Source: §1 · p. 2

Representation analyses

  • Across physical-quantity probing and distracted-to-clean retrieval, ConfigWM’s representations remain more stable than baseline representations as distractors are introduced.These analyses indicate better preservation of task-relevant information while suppressing noise.Source: §1 · p. 2

Approach

  • ConfigWM uses task instructions to construct task-relevant latent states and learn action-conditioned dynamics over them.The task query steers visual prediction, and the resulting query-conditioned representations define the state space for planning.Source: §1 · pp. 1–2

Evaluation

  • The evaluation adds progressively increasing dynamic, target-like distractors to Two-Room, Push-T, and OGBench-Cube.The controlled setting tests planning under noisy observations.Source: §1 · p. 2

2 Related Works

Prior world models use pretrained features, task-shaped latent states, or compact representations, while steerable-visual-representation work conditions features on task or language signals. ConfigWM applies language during state construction for action-conditioned world modeling and planning.

World State Representations

  • World models have used pretrained visual features, jointly learned states, latent regularization, physical grounding, and compact control-relevant representations.These approaches shape or reuse a state space for prediction and planning.Source: §2 · p. 2

Steerable Representations

  • Steerable representation methods condition visual features on task or language signals, but their focus leaves open whether those states improve action-conditioned world modeling and planning.Language has also conditioned future prediction and action generation.Source: §2 · p. 2
  • ConfigWM applies language at state construction to obtain task-relevant states for planning.The figure depicts increasing dynamic, target-like distractors while holding the task, action sequence, goal, and target trajectory fixed.Source: §2 · p. 2, Figure 2 · p. 3

3 Constructing Noisy Worlds

The paper creates controlled noisy variants of three visual-control environments by adding dynamic, target-like distractors while preserving the underlying task and target motion. Matched clean and noisy observations provide a difficulty axis and training pairs for configurable perception.

3 Constructing Noisy Worlds

  • The evaluation spans Two-Room navigation, Push-T planar manipulation, and OGBench-Cube 3D robotic manipulation.Together, the environments cover representative goal-conditioned planning settings.Source: §3 · p. 3
  • Dynamic, target-like distractors move independently and compete with the real target while preserving the original task and target motion.The nine difficulty levels vary the number of distractors without changing the underlying control problem.Source: §3 · p. 3

3.1 Dynamic Distractor Construction

  • Distractors are segmented and tracked from valid rollouts, recolored, and composited onto another rollout with their trajectories preserved.The procedure modifies visual observations without modifying simulator state or task dynamics, and can be adapted to other environments.Source: §3.1 · p. 3
  • Each clean rollout yields nine matched conditions, from d0 to d8, sharing the task query, action sequence, goal, and target trajectory.Every noisy observation is paired with a clean observation from the same rollout and time step; these pairs also train the configurable encoder.Source: §3.1 · p. 3

3.2 Environment Instantiations

  • The environment-specific distractors are similar-looking agents in Two-Room, recolored T-shaped blocks in Push-T, and tiled arm–cube scenes in OGBench-Cube.In each setting, the task query identifies the agent, block, or scene relevant to prediction and planning.Source: §3.2 · p. 3

4 Methodology

ConfigWM uses language to configure perception and construct task-relevant predictive states for planning in noisy environments. A configurable encoder grounds query-conditioned representations in visual embeddings, after which the model learns action-conditioned dynamics over projected states.

4 Methodology

  • ConfigWM uses language to configure perception, constructing task-relevant predictive states as the state space for action-conditioned world modeling and planning.The training pipeline first predicts the query-specified target representation, then learns dynamics over projected states.Source: §4 · pp. 3–4, Figure 3 · p. 4

4.1 Configurable Encoder

  • The configurable encoder uses a task query to extract task-relevant visual representations while retaining fine-grained visual information needed for low-level planning.It is trained on noisy observations, task queries, and corresponding distractor-free visual targets; a predictor produces a representation trained to match the target representation.Source: §4.1 · p. 4

4.2 ConfigWM: Modeling Noisy Worlds

  • ConfigWM encodes current and next observations under the same query and learns how the projected state evolves under the action.The configurable encoder is frozen for this stage, and a shared state projector is applied to both observations.Source: §4.2 · p. 4, Figure 3 · p. 4

𝑆𝑞𝑡= 𝑃𝜙(𝐸Configurable-Encoder(𝑂𝑡, 𝑞)),

ConfigWM forms task-relevant latent states and learns action-conditioned dynamics, then plans by optimizing candidate actions using terminal distance to a query-matched goal state.

𝑆𝑞𝑡= 𝑃𝜙(𝐸Configurable-Encoder(𝑂𝑡, 𝑞)),

  • An action-conditioned causal Transformer predicts the next task-relevant state, while the state projector and dynamics model are jointly optimized using prediction error and SIGReg regularization.The predicted state and encoded next state occupy the same task-relevant latent space, where distractor dynamics are suppressed.Source: 𝑆𝑞𝑡=… · p. 4

4.3 Planning with ConfigWM

  • A frozen configurable encoder and learned projector encode initial and goal observations under the task query, and the world model rolls out candidate actions from the initial latent state.Each predicted state is produced by applying the dynamics model to the previous predicted state and the candidate action.Source: §4.3 · p. 5, Figure 4 · p. 5
  • CEM optimizes candidate action sequences within MPC using terminal latent distance to the goal, then planning repeats after the first action is executed.The predicted and goal states use the same query, so terminal distance measures progress toward the task-relevant goal.Source: §4.3 · p. 5

5 Experiments

ConfigWM constructs query-conditioned predictive states and learns action-conditioned dynamics, outperforming baselines under dynamic distractors. Its representations retain task-relevant information, and changing the query changes both decoded state content and planning performance.

5.1 Experimental Setup

  • The comparison includes LeWM, TC-LeWM, Sub-JEPA, EB-JEPA, and a controlled DINOv2-WM baseline, using shared data and planning protocols.DINOv2-WM retains ConfigWM’s projector, dynamics model, objective, and planning protocol, changing only visual predictive-state construction; EB-JEPA is evaluated only on Two-Room.Source: §5.1 · p. 5

5.2 Planning under Dynamic Distractors

  • At d8, ConfigWM retains 83.8% of its d0 success rate versus 42.0% for baselines, and outperforms competing baselines across all three environments.It remains robust even when its dynamics model is trained only on d0 transitions, achieving performance comparable to level-matched training.Source: §5.2 · p. 5, Figure 5 · p. 6

5.3 Is Task-Relevant Physical Information Preserved?

  • From d0 to d8, ConfigWM’s probing score decreases by only 0.005 on average, compared with an average decrease of 0.497 for baselines in each environment.The score is the macro-average Pearson correlation across linearly probed task-relevant physical quantities.Source: §5.3 · p. 6, Figure 6 · p. 6

5.4 Do Distracted Representations Align with Clean Counterparts?

  • ConfigWM maintains strong clean–distracted correspondence across Two-Room, Push-T, and OGBench-Cube as distractor levels increase.Figure 6 reports distracted-to-clean Top-1 retrieval accuracy across levels d1–d8, alongside physical-quantity probing.Source: §5.4 · pp. 6–7, Figure 6 · p. 6

5.6 Does Perceptual Configuration Affect Planning?

  • 67.8% correct-query planning success falls to 31.6% with mismatched queries, near the 29.1% random-policy rate.The decrease is averaged over distractor levels d1–d8 and the three environments; only the task query is changed.Source: Figure 8 · p. 7, §5.6 · pp. 7–8

5.5 Does the Query Configure the Predictive State?

  • Changing only the color query produces distinct decodings that retain the queried block and task-relevant agent and target pose while largely suppressing other distractors.A shared decoder has no direct access to the query, so the differing decoded content provides qualitative evidence that the query configures the predictive state.Source: §5.5 · p. 7, Figure 7 · p. 7

5.7 Do Representation Metrics Reflect Planning Performance?

  • Physical-quantity probing and distracted-to-clean retrieval each have an average Pearson correlation of 0.89 with planning success across the three environments.The correlations are computed within each environment across methods and distractor levels d1–d8.Source: Figure 9 · p. 7, §5.7 · p. 8

6 Scaling the configurable encoder with real-world data

The paper explores training configurable perception from real-world video by generating task-irrelevant visual distractors while preserving the task description and clean observation. Across several visual backbones, the configurable encoder reduces distractor responses while keeping task-core prediction nearly unchanged.

6 Scaling the configurable encoder with real-world data

  • A VLM generates prompts for inserting task-irrelevant objects into clean observations, and the encoder learns to predict the clean target representation from the noisy observation conditioned on the task description.The task-relevant scene content and task description remain unchanged during this procedure.Source: §6 · p. 8
  • Across DINOv2, DINOv3, V-JEPA 2.1, and LingBot-Vision, the configurable encoder reduces distractor average precision by 17.9 percentage points while task-core AP remains nearly unchanged.The evaluation also includes held-out PCA visualizations and patch-level linear probing.Source: §6 · p. 8
  • The results suggest a practical route to scaling configurable perception to large real-world video datasets without manual annotation.Source: §6 · p. 8

7 Conclusion

ConfigWM uses task queries to construct task-relevant states for action-conditioned world modeling, and maintains stronger planning performance under dynamic, target-like distractors. Representation analyses and query interventions support the role of query-configured states, while real-world results suggest a path to scaling the encoder across visual backbones.

7 Conclusion

  • Across three environments with increasing dynamic, target-like distractors, ConfigWM maintains stronger planning performance than baselines while preserving task-relevant physical information and clean-state correspondence.Different queries yield different decoded task-relevant states, and mismatched queries substantially reduce planning success.Source: §7 · p. 8

A. Details on Noisy-World Construction

  • The noisy-world construction appendix details distractor generation, environment-specific construction, query templates, and dataset examples.The generation pipeline includes object extraction and tracking, environment-matched reconstruction, cumulative distractor levels, and color balancing.Source: A. Details on Noisy-World Construction · p. 13

B. Implementation Details

  • The implementation appendix covers configurable-encoder training, state projection and dynamics training, baseline implementations, and the shared CEM-based planning protocol.Source: B. Implementation Details · p. 13

C. Additional Experimental Results

The additional experiments report detailed planning, probing, retrieval, query-conditioning, and rollout analyses, alongside real-world encoder scaling and noisy-world dataset construction.

C.1 Detailed Planning Results

  • The appendix provides complete matched and clean-trained transfer planning results, including per-level success rates and aggregate performance.The qualitative rollouts compare ConfigWM with the strongest matched-planning baseline in each environment under clean and severe distractors.Source: C. Additional Experimental Results · p. 13

C.2 Quantity-Wise Physical Probing Results

  • Per-quantity linear-probing results are reported across environments and distractor levels.The appendix also reports Top-5 and MRR results for distracted-to-clean state retrieval.Source: C. Additional Experimental Results · p. 13

C.4 Detailed Query-Conditioning Results

  • Additional decoding results visualize how changing only the task query changes the visual content retained in the predictive state.The appendix also decomposes query-conditioned planning by distractor level and individual mismatched query.Source: C. Additional Experimental Results · p. 13

D. Scaling the Configurable Encoder with Real-World Data

  • Real-world data experiments extend the configurable encoder to four large visual backbones and evaluate distractor suppression and preservation of task-relevant regions.The appendix supplies the data-construction prompts used to generate paired noisy and clean observations.Source: D. Scaling the Configurable Encoder… · p. 13

A.1 Dynamic Distractor Generation

  • Noisy-world datasets are built by extracting and replaying tracked target-object trajectories as distractors, producing temporally stable, visually plausible additions.Distractors are added cumulatively across levels, while color-balanced, query-conditioned targets select the requested object and retain relevant context.Source: §A.1 · p. 14

A.2 Environment-Specific Construction

  • The environment-specific constructions preserve task context while adding target-like distractors: room balls in Two-Room, T-blocks in Push-T, and coherent arm–cube scenes in OGBench-Cube.OGBench-Cube uses 2 × 2 tiles for d1–d3 and 3 × 3 tiles for d4–d8; the layout changes to accommodate added distractors.Source: §A.2 · pp. 14–15

A.3 Dataset Scale

The dataset pairs noisy observations with query-conditioned clean targets and includes multiple queries for the same source frame.

A.3 Dataset Scale

  • Source–clean pairs combine a source observation, language query, and corresponding query-conditioned clean target at the same time step.Pairs cover d0–d8, and some source frames are paired with multiple queries and corresponding clean targets.Source: §A.3 · p. 15
  • Figure 11 depicts distractors accumulating across d1–d8 within each trajectory frame, with the rightmost column showing the d0 target selected by the query.Source: Figure 11 · p. 15

B.1 The Configurable Encoder Training

The appendix describes configurable-encoder training, ConfigWM dynamics, baselines, and evaluation, then reports additional planning and representation analyses. Unified ConfigWM is comparable to the single-domain model on Two-Room and Push-T, and slightly lower on OGBench-Cube while outperforming the other baselines.

B.1 The Configurable Encoder Training

  • The configurable encoder is trained on noisy observations paired with task queries and clean targets, including same-source samples with multiple valid queries.Its spatially aligned patch-level multi-positive InfoNCE objective treats matching query-conditioned targets as positives and same-source samples with different queries as negatives.Source: §B.1 · p. 16

B.2 ConfigWM Training

  • ConfigWM uses a trainable spatial projector to aggregate configurable-encoder patch features into 192-dimensional states, while keeping the encoder frozen.A causal Transformer predicts future states conditioned on actions, using latent prediction error together with SIGReg.Source: §B.2 · p. 17

B.3 Baseline Implementation Details

  • The comparison methods share training trajectories and planning settings; DINOv2-WM uses a frozen pretrained DINOv2 encoder without language conditioning.Source: §B.3 · p. 17

B.4 Planning and Evaluation Protocol

  • All methods are evaluated with the same CEM-based model predictive control protocol, which rolls candidate actions through learned dynamics and minimizes terminal distance to the encoded goal.Reported results average evaluation seeds 40, 42, and 45.Source: §B.4 · p. 17

C.1 Detailed Planning Results

  • Unified ConfigWM achieves comparable planning performance to single-domain ConfigWM on Two-Room and Push-T, and slightly lower performance on OGBench-Cube while still outperforming all other baselines.Both variants maintain relatively stable physical-quantity probing across distractor levels.Source: §C.1 · p. 19, Figure 14 · p. 20

C.2 Quantity-Wise Physical Probing Results

  • Quantity-wise linear probes on frozen representations report test-set Pearson correlation r for physical quantities across distractor levels d0–d8, while distracted-to-clean retrieval reports Top-5 accuracy and MRR against 400 clean candidates.Source: §C.2 · p. 20, Figure 15 · p. 20, §C.3 · p. 21

C.4 Detailed Query-Conditioning Results

  • With the input observation held fixed, query-varying decoding visualizes query-dependent content already encoded in the predictive state.The decoder receives no direct access to the query.Source: §C.4 · p. 22, Figure 19 · p. 22

C.6 Qualitative Planning Rollouts

  • Qualitative rollouts show both methods completing tasks under clean observations, while under severe distractors ConfigWM reaches the goal and the comparison model fails.The comparison is against DINOv2-WM on OGBench-Cube and the strongest baseline for each of the other environments.Source: Figure 20 · p. 24, §C.6 · p. 24, Figure 21 · p. 25

D Scaling the configurable encoder with Real-World Data

The real-world study applies configurable perception to task-annotated videos and four large visual backbones. Its reported evaluations indicate distractor suppression while largely preserving task-relevant regions.

D Scaling the configurable encoder with Real-World Data

  • Task-annotated clips from egocentric, robotic-manipulation, and instructional-video datasets supply clean target observations and task-description queries.Source: D Scaling the configurable encoder… · p. 25
  • A VLM proposes task-irrelevant objects and edit instructions, an image-editing model creates noisy inputs, and a second VLM pass rejects edits that alter task-relevant content or scene geometry.Source: D Scaling the configurable encoder… · p. 25
  • The encoder is instantiated with DINOv2-Giant, DINOv3-7B, V-JEPA 2.1-Gigantic, and LingBot-Vision-Giant for held-out qualitative evaluation.Figure 22 shows distractor-associated feature regions suppressed while task-relevant entities and spatial structure are retained.Source: D Scaling the configurable encoder… · p. 25, Figure 22 · p. 26
  • The configurable encoder consistently suppresses distractors while largely preserving task-relevant spatial information across four visual backbones.The quantitative evaluation uses 300 cases, split 80% for training linear classifiers and 20% for testing.Source: Table 7 · p. 26, D Scaling the configurable encoder… · p. 26
Loading 2610.27914v1…