PaperlayerView exact revision on arXiv

Source-linked AI summary

FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment

Filip Grigorov, Kourosh Darvish, Nandita Vijaykumar

arXiv:2610.11119v1cs.RO

TL;DR

RGB-D observations are high-dimensional and noisy, while privileged state can accelerate training but is unavailable at test time, creating a modality gap. FOCUS uses a single PPO run to align the two actor pathways and regulate rollout modality from their policy disagreement. Across five manipulation tasks, it improves test success and complete-pipeline training-success AUC over RGB-D-at-test baselines.

  • Problem

    RGB-D manipulation learning is challenging because visual inputs are high-dimensional and noisy, while privileged simulator state can help determine actions but is unavailable at test time.

    Source: §1 · p. 1

  • Method

    FOCUS uses a single-stage PPO framework that aligns privileged-state and RGB-D actor representations and regulates rollout modality using cross-modal action-distribution disagreement.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Across five IsaacLab manipulation tasks, average test success rises from 0.71 to 0.93 and complete-pipeline budget-normalized training-success AUC rises from 0.47 to 0.65 versus the strongest RGB-D-at-test baselines.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Ablations identify controlled RGB-D rollout exposure and policy-output KL as the most reliable contributors, while other auxiliary losses have task-dependent effects.

    Source: §5 · p. 9

    Evaluation is limited to simulation, relatively short-horizon manipulation, and two training seeds; task-specific reward engineering may affect behavior and cross-task comparability.

    Source: §6 · p. 10

Abstract

from arXiv · show

Vision-based reinforcement learning for robotic manipulation is sample-inefficient because RGB-D observations are high-dimensional and noisy. Privileged state information available in simulation can accelerate training, but its absence at test time creates a train-test modality gap. We propose FOCUS, a single-stage PPO framework that trains the critic on privileged state while automatically regulating whether the actor collects rollouts from RGB-D or privileged-state latents. Regulation is driven by the KL divergence between the action distributions induced by the two modalities, while representation alignment encourages consistent action selection across them. Together, these mechanisms limit RGB-D rollouts when the actor's action distributions from RGB-D and privileged-state latents disagree. As they align, RGB-D exposure increases, shifting on-policy training toward the RGB-D inputs used at test time. Across five manipulation tasks, FOCUS raises average test success from 0.71 to 0.93 relative to the strongest RGB-D-at-test baseline on each task. When accounting for each method's complete training pipeline, budget-normalized training-success AUC increases from 0.47 to 0.65. On Pick-and-Place, test success rises from 0.47 to 0.86, while AUC increases from 0.12 to 0.61, a 5.0x improvement in learning efficiency over the fixed interaction budget.

1 Introduction

FOCUS addresses the tension between sample-inefficient RGB-D training and privileged-state training that may not match test-time vision. It aligns the actor’s modalities and uses their policy disagreement to regulate RGB-D rollouts during a single PPO run.

1 Introduction

  • RGB-D manipulation learning is difficult because visual inputs are high-dimensional and noisy, while privileged simulator state can provide compact task-relevant information.Simulation also offers resettable environments, parallel rollouts, controlled task variation, and training-only privileged information.Source: §1 · p. 1
  • Privileged-state rollouts can produce decisions poorly matched to test-time RGB-D, whereas RGB-D-only training can be sample-inefficient.FOCUS aims to shift training toward RGB-D without requiring RGB-D rollouts from the start or separate transfer stages.Source: §1 · p. 2
  • FOCUS is a single-stage PPO framework that aligns representations and uses cross-modal action-distribution disagreement to regulate whether actor rollouts use RGB-D or privileged-state latents.An upper-quantile KL budget reduces RGB-D exposure when rare high-disagreement states remain.Source: §1 · p. 2
  • Across five IsaacLab manipulation tasks, average test success rises from 0.71 to 0.93 and complete-pipeline budget-normalized training-success AUC rises from 0.47 to 0.65 versus the strongest RGB-D-at-test baselines.On Pick-and-Place, test success rises from 0.47 to 0.86 and AUC from 0.12 to 0.61 (5.0×).Source: §1 · p. 2
  • FOCUS-simple matches or exceeds FOCUS on two of three discriminative tasks for each metric, suggesting simplified future variants.Source: §1 · p. 3

2 Related Work

Prior work uses privileged information in teacher–student transfer, asymmetric actor–critic learning, or visual representation methods. FOCUS distinguishes itself by using current cross-modal policy disagreement to regulate on-policy rollout generation within a single actor.

Teacher–Student and Distillation

  • Teacher–student pipelines commonly separate privileged training from visual transfer, while FOCUS regulates privileged-state and RGB-D paths online according to their behavioral disagreement.Some newer methods use joint teacher–student training or adaptive data sharing, but still organize learning around distinct policies.Source: §2 · p. 3

Asymmetric Actor–Critic and Privileged Critics

  • Asymmetric actor–critic methods give the critic privileged information while the actor learns from test-time observations [7] [25] [26], but the actor still collects RGB-D experience from the start.FOCUS retains a privileged-state critic and additionally regulates which actor path generates rollouts.Source: §2 · p. 3

Privileged-Information-Guided Learning and Visual RL

  • Privileged-information-guided methods use training signals for visual attention, auxiliary learning, or input-reliance regularization [17], but do not use action agreement to regulate rollout generation.Source: §2 · p. 3
  • Visual RL methods use reconstruction, latent dynamics, contrastive learning, and augmentation [28] [19] [20] [3], but still rely on RGB-D-generated experience from the start.FOCUS combines visual-policy optimization with privileged-state actor paths and action-level alignment.Source: §2 · p. 3

Fusion and Modality Routing

  • Fusion and modality-routing methods offer mechanisms for combining or selecting representations [29] [30] [32], but do not address replacing training-only privileged state with RGB-D action selection at test time.Source: §2 · p. 3

3 Methodology

FOCUS trains a privileged-state and RGB-D actor pathway in one PPO loop, with the critic anchored to privileged state. It regulates rollout modality using cross-modal policy disagreement and trains the RGB-D representation toward compatible actions.

3.1 Problem setup and method overview

  • FOCUS aligns privileged-state and RGB-D representations so their fused actor inputs induce compatible action distributions, while using KL-guided exposure and additional control-agreement objectives.At test time, action selection uses RGB-D observations, proprioception, and the previous action rather than privileged state.Source: §3.1 · p. 4

3.2 KL-guided modality switching

  • RGB-D exposure increases as the modalities agree and decreases when disagreement exceeds configured levels; an upper-quantile term limits exposure when some states remain highly discordant.The exposure factors use running mean and upper-quantile estimates; a disagreement of 2ε yields a factor of 0.5.Source: §3.2 · p. 5

3.3 Representation alignment objective

  • With the actor and fusion network frozen, the policy-output alignment loss updates the RGB-D encoder to reduce disagreement between action distributions for corresponding modality inputs.Its KL(πo∥πs) term matches the directional disagreement monitored by the switch.Source: §3.3 · p. 6

3.4 Cross-modal control regularization

  • Auxiliary control regularizers encourage agreement in policy outputs and value predictions, using privileged-state outputs as stop-gradient anchors.RGB-D-conditioned value predictions are used only for this auxiliary agreement; rollout values, returns, and PPO advantages remain anchored to privileged state.Source: §3.4 · p. 6

3.5 Training procedure

  • Training uses separate RL and representation optimizers in one on-policy PPO loop, with each rollout’s actor modality fixed and the critic anchored to the privileged-state pathway.Disagreement measured during training updates the running statistics that regulate RGB-D exposure in later rollouts.Source: §3.5 · pp. 6–7

4 Experimental Setup

FOCUS is evaluated against visual and privileged-observability baselines across five IsaacLab manipulation tasks, with test success and full-pipeline training efficiency as key measures. Its average results improve over RGB-D-at-test baselines, while ablations highlight controlled visual exposure and policy-output alignment as reliable components.

4 Experimental Setup

  • The evaluation covers five IsaacLab tasks and compares FOCUS with states-only, RGB-D-only, RGB-D-asymmetric, and a three-stage teacher–student baseline.Methods use two training seeds and five evaluation seeds per checkpoint; end-to-end AUC includes each method’s complete training procedure.Source: §4 · p. 7

4.1 Main Results

  • Pick-and-Place improves from 0.47 to 0.86 test success and 0.12 to 0.61 normalized training success AUC (5.0×); Reach-Cube is solved by all methods.On Lift-Cube, test success rises from 0.35 to 0.98 and AUC from 0.16 to 0.45; on Push-Cube, the respective values rise from 0.75 to 0.85 and 0.29 to 0.34.Source: §4.1 · p. 7
  • Open-Drawer is less diagnostic because dense reward shaping brings several methods near saturation; Figure 2 shows representative successful RGB-D test rollouts on three tasks.The test-time policy uses RGB-D observations, proprioception, and the previous action, without privileged state or modality switching.Source: §4.1 · pp. 7–8, Figure 2 · p. 8
  • Against the strongest RGB-D-at-test baselines, average test success rises from 0.71 to 0.93; across complete pipelines, budget-normalized Training Success AUC rises from 0.47 to 0.65.The averages exclude the states-only reference, which uses privileged state at test time.Source: §4.1 · p. 7, Table 1 · p. 8, Table 2 · p. 9

4.2 Ablation Study

  • A fixed exposure ramp yields lower Pick-and-Place normalized AUC than FOCUS (0.361 vs. 0.612) and lower Push-Cube test success (0.822 vs. 0.850).The comparison indicates a better sample-efficiency and success trade-off for adaptive exposure on these tasks.Source: §4.2 · p. 9
  • On Push-Cube, KL-guided switching keeps RGB-D exposure from saturating early and coincides with lower upper-tail KL mismatch and higher training success than two variants.The variants are a fixed ramp and one without the upper-quantile budget.Source: §4.2 · p. 9
  • FOCUS-simple reaches 1.000 test success on Lift-Cube and 0.994 on Pick-and-Place while retaining only the KL-guided switch and image-encoder policy-output KL term.The ablation suggests these are the most reliable contributors in the evaluation; other auxiliary losses are treated as optional, task-dependent terms.Source: §4.2 · p. 9

5 Conclusion

FOCUS regulates privileged-state versus RGB-D actor rollouts during PPO training while using RGB-D for test-time action selection. Across five IsaacLab tasks, it improves average test success and complete-pipeline budget-normalized training-success AUC over the strongest RGB-D-at-test baselines.

5 Conclusion

  • Average test success rises from 0.71 to 0.93 and budget-normalized training-success AUC from 0.47 to 0.65 across five IsaacLab manipulation tasks.The comparisons use the strongest RGB-D-at-test baseline on each task for test success and account for each method's complete training pipeline for AUC.Source: §5 · p. 9
  • Table 3 reports ablations on Push, Lift, and Pick-and-Place with 1,500/1,500/2,500 training iterations, two training seeds, and five test seeds.Its AUC is budget-normalized by each task horizon.Source: Table 3 · p. 10
  • Pick-and-Place evaluation requires four consecutive on-target steps, whereas training uses a one-step success signal.Source: §5 · p. 10
  • The KL-guided switch reduces exposure-weighted upper-tail cross-modal KL mismatch versus a fixed ramp and a variant without the upper-quantile budget, while improving training success.The diagnostic is representative of Push-Cube.Source: Figure 4 · p. 10
  • Ablations identify controlled RGB-D rollout exposure and image-encoder alignment through policy-output KL as the most reliable contributors; other auxiliary losses have task-dependent effects.The conclusion treats the remaining losses as motivating simpler future variants, not universally necessary components.Source: §5 · p. 10

6 Limitations

FOCUS is evaluated only in simulation on relatively short-horizon manipulation tasks with two training seeds, and its results depend in part on task-specific reward engineering. The authors caution that the limited evaluation does not establish universal ablation conclusions or address the separate sim-to-real gap.

6 Limitations

  • Evaluation is limited to simulation, relatively short-horizon manipulation, and two training seeds; task-specific reward engineering may affect behavior and cross-task comparability.Some runs also converge to locally successful but suboptimal behaviors.Source: §6 · p. 10
  • With a limited evaluation suite, ablations support component-level trends, not universal claims about which auxiliary losses are necessary.More task and seed coverage is needed to distinguish when auxiliary alignment is necessary from when FOCUS-simple suffices.Source: §6 · p. 11
  • FOCUS addresses the privileged-state-to-RGB-D observation gap, not the sim-to-real gap; appearance shift, sensing noise, calibration error, latency, and dynamics mismatch need separate methods or real-world fine-tuning.The authors identify physical-robot evaluation, broader manipulation suites, and less engineered training signals as future work.Source: §6 · p. 11

E Train Step@50 Results

Table 4 reports when training success occupancy first reaches 50%, with lower training steps indicating earlier attainment; teacher–student results distinguish final-student training from the full pipeline.

E Train Step@50 Results

  • Table 4 reports the first training point reaching 50% success occupancy, with lower values better; teacher–student entries distinguish final-student fine-tuning from the full end-to-end pipeline.Parentheses give the number of training seeds reaching the threshold; “Not reached” means no seed reached it.Source: Table 4 · p. 21

F Additional Training Curves

The appendix supplements training-curve views with qualitative representation probes and details of the five-task evaluation protocol. The visualizations are supporting diagnostics, not the paper’s main quantitative evidence.

F Additional Training Curves

  • The full-training EMA curves show training-time success for all tasks; for teacher–student they cover only the final student PPO phase, while Table 2 reports end-to-end normalized AUC.The caption distinguishes the plotted phase from the complete-pipeline metric.Source: Figure 5 · p. 21

G Per episode success rate curves

  • Per-episode curves cover all five tasks and expose short-term variance, instability, and plateau behavior; earlier rises and higher final plateaus indicate better sample efficiency and more reliable training.Figure 6 shows the raw training success signal, unlike the main-text EMA curves.Source: G Per episode success rate curves · p. 22, Figure 6 · p. 22

H Visualizations

  • The appendix’s visual probes are supporting diagnostics only and are not part of the main quantitative evidence.They collect qualitative views of learned visual and latent representations.Source: H Visualizations · p. 23

H.1 Activation probe

  • Activation probes qualitatively show depth emphasizing geometric structure near task-relevant regions and RGB responding more broadly to appearance, contours, and scene boundaries.The patterns remain broadly similar before and after image exposure; the authors treat them as a check, not a formal explanation of policy behavior.Source: §H.1 · p. 23, Figure 7 · p. 23

H.2 Additional activation visualizations

  • Figure 8 adds qualitative activation maps intended to check whether the encoder continues attending to task-relevant regions after transition to RGB-D control.The figure covers Lift, Push, Pick-and-Place, and Open-Drawer.Source: §H.2 · p. 24, Figure 8 · p. 24

H.3 t-SNE latent visualizations

  • Later t-SNE projections often show tighter cross-modal grouping on harder tasks, consistent with improved latent compatibility, but the visualization is qualitative and projection-dependent.The authors treat these plots as supporting diagnostics rather than quantitative alignment evidence.Source: §H.3 · p. 25, Figure 9 · p. 25

I Protocol details

  • The five IsaacLab tasks span visual and control difficulty, from Reach-Cube as a sanity check to Pick-and-Place requiring grasping, transport, and placement.The suite also tests visually grounded lifting, contact-rich pushing, and articulated-object interaction.Source: I Protocol details · p. 26

J Implementation details

The implementation addresses artifacts from tiled-camera rendering with shared preprocessing, while preserving a deployable observation interface that excludes privileged simulator information.

J Implementation details

  • The deployable actor receives RGB-D, proprioception, and the previous action, but not object poses, privileged simulator state, contact state, semantic labels, segmentation IDs, or task variables.Source: J Implementation details · p. 26
  • The cleanup mask removes pixels outside the agent’s rendered environment before RGB-D construction and is applied identically to vision-based methods on each task.The passage characterizes cleanup as rendering-artifact correction rather than a source of task information.Source: J Implementation details · p. 26
  • When simulator-specific cleanup is unavailable, depth truncation and workspace cropping alone can implement the same observation contract.The stated purpose is shared rendering-artifact correction, not providing privileged task information.Source: J Implementation details · p. 26

K Additional implementation defaults

The appendix lists implementation defaults and architecture specifications in three supplementary tables.

K Additional implementation defaults

  • Table 5 lists the main default hyperparameters for FOCUS unless otherwise noted.Source: Table 5 · p. 27
  • Table 6 provides additional implementation defaults that complement Table 5.Source: Table 6 · p. 27
  • Table 7 specifies Torch-level architecture components used in the main experiments, with latent dimension d=256.Source: Table 7 · p. 28

L Logged metrics and diagnostics

The appendix identifies a table of train/test metrics and additional diagnostics logged by the evaluation protocol.

L Logged metrics and diagnostics

  • Table 8 lists train/test metrics and additional diagnostics logged by the protocol.Source: Table 8 · p. 29

M Engineered rewards for each task

This section describes task-specific engineered rewards for Lift-Cube, Pick-and-Place, Push-Cube, Reach-Cube, and Open-Drawer, shaping progress toward task goals through object approach, alignment, grasping, or movement. For Open-Drawer, it also describes staged bonuses for increasing drawer displacement.

Lift-Cube rewards

  • Lift-Cube combines a sigmoid lift-progress reward and above-target bonus with proximity-based grasp shaping tied to gripper closure.The grasp term rewards closing near the object and opening when far away.Source: M Engineered rewards for each task · p. 31

Pick-and-Place rewards

  • Pick-and-Place rewards near-object closing, gated carrying and transport, and opening to release the object within placement thresholds.The reward also combines these stages with reach, lift, hover, and release terms.Source: M Engineered rewards for each task · p. 33
  • The Pick-and-Place state-reward variant shapes reaching, closing, lifting, transport, and placement, with closing and transport terms conditioned differently from the main reward.Its closing reward depends on proximity and gripper opening, while transport is gated by grasping and lifting.Source: M Engineered rewards for each task · pp. 33–34

Push-Cube rewards

  • Push-Cube rewards end-effector proximity to the cube, cube proximity to its target, and gripper-finger closure.The object-to-target distance shaping uses a tanh transform; finger closure is rewarded as the fingertips approach one another.Source: M Engineered rewards for each task · p. 35

Reach-Cube rewards

  • Reach-Cube uses end-effector-to-object distance shaping and tracks success when that distance falls below 0.3.The success function records the thresholded indicator in the environment’s success-rate state.Source: M Engineered rewards for each task · pp. 36, 38

M Engineered rewards for each task

  • Handle-approach reward is the squared reciprocal of distance and doubles when distance is within the threshold.The reward is defined as (1 / (1 + distance^2))^2, with a 2x multiplier when distance <= threshold.Source: M Engineered rewards for each task · p. 37
  • Handle alignment rewards matching end-effector orientation, while graspability checks whether the two fingers straddle the handle vertically.A separate gripper-approach reward is gated by that graspability condition and reflects both fingers’ vertical distances to the handle.Source: M Engineered rewards for each task · pp. 37–38
  • Drawer-opening rewards combine progress with graspability and staged bonuses at displacement thresholds above 0.01, 0.2, and 0.3.The open-drawer bonus multiplies drawer position by (is_graspable + 1.0); the multistage reward adds 0.5 for opening past 0.01 and graspability-weighted rewards past 0.2 and 0.3.Source: M Engineered rewards for each task · p. 38
Loading 2610.11119v1…