Source-linked AI summary
Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation
Seungyeon Yoo, Gawon Lee, Seungwoo Jung, Inkyu Jang, H. Jin Kim
TL;DR
RGB-only visual navigation remains vulnerable to collisions, while existing visual CBF approaches often depend on scene reconstruction or rendering and are primarily designed for static scenes. This paper distills a privileged CBF teacher into an RGB-only student safety filter for dynamic environments. Experiments report improved safety and navigation outcomes across simulation and real-world settings, with occasional timeout failures when filtering slows the robot in densely occupied regions.
Problem
RGB-only navigation policies remain vulnerable to collisions, while visual CBF approaches often rely on scene representations and are designed for static scenes.
Source: §I · p. 1Method
A privileged state-based CBF teacher supervises an RGB-only student filter, with teacher constraints limited to obstacles visible in the student’s RGB history and accounting for obstacle-velocity uncertainty.
Source: §I · p. 2, §V · p. 3, A. Privileged State-Based CBF Teacher · p. 3Results
Experiments report that the distilled filter improves success and collision-free rates and increases obstacle clearance, while outperforming visual CBF baselines in nearly every reported cell.
Source: C. Navigation Results · p. 6, B. Navigation Results · p. 7, §VIII · p. 8Takeaways & Limitations
The student filters nominal actions using RGB observations and robot velocity without requiring localization, obstacle-state estimation, or a 3D scene representation at deployment.
Source: §I · p. 2, §VIII · p. 8In densely occupied regions, the filter can increase timeout failures because it reduces speed.
Source: B. Navigation Results · p. 7
Abstract
from arXiv · showhide
RGB-only end-to-end visual navigation policies remain vulnerable to collisions in real-world dynamic environments, motivating a dedicated safety layer. Existing visual Control Barrier Function (CBF) approaches seek to provide safety from RGB observations, but often rely on real-time rendering or explicit scene reconstruction and are primarily designed for static scenes, limiting their practicality for onboard deployment. We propose a teacher-student visual distillation framework that transfers the safety behavior of a privileged CBF teacher to an RGB-only student filter for dynamic environments. The student maps a short RGB history, robot velocity, and a nominal control action directly to a safe action, while the teacher uses ground-truth robot and obstacle states in a real-to-sim dynamic Gaussian Splatting environment. To reduce the teacher-student information gap, the teacher constructs safety constraints only from obstacles observable within the student's RGB history. It also accounts for obstacle-velocity uncertainty to improve robustness to motion variations, while action augmentation exposes the student to diverse safe and unsafe nominal actions to better capture the safety boundary. At deployment, the student requires only RGB observations and robot velocity, without explicit 3D reconstruction or online rendering. Experiments show that the proposed method outperforms visual CBF baselines and improves the safety of RGB-based navigation policies under dynamic obstacle motion. Project page: https://syeon-yoo.github.io/distill-cbf-site/.
I. INTRODUCTION
RGB-only navigation policies offer practical deployment advantages but remain vulnerable to collisions, especially around dynamic obstacles. The paper distills a privileged CBF teacher into an RGB-only safety filter for dynamic environments.
I. INTRODUCTION
- RGB-only visual navigation policies simplify deployment, but their susceptibility to collisions—particularly in dynamic environments—remains a major obstacle.They avoid explicit mapping or reconstruction in the control loop and can provide low-latency visual feedback.Source: §I · p. 1
- Existing safety mechanisms may require localization, obstacle-state estimation, depth, or 3D scene representations, while dynamic obstacles add motion and velocity uncertainty.Prior visual CBF approaches assume static scenes and require a 3D scene representation at deployment.Source: §I · p. 1
- The method distills a privileged state-based CBF teacher into an RGB-only student that maps nominal actions to safe actions.The teacher uses simulated ground-truth states, while the student uses RGB inputs and robot velocity at deployment.Source: §I · pp. 1–2
- The teacher limits its constraints to obstacles visible in the student’s short RGB history and accounts for uncertainty in obstacle velocities.Training also supplies multiple teacher-labeled candidate actions per observation to help the student learn the safe-action boundary.Source: §I · p. 2
- Experiments show the filter outperforms prior visual CBF methods in simulation and reduces collisions for multiple nominal policies in real-world dynamic environments.The student uses RGB observations and robot velocity at deployment, without a 3D scene representation, localization, or obstacle-state estimation.Source: §I · p. 2
A. RGB-Only Visual Navigation
RGB-only navigation training and real-to-sim pipelines address deployment challenges, but safety around moving obstacles remains insufficiently addressed. Visual CBFs and privileged learning offer relevant approaches, while CBF-QPs provide a mechanism for minimally correcting unsafe actions.
A. RGB-Only Visual Navigation
- RGB-only navigation and real-to-sim pipelines improve real-world applicability, yet safety around moving obstacles remains insufficiently addressed.General navigation models leverage cross-embodiment data, while Gaussian Splatting pipelines generate photorealistic navigation data, including dynamic scenes.Source: A. RGB-Only Visual Navigation · p. 2
B. Visual CBF-Based Safety Filter
- Visual CBF baselines rely on scene representations or depth and assume static obstacles; this work instead distills a privileged CBF filter for RGB-only inference with obstacle motion considered.Prior methods include NeRF rendering, online splat maps, RGB-D barriers, and point-cloud barriers.Source: B. Visual CBF-Based Safety Filter · p. 2
C. Learning from Privileged Teachers
- Privileged teacher-student learning is common in policy learning and perception, but rarely distills corrective behavior from a model-based safety filter.This work addresses the information gap between a state-based CBF teacher and a student operating from partial visual observations.Source: C. Learning from Privileged Teachers · p. 2
III. PRELIMINARIES: CONTROL BARRIER FUNCTIONS
- A CBF-QP returns a reference action unchanged when safe and otherwise makes the smallest correction that restores safety.Multiple obstacle constraints can be stacked, provided the resulting QP remains feasible.Source: §III · p. 3
IV. PROBLEM FORMULATION
The problem is to filter a visual navigation policy’s command using only onboard RGB observations and robot speed, while reproducing the behavior of a privileged safety filter. Training uses a photorealistic target-environment simulation where privileged states are available.
IV. PROBLEM FORMULATION
- The setting is a wheeled robot with a forward-facing RGB camera navigating among static obstacles and a moving human, modeled with planar position, heading, speed, acceleration, and yaw rate.The state includes robot and human states and static obstacles; safety requires minimum clearance from obstacles and the human.Source: §IV · p. 3
- At deployment, the filter can use a short RGB-frame history, the nominal command, and forward speed from the wheel encoder.The nominal navigation policy maps the current image and robot speed to a reference command and provides no safety guarantee of its own.Source: §IV · p. 3
- The learning objective is an RGB-only filter that reproduces the privileged filter’s behavior without access to the full state.The privileged filter’s evaluation requires robot pose, human position and velocity, and scene geometry.Source: §IV · p. 3
- Training uses a photorealistic digital twin of the target environment, where privileged state is available and dynamic scenarios can be generated at scale.The teacher’s decisions must be inferable from the student’s short RGB history, so the teacher should depend on privileged state only through visually observable quantities.Source: §IV · p. 3, Figure 2 · p. 3
V. RGB-ONLY SAFETY FILTER DISTILLATION
The framework pairs a privileged CBF teacher, which filters nominal navigation actions using simulator information, with an RGB-only student trained to reproduce its safe actions.
V. RGB-ONLY SAFETY FILTER DISTILLATION
- A privileged CBF teacher maps nominal actions from RGB-only navigation policies to safe actions using simulator-provided privileged information.The student distills this behavior in a photorealistic simulator for deployment without privileged information and with multiple nominal policies.Source: §V · p. 3
- The student maps a stack of RGB observations and a nominal action to a safe action.The teacher-to-student setup is designed to transfer a ground-truth-state safety filter into an RGB-only filter.Source: §V · p. 3
- The student is distilled in a photorealistic simulator and can be deployed without privileged information across multiple nominal policies.Source: §V · p. 3
A. Privileged State-Based CBF Teacher
The teacher limits its inputs to obstacle information inferable from the student's short RGB history, then uses dynamic CBF constraints that account for stale human-velocity estimates and their uncertainty.
A. Privileged State-Based CBF Teacher
- Static geometry is represented by the nearest raycast hit per bearing, while a human is included only if visible in one of the student's three RGB-history frames.The teacher estimates velocity from the most recent visible frame and extrapolates the human's position; its velocity estimate can therefore be stale.Source: A. Privileged State-Based CBF Teacher · p. 3
- The teacher augments the dynamic parabolic CBF with an uncertainty-dependent tightening margin because the observed human velocity can become stale as motion changes.Using the exact privileged velocity would widen the teacher-student information gap.Source: A. Privileged State-Based CBF Teacher · pp. 3–4
- The human-velocity error is modeled with anisotropic covariance, assigning greater uncertainty along the current motion direction than across it.This reflects the assumption that pedestrians are more likely to vary speed than abruptly change heading, encouraging yielding to crossing pedestrians without uniformly enlarging the margin.Source: A. Privileged State-Based CBF Teacher · p. 4
- A first-order approximation propagates velocity uncertainty into the barrier, and the margin is treated as uncertainty-aware tightening rather than a probabilistic guarantee.The covariance is fixed to avoid excessive conservatism under prolonged occlusions.Source: A. Privileged State-Based CBF Teacher · p. 4
- At each step, the teacher's QP uses barriers for the static obstacles and, when present, the uncertainty-aware human barrier.The QP is strictly convex.Source: A. Privileged State-Based CBF Teacher · p. 4
B. RGB-Only Student Safety Filter
The RGB-only student learns to reproduce the teacher's filtered actions from onboard observations and nominal actions, using varied rollout data and offline action augmentation to broaden training coverage.
B. RGB-Only Student Safety Filter
- Training data log camera frames, robot speed, nominal actions, and the teacher's QP solution in dynamic Gaussian Splatting simulation.The dataset includes nominal and filtered rollouts, using imitation-learning, reinforcement-learning, and general navigation policies as nominal controllers.Source: B. RGB-Only Student Safety Filter · p. 4
- Offline action augmentation perturbs logged actions and relabels them with teacher outputs, broadening candidate-action coverage at visited states.Candidates come from Gaussian and uniform distributions, with additional samples from any CBF constraint's intersection with the action space.Source: B. RGB-Only Student Safety Filter · p. 4
- A residual CNN encodes each frame, and an MLP maps concatenated frame features and scalar inputs to the predicted safe action.Source: B. RGB-Only Student Safety Filter · p. 4
- An auxiliary image-only head predicts whether a human is in view, encouraging the CNN to represent human presence; the head is discarded at deployment.Its training labels are computed in simulation from the human's ground-truth position and robot pose.Source: B. RGB-Only Student Safety Filter · p. 4
VI. SIMULATION EXPERIMENTS
The simulation experiments evaluate whether the onboard-observation safety filter improves both safety and task performance for nominal navigation policies in dynamic scenes.
VI. SIMULATION EXPERIMENTS
- The filter is evaluated in the ReaDy-Go simulator, which renders reconstructed scenes and dynamic human avatars with Gaussian splatting.Source: §VI · p. 4
- The evaluation tests the filter using only onboard observations.Source: §VI · p. 4
- The experiments assess whether filtering improves both safety and task performance of nominal policies.Source: §VI · p. 4
A. Experimental Setup
The experiments evaluate point-goal navigation around static obstacles and a moving human, using shared trials across nominal policies and safety filters. They report navigation, collision, clearance, timing, and failure metrics for imitation-learning, reinforcement-learning, and ViNT policies.
1) Task description:
- Trials use random start-goal pairs in each scene; success requires reaching the goal within 50 seconds, while collision or timeout counts as failure.Start-goal pairs and human trajectories are identical across nominal policies and safety filters.Source: 1) Task description · p. 4
2) Evaluation metrics:
- Evaluation reports success rate, collision-free rate, mean safety margin, average reaching time, and static-collision, human-collision, and timeout rates.Failed trials count as zero clearance for mean safety margin and as the maximum time for average reaching time.Source: 2) Evaluation metrics · p. 5
3) Nominal visual navigation policies: •
- The nominal policies are an imitation-learning policy trained by behavior cloning, per-scene SAC policies, and ViNT using its public checkpoint without fine-tuning.The imitation policy follows the ReaDy-Go pipeline; ViNT's third waypoint is mapped to velocity commands and then to an acceleration reference.Source: 3) Nominal visual navigation policies: • · p. 5
4) Visual safety filter baselines: •
The evaluation compares visual safety-filter baselines and describes the student training data. The distilled filter improves navigation outcomes over nominal policies, closely tracks the teacher, and reproduces its action-space intervention boundary.
4) Visual safety filter baselines: •
- SAFER-Splat receives simulated Gaussian primitives and ground-truth human Gaussians, while NeRF-CBF uses per-scene NICE-SLAM and rendered human depth for candidate-action scoring.Both baselines use the same nominal policies, scenarios, dynamics, and success and collision criteria as the proposed method.Source: 4) Visual safety filter baselines: • · p. 5
B. Implementation Details
- Each scene's student is trained on data from three nominal policies and two rollout types, with roughly 1.5 million transitions expanded to 12 million by action augmentation.The rollout types use unfiltered nominal policies and the teacher filter, respectively.Source: B. Implementation Details · p. 5
C. Navigation Results
- Across all scenes and nominal policies, the distilled filter raises success rate, collision-free rate, and safety margin over the unfiltered policy, closely tracks the teacher, and outperforms both visual CBF baselines in nearly every cell.The student acts on RGB images, the nominal command, and robot speed; SAFER-Splat and NeRF-CBF use ground-truth human pose at test time and depend on a 3D scene representation.Source: C. Navigation Results · pp. 5–6
D. Filtering Boundary Reproduction by the Student
- The student boundary closely follows the teacher's constraint boundary as the scene changes, capturing local intervention structure rather than only individual safe-action labels.The figure maps observations in the top row to action-space boundaries below; its left snapshot rejects right turns, while the right leaves only decelerating actions admissible.Source: D. Filtering Boundary Reproduction by… · p. 6, Figure 3 · p. 6
E. Ablation Study
The ablations show that combining dataset aggregation with action augmentation improves performance across scenes and nominal policies, while the uncertainty-aware margin reduces human collisions.
1) Effect of the uncertainty-aware margin:
- Without the margin, teacher human collisions rise to 12.1% from 7.3%, and student human collisions rise to 15.3% from 10.8%.The passage attributes this to stale velocity estimates when a human changes pace or heading outside the field of view.Source: E. Ablation Study · p. 6
- The student reproduces the uncertainty-aware tightening from RGB history without observing human velocity or its covariance.The margin trades modest conservatism for robustness to motion changes.Source: E. Ablation Study · p. 6
2) Ablation on dataset aggregation and augmentation:
- 74.0% mean success and 86.1% collision-free rate with aggregation and augmentation, close to the teacher’s 76.8% and 91.5%.Either ingredient alone does not improve over the per-nominal, non-augmented student; together they provide diverse visited states and broader nominal-action coverage.Source: 2) Ablation on dataset aggregation and… · p. 7
VII. HARDWARE EXPERIMENTS
Hardware experiments test the RGB-only filter across nominal policies and dynamic environments, including under persistently unsafe forward commands. The filter improves real-world navigation outcomes, and the conclusion identifies transfer to unseen environments and policies as future work.
A. Hardware Platform
- The filter runs in 9.2 ms per inference on a Jetson Orin NX; wheel odometry supplies relative goals only to nominal policies that need them.The safety filter itself receives no odometry.Source: A. Hardware Platform · p. 7
B. Navigation Results
- Across 180 real-world episodes with five unseen human obstacles, the filter improved success, collision-free rates, and clearance across nominal policies, but occasionally increased timeouts.The reported failure analysis says the filter reduces speed in densely occupied regions.Source: B. Navigation Results · p. 7
C. Safety Under Aggressive Nominal Inputs
- Under persistent 0.75 m/s forward commands with zero angular velocity, the filter brakes for crossing humans and steers around static obstacles and walls.The behaviors are shown in the Library and Lobby environments.Source: C. Safety Under Aggressive Nominal… · pp. 7–8
VIII. CONCLUSION
- Future work aims to scale training across scenes toward one RGB-only filter that transfers to unseen environments and nominal policies.The conclusion reports that the current filter improves success, collision-free rates, and clearance in simulation and real-world experiments.Source: §VIII · p. 8