Source-linked AI summary
REDACT: Robust Perceptive Locomotion under Unseen Visual Corruption
Natapat Kirdwichai, Tobias Driskell-Poole, Andrei Sontea, Jadu Dash, Muhammad Burhan Hafez, Danesh Tarapore
TL;DR
Depth-conditioned locomotion policies face unfamiliar visual corruption that may be missing from training coverage. REDACT combines an improved encoder, persistent masking, and clean-calibrated consensus gating; it reports higher success on unseen corruptions and zero-shot traversal in real-world structured and forest environments.
Problem
Visual changes from clutter, occlusion, and sensor responses are difficult to capture fully during training, leaving robustness to unmodeled corruption uncertain.
Source: §I · p. 1Method
REDACT combines an improved visual encoder, persistent feature masking, and a consensus gate with per-cell thresholds calibrated on clean rollouts alone.
Source: §I · p. 2Results
At maximum tested severity on corruptions absent from training, REDACT achieved 74% mean success versus 37% for augmented baselines, with further gains from augmentation.
Source: §I · p. 2Takeaways & Limitations
Real-world trials demonstrate zero-shot traversal in structured and forest environments with unfamiliar scene content.
Source: §V · p. 8
Abstract
from arXiv · showhide
Depth-conditioned locomotion policies have demonstrated impressive agile maneuvers, but can be steered to unpredictable actions when observations are outside their training distribution. Occlusion, invalid returns, sensor noise, and visual distractors can shift deployment observations away from nominal simulated depth. While synthetic sensor augmentation targets specified degradations, it does not by itself define behavior under corruption families omitted from training. To address gaps in training-time coverage, we present REDACT (Retaining Evidence Despite Artifacts for Continued Traversal), a teacher-student framework combining an improved visual encoder architecture, persistent feature masking, and a novel consensus-gating algorithm to retain useful depth information under unmodeled corruption. The gate uses approximate conformal calibration on clean observations alone, requiring no prior knowledge of the corruption type. Trained on clean simulated depth, REDACT retains useful visual information under unseen corruption, supporting higher traversal success than existing parkour baselines. Evaluation of depth augmentation across corruption families further shows that REDACT improves robustness where augmentation coverage is missing. Real-world trials demonstrate zero-shot transfer to structured and forested environments with unfamiliar scene content.
I. INTRODUCTION
REDACT addresses the gap between training-time visual coverage and the unfamiliar depth changes encountered in unstructured terrain. Its clean-calibrated consensus gate and other design elements support traversal under unseen corruption, including in real-world settings.
I. INTRODUCTION
- Forest traversal involves roots, deadwood, vegetation, and abrupt elevation changes, while overhanging vegetation, clutter, and weather-dependent sensor responses challenge visual coverage.Terrain information helps legged robots anticipate footholds and adjust body motion before contact.Source: §I · p. 1
- REDACT combines improved visual encoding, persistent feature masking, and consensus gating to retain useful terrain information under corruption absent from training.The gate uses disagreement among local predictions of shared visual features, with per-cell thresholds calibrated on clean rollouts alone.Source: §I · p. 2
- 74% mean success versus 37% for augmented baselines on corruptions absent from training at maximum tested severity; augmentation provides further gains.The comparison assesses REDACT’s utility where augmentation coverage is missing.Source: §I · p. 2
- Real-world trials demonstrate zero-shot traversal over structured obstacles and forest terrain with unfamiliar visual content.The illustrated trials include hurdles with adjacent pillars, logs with distractors, and logs amid vegetation and leaf litter.Source: Figure 1 · p. 1, §I · p. 2
II. RELATED WORK
Prior work uses augmentation and perceptual gating to address degraded observations, but robustness to unmodeled corruption remains uncertain. REDACT’s related-work context also identifies open questions about selective rejection and visual encoder design.
II. RELATED WORK
- Prior deployment results do not establish whether robustness extends to unmodeled corruptions arising under different environmental and weather conditions.Preprocessing and synthetic sensor augmentation target discrepancies or anticipated degradations.Source: §II · p. 2
- Existing locomotion gates regulate vision or switch to proprioceptive fallback, while learned selection for egocentric depth must retain useful information under degradations absent from gate training.RENet discards useful obstacle information when only part of the observation is corrupted; Miki et al. train selection using hand-designed terrain-height perturbations.Source: §II · p. 2
- PatchGuard masks suspicious image regions before aggregation but requires a bound on corruption extent, which is difficult to specify for variable-span depth disturbances.The passage identifies noise, occlusion, and unfamiliar geometry as disturbances with varying spatial extents.Source: §II · p. 2
- Whether deeper residual encoder improvements confer robustness to depth corruptions absent from training remains open in perceptive locomotion.Depth-conditioned methods commonly use shallow convolutional encoders, and their encoding errors can propagate to downstream components.Source: §II · p. 2, Figure 2 · p. 2
III. METHOD
REDACT trains a privileged teacher and depth-conditioned student to retain terrain information while limiting unseen-corruption influence. Teacher training uses multi-query attention and regularization to shape terrain representations for student fine-tuning.
III. METHOD
- The framework trains a privileged teacher and depth-conditioned student, using clean-calibrated thresholds to guide feature removal at deployment.The teacher’s terrain representation is regularized to reduce dependence on privileged detail that onboard depth cannot recover; persistent masking prepares control from partial feature grids.Source: §III · p. 3
A. Teacher Training
- Multiple queries are formed by an MLP fusing proprioceptive history encoded by a temporal convolutional network with pooled scandot features; keys and values come from 2×2 scandot patches for retrieval.Separate retrievals allow the teacher to represent terrain regions jointly relevant to a maneuver, such as takeoff and landing surfaces.Source: A. Teacher Training · p. 3
- Attention-entropy and overlap penalties concentrate retrievals and encourage distinct query regions; PPO training also penalizes changes between consecutive terrain representations.The retrieved features are concatenated and projected into a tanh-bounded terrain latent conditioning the actor alongside proprioception and privileged states.Source: A. Teacher Training · p. 3
B. Student Training
The student combines a residual depth encoder and selective feature aggregation with recurrent control. Training uses shared full-map targets for local predictions and intermittent feature masks to prepare for partial visual evidence.
B. Student Training
- A 4×6 depth-feature grid supports selective aggregation, while mean–max pooling and a spectrally capped projection form the visual latent.The student uses an IMPALA-style residual architecture, group normalization, and strided convolutions for downsampling.Source: B. Student Training · pp. 3–4
- Finite scalar quantization maps each latent coordinate to five levels, and the quantized latent and proprioception enter a GRU that supplies heading errors and a learned representation to the actor.Perturbations within quantization bins leave the visual code unchanged.Source: B. Student Training · p. 4
- Each cell predicts the shared unmasked full-map latent, enabling disagreement-based rejection of inconsistent spatial features before visual information reaches the recurrent policy.A feature-wise standard-deviation penalty discourages latent collapse.Source: B. Student Training · p. 4
- Intermittent contiguous and scattered feature masks prepare the recurrent policy for partial visual evidence, while leaving cell heads and full-map supervision unchanged.Masks are held fixed across several camera updates and sampled independently of consensus scores.Source: B. Student Training · p. 4
- The student collects trajectories under its own policy while the teacher supervises visited states through DAgger-style behavioral cloning, with the visual encoder, GRU, and initialized actor optimized jointly.The terrain set includes tilted ramps, gaps, hurdles, and steps.Source: B. Student Training · p. 4, Figure 4 · p. 4
C. Conformal Calibration and Deployment
REDACT calibrates a separate disagreement threshold for each spatial cell using clean observations, then excludes cells that exceed their thresholds during deployment. The calibration is approximate because deployment observations change under gating, and the fallback can retain above-threshold evidence.
C. Conformal Calibration and Deployment
- With at most 11 corrupted cell predictions, each leave-one-out reference retains an uncorrupted majority, but corruption that shifts predictions together may produce little disagreement and remain undetected.The coordinatewise median of other cells’ predictions prevents a cell from influencing its own reference.Source: C. Conformal Calibration and Deployment · p. 4
- Fig. 6 plots success rate across four depth-corruption sweeps and pillar-hurdle trials, with means and 95% confidence intervals.Pillar and hurdle heights vary together, with pillars three times the hurdle height.Source: Figure 6 · p. 5
- Separate cell thresholds are calibrated as empirical 99th percentiles of disagreement scores from clean rollouts collected by the frozen student with gating disabled.Calibration samples one frame per episode from approximately 3,500 episodes.Source: C. Conformal Calibration and Deployment · p. 5
- Exact conformal guarantees require exchangeability between calibration and nominal test scores, while gating-related observation changes make deployment calibration approximate.Episode-level sampling avoids within-episode temporal dependence.Source: C. Conformal Calibration and Deployment · p. 5
- At deployment, cells exceeding their thresholds are excluded from pooling; if more than 18 are flagged, the gate removes only the six highest-scoring cells and pools the remaining 18.This fallback limits feature removal but may retain evidence above its calibrated threshold.Source: C. Conformal Calibration and Deployment · p. 5
IV. RESULTS
The results section evaluates REDACT against parkour baselines and augmentation-trained policies under observations beyond training coverage. It also examines component contributions and assesses zero-shot transfer in real environments.
IV. RESULTS
- Comparisons with parkour baselines and augmentation-trained policies assess traversal robustness beyond training coverage.The section evaluates whether REDACT maintains successful traversal when depth observations differ from those encountered during training.Source: §IV · p. 5
- Component ablations and gate analysis examine the contributions of the encoder and selective rejection.Source: §IV · p. 5
- Real-world trials assess zero-shot transfer to structured and forest environments.Source: §IV · p. 5
A. Experimental Setup
REDACT and its baselines share a simulated training setup, while evaluation tests locomotion under several visual corruptions and distractions. Performance is reported using success and displacement metrics across five seeds.
A. Experimental Setup
- All policies use the same terrain generation, curriculum, rewards, domain randomization, and 15,000-iteration training budget for a simulated Unitree Go2.Training uses depth images rendered at 106 × 60 pixels and preprocessed to 87 × 58 pixels.Source: A. Experimental Setup · p. 5
- Evaluation covers Gaussian noise, flying pixels, Perlin dropout, simulated vine occlusion, and pillar distractions across four terrain types and five seeds.Each condition uses 3,200 environments; corruption sweeps range from mild degradation to severe stress beyond typical sensor conditions.Source: A. Experimental Setup · p. 5
- Success rate measures reaching all eight randomized checkpoints within 0.2 m, while mean x-displacement records the fraction reached; results report means and 95% Student-t confidence intervals.SR is a percentage and MXD is in [0,1]; corruption tables use maximum severity.Source: A. Experimental Setup · pp. 5–6
B. Comparison to Baselines
REDACT is compared with Extreme Parkour and REAL without depth-image augmentation. It has higher success rates under unseen depth corruption, including pillar hurdles that add unfamiliar geometry beside the route.
B. Comparison to Baselines
- REDACT, Extreme Parkour, and REAL are trained without depth-image augmentation, retaining the baselines’ original teacher–student architectures.All three support locomotion under nominal observations, but their success rates diverge under unseen depth corruption.Source: B. Comparison to Baselines · p. 6
- REDACT has higher success rates across the tested corruption sweeps, indicating policy-design differences accompany robustness beyond training visual conditions.The passage reports this comparison without numerical success-rate values.Source: B. Comparison to Baselines · p. 6
- Under pillar hurdles, both baselines show foot-placement and steering errors despite retaining the depth information needed for traversal.Pillars are placed beside the route without changing the optimal route or required maneuver.Source: B. Comparison to Baselines · p. 6
C. Generalization Beyond Training Corruptions
Depth-image augmentation transfers unevenly across corruption families. REDACT outperforms augmentation-trained ConvNet policies on corruptions absent from training, while targeted augmentation provides additional, disturbance-specific gains.
C. Generalization Beyond Training Corruptions
- The transfer evaluation trains the ConvNet separately on each of four corruption types and tests each policy on all four corruptions and nominal observations.Augmented REDACT variants use Perlin dropout or vine occlusion, with GroupNorm scale and shift learned only from clean depth.Source: C. Generalization Beyond Training… · p. 6
- Flying-pixel and Gaussian augmentation improve performance under both noise types, but vine-occlusion training provides little benefit under Gaussian noise or Perlin dropout.Vine-trained ConvNet SR under flying pixels falls to 1.9%.Source: C. Generalization Beyond Training… · p. 6
- Perlin augmentation reduces nominal ConvNet SR from 85.9% to 77.9%, showing that gains on corrupted inputs need not preserve nominal performance.The authors suggest extensive depth removal during training can impair nominal performance.Source: C. Generalization Beyond Training… · p. 6
- Without augmentation, REDACT has higher SR and MXD than augmentation-trained ConvNet policies on corruptions absent from their training.Its advantage under flying pixels and Gaussian noise persists against policies trained with the corresponding augmentation.Source: C. Generalization Beyond Training… · p. 6
- Targeted augmentation further improves REDACT under Perlin dropout and vine occlusion, although Perlin training reduces its flying-pixel robustness.The findings distinguish broader policy-design benefits from augmentation gains on specific disturbances.Source: C. Generalization Beyond Training… · p. 7
D. Component Ablations
Component ablations assess the encoder, consensus gate, and training-time masking; gate analysis examines corruption discrimination and depth reconstruction. Gating improves traversal under spatially concentrated disturbances, while its detection varies by corruption.
D. Component Ablations
- Replacing the residual encoder with the ConvNet lowers SR under nominal observations and all tested disturbances, while removing group normalization selectively harms Perlin, vine, and pillar-hurdle performance.Nominal traversal and pixelwise-noise performance remain high without group normalization.Source: D. Component Ablations · p. 7
- 65.7% vine-occlusion SR with gating versus 39.0% without it; removing training-time masking sharply reduces pillar-hurdle SR regardless of gating.Gating also raises performance under Perlin dropout and pillar hurdles while leaving nominal SR largely unchanged.Source: D. Component Ablations · p. 7
E. Gate Analysis
- Flag-count AUROC is 0.94 for Perlin dropout and 0.95 for vine occlusion, versus 0.73 for flying pixels and 0.51 for Gaussian noise.The authors note spatially distributed perturbations may affect both local predictions and their leave-one-out median reference; REDACT remains robust to Gaussian noise with gating disabled.Source: E. Gate Analysis · p. 7
- For simulated vine occlusion, gated reconstructions more closely match clean-input reconstructions and have lower L1 error than ungated reconstructions.The decoder is trained on clean depth frames, and flagged cells align approximately with the occluder.Source: E. Gate Analysis · p. 7, Figure 8 · p. 7
F. Real-World Traversal
In real-world trials, REDACT showed little gait-quality change with adjacent pillars and completed seven of ten trials at each forest site; the conclusion also reports zero-shot traversal in structured and forest environments with unfamiliar scene content.
F. Real-World Traversal
- Median visual-update latency was 8.8 ms with gating, 6.2 ms without gating, and 1.9 ms with the ConvNet.The depth backbone ran at 10 Hz and the MLP actor at 50 Hz.Source: F. Real-World Traversal · p. 7
- Fig. 9 follows one log traversal in order, pairing external traversal-stage views with onboard depth crops showing vegetation dropouts and added tree and branch geometry.Source: Figure 9 · p. 8
- All policies traversed nominal hurdles; with adjacent pillars, baselines often collided while REDACT's gait changed little, and REDACT completed seven of ten forest trials at each site.Baseline log evaluations were not extended because of observed collisions and hardware-safety concerns.Source: Table IV · p. 8, F. Real-World Traversal · p. 8
V. CONCLUSION
- REDACT combines improved visual encoding, persistent feature masking, and consensus gating; evaluations report improved robustness to unseen disturbances and benefits from targeted depth augmentation.Real-world trials demonstrate zero-shot traversal in structured and forest environments with unfamiliar scene content.Source: §V · p. 8