Source-linked AI summary
Recovering the View: Benchmarking Physical Active Vision for Occlusion Recovery in Robotic Manipulation
Kaijun Luo, Yudi Huang, Qijun Zhong, Xinshuai Song, Yang Liu, Liang Lin
TL;DR
Manipulation policies often rely on passive views, leaving recovery from externally disrupted visibility during execution under-supported in existing benchmarks. This paper introduces BAVO-Bench and A-FAR, and reports improved performance over ManiFlow across visibility conditions, including unexpected occlusion timing.
Problem
Existing benchmarks provide limited support for studying how policies recover when task-relevant visibility is disrupted unexpectedly during manipulation.
Source: §1 · p. 2Method
BAVO-Bench controls external occlusion timing, while A-FAR uses robot-centric 3D observations and training-time 4D relational distillation for joint viewpoint and manipulation control.
Source: §1 · pp. 2–3Results
Across five tasks, A-FAR improves over ManiFlow under all visibility conditions, including Random-time Occlusion (SR 18.7% vs 15.0%), and transfers to the real-world active-camera platform.
Source: §5.2 · p. 8, §6 · p. 10Takeaways & Limitations
Recovering visibility does not necessarily ensure task success; exploiting newly acquired observations for subsequent closed-loop control remains a challenge.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Physical active vision allows robots to change their viewpoint when task-relevant observations become unreliable, yet existing manipulation benchmarks provide limited support for studying how policies recover from occlusion during execution. We introduce BAVO-Bench (Bimanual Active Vision under Occlusion), a bimanual active-vision benchmark that systematically controls external visibility through Clean, Stage Occlusion, and Random-time Occlusion conditions, enabling evaluation of both manipulation performance and active visual recovery. Building on this setting, we present A-FAR (Active Future-Aware Recovery), an active-vision policy for joint viewpoint and manipulation control. A-FAR represents moving-camera observations in a unified robot-centric 3D frame and distills relational structure together with its future evolution from a pretrained 4D model, providing the policy with future-aware geometric guidance without requiring future observations at deployment. Experiments across multiple manipulation tasks show that A-FAR improves robustness to both structured and temporally shifted occlusions while maintaining strong performance under clean observations.
1 INTRODUCTION
The paper frames physical active vision as closed-loop recovery when externally induced occlusion makes views unreliable during manipulation. It introduces a benchmark and policy to evaluate and address this setting.
1 INTRODUCTION
- Existing manipulation policies often rely on passive views, while physical active vision can acquire fresh measurements after task-relevant visibility is lost.Computational alternatives instead select or synthesize observations from existing scene evidence.Source: §1 · p. 2
- The paper studies recovery from externally induced visibility failures during execution, including disruptions that occur unexpectedly rather than at planned task stages.The policy must recover an informative viewpoint before continuing manipulation.Source: §1 · p. 2
- BAVO-Bench evaluates five bimanual tasks under Clean, Stage Occlusion, and Random-time Occlusion, separating nominal performance, demonstrated recovery, and unexpected-timing generalization.Camera-attributable Visibility Gain measures whether camera motion itself restores task-relevant visibility.Source: §1 · p. 2
- A-FAR combines robot-centric 3D observations with training-time 4D relational supervision to guide joint viewpoint and manipulation control without a 4D teacher at deployment.The supervision includes current point relations and their future evolution.Source: §1 · pp. 2–3
- The authors also develop a low-cost VR-enabled active-vision stack for demonstrations in simulation and the real world, with a physical camera platform for deployment.Source: §1 · p. 3
2 RELATED WORK
Related work covers physical active vision, benchmarks that evaluate information acquisition, and representations for viewpoint-consistent, future-aware control. This paper distinguishes its focus as recovery after externally disrupted visibility.
Physical active vision for robotic manipulation
- AV-ALOHA, Vision in Action, and SaPaVe learn coordinated viewpoint control, while this work focuses on recovery after visibility is externally disrupted during manipulation.Computational methods such as ActiveVLA select viewpoints from reconstructed scene representations.Source: §2 · p. 3
Benchmarks for active-view manipulation
- Existing benchmarks evaluate exploration, occlusion, out-of-view conditions, or procedural shifts; BAVO-Bench instead applies controllable visibility failures within an otherwise unchanged task.Its CVG metric compares the recovered view with a same-state frozen-camera counterfactual.Source: §2 · p. 3
Viewpoint-consistent and future-aware representations
- Related representation methods use camera-pose supervision, predict future 3D flow and camera trajectories, or distill temporal 4D relations; A-FAR uses shared robot-centric coordinates and D4RT queries.A-FAR conditions joint viewpoint–manipulation control without future observations or the 4D teacher at deployment.Source: §2 · p. 3
3 BAVO-BENCH: PHYSICAL ACTIVE-VIEW RECOVERY UNDER EXTERNAL OCCLUSION
BAVO-Bench converts external visibility loss into a controlled intervention in five multi-stage bimanual tasks. It separates nominal execution, stage-aligned recovery, and recovery under unexpected timing, with task-performance and visibility-recovery metrics.
3 BAVO-BENCH: PHYSICAL ACTIVE-VIEW RECOVERY UNDER EXTERNAL OCCLUSION
- BAVO-Bench builds on AV-ALOHA with five multi-stage bimanual tasks where external occlusion is a controlled intervention over otherwise unchanged tasks.Source: §3 · p. 3
3.1 CONTROLLED VISIBILITY INTERVENTIONS
- Clean provides the nominal reference; Stage Occlusion uses predefined task transitions, while Random-time samples interventions independently and may include more events.The figure shows representative rollouts, with red borders marking occlusion events and timelines illustrating recovery and manipulation.Source: §3.1 · p. 4, Figure 2 · p. 4
3.2 DEMONSTRATION AND EVALUATION PROTOCOL
- Operators provide 100 Stage-Occlusion demonstrations per task, chiefly repositioning the observation arm after occlusion; the same Stage-trained policy is evaluated in all three conditions.Source: §3.2 · p. 4
3.3 EVALUATION METRICS
- Success Rate measures full-task completion, while Task Progress Score captures the deepest milestone reached as a fraction of five.Together with CVG, these metrics cover task performance and visibility recovery.Source: §3.3 · p. 5
- CVG compares normalized target visibility along the policy’s camera trajectory with an event-start fixed-camera counterfactual over H steps, using the same scene state for both views.Counterfactual renders are used only for privileged simulation-time evaluation, not policy observations or actions.Source: §3.3 · p. 5
4 A-FAR: ACTIVE FUTURE-AWARE RECOVERY
A-FAR combines robot-base-frame 3D observations with training-time 4D relational supervision to predict future-aware representations for joint manipulation and viewpoint control. Its teacher and future observations are used only during training.
4.1 VIEWPOINT-STABLE 3D OBSERVATION
- RGB-D points are transformed into a shared robot-base frame, keeping static geometry fixed across camera motion while preserving object motion, occlusion, and newly revealed surfaces.The camera-to-base transform composes the observation-arm-to-base transform with the calibrated camera-to-end-effector transform.Source: §4.1 · pp. 5–6
4.2 TRAINING-ONLY 4D RELATIONAL TEACHER
- A frozen D4RT teacher queries aligned points across current and future contexts; pairwise-distance targets transfer current geometry and its evolution, weighting larger temporal changes more.The dynamics-aware weighting emphasizes pairs with stronger relational variation while retaining supervision over the remaining structure.Source: §4.2 · pp. 6–7
4.3 FUTURE-AWARE ACTION POLICY
- Conditioned on current point tokens and robot state, WorldQueryFormer predicts point-aligned future tokens that are gated into the current representation for ManiFlow action generation.The zero-initialized feature-wise residual preserves the original current-observation pathway at training start.Source: §4.3 · p. 7
4.4 TRAINING OBJECTIVE AND INFERENCE
- Current-relation and dynamics-weighted evolution losses train the representation and future-token predictor alongside the action objective; deployment uses only current RGB-D observations and robot state.The action objective is evaluated on the fused representation, while the D4RT teacher and future observations are removed at deployment.Source: §4.4 · p. 7
5 EXPERIMENTS
The experiments evaluate A-FAR on BAVO-Bench across clean, stage-aligned, and unexpected occlusions, measuring task performance and visibility recovery. They find stronger task performance than the ManiFlow backbone, analyze how visibility recovery relates to manipulation, and demonstrate real-world deployment.
5.1 EXPERIMENTAL SETUP
- ACT, ManiFlow, and A-FAR are trained per task with three seeds, while π0.5 is trained once on the mixed five-task dataset; evaluation uses 20 episodes per condition.The comparison covers five tasks and Clean, Stage Occlusion, and Random-time Occlusion; reported metrics include SR and TPS, with CVG under Random-time Occlusion.Source: §5.1 · p. 8
5.2 MAIN RESULTS
- A-FAR achieves the highest macro-averaged success rate and task progress under all three visibility conditions, improving success rate over ManiFlow by 5.6, 3.0, and 3.7 pp under Clean, Stage Occlusion, and Random-time Occlusion.Task progress also improves consistently over ManiFlow.Source: §5.2 · p. 8, Table 1 · p. 8
- Under Random-time Occlusion, A-FAR improves success rate from 15.0% to 18.7% over ManiFlow, with gains across all five tasks; π0.5 falls from 42.0% under Stage to 4.0% under Random-time.A-FAR’s task progress rises from 50.5% to 51.9% over ManiFlow under Random-time Occlusion.Source: §5.2 · p. 8
5.3 ACTIVE-VIEW RECOVERY ANALYSIS
- A-FAR raises average Camera-attributable Visibility Gain from 28.57 pp to 29.64 pp over ManiFlow, while π0.5 has the highest average gain at 34.42 pp.The analysis distinguishes visibility acquisition from downstream task performance.Source: §5.3 · pp. 8–9, Table 2 · p. 8
- A-FAR has the highest average SR–CVG Joint score at 5.78, versus 4.42 for ManiFlow, 2.97 for ACT, and 1.41 for π0.5, and leads on four of five tasks.The authors interpret this as stronger coupling of visibility recovery with subsequent manipulation, despite π0.5’s higher raw CVG.Source: Table 2 · p. 8, §5.3 · p. 9
5.4 ABLATION STUDIES
- Future relational modeling raises three-condition average success rate from 41.0% for ManiFlow to 44.7% for the full model, while moving-camera-frame observations reduce A-FAR’s average from 44.7% to 27.0%.The ablation identifies viewpoint-stable geometry in a shared base frame as a prerequisite for exploiting temporal relational cues.Source: §5.4 · p. 9
5.5 REAL-WORLD FEASIBILITY
- On the physical platform, A-FAR changes the observation camera’s viewpoint after an external occluder blocks task-relevant information, then continues manipulation without a separate perception or view-planning module.The rollout illustrates closed-loop active-view behavior on real hardware.Source: §5.5 · p. 9, Figure 4 · p. 9
6 CONCLUSION
The paper frames physical active vision as closed-loop recovery from visibility failures during manipulation and concludes that visibility recovery alone does not guarantee task success. BAVO-Bench uses five multi-stage bimanual tasks and externally introduced occlusions that leave the underlying task unchanged.
6 CONCLUSION
- Across five bimanual tasks, A-FAR improves over its ManiFlow backbone, including under Random-time Occlusion, and transfers to a real-world active-camera platform; recovered visibility does not necessarily translate into task success.The conclusion identifies effective use of newly acquired observations in subsequent closed-loop control as a remaining challenge.Source: §6 · p. 10
A.1 TASK AND MILESTONE DEFINITIONS
- Each BAVO-Bench task has five ordered milestones, with the fifth defining task success; milestones are determined from privileged simulator states and contact predicates for evaluation only.The five tasks are Cube Handoff, Block Stacking, Drawer Stowing, Slide Retrieval, and Dustpan Sweeping.Source: §A.1 · p. 13
A.2 OCCLUSION PROTOCOL
- Both occlusion conditions keep the manipulation task unchanged, and their fixed-in-world occluders require active camera movement to recover visibility.Stage Occlusion updates targets at two predefined task-stage transitions; Random-time Occlusion samples intervention times independently of those transitions and may interrupt ongoing behavior.Source: §A.2 · p. 13
- Occluders are placed between the active camera and the task-relevant target, accepted when they substantially suppress target visibility while preserving robot clearance, and held fixed until the next intervention.Source: §A.2 · p. 13
A.3 METRIC IMPLEMENTATION DETAILS
The metric implementation defines recovery windows for Camera-attributable Visibility Gain and specifies how episodes and events contribute to its reported average.
A.3 METRIC IMPLEMENTATION DETAILS
- Camera-attributable Visibility Gain uses a 60-control-step recovery horizon, equivalent to 2.4 seconds at 25 Hz; Random-time interventions are separated by more than 60 steps so recovery windows do not overlap.CVG is reported only under Random-time Occlusion.Source: §A.3 · p. 13
- Only events with a complete recovery window are retained, and episodes ending before that window closes are excluded rather than padded.For episodes with multiple retained events, event-level values are averaged within each episode before averaging across episodes.Source: §A.3 · p. 13
- If the target is not visible in the no-occluder rendering, its normalized visibility is set to zero; counterfactual views change only evaluation camera pose and occluder presence, not policy observations, actions, or environment dynamics.Source: §A.3 · pp. 13–14
A.4 BENCHMARK ROLLOUT VISUALIZATIONS
The rollout visualizations show representative sequences for all five benchmark tasks under Clean, Stage Occlusion, and Random-time Occlusion. They pair third-person and active-camera views over time.
A.4 BENCHMARK ROLLOUT VISUALIZATIONS
- Figures 5–7 show representative rollout sequences for five tasks, with the third-person scene view above the active-camera view and frames ordered left to right.The task order is Cube Handoff, Block Stacking, Drawer Stowing, Slide Retrieval, and Dustpan Sweeping.Source: §A.4 · p. 14, Figure 5 · p. 15
- Stage Occlusion sequences show visibility disruptions introduced at predefined task transitions, whereas Random-time Occlusion sequences show disruptions that may occur during ongoing manipulation.Source: Figure 6 · p. 16, Figure 7 · p. 17
B A-FAR AND IMPLEMENTATION DETAILS
The implementation precomputes current and future relational targets from a frozen OpenD4RT teacher, then combines point-cloud and robot-state features with a future-query transformer. The additional-results section specifies that its task-level tables report SR and TPS across five tasks and three visibility conditions.
B.1 RELATIONAL DISTILLATION DETAILS
- Current relations and relational evolution are extracted from the teacher’s final normalized decoder representation and cached for policy training.The teacher uses 32-frame RGB clips, with frame 16 as the anchor and frame 31 as the future target.Source: §B.1 · p. 18
- Dynamics-aware weighting applies only to relational evolution; current relations use uniform weighting, and both terms contribute equally with λrel = 0.01.The weighting retains supervision for relations with weaker temporal variation.Source: §B.1 · p. 18
B.2 ARCHITECTURE AND TRAINING CONFIGURATION
- Each observation uses 256 colored 3D points, encoded into 128-dimensional point features, while the 21-dimensional robot state becomes a 64-dimensional representation.WorldQueryFormer has two transformer blocks with four attention heads each; robot state initializes future queries, which attend to current point features.Source: §B.2 · p. 18
C ADDITIONAL EXPERIMENTAL RESULTS
- The additional-results section reports task-level SR and TPS for ACT, π0.5, ManiFlow, and A-FAR across five tasks and Clean, Stage Occlusion, and Random-time Occlusion conditions.π0.5 uses the monocular configuration from the main experiments; CVG and joint SR–CVG scores are reported separately in Table 2.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 18
C.1 TASK-LEVEL SUCCESS RATE
The task-level success-rate results show that A-FAR improves over ManiFlow under Random-time Occlusion across all five tasks, with gains especially visible on Cube, Slide, and Sweep. Stack remains challenging for all evaluated methods.
C.1 TASK-LEVEL SUCCESS RATE
- A-FAR improves over ManiFlow on all five tasks under Random-time Occlusion, with gains especially visible on Cube, Slide, and Sweep.Stack remains challenging for all evaluated methods.Source: §C.1 · pp. 18–19, Table 7 · p. 19
C.2 TASK-LEVEL PROGRESS
Under Random-time Occlusion, A-FAR improves TPS over ManiFlow on four of five tasks, while the methods remain close on Stack. The real-world platform pairs a CamBot camera arm with a Piper manipulation arm, oriented parallel along a table edge.
C.2 TASK-LEVEL PROGRESS
- Under Random-time Occlusion, A-FAR improves TPS over ManiFlow on four of five tasks, while the methods remain close on Stack.Together with the SR results, the gains are distributed across multiple manipulation tasks rather than dominated by one task.Source: §C.2 · p. 19, Table 8 · p. 20
D REAL-WORLD ACTIVE-VISION PLATFORM
- The real-world setup uses a CamBot equipped with an Intel Realsense D435i camera as the observation arm and a Piper arm for manipulation.The arms are mounted along the table edge, parallel to each other; at home position, the gripper and camera point toward the workspace in the same direction.Source: D REAL-WORLD ACTIVE-VISION PLATFORM · p. 20, Figure 8 · p. 20