PaperlayerView exact revision on arXiv

Source-linked AI summary

VP-EVT: Active Visibility Preservation for Embodied Visual Tracking

Yanci Wen, Tianle Zeng, Jianwei Peng, Weixi Situ, Hanjing Ye, Hong Zhang

arXiv:2610.21999v1cs.RO

TL;DR

Persistent pedestrian occlusion can leave EVT agents following without addressing blocked visibility. VP-EVT estimates target–pedestrian motion and predicts whether following will sustain obstruction, then selectively adjusts viewpoint; on the Risk split, it improves success rate and reduces visibility deficit and target-loss duration relative to a competitive baseline.

  • Problem

    Existing EVT methods may treat target visibility as a by-product of following, leaving the agent without an explicit decision about whether following will sustain pedestrian occlusion.

    Source: §I · p. 1

  • Method

    VP-EVT combines language-guided target and pedestrian observations with ego-motion-based temporal beliefs and waypoint-conditioned occlusion prediction to selectively adjust viewpoint and resume stable following.

    Source: §I · p. 1, B. Target–Pedestrian Temporal Belief · p. 3

  • Results

    121.4% relative success-rate improvement, with visibility deficit reduced by 78.8% and longest target-loss duration by 74.4% on the Risk split versus a competitive baseline.

    Source: B. Main Closed-Loop Results · p. 6

  • Takeaways & Limitations

    OccEVT-Bench evaluates closed-loop persistent occlusion, and OccEVT-Expert supplies demonstrations for training proactive visibility-preserving policies.

    Source: §I · p. 2

Abstract

from arXiv · show

Embodied Visual Tracking (EVT) is an essential capability for intelligent robots in human-robot collaboration, requiring an agent to follow a designated target person in dynamic environments relying solely on egocentric visual perception. However, persistent pedestrian occlusions in dynamic scenes pose critical challenges to existing EVT methods. Lacking proactive visibility preservation, reactive tracking methods frequently experience prolonged target loss under severe occlusions. Therefore, we propose VP-EVT, a framework that treats target visibility as an explicit objective of motion control rather than a by-product of following. Acting on this objective requires understanding both what obstructs the target and how robot motion can change that obstruction. VP-EVT therefore uses language-guided perception to extract separate target and pedestrian observations. When occlusion makes current observations incomplete, VP-EVT combines visual history with ego-motion to maintain estimates of their relative positions and motion. Using these estimates and the current following waypoints, VP-EVT predicts whether continued following will sustain the obstruction. When the predicted risk is low, the original following motion is retained; otherwise, a recurrent controller combines updated estimates with maneuver context from preceding decisions to coordinate viewpoint adjustment and the return to stable following. Moreover, we introduce OccEVT-Bench for closed-loop evaluation of persistent occlusion and collect expert demonstrations for training. On its Risk split, VP-EVT improves success rate by 121.4% relative to a competitive baseline and reduces visibility deficit and longest target-loss duration by 78.8% and 74.4%, respectively.

I. INTRODUCTION

EVT can lose a target for prolonged periods when following continues behind an occluding pedestrian; target estimation and recovery alone do not decide whether following motion will sustain the blockage. VP-EVT makes visibility an explicit motion objective and evaluates whether to adjust the viewpoint while continuing to follow.

I. INTRODUCTION

  • Persistent pedestrian occlusion can prolong target loss even when a robot maintains following distance and avoids collisions.The challenge is to anticipate whether current following will sustain the obstruction and adjust viewpoint while following.Source: §I · p. 1
  • Target estimation and recovery preserve information or respond to degraded visibility, but do not decide whether current following will sustain an obstruction.The proposed missing decision is whether to change the following motion to prevent avoidable persistent occlusion.Source: §I · p. 1
  • VP-EVT uses separate language-conditioned target and pedestrian observations, temporal beliefs integrating observations with ego-motion, and waypoint-conditioned occlusion prediction for selective visibility-preserving motion.It adjusts motion when intervention is needed and otherwise aims to restore stable following.Source: §I · p. 1
  • OccEVT-Bench measures visibility loss alongside tracking safety, and OccEVT-Expert provides 20,000 dynamic-occlusion episodes for training.The benchmark is intended to assess persistent occlusion in closed-loop tracking.Source: §I · p. 2

A. Embodied Visual Tracking

  • VP-EVT takes a relational approach to predicting persistent obstruction from target beliefs, pedestrian estimates, and current following waypoints.It coordinates selective visibility-preserving motion with a return to stable following.Source: A. Embodied Visual Tracking · p. 2

B. Visual Tracking Benchmarks

Existing visual tracking benchmarks characterize perception robustness, but pre-recorded evaluation cannot capture how an agent changes subsequent viewpoints. OccEVT-Bench evaluates closed-loop occlusion handling, while VP-EVT seeks to minimize visibility deficit subject to following and collision constraints.

B. Visual Tracking Benchmarks

  • Pre-recorded tracking evaluations do not let agents alter subsequent viewpoints, while OccEVT-Bench measures persistent target-loss and visibility degradation alongside recovery and collision safety.Matched Safe episodes assess whether occlusion handling disrupts otherwise valid following.Source: B. Visual Tracking Benchmarks · p. 2

III. METHODOLOGY

  • The visibility-preserving EVT objective is to minimize cumulative visibility deficit while avoiding collisions and maintaining or restoring prescribed following distance and target-relative pose.Source: §III · p. 2
  • VP-EVT has three modules: target-conditioned visual grounding, target–pedestrian temporal belief, and plan-conditioned risk and recurrent control.Source: §III · p. 2
  • A causal RGB history, target description, and ego-motion inform the system; a frozen follower supplies robot-local following waypoints.The waypoints are denoted Wt.Source: §III · p. 2
  • Target and pedestrian observations and prior estimates update persistent target and short-term pedestrian beliefs; waypoint-conditioned risk prediction triggers recurrent viewpoint adjustment when needed.Otherwise, standard following is retained, and updated observations and ego-motion feed back into belief estimation.Source: §III · p. 2

A. Target-Conditioned Visual Grounding

Target-conditioned visual grounding uses a language prompt and causal visual history to derive separate evidence for the target and potential occluding pedestrians. The resulting observations support robot-local spatial estimates for later belief updates and control.

A. Target-Conditioned Visual Grounding

  • A frozen visual encoder processes a causal five-frame RGB history, while a language-conditioned relation encoder produces separate target and other-person maps.The maps may overlap because a coarse image region can contain both target and occluding pedestrian during partial occlusion.Source: A. Target-Conditioned Visual Grounding · pp. 2–3
  • Image-space maps weight relation tokens into separate target and pedestrian features, with centroid, spread, and confidence statistics retained.Source: A. Target-Conditioned Visual Grounding · p. 3
  • Temporal observation heads combine visual-history features with robot motion to estimate target and pedestrian locations relative to the robot.The observations connect image-space grounding to robot-local spatial estimation.Source: A. Target-Conditioned Visual Grounding · p. 3
  • Each observation head predicts a joint distribution over discretized bearing and distance, learned from robot-relative position supervision rather than geometric projection of image heatmaps using depth.Source: A. Target-Conditioned Visual Grounding · p. 3

B. Target–Pedestrian Temporal Belief

VP-EVT maintains robot-local temporal beliefs for the target and potential occluding pedestrian, preserving spatial estimates when visual evidence is unavailable. Target motion is propagated and corrected when observations return, while pedestrian estimates prioritize available observations and bridge temporary gaps.

B. Target–Pedestrian Temporal Belief

  • Target and pedestrian positions and velocities are represented as Gaussian beliefs, with the mean as estimated state and covariance as uncertainty.The target belief is propagated using velocity and transformed into the current robot frame using ego-motion, then corrected by reliable visual observations.Source: B. Target–Pedestrian Temporal Belief · p. 3
  • When target evidence is unavailable, its belief continues propagating with increasing uncertainty until observations resume.Visual examples illustrate how reduced evidence affects spatial estimates and uncertainty.Source: B. Target–Pedestrian Temporal Belief · pp. 3–4
  • The pedestrian belief prioritizes available observations and uses temporal propagation to bridge temporary gaps without unnecessarily smoothing the occluder's motion.Source: B. Target–Pedestrian Temporal Belief · p. 4
  • Filtered position distributions become coarse robot-centric belief maps on a polar grid discretized by bearing and distance.Spatial weights encode plausible locations; confident estimates concentrate them, while uncertainty spreads them across cells.Source: B. Target–Pedestrian Temporal Belief · p. 4
  • Belief tokens combine coarse maps with position, velocity, uncertainty, and visibility or observation availability for subsequent risk prediction and control.Source: B. Target–Pedestrian Temporal Belief · p. 4

C. Plan-Conditioned Risk and Recurrent Control

VP-EVT predicts persistent occlusion from target and pedestrian beliefs, following waypoints, and robot-motion history, then selectively intervenes with recurrent control and resumes standard following when checks pass.

C. Plan-Conditioned Risk and Recurrent Control

  • Persistent-occlusion risk is predicted from target and pedestrian beliefs, current following waypoints, and recent robot-motion history.The predictor is trained on subsequent physical target-loss duration; temporally confirmed high risk activates visibility-preserving control, while low risk retains standard following.Source: C. Plan-Conditioned Risk and Recurrent… · p. 4
  • When intervention is needed, a recurrent controller combines updated beliefs with maneuver context to generate motion that adjusts the viewpoint and restores following.Its recurrent state persists across successive control chunks, and the complete chunk of four predicted controls is executed before the next decision.Source: C. Plan-Conditioned Risk and Recurrent… · p. 4
  • Control references combine resampled following waypoints with extrapolated target and pedestrian positions, relative displacement, and velocities.The controller predicts control increments subject to velocity and acceleration limits.Source: C. Plan-Conditioned Risk and Recurrent… · p. 4
  • A separate handoff predictor returns the robot to waypoint following only after visibility and following-geometry checks pass across consecutive decisions.The handoff predictor also checks predicted relations along the following waypoints.Source: C. Plan-Conditioned Risk and Recurrent… · p. 4
  • Training uses expert maneuver sequences for the recurrent controller and privileged annotations for supervision, not as policy inputs at inference.The perception and decision modules are fixed during controller training, which combines motion imitation with human-clearance and following-distance penalties.Source: C. Plan-Conditioned Risk and Recurrent… · pp. 4–5

IV. BENCHMARK DESIGN

OccEVT-Bench is a closed-loop benchmark centered on persistent pedestrian occlusion. It evaluates whether an embodied tracker can reduce sustained obstruction through its own motion while maintaining effective following.

IV. BENCHMARK DESIGN

  • OccEVT-Bench evaluates active reduction of sustained pedestrian obstruction through robot motion while maintaining effective following.The benchmark distinguishes active visibility preservation from eventual target recovery.Source: §IV · p. 5
  • The benchmark is built on Habitat-GS and supports 3D Gaussian Splatting scene rendering and custom-scene import.Source: §IV · p. 5
  • Table I summarizes the benchmark’s scope relative to existing benchmarks.Source: §IV · p. 5

A. Episode Design

OccEVT-Bench separates persistent-occlusion Risk episodes from Safe episodes without such occlusion, and evaluates following, visibility loss, and loss attribution alongside collision safety.

A. Episode Design

  • Risk episodes test persistent pedestrian-occlusion handling, while Safe episodes test continued following without unnecessary intervention.Episode labels remain fixed across evaluated policies; the benchmark contains 400 evaluation episodes spanning diverse scenes and pedestrian interactions.Source: A. Episode Design · p. 5

B. Evaluation Metrics

  • Success Rate measures stable following without collisions, while Collision-Free Rate measures episodes without collisions.Risk success requires a stable-following window after the predefined interaction; Safe success requires stable following during the final evaluation window.Source: B. Evaluation Metrics · p. 5
  • Visibility Deficit accumulates visibility degradation, and Longest Target Loss measures the longest continuous period below the visibility threshold.Lower values indicate better visibility preservation and less persistent target loss, respectively.Source: B. Evaluation Metrics · p. 5
  • Loss Attribution separates physical-occlusion loss from out-of-FOV loss, revealing whether reduced in-view occlusion comes with out-of-view failures.The reported metric descriptions distinguish performance measures from dataset statistics or annotations; an unreported criterion does not imply occlusion is absent.Source: B. Evaluation Metrics · p. 5

C. Expert Demonstrations

OccEVT-Expert provides 20,000 dynamic-occlusion training episodes with expert demonstrations and privileged annotations for training proactive visibility-preserving policies.

C. Expert Demonstrations

V. EXPERIMENTS

VP-EVT is evaluated for tracking, visibility preservation, and safety on held-out OccEVT-Bench episodes, with Risk and Safe results reported separately. It improves Risk success and visibility metrics over OA-VAT while retaining similar Safe-episode performance to the base follower, though its Risk collision-free rate remains below OmTrackVLA’s.

A. Experimental Setup

  • The evaluation uses 400 held-out episodes with matched configurations, camera settings, robot dynamics, and termination rules, and reports Risk and Safe separately.Baselines include OA-VAT, Mono-RPF, MiniCPM-RobotTrack, and OmTrackVLA; VP-EVT is trained on OccEVT-Expert with the visual encoder and base follower frozen.Source: A. Experimental Setup · p. 6

B. Main Closed-Loop Results

  • Risk SR is 77.5% for VP-EVT, versus 35.0% for OA-VAT and 1.0% for OmTrackVLA’s following waypoints.The comparison evaluates the proposed visibility-preserving components on top of OmTrackVLA’s waypoints.Source: B. Main Closed-Loop Results · p. 6
  • Relative to OA-VAT, VP-EVT reduces VD by 78.8% to 4.235 s and LTL by 74.4% to 3.442 s.These metrics capture accumulated visibility degradation and longest uninterrupted target loss, respectively.Source: B. Main Closed-Loop Results · p. 6
  • VP-EVT reduces OA-VAT’s POL of 3.755 s and OOF of 16.041 s to 1.715 s and 1.747 s, respectively.The largest absolute reduction is in out-of-FOV loss, while in-view loss also decreases.Source: B. Main Closed-Loop Results · p. 6
  • On Safe episodes, VP-EVT matches OmTrackVLA in SR (90.0%) and CFR (99.0%), with visibility metrics close to its values.The reported comparison supports retaining the base follower’s aggregate performance when persistent occlusion is absent, but does not establish how often unnecessary interventions occur.Source: B. Main Closed-Loop Results · pp. 6–7
  • Fig. 4 compares target visibility and cumulative visibility deficit over selected Risk episodes, with τloss = 0.35 marking the target-loss threshold.Higher visibility and lower cumulative deficit indicate better visibility preservation.Source: Figure 4 · p. 7
  • On Risk episodes, VP-EVT’s CFR is 90.5%, above OA-VAT’s 69.0% but below OmTrackVLA’s 97.5%.Collision-free execution alone does not ensure successful tracking: OmTrackVLA has high CFR but low Risk SR.Source: B. Main Closed-Loop Results · p. 7

C. Ablation Study

The ablations test VP-EVT’s belief representation, temporal evidence, intervention strategy, waypoint conditioning, and maneuver memory on Risk episodes. They support the full model’s combination of persistent state estimation and selective, context-conditioned intervention.

C. Ablation Study

  • The ablation variants are evaluated on 200 Risk episodes, predicting and executing four controls per decision.SR and CFR are fractions, and duration metrics are in seconds.Source: C. Ablation Study · p. 7
  • Removing both target–pedestrian beliefs reduces SR from 77.5% to 11.0%, while POL rises from 1.715 to 12.974 s and LTL from 3.442 to 18.762 s.This joint ablation supports the combined representation’s importance but does not isolate either belief’s contribution.Source: C. Ablation Study · p. 7
  • In the real-world deployment illustration, green dotted paths show visibility-preserving motion around the obstructing pedestrian, while gray dotted paths show the following reference.Frames run chronologically from left to right, then top to bottom.Source: Figure 5 · p. 7
  • Disabling occlusion-time belief propagation reduces SR to 47.5% and CFR to 54.5%; the single-frame variant reaches 43.5% SR, with LTL of 6.611 s and OOF of 5.647 s.Because the single-frame variant also disables belief propagation, it does not isolate visual history alone.Source: C. Ablation Study · p. 7
  • Reactive triggering lowers SR from 77.5% to 68.5% and raises VD from 4.235 to 9.675 s and LTL from 3.442 to 6.246 s.Continuous intervention instead yields 46.5% SR and 52.5% CFR, with OOF increasing from 1.747 to 4.975 s.Source: C. Ablation Study · p. 7
  • Removing waypoint conditioning reduces SR to 42.5% and CFR to 48.5%, while resetting maneuver memory lowers them to 69.5% and 75.5%, respectively.The memory-reset variant worsens all four visibility-loss metrics despite retaining visual history, belief propagation, and the four-control execution horizon; duration metrics should be interpreted alongside SR and CFR because collisions shorten evaluation.Source: C. Ablation Study · pp. 7–8

D. Real-World Deployment

VP-EVT is deployed on a quadruped robot to maneuver around an obstructing pedestrian, then resume following. The conclusion reports improved tracking and visibility outcomes with comparable Safe-episode performance to the base follower.

D. Real-World Deployment

  • On a Unitree Go2, VP-EVT moves laterally around an obstructing pedestrian, reduces obstruction while advancing, then resumes following behind the target.Source: D. Real-World Deployment · p. 8

VI. CONCLUSION

  • Experiments show higher tracking success and reduced visibility degradation and continuous target loss, with comparable Safe-episode performance to the base follower.Source: §VI · p. 8
  • VP-EVT combines language-guided temporal beliefs, waypoint-conditioned occlusion prediction, and selective recurrent control to preserve visibility and restore stable following.Source: §VI · p. 8
  • The conclusion introduces OccEVT-Bench for evaluation and OccEVT-Expert for training, and identifies intervention safety and robustness in denser crowds and diverse real-world settings as future work.Source: §VI · p. 8

Related papers

Loading 2610.21999v1…