PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Current Scene: Event-Referential Grasping with Active View Selection

Hyunjoon Lee, Haebeom Jung, Eunsung Cha, Daeun Lee, Yu-Chiang Frank Wang, Jaesung Choe, Jaesik Park

arXiv:2609.39375v1cs.ROcs.CV

TL;DR

Event-referential requests require identifying a target from past interactions, including when it is no longer visible. BeyondCSe combines video reasoning with event-conditioned view selection; in heavy occlusion, it reaches 95% grasp success with 2.20 views on average, versus 75% and 3.35 views for a baseline given the target’s ground-truth 3D box.

  • Problem

    A target identified by its role in a past event cannot be identified from the current scene alone, and identifying it does not provide the current visual evidence needed to grasp it when occluded.

    Source: §I · pp. 1–2

  • Method

    BeyondCSe uses off-the-shelf models to recover event-based target identity and 3D observations, then combines the event prior with scene geometry for volumetric belief updates and viewpoint search.

    Source: §I · p. 2

  • Results

    95% grasp success with 2.20 views on average, versus 75% success and 3.35 views for Breyer et al. despite its privileged target box.

    Source: 2) Belief Ablations · p. 7

  • Takeaways & Limitations

    Past target observations provide spatial evidence for selecting views when the requested object is hidden from the current view.

    Source: §I · pp. 1–2

Abstract

from arXiv · show

A robot that observes people interacting with objects should be able to carry out later requests that refer back to those interactions. Such requests may specify a grasp target by the role it played in a past event rather than by its name or appearance. Moreover, the target may no longer be visible when the robot is asked to act. We present BeyondSCe, a zero-shot robotic grasping system for this event-referential setting. Given the event history and the current scene, the system identifies the requested object or part and localizes it for grasping. If the target is occluded, it combines an event prior recovered from the history with current scene geometry to select camera viewpoints likely to reveal the target. The system uses pretrained models without additional task-specific training. In real-robot experiments with a single wrist-mounted RGB-D camera, it achieves grasp success rates of 76% and 77% for initially visible and occluded targets, respectively, compared with 40% and 55% for the strongest baseline in each condition. On four additional scenes with heavy occlusion, it increases grasp success rates from 75% to 95% while reducing the mean number of views from 3.35 to 2.20, compared with an active-perception baseline given the target's ground-truth 3D bounding box.

I. INTRODUCTION

Event-referential requests require identifying a target from its role in a past event, and target identification alone does not provide the current visual evidence needed to grasp an occluded target. BeyondCSe links event-history reasoning to active view selection, using historical target observations and current geometry to search for a view that reveals the target.

I. INTRODUCTION

  • Targets specified by past-event roles cannot be identified from the current scene alone, and recognizing a hidden target does not provide the visual evidence needed for grasping.Earlier history-based approaches can recover target identity, but downstream grasping still needs a view that reveals the target.Source: §I · pp. 1–2
  • Its event-conditioned prior uses the target’s past observations to estimate where it went, while view scoring discounts rays through unobserved space.This contrasts with active-perception guidance based only on the current scene, such as geometric information gain or predicted grasp affordance.Source: §I · p. 2
  • BeyondCSe grounds event-referential targets using history and the current scene, then invokes active view selection when initial grounding fails.The system combines historical 3D target observations with current geometry to initialize a volumetric target belief and updates it as new observations arrive.Source: §I · p. 2
  • The contributions include zero-shot object-or-part grounding, probabilistic volumetric belief updates and view selection, and real-robot evaluation with a wrist-mounted RGB-D camera.The pipeline uses an off-the-shelf MLLM and point tracker to recover an event prior; the evaluation examines search cost and grasp success.Source: §I · p. 2

A. Language-Guided Grasping and Grounding

The cited work spans language-guided grounding, history-dependent manipulation, and active view planning. This paper’s setting identifies an object or part through an event history and may require searching when it is indistinguishable or occluded in the initial view.

A. Language-Guided Grasping and Grounding

  • Point-based grounding includes GraspMolmo, which predicts a task-oriented grasp point from a single image, and Point2Act, which distills multiview MLLM point predictions into a 3D relevancy field.LERF-TOGO and GraspSplats ground language queries in 3D representations built from multiple views. VoLo [1] coordinates VLAs, perception models, and action primitives for long-horizon manipulation, including tasks involving memory and complex references.Source: A. Language-Guided Grasping and… · p. 2

B. History-Dependent Manipulation and Episodic Memory

  • History-dependent systems use past observations for policies, keyframe-based instructions, or memory evaluation, whereas this system recovers explicit 3D observations of event-referred targets without task-specific training.Related work also includes recovering past object locations from image queries and retrieving video frames to answer questions.Source: B. History-Dependent Manipulation and… · p. 2

C. Active Perception for Occluded-Target Search

  • Prior active-view methods use geometric information gain, grasp affordance, grasp-success uncertainty, or object–occluder relations; this system instead searches using an instance-specific event-history belief.Its view selection considers visibility of likely target locations and discounts rays through unobserved space.Source: C. Active Perception for… · p. 2

A. Problem Formulation

  • The task uses a wrist-mounted camera’s final event-video frame as the initial RGB-D observation, but the target may be indistinguishable or occluded there.The system can then move the camera to acquire additional observations, with camera intrinsics and camera-to-base transform known at each step.Source: A. Problem Formulation · pp. 2–3

B. System Overview

The system connects video reasoning to active perception through a target description and historical spatial observations. It first tries to ground the target in the initial image, then searches with new views if needed.

B. System Overview

  • If video reasoning yields a valid initial 3D action point, the system sends it to grasp validation; otherwise, it initializes active perception from historical observations and initial scene geometry.The pipeline uses an off-the-shelf MLLM and point tracker without additional task-specific training.Source: B. System Overview · p. 3

1) Selecting the referent:

  • RECORD forms a time-ordered event record; SELECT uses it with the instruction to produce a target description and a distinguishing event cue.Each recorded event describes an action, its object, and another involved object if any; the target goes to POINT and the cue to LOCATE.Source: 1) Selecting the referent · p. 3

2) Grounding in the initial image:

  • LOCATE proposes an initial-image box from the cue, and POINT uses the target and crop to return an action pixel, expanding the region or using the full image if needed.The pixel is mapped to the full initial image and lifted to 3D using valid depth and camera pose.Source: 2) Grounding in the initial image · p. 3, Figure 3 · p. 3

3) Recovering historical locations:

Historical 3D observations initialize an event-conditioned target belief alongside a map of current geometry. Belief-guided views update both until the target is confirmed, after which grasp validation may request refinement.

3) Recovering historical locations:

  • When initial-image pointing fails, the system searches past frames from recent to earlier, rejects tracks still visible at video end, and selects the most recent appearance match.It lifts valid-depth track samples into the robot base frame; the resulting 3D event track initializes the target belief.Source: 3) Recovering historical locations · pp. 3–4

D. Event-Conditioned Active Perception

1) Event-Conditioned Belief Initialization:

  • The initial map is a TSDF from the first RGB-D observation, while the target belief is a probability density over the geometrically admissible workspace.The workspace retains regions unresolved by occlusion or missing depth, and belief initialization does not require a current target mask, box, or center.Source: 1) Event-Conditioned Belief… · p. 4
  • The event prior is centered on the last 3D observation and shaped by the terminal path segment without extrapolating beyond it.The path is parameterized by arc length; the minimum spread is positive, and Ctail is zero for a single observation or zero-length path.Source: 1) Event-Conditioned Belief… · p. 4
  • A uniform component assigns nonzero probability throughout the workspace, allowing recovery when the historical estimate is inaccurate; without valid 3D history, the initial belief is uniform.The Gaussian component is restricted and normalized over the workspace.Source: 1) Event-Conditioned Belief… · p. 4

2) Belief-Guided View Search:

Belief-guided search evaluates feasible camera views by how likely a negative observation is to reduce belief in possible target locations. It executes the highest-scoring view with a valid motion plan.

2) Belief-Guided View Search:

  • For each feasible view, the system renders a negative-observation likelihood at each target-location hypothesis, using depth consistency and target-miss factors.Candidate views are generated around high-probability belief regions and filtered for kinematic, self-collision, and scene-collision feasibility.Source: 2) Belief-Guided View Search · p. 4
  • Ray visibility confidence is attenuated with distance through unobserved space, while a nonzero transmittance floor prevents such views from being treated as impossible.The attenuation discounts views whose apparent informativeness depends on unobserved space.Source: 2) Belief-Guided View Search · p. 4
  • Each view is scored by the target-belief mass that a negative observation is expected to downweight, and the highest-scoring view admitting a valid motion plan is executed.The event-conditioned prior and current scene geometry inform view selection.Source: 2) Belief-Guided View Search · pp. 4–5

3) Map and Belief Update:

After each view, the system integrates the new depth and updates the target-location belief. Negative observations revise that belief without ruling out locations, and search repeats until a target is found or no feasible view remains.

3) Map and Belief Update:

  • After a view, the system integrates its depth into the map and updates the same belief used to score views, combining depth-consistency and target-miss likelihoods when POINT returns NOT-VISIBLE.The depth likelihood remains neutral where the observation supplies no valid geometric evidence.Source: 3) Map and Belief Update · p. 5
  • Both likelihoods remain in (0, 1], so negative observations downweight target locations rather than ruling them out.The target-miss likelihood combines predicted visibility with the probability that POINT misses a visible target.Source: 3) Map and Belief Update · p. 5
  • POINT queries the target description at each belief-guided view; a lifted pixel ends search, otherwise the posterior is rescored until no feasible view or the view budget remains.Source: 3) Map and Belief Update · p. 5

4) Target Confirmation and View Refinement:

Once the target is confirmed, the system generates and validates grasp candidates. Incomplete target geometry triggers a bounded cycle of target-centered viewing, map integration, and grasp regeneration, ending with a grasp or abstention.

4) Target Confirmation and View Refinement:

  • Following Breyer et al., incomplete target geometry triggers a feasible target-centered view, map integration, and grasp regeneration; here the target region is available only after event-conditioned search confirms it.Source: 4) Target Confirmation and View… · p. 5
  • The bounded loop terminates with an executable grasp or abstains if none is found within the active-view budget.Grasp candidates are validated for target association and motion feasibility.Source: 4) Target Confirmation and View… · p. 5

IV. EXPERIMENTS

The experiments use a wrist-mounted RGB-D camera and tabletop episodes that pair interaction histories with initial observations and event-referential instructions. They evaluate visible and occluded targets, with additional heavily occluded scenes for active-perception comparisons.

1) Hardware Setup:

  • All methods use the same OMY-F3M robot, wrist-mounted Intel RealSense D435i RGB-D camera, calibration, joint limits, collision checks, and grasp generation and execution.Perception models run on a workstation with one NVIDIA GeForce RTX 4090 GPU.Source: 1) Hardware Setup · p. 5

2) Dataset:

  • Episodes pair an interaction video and initial wrist-camera RGB-D observation with instructions referring to event order or object parts, under visible and occluded target conditions.Tabletop scenes use common objects and distractors; occluders hide queried targets from the initial view in the occluded condition.Source: 2) Dataset · p. 5
  • The evaluation includes 10 visible scene–query pairs across four scenes and 20 occluded pairs across 10 scenes, with five trials per pair for 50 visible and 100 occluded trials.Source: 2) Dataset · pp. 5–6
  • Four additional heavily occluded scenes test active perception; every queried target begins out of sight inside a basket or behind objects and remains hidden from a fixed bird’s-eye view.Methods within each comparison use the same scenes and queries.Source: 2) Dataset · p. 6

3) Implementation Details:

The evaluation defines stage-specific success metrics and compares the method with zero-shot grasping baselines under visible and occluded conditions. It also describes the active-perception evaluation setup and implementation details.

3) Implementation Details:

  • Table II evaluates four heavily occluded scenes (S1–S4), with view counts averaged over trials including failures and grasp success aggregated across all 20 trials.AnyGrasp candidates from virtual views are pooled, then non-maximum suppression removes duplicate poses.Source: Table II · p. 6, 3) Implementation Details · p. 6

4) Metrics:

  • Localization succeeds when the predicted 3D point falls within the annotated target box, while planning requires a feasible grasp plan and grasp success requires grasping the instructed target.The three metrics represent successive stages: localization enables planning, and a feasible plan enables execution.Source: 4) Metrics · p. 6

B. Comparison with Zero-Shot Grasping Methods

  • Baselines are evaluated with original and reasoned instructions; multiview methods receive 30 predefined observations, while GraspMolmo uses only the initial RGB-D view and is not evaluated for occluded targets.The method starts from the same initial scene and selects additional views as needed.Source: B. Comparison with Zero-Shot Grasping… · pp. 6–7
  • Reasoned instructions improve localization and grasping success for every evaluated baseline, although none receives event history.With original instructions, Point2Act and GraspMolmo achieve higher visible-target localization success than LERF-TOGO and GraspSplats.Source: B. Comparison with Zero-Shot Grasping… · p. 7
  • Across all 150 trials, grasp success is 76.7% for our method versus 50.0% for the strongest baseline using reasoned instructions.For visible targets, grasping succeeds in 38/50 trials versus 20/50 for the strongest baseline; for occluded targets, it succeeds in 77/100 versus 55/100.Source: B. Comparison with Zero-Shot Grasping… · p. 7

EPIC-KITCHENS

The paper describes video-reasoning examples from EPIC-KITCHENS and EgoDex, and details an active-perception comparison on heavily occluded scenes. The Breyer et al. baseline receives a ground-truth target box and uses the shared grasping stack.

EPIC-KITCHENS

  • The EPIC-KITCHENS and EgoDex examples pair sampled event histories and instructions with pipeline excerpts, including proposal crops and final-frame regions with predicted points.Source: Figure 8 · p. 7

C. Active Perception and Belief Ablations

1) Breyer et al.:

  • Breyer et al. receives the target’s ground-truth 3D bounding box and retains its TSDF ray-casting objective, with VGN replaced by the AnyGrasp generator and planning stack shared with the proposed method.All methods stop at the first feasible target grasp; target association uses the provided box for the baseline and the confirmed target region for the proposed method and its ablations.Source: 1) Breyer et al · p. 7

2) Belief Ablations:

On heavily occluded scenes, the full method improves grasp success and reduces views relative to the active-perception baseline, while belief ablations show increased search costs. Egocentric video examples extend the reasoning pipeline beyond the robot scenes, and the formulation assumes targets remain stationary during search.

2) Belief Ablations:

  • 95% grasp success with 2.20 views on average compares with 75% success and 3.35 views for Breyer et al., despite its privileged target box.The baseline box removes localization ambiguity but neither reveals occluded target geometry nor guarantees grasp success.Source: 2) Belief Ablations · p. 7
  • Without belief weights, mean views rise to 4.65 and grasp success falls to 80%; without transmittance, mean views rise to 3.10, and without the event prior, views rise from 2.20 to 4.00 while grasp success matches the full method.The belief ablations retain candidate views, feasibility checks, target confirmation, and the grasp pipeline; they change view weighting, transmittance, or the initial event-derived belief.Source: 2) Belief Ablations · pp. 7–8

D. Qualitative Video Reasoning on Egocentric Videos

  • In selected EgoDex and EPIC-KITCHENS clips, the pipeline selects targets referenced by plate-placement or washing order and produces final-frame points, though final-frame region estimation can remain imprecise.In the EPIC-KITCHENS example, LOCATE proposes an inaccurate region after large camera viewpoint changes, but POINT identifies the carrot within the expanded crop.Source: D. Qualitative Video Reasoning on… · p. 8

V. CONCLUSION

  • The formulation assumes the target remains stationary during search; handling interactions in which objects continue moving is left for future work.The conclusion reports that heavy-occlusion experiments reduce observations while achieving higher grasp success than the active-perception baseline.Source: §V · p. 8
Loading 2609.39375v1…