Source-linked AI summary
Beyond Object Selection:Markerless Gaze-based Robot Placement at Arbitrary Position
Yuzhi Lai, William Marx, Shenghai Yuan, Peizheng Li, Zhuoyu Ran, Andreas Zell
TL;DR
Gaze-based assistive manipulation must move beyond object selection to arbitrary-position placement, but pose accuracy alone may not predict the resulting gaze target error. The paper presents a markerless benchmark framework, dataset, reference-selection strategy, and task-oriented metric, finding that conventional pose rankings do not always match GSIE performance.
Problem
Arbitrary-position gaze placement requires headset–robot alignment, yet translation and rotation errors can jointly alter the gaze ray and make pose metrics insufficient for task accuracy.
Method
The paper presents a markerless interaction framework with Graph-based Reference Selection, a dedicated cross-device dataset, and benchmarked alignment pipelines evaluated with GSIE.
Results
Alignment methods ranked highly by conventional pose metrics are not always optimal in GSIE, which directly measures gaze–surface intersection displacement.
Takeaways & Limitations
Gaze-based manipulation should be evaluated at the task level because pose metrics do not fully characterize downstream placement error.
Abstract
from arXiv · showhide
Gaze-based assistive manipulation typically supports object selection, while arbitrary-position placement requires accurate spatial alignment between the headset and robot. However, for gaze-based manipulation, pose accuracy does not necessarily translate into task accuracy: translational and rotational errors jointly affect the transformed gaze ray and may compensate for each other. To study cross-device alignment from this task-oriented perspective, we present a markerless interaction framework and a dedicated cross-device dataset. We propose Graph-based Reference Selection to address sparse robot references. We further develop and benchmark multiple task-specific alignment pipelines under a unified protocol. Specifically, we introduce Gaze--Surface Intersection Error (GSIE), which directly measures the spatial error of the gaze-specified target. Experiments show that alignment methods ranked highly by conventional pose metrics are not always optimal in GSIE, demonstrating the importance of evaluating gaze-based manipulation at the task level.
I. INTRODUCTION
Gaze-based assistive manipulation must extend beyond object selection to arbitrary-position placement, which requires reliable headset–robot alignment. This work introduces a markerless, task-oriented benchmark framework, Graph-based Reference Selection, a cross-device dataset, and GSIE for evaluating placement accuracy.
- Motivation: Arbitrary-position gaze interaction requires transferring a gaze-specified target from the user’s egocentric view into the robot coordinate frame.Object selection alone does not require recovering arbitrary workspace positions.
- Limitations of Existing Systems: Existing systems commonly restrict gaze to object selection or predefined targets, while fiducial markers require persistent visibility and remain vulnerable to occlusion.These constraints limit free placement and user range of motion.
- Evaluation Gap: Translation and rotation errors can compensate for or amplify each other, so pose accuracy alone does not fully characterize gaze-based placement accuracy.Sparse robot references and large viewpoint changes also make cross-device feature matching difficult.
- Approach: The proposed markerless framework combines graph-based reference selection with feature-based geometric pose estimation using semantic and spatial relationships.The approach selects a compatible robot view before geometric alignment.
- Benchmark Contributions: The benchmark includes a dedicated robot–headset dataset and GSIE, which measures spatial deviation at the gaze–surface intersection.Together they support systematic evaluation from both pose- and task-level perspectives.
II. RELATED WORKS
Prior gaze systems support selecting known objects but not arbitrary free-space placement, while cross-device alignment methods face viewpoint, occlusion, and workspace constraints. The proposed framework addresses this setting through robot reference scanning, 3D scene reconstruction, and headset–robot transformation estimation.
- Gaze-Based Manipulation: Existing gaze systems associate attention with known objects or predefined targets, but cannot directly specify arbitrary free-space placement.Feature matching degrades under large viewpoint changes, and fiducial markers can be occluded.
- Cross-View Alignment: SLAM-style continuous scanning is poorly suited to manipulator-mounted cameras because robot geometry, limited workspace, and self-occlusion cause appearance changes and unstable tracking.These constraints motivate sparse multi-view reference acquisition.
- Framework Setup: The framework records robot reference images and headset egocentric observations, including intrinsics, gaze targets, and known robot-camera extrinsics.Its objective is to estimate the robot-base-to-headset-camera transformation and the gaze target in robot coordinates.
B. Interaction Framework
The framework selects compatible robot reference views using semantic and spatial relationships, then supports gaze interaction through a reconstructed 3D scene aligned to the robot frame.
- Scene Acquisition: The scene acquisition stage reconstructs a 3D scene and aligns it to the robot reference frame for task-level evaluation.The reconstruction is fused in the robot base frame after SIM(3) alignment and ICP refinement.
- Gaze-Based Interaction: During interaction, gaze samples within the temporal interval of a deictic speech command are aggregated in the headset coordinate frame.The headset records egocentric RGB images, eye-tracking signals, and speech commands.
- Graph-Based Reference Selection: Graph-based Reference Selection uses semantic classes and angular relationships between neighboring objects to compare headset and robot image graphs.Each object forms a graph node, while edges encode relative image-plane angles and descriptors count neighboring object classes across angular bins.
- Graph-Based Reference Selection: The graph descriptors are compared with matrix cosine similarity, and globally optimal one-to-one object matching is computed with the Hungarian Algorithm.The compatibility score incorporates the number of detected objects in the headset image.
- Reference Selection: The robot image with the highest compatibility score is selected as the reference for subsequent feature matching and pose estimation.The selected view provides the most compatible spatial configuration for deterministic cross-device correspondence.
- Pose Refinement: Object-level ellipsoid refinement is illustrated as improving alignment between projected 3D ellipsoids and observed 2D ellipses after pose optimization.This refinement is part of the broader alignment pipeline following reference selection.
D. Object-Level Refinement via Enclosing Ellipsoid
The object-level refinement pipeline represents 3D objects with minimum-volume enclosing ellipsoids, matches them to 2D image ellipses, and refines the initial pose estimate.
- Motivation: The method introduces markerless object-level refinement to align 3D object ellipsoids with their 2D observations when PnP remains inaccurate.This addresses sparse or unstable feature correspondences after reference selection.
- 3D Object Representation: Each object is represented by a minimum-volume enclosing ellipsoid whose center and positive-definite matrix determine its geometry.The matrix encodes orientation and semi-axis lengths.
- 3D Object Representation: MVEE minimizes enclosing volume while enforcing point-wise containment, unlike PCA-based fitting, which does not guarantee that all observed points are enclosed.The ellipsoid is obtained using the Khachiyan algorithm.
- 2D–3D Matching: Object masks in the headset image are fitted with 2D ellipses, and the initial pose projects the 3D quadrics into the headset image.The projected and observed ellipses are represented as Gaussian distributions for matching.
- Pose Refinement: Matched projected ellipses and observed ellipses are assigned by minimizing squared Wasserstein distance, followed by pose refinement.The globally optimal one-to-one assignment is obtained from the graph matching stage.
- Outcome: The refinement introduces object-level geometric constraints and corrects the initial point-based pose estimate.The pipeline also includes a PnP+Line alignment stage that extracts and matches structural lines across robot and headset views.
E. Perspective-n-Point-Line
The PnP+Line pipeline augments point-based pose estimation with 3D–2D line correspondences and jointly optimizes point and line reprojection errors. Line extraction and matching use normalized image coordinates, geometric filtering, and mutual-best constraints.
- 3D–2D line correspondences extend the initial PnP estimate for scenes with weak textures but structurally salient regions.
- Line segments are extracted with LSD, normalized by image diagonal, and filtered or merged using angular, midpoint, and endpoint distances.
- Candidate 3D lines are formed from points satisfying a point-to-line distance threshold, with a unit direction vector defining each candidate line.
- A line pair is retained when matched point pairs lie near both lines and its similarity exceeds τL while remaining the distinctive mutual-best match.
- The initial pose is refined by jointly minimizing point reprojection and projected-line errors, with λ balancing the two constraints.
IV. CROSS-DEVICE DATASET
The authors construct a cross-device dataset with robot-mounted RGB-D reference views and egocentric headset queries across varied tabletop scenes. Ground-truth poses are obtained using ArUco markers and validated against robot kinematics.
- A Franka Emika Panda carries a calibrated RGB-D camera that captures 12 tabletop reference views at approximately 30° intervals.
- Meta Aria glasses record egocentric query images from different viewpoints, while ArUco markers provide ground-truth headset poses under occlusion.
- The dataset contains 3 tabletop-scene categories with 10 random-object sequences per category.
- Each sequence includes 12 calibrated robot RGB-D images, more than 40 headset RGB frames, camera intrinsics, and ground-truth cross-device poses.
V. EXPERIMENTS
The experiments evaluate cross-device alignment with pose-level metrics and the task-oriented GSIE metric. The setup combines calibrated transformations, reconstructed-scene gaze targets, reference-selection evaluation, and controlled acquisition conditions.
- ATE measures translational transformation error, while ARE measures rotational transformation error between estimated and ground-truth cross-device poses.
- GSIE compares gaze targets obtained from estimated and ground-truth poses after transforming a headset-image gaze ray into the robot frame.
- GSIE directly measures the spatial displacement of the gaze target caused by cross-device alignment error.
- Reference-selection evaluation reports reprojection error and selection accuracy, with the ground-truth reference defined by the smallest ground-truth viewing-angle difference.
- Robot reference images are sampled at approximately 30° viewpoint intervals, while intermediate frames serve as headset queries in an asymmetric cropped-view setup.
A. Benchmark of Cross-Device Alignment
The benchmark compares alignment pipelines across point, line, and object-level methods and exposes a mismatch between pose metrics and gaze-placement accuracy. Ellipsoid refinement achieves the best GSIE consistently despite not always minimizing ATE or ARE.
- Benchmark scope: Four cross-device alignment pipelines are benchmarked under a unified protocol designed for gaze-based manipulation and downstream placement evaluation.
- Point-based methods: Point-based PnP pipelines are relatively stable, but their performance depends on the quality and spatial distribution of cross-view correspondences.
- Point-based methods: SuperPoint+LightGlue is used as the default point-based pipeline because of its substantially lower runtime, despite scene-dependent performance differences.
- Line-based refinement: Additional line constraints do not consistently improve pose estimation: PnP+Line performs well in SC but degrades in SB, fr2/dishes, and fr3/long.
- Task-oriented evaluation: Ellipsoid refinement achieves the lowest GSIE in every scene, although it does not consistently achieve the lowest ATE or ARE.
- Task-oriented evaluation: ATE and ARE ignore error direction relative to the gaze ray and scene geometry, whereas GSIE captures translation–rotation coupling directly in task space.
B. Performance of Graph-based Reference Selection
Graph-based Reference Selection selects robot views using semantic and spatial compatibility rather than appearance similarity alone. It achieves consistently high selection accuracy while maintaining low reprojection error and approximately 15 ms runtime.
- SAcc. reaches 0.95–1.00 across all scenes, with the lowest RE and approximately 15 ms runtime.The method combines efficient reference selection with geometrically reliable matches.
- HF-Net and MegaLoc often retrieve globally similar images whose viewing directions are incompatible with the headset query.Appearance similarity does not reliably identify the robot reference that best supports cross-device alignment.
- Exhaustive matching can favor feature-rich background regions, producing many PnP inliers without aligning the relevant foreground objects.This failure is especially evident in SB and fr2/dishes, where target objects have few detectable features.
- The semantic-spatial graph emphasizes object configuration and viewing-direction compatibility without exhaustively matching every candidate.
C. Baseline Comparison
The baseline comparison evaluates real-world gaze placement against marker-, graph-, feature-, and label-based alternatives. The methods expose trade-offs among marker visibility, cross-view matching robustness, placement stability, and runtime.
- Placement Success Rate counts trials successful when the object center lies within the 5 cm target region, while Selection Acc. measures correct object identification.An operator recorded 100 independent gaze–speech commands in each scene, and target regions were masked for all methods.
- ARmarker offers ground-truth alignment when detected, but occlusion and missed detections restrict visibility and usable motion range.
- GOReloc uses coarse 2D-object-center to 3D-centroid correspondences, resulting in relatively low physical placement success rates.
- FAM-HRI becomes unreliable under large viewpoint changes because sparse or incorrect local matches destabilize the inferred target position.Relaxing its matching threshold increases coverage but also increases incorrect associations.
- Complete pipelines operate at approximately 22 FPS, whereas LoMa requires approximately 126 ms per image pair even with ONNX acceleration.Graph-based reference selection adds only about 3 ms, while SuperPoint+LightGlue accounts for most computation.
E. Ablation Study
The ablations show that ellipsoid refinement and semantic-angular graph cues materially affect alignment and reference selection. MVEE gives the strongest reported alignment refinement, while conventional pose metrics do not fully capture gaze-placement error.
- Alignment refinement: MVEE reduces ATE from 31.2 mm to 26.7 mm and GSIE from 18.5 mm to 16.4 mm over the PnP baseline.It achieves the lowest reported values across all alignment metrics.
- Alignment refinement: ORB slightly improves ATE but substantially degrades ARE and GSIE, while SIFT+LightGlue underperforms the default feature pipeline.
- Graph descriptor: Removing semantic or angular information substantially degrades reference selection, and distance-bin replacements remain considerably worse than the full descriptor.
- Study scope: Semantic structural reasoning and feature-based geometric alignment are benchmarked under a unified protocol for arbitrary-position gaze placement.
- Task-level evaluation: GSIE measures gaze–surface intersection deviation because ATE and ARE do not fully characterize downstream placement error.