PaperlayerView exact revision on arXiv

Source-linked AI summary

SPOT: Spatial Perception-Oriented Long-Horizon Humanoid Teleoperation

Lixing Fang, Ziyan Xiong, Sunli Chen, Zhiyang Dou, Chuang Gan

arXiv:2609.07933v1cs.RO

TL;DR

Long-horizon humanoid teleoperation requires sustained task-relevant spatial awareness, which existing interfaces can limit. SPOT provides a wide, stable, actively inspectable robot-centric view with visual exploration decoupled from robot actuation. Across perception-critical tasks, it improves efficiency and accuracy and reduces search and recovery time.

  • Problem

    Long-horizon loco-manipulation requires sustained task-relevant spatial awareness, but existing egocentric interfaces can have narrow views, unstable camera views, and looking motions coupled to robot actuation.

    Source: §1 · p. 2

  • Method

    SPOT combines binocular fisheye sensing, a wide-field stereoscopic robot-centric display, viewpoint-decoupled free-looking, and visual stabilization.

    Source: §1 · pp. 2–3

  • Results

    SPOT improves task efficiency and accuracy and reduces search and recovery time across perception-critical humanoid loco-manipulation tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results suggest that perception transfer is an important complement to motion transfer for scalable long-horizon humanoid teleoperation.

    Source: §1 · p. 3

    End-to-end camera streaming latency remains a bottleneck for responsive teleoperation, particularly during rapid manipulation or recovery.

    Source: §6 · p. 9

Abstract

from arXiv · show

High-quality demonstration data is becoming a central bottleneck for training general-purpose humanoid robots. While recent humanoid teleoperation systems have made substantial progress in retargeting human motion to robot motion, long-horizon loco-manipulation requires another capability: operators must maintain task-relevant spatial awareness over time, e.g., object locations, surrounding environments, the robot's pose. We call the extent of this awareness the operator's perceptual horizon. However, existing methods often shorten this: narrow views miss peripheral events, robot-mounted cameras become unstable during locomotion, and coupled head-view control makes looking around interfere with robot motion. We present SPOT, a Spatial Perception-Oriented VR Teleoperation system for collecting long-horizon humanoid demonstration data by providing extended perceptual horizon. SPOT combines a robot-mounted binocular fisheye camera, a wide-field stereoscopic display, viewpoint-decoupled free-looking, and visual stabilization to provide a robot-centric view that is wide, stable, and actively inspectable. Unlike conventional egocentric interfaces, SPOT decouples visual exploration from robot actuation: the egocentric stereo observation is rendered on a virtual hemisphere around the operator, so natural head rotations change where the operator looks within the wide-field view rather than commanding the robot head, camera, or torso. We evaluate SPOT on perception-critical humanoid data-collection tasks spanning drop recovery, peripheral retrieval, large-workspace bimanual manipulation, fine alignment, and dynamic interaction. SPOT improves efficiency, accuracy, and recovery speed, demonstrating its effectiveness for user-friendly and scalable long-horizon humanoid data collection.

1 Introduction

Long-horizon humanoid teleoperation depends on sustained spatial awareness as well as accurate motion transfer. SPOT combines a wide, stable, actively inspectable robot-centric view with viewpoint-decoupled looking, and improves efficiency, accuracy, and recovery across perception-critical tasks.

1 Introduction

  • Long-horizon data collection involves mobility, bimanual interaction, unexpected-event recovery, and precise spatial alignment, beyond transferring human motion to robot motion.Existing systems have advanced motion tracking, retargeting, and whole-body control.Source: §1 · p. 2
  • Existing egocentric interfaces may miss peripheral events, become unstable during locomotion, or couple looking around to robot motion.The illustrated limitations also include weakened depth perception and turning difficulties in exocentric interfaces, and hidden manipulators or off-camera objects in egocentric interfaces.Source: §1 · p. 2, Figure 2 · p. 2
  • SPOT improves task efficiency and accuracy and reduces search and recovery time across perception-critical humanoid loco-manipulation tasks.The evaluated suite includes unexpected drop recovery, peripheral object retrieval, wide-range bimanual manipulation, fine spatial alignment, and dynamic interaction.Source: §1 · p. 3
  • The operator’s perceptual horizon—the spatial and temporal extent of reliably maintained task-relevant awareness—is a practical bottleneck complementary to motion transfer.Limited awareness can make object search repetitive, recovery slower, and precise alignment more difficult.Source: §1 · pp. 2–3
  • SPOT combines binocular fisheye sensing, wide-field stereoscopic viewing, viewpoint-decoupled free-looking, and visual stabilization in a robot-centric interface.Head rotations change where the operator looks within the observation rather than commanding the robot; stabilization compensates for camera rotations induced by robot motion.Source: §1 · pp. 2–3

2 Related Works

Related work covers bimanual teleoperation, whole-body humanoid teleoperation, and immersive visual feedback with viewpoint control. The approaches address different challenges, from arm-trajectory mapping to whole-body tracking and robot-view rendering.

Bimanual teleoperation

  • Bimanual systems map human-operated bi-arm kinematics to robot arms, use exoskeletons to collect arm trajectories, or track relative hand positions with VR/AR headsets.The described setups attach two arms to a static or moving platform.Source: §2 · p. 3

Humanoid teleoperation

  • Humanoid teleoperation requires whole-body joint collection and retargeting without an isomorphic robot under direct operator control.Prior approaches include MoCap, exoskeletons, inertial motion capture, and more recently VR interfaces.Source: §2 · p. 3

Immersive visual feedback and viewpoint control

  • Immersive feedback work includes spherical re-rendering, decoupled viewpoints using live point clouds and SLAM geometry, and rotation compensation for comfort and VR sickness.The cited approaches respectively reduce apparent latency, extend the view beyond the current camera view, and improve comfort.Source: §2 · p. 3

3 Methods

SPOT pairs a robot-centric immersive perception interface with a VR teleoperation pipeline for long-horizon humanoid loco-manipulation. Operators can explore a stabilized wide-field view while tracked headset and controller signals provide separate robot commands.

3 Methods

  • SPOT combines a stabilized immersive perception interface with a VR teleoperation pipeline that converts operator inputs into robot control targets.The pipeline uses a target-conditioned policy to execute those targets.Source: §3 · pp. 3–4, §3.2 · p. 5

3.1 Immersive Perception Interface

  • The egocentric observation surrounds the operator on a virtual hemisphere, so head turns redirect viewing without commanding robot camera or head motion.Robot commands instead come separately from tracked headset and controller signals.Source: §3.1 · p. 4
  • Camera-motion compensation rotates the displayed observation relative to the initial camera orientation, stabilizing the scene during locomotion, balancing, or external interference.The implementation uses an IMU attached to the stereo camera to estimate orientation.Source: §3.1 · p. 4

3.2 Teleoperation Pipeline

  • An OpenXR interface acquires headset and controller poses, and has been validated on Meta Quest and PICO headsets without pipeline changes.It streams tracked poses, joystick inputs, and button states for teleoperation.Source: §3.2 · p. 5
  • A streaming network estimates torso orientation from VR pose history, while calibrated Cartesian retargeting maps torso and wrist targets for a policy tracking hand and locomotion commands.The operator commands locomotion separately through a joystick; the target-conditioned policy is trained in simulation with domain randomization.Source: §3.2 · p. 5

4 Experimental Results

SPOT is evaluated on long-horizon tasks that require spatial awareness during locomotion and manipulation. Across the reported tasks, it improves efficiency, while task-specific comparisons describe faster recovery, peripheral search, and viewpoint realignment.

4.1 Experimental Setup

  • Ten operators completed 20 trials per task and interface; methods shared the robot, pipeline, environments, and protocols, with interface order randomized.The study compares exocentric observation with a matched active-view egocentric baseline whose viewport is limited to 110°×70°.Source: §4.1 · p. 6
  • SPOT improves efficiency across recovery, peripheral awareness, bimanual coordination, and long-horizon loco-manipulation tasks.The evaluation includes unexpected-drop recovery, peripheral retrieval, multi-object bimanual retrieval, and a light-switch task.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 Awareness of Unexpected Events and Out-of-View Objects

  • SPOT reduces recovery time versus both baselines and visual search time versus conventional egocentric teleoperation, while supporting dynamic-object tracking across a large field of view.The conventional egocentric view loses awareness of objects outside its narrow field, requiring additional search and relocalization.Source: §4.2 · p. 7

4.3 Bimanual Spatial Awareness

  • SPOT improves bimanual task efficiency and reduces grasp attempts, while conventional egocentric viewing can force sequential handling and exocentric viewing weakens depth judgment.Qualitative demonstrations include rope knotting and toy-sword insertion requiring monitoring of both manipulators and spatial alignment.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 Spatial Consistency Across Locomotion and Manipulation

  • SPOT maintains stable robot-centered perception across locomotion and manipulation, reducing viewpoint-alignment time compared with conventional egocentric viewing after robot turns.The Light Switch task measures viewpoint realignment after turning; conventional egocentric frames rotate with the robot during navigation.Source: §4.4 · p. 8, Figure 6 · p. 8

5 Conclusion

SPOT extends operators’ perceptual horizon with an immersive, stabilized, robot-centric interface that decouples visual exploration from robot actuation. Across perception-critical loco-manipulation tasks, it improves efficiency, accuracy, and recovery speed relative to conventional egocentric teleoperation.

5 Conclusion

  • SPOT combines wide-field stereo observation, viewpoint-decoupled free-looking, and visual stabilization to support long-horizon humanoid data collection.The interface allows operators to inspect the environment without interfering with robot motion.Source: §5 · p. 9
  • Across tasks involving drop recovery, peripheral retrieval, bimanual manipulation, fine alignment, and dynamic interaction, SPOT improves efficiency, accuracy, and recovery speed.The comparison is with conventional egocentric teleoperation interfaces.Source: §5 · p. 9
  • SPOT frames spatial perception as an explicit interface design objective and treats perception transfer as a complement to motion transfer.The conclusion links improved perception, search, and spatial context with the quality, consistency, and scalability of demonstrations.Source: §5 · p. 9

6 Limitations and Future Work

SPOT currently emphasizes visual perception through immersive stereo observation, while several capabilities and impacts remain unaddressed. The authors identify sensory integration, visual-stream latency, and downstream policy learning as future directions or open questions.

6 Limitations and Future Work

  • SPOT has not yet been evaluated for its impact on large-scale downstream policy learning.The authors identify how operator perception affects learned-policy scalability and quality as an open direction.Source: §6 · p. 9
  • The current system focuses primarily on visual perception, leaving tactile feedback, force interaction, and audio as future sensory channels to incorporate.Source: §6 · p. 9
  • End-to-end camera-streaming latency remains a bottleneck that can reduce responsiveness during rapid manipulation or recovery.The system remains usable under current latency conditions, but the authors identify lower visual-feedback latency as important for highly responsive whole-body teleoperation.Source: §6 · p. 9

A.1 Ablation Study

The ablation study finds complementary benefits from stereo perception, wide-field observation, and stabilized visual feedback. Their effects vary by task, with wide-field removal most harmful in multi-object retrieval.

A.1 Ablation Study

  • The study compares the full system with monocular display, narrow-field display, and no-stabilization variants using average task completion time.The two evaluated tasks are Multi-object Bimanual Retrieval and Light Switch.Source: §A.1 · p. 13, Table 2 · p. 13
  • Removing stereo increases completion time in both tasks, while removing the wide field causes the largest degradation in Multi-object Bimanual Retrieval.Without stereo, operators struggle to estimate gripper-target depth; without the wide field, they lose sight of an object and need extra visual search.Source: §A.1 · p. 13
  • Removing stabilization primarily affects Light Switch, where robot turning and locomotion change the visual reference frame and require extra viewpoint corrections.Source: §A.1 · p. 13
  • Stereo perception, wide-field observation, and stabilized visual feedback provide complementary benefits for spatial perception and teleoperation efficiency.Source: §A.1 · p. 13

A.2 User Study

In a user study of three teleoperation interfaces, SPOT received the highest ratings for visual perception and manipulation ease. Exocentric teleoperation scored highest for comfort, while SPOT was rated more comfortable than conventional egocentric teleoperation.

A.2 User Study

  • Ten participants rated SPOT, conventional egocentric, and exocentric teleoperation on a five-point scale after completing the task with each interface.The interface order was randomized; participants had varied prior teleoperation experience.Source: §A.2 · p. 13
  • SPOT received the highest user-study scores for visual perception and manipulation ease, while exocentric teleoperation scored highest for comfort.Source: §A.2 · p. 13, Table 3 · p. 14
  • SPOT was rated substantially more comfortable than conventional egocentric teleoperation, while exocentric teleoperation’s comfort advantage was attributed as likely related to avoiding immersive head-mounted viewing.The authors describe this as preliminary subjective evidence of improved comfort with stabilized visual feedback.Source: §A.2 · p. 13

A.3 Experienced and Novice Operators

SPOT remained competitive for both experienced and novice operators, achieving the lowest completion time in most settings. The comparison included 3 participants with prior teleoperation or immersive-interface experience and 7 new to robot teleoperation.

A.3 Experienced and Novice Operators

  • SPOT achieved the lowest completion time in most settings and remained competitive across experienced and novice operators.The comparison included 3 experienced and 7 novice participants.Source: §A.3 · p. 14
  • Among the 10 participants, 3 had prior teleoperation or immersive-interface experience, while 7 were new to robot teleoperation.Source: §A.3 · p. 14
  • Table 4 evaluates experienced and novice operators.Source: Table 4 · p. 14
Loading 2609.07933v1…