Source-linked AI summary
Event-based Agile Object Catching with a Quadrupedal Robot
Benedek Forrai, Takahiro Miki, Daniel Gehrig, Marco Hutter, Davide Scaramuzza
TL;DR
High-speed object catching with onboard sensing remains a challenge for quadrupedal robots, and standard cameras face a bandwidth-latency tradeoff. This paper uses event-based perception and a learning-based planner for quadruped catching, reporting catches up to 15 m/s from 4 meters with an 83% success rate.
Problem
High-speed object catching with onboard sensing remains a significant challenge for quadrupedal robots, while RGB cameras and LiDARs face a bandwidth-latency tradeoff.
Source: §I · p. 1Method
An event-camera visual frontend estimates object trajectories, and a learning-based planner uses those estimates for quadruped catching.
Source: §VI · p. 6, §VII · p. 7Results
15 m/s from 4 meters with an 83% success rate is reported for the quadruped catching system.
Source: §I · p. 2Takeaways & Limitations
The event-based pipeline tracks objects at up to 100 Hz on an NVIDIA Jetson Orin.
Source: §I · p. 2
Abstract
from arXiv · showhide
Quadrupedal robots are conquering various indoor and outdoor applications due to their ability to navigate challenging uneven terrains. Exteroceptive information greatly enhances this capability since perceiving their surroundings allows them to adapt their controller and thus achieve higher levels of robustness. However, sensors such as LiDARs and RGB cameras do not provide sufficient information to quickly and precisely react in a highly dynamic environment since they suffer from a bandwidth-latency tradeoff. They require significant bandwidth at high frame rates while featuring significant perceptual latency at lower frame rates, thereby limiting their versatility on resource-constrained platforms. In this work, we tackle this problem by equipping our quadruped with an event camera, which does not suffer from this tradeoff due to its asynchronous and sparse operation. In leveraging the low latency of the events, we push the limits of quadruped agility and demonstrate high-speed ball catching for the first time. We show that our quadruped equipped with an event camera can catch objects with speeds up to 15 m/s from 4 meters, with a success rate of 83%. Using a VGA event camera, our method runs at 100 Hz on an NVIDIA Jetson Orin.
MULTI-MEDIA MATERIAL
High-speed catching with onboard sensing remains a challenge for quadrupeds. This work uses event-based trajectory estimates to guide a quadruped’s catching policy, and reports successful catches up to 15 m/s.
MULTI-MEDIA MATERIAL
- The multimedia material provides experiment videos and links to the open-source code.The video includes visual results; the code repository is named in the paper.Source: MULTI-MEDIA MATERIAL · p. 1
I. INTRODUCTION
- High-speed object catching with onboard sensing remains a significant challenge for quadrupeds, while RGB cameras and LiDARs face a bandwidth-latency tradeoff.High frame rates require substantial bandwidth; lower frame rates increase perceptual latency and can affect reaction time and success rate.Source: §I · p. 1
- The pipeline detects independently moving objects in events, fits ballistic trajectories, and passes the estimate to a data-driven catching policy.The paper reports object tracking at up to 100 Hz on an NVIDIA Jetson Orin.Source: §I · p. 2
- 15 m/s from 4 meters with an 83% success rate is the reported maximum for the event-camera quadruped catching system.The work presents this as the first successful event-camera-based high-speed object catching demonstration with a net-equipped quadruped.Source: §I · p. 2
- The authors extend latency analysis from to compare standard- and event-camera algorithms, both of which use at least two 3D detections to fit a ballistic parabola.The fitted parabola is used to infer the object’s impact range for catching.Source: §I · p. 2, Figure 3 · p. 2
A. Dynamic Object Catching in Robotics
Prior dynamic-object catching work spans known trajectories, onboard drone perception, and a learning-based detector on a linear slider. This work applies the catching concept to a legged robot.
A. Dynamic Object Catching in Robotics
- Earlier catching systems used known trajectories detected by motion-capture rigs, with implementations ranging from robotic arms to drones.Onboard drone catching with a standard camera was limited to 6 m/s and used bright LED lights on the balls.Source: A. Dynamic Object Catching in Robotics · p. 2
- Falanga et al. targeted objects flying at up to 10 m/s with a drone, focusing on evasion rather than catching, which imposes a lower constraint on the robot policy.Source: A. Dynamic Object Catching in Robotics · p. 2
- After a learning-based detector was demonstrated for catching with a linear slider, this work applies the concept to a complex legged robot.Source: A. Dynamic Object Catching in Robotics · p. 2
B. Advancements in Legged Robotics
Legged robots present a more complex control problem than lightweight, easily modeled quadcopters. The section describes the tradeoffs between model-based planning, learning-based control, and delayed exteroceptive mapping for dynamic tasks.
B. Advancements in Legged Robotics
- Compared with lightweight quadcopters, legged robots pose a more sophisticated control problem because of their size and more complex environmental interactions.Source: B. Advancements in Legged Robotics · p. 2
- Model-based control can require computationally heavy planning, while deep reinforcement learning has shown complex agile maneuvers without compromising real-time performance.The cited examples for deep reinforcement learning are,,, [16], and.Source: B. Advancements in Legged Robotics · p. 2
- Depth-sensor mapping pipelines introduce latency for dynamic environments, and high-speed catching with onboard vision remains an open question despite work on raw-depth locomotion.Source: B. Advancements in Legged Robotics · p. 2
- An off-board hierarchical reinforcement-learning approach has demonstrated a small quadruped acting as a goalie by selecting behaviors based on incoming ball trajectories [22].Source: B. Advancements in Legged Robotics · p. 2
III. WORST CASE LATENCY ANALYSIS
The analysis compares worst-case time to collect enough measurements to estimate a ballistic trajectory. Event-based processing avoids waiting for the next camera frame, and one-shot extraction of position and velocity could reduce latency further.
III. WORST CASE LATENCY ANALYSIS
- Latency is defined as the worst-case wait to collect the minimum measurements needed to describe the object’s parabola.The analysis focuses on measurements of 3D position and/or 3D velocity.Source: §III · p. 3
- For standard cameras, the algorithm must wait for a frame or the previous detection cycle before starting, then perform two consecutive detections.The waiting interval is determined by the larger of processing time and inter-frame time.Source: §III · p. 3
- For event algorithms, the initial wait is approximately free of frame-interval delay and is instead dominated by the previous detection cycle.The paper sets ∆tFPS ≈ 0 for event-based algorithms.Source: §III · p. 3
- One-shot event methods could extract 3D position and velocity together, reducing latency to ∆tl = 2∆tc.Source: §III · p. 3
- The presented catching approach is two-shot, so it benefits from fast execution; the authors leave one-shot methods for future work.They state that it can outperform RGB-camera approaches at similar inter-frame intervals.Source: §III · p. 3
IV. APPROACH
The approach pairs an event-based visual frontend that estimates object trajectories with a planning backend that converts predicted trajectories into low-level motor commands for the quadruped.
IV. APPROACH
- The method pairs event-based object-trajectory estimation with a planning backend that converts predictions into low-level motor commands for the quadruped.The full method is illustrated in Fig. 4.Source: §IV · p. 3
- The visual frontend estimates object trajectories from event-based visual input.Source: §IV · p. 3
- The planning backend converts the predicted trajectory into low-level motor commands.Source: §IV · p. 3
A. Visual Frontend
The visual frontend detects independently moving objects in event data, estimates their ballistic trajectories, and predicts where they intersect a plane for catching.
A. Visual Frontend
- Event data and IMU measurements feed a motion-compensated timestamp image whose thresholded values segment independently moving objects.The estimator processes an asynchronous event buffer and uses an adaptive threshold that depends on angular rate.Source: A. Visual Frontend · pp. 3–4
- DBSCAN clusters moving-object pixels using a dissimilarity score and optical-flow similarity, then bounding boxes are passed to trajectory estimation.The score formulation follows.Source: A. Visual Frontend · p. 4
- For monocular depth, the method assumes a known object size, estimates depth from its image width and camera focal length, then backprojects and transforms 3D positions using robot odometry.Source: A. Visual Frontend · p. 4
- RANSAC fits a ballistic parabola to noisy 3D positions by testing point pairs and counting inliers under a weighted distance criterion.The fitting minimizes weighted squared error to inlier points; the detector accounts for greater noise in the camera z-direction.Source: A. Visual Frontend · p. 4
- The backend target is the parabola's intersection with a plane through the robot base, and a median filter smooths impact-point detections until catching begins.Source: A. Visual Frontend · pp. 4–5
B. Robot Locomotion Backend
The backend uses a reinforcement-learned joint-control policy to move the net toward the estimated impact point, trained in simulation with target tracking and motion penalties.
B. Robot Locomotion Backend
- The robot's joint-level policy is trained with reinforcement learning to reach the estimated impact point with its net as quickly as possible.The method models catching as an MDP and uses PPO.Source: B. Robot Locomotion Backend · p. 5
- The policy takes a 48-dimensional state and outputs differences between desired and nominal joint positions; its inputs include the impact point and proprioceptive state.The proprioceptive terms include gravity, body velocity, angular velocity, joint positions, and joint velocities.Source: B. Robot Locomotion Backend · p. 5
- Simulation training randomizes target net positions within specified base-frame ranges, adds observation noise and rough terrain, and ends episodes when the base or thigh hits the ground.Target positions are resampled every second with probability 0.2.Source: B. Robot Locomotion Backend · p. 5
- The reward combines target tracking with penalties intended to make the robot's motion smooth.The target-tracking term decreases exponentially with squared distance between the impact point and net position.Source: B. Robot Locomotion Backend · p. 5
V. EXPERIMENTS
Real-world tests evaluate both impact-point detection and catching under manually thrown balls, using an onboard event-camera system. Catch success stayed above 80% across tested speeds, while vision accuracy degraded at high speeds and larger deviations.
Experimental setup
- Manual tests used two 10 cm rubber balls thrown at 5–15 m/s; the VGA DVXplorer and Jetson Orin ran vision at up to 100 Hz.The ANYmal C carried a 0.3 m landing net; catching was deployed without additional real-world fine-tuning.Source: §V · p. 5
Results
- Vision accuracy degraded above 9 m/s and beyond 0.4 m deviation, as faster throws yield fewer trajectory samples and larger deviations move balls out of view sooner.At low speeds, steeper impact angles reduce effective net area, and the policy does not directly observe impact angle.Source: §V · p. 6
- For ball catching across the tested object speeds, the reported success rate remained above 80%, with its maximum at 12 m/s, as summarized in Table I.Source: §V · p. 6, Table I · p. 6
- At 15 m/s and 0.6 m deviation, 50.25% of impact points were detected within a net radius of ground truth.This is the vision algorithm's success criterion.Source: Table II · p. 6, §V · p. 6
VI. DISCUSSION
The discussion presents onboard event-based catching of 5–15 m/s objects and identifies sensing and control changes that could extend the demonstrated capability. The current monocular setup requires object-size knowledge, and steep impact trajectories can reduce effective net area.
Demonstrated capability
- The system demonstrates quadruped catching of 5–15 m/s objects, including throws deviating 0.4 m from the net center.The onboard event-based tracker provides adequate estimates over a 0.6 m range.Source: §VI · p. 6
Limitations and extensions
- Monocular depth estimation requires known object size; stereo is proposed to remove that requirement and reduce depth uncertainty.A higher-FoV camera is suggested to extend the robot's lateral range.Source: §VI · p. 6
- The current controller reaches the target quickly; training with ball motion, sensor properties, and detection could improve it, while impact-point velocity is proposed for steep-trajectory failures.Steep trajectories reduce effective net area.Source: §VI · pp. 6–7
VII. CONCLUSION
The paper addresses the bandwidth-latency tradeoff of conventional exteroceptive sensors for agile quadruped catching. It combines an event camera with a high-speed visual frontend and learning-based planner, reporting an 83% success rate for objects thrown at up to 15 m/s.
Conclusion
- Event cameras avoid the bandwidth-latency tradeoff attributed to commonly used RGB cameras and LiDARs.The conclusion links this motivation to agile catching with a quadruped.Source: §VII · p. 7
- The method combines an event-camera high-speed visual frontend with a learning-based planner for agile ball-catching.Source: §VII · p. 7
- 83% success was reported while catching objects thrown at up to 15 m/s.Source: §VII · p. 7