R2RI: A Multi-View Event and RGB Dataset for Robot-to-Robot Interaction

1. Introduction

• Robot-robot interaction lacks large-scale, diverse, publicly available datasets, while real-data collection is costly, difficult to scale, and safety-constrained.Existing collections often limit viewpoint diversity, robot types, task definitions, or sensing modalities.Source: §1 · p. 1
• R2RI combines egocentric onboard streams and exocentric fixed-camera views with synchronized RGB and event data.Egocentric data reflects information available to a robot at deployment; exocentric data provides global spatial context.Source: §1 · p. 2
• The dataset adapts real human interactions to multiple robot models using a URDF-based human-to-robot mapping algorithm, enabling automatic frame annotation.Simulation supports varied robots, interactions, backgrounds, illumination, and simultaneous RGB and event rendering at 120 fps.Source: §1 · p. 2
• R2RI is presented as the first large-scale multimodal dataset for robot-robot interaction, with heterogeneous humanoid platforms, two viewpoints, and baseline evaluations across eight perception tasks.The authors release the dataset and annotations for the modalities and tasks.Source: §1 · p. 2

Robot-Robot Interaction Datasets

• To the authors’ knowledge, no literature dataset explicitly targets robot-robot interaction; many robot datasets instead focus on manipulation with non-humanoid arms and static objects.Other collections may contain few annotated frames or robot models, limiting their usefulness for recent deep learning methods.Source: §2 · pp. 2–3
• Robot pose-estimation datasets typically show only a single robot, annotate its pose, and include no interactions; some also have too few annotated frames or robot models for effective use of recent deep-learning techniques.Simba includes RGB and depth; DREAM detects 2D keypoints and uses PnP to recover extrinsics and then 3D pose.Source: §2 · p. 3

Robot Detection and Pose Estimation

• Robot detection has progressed from markers, specialized sensors, or proprietary software toward CNN-based methods adapted from architectures such as Faster R-CNN and YOLO [30].Earlier approaches could be reliable under controlled conditions for known robot models, but their requirements are not always feasible.Source: §2 · p. 3
• RGB pose-estimation approaches include marker-based hand–eye calibration and render-and-compare methods that iteratively minimize discrepancies between projected models and camera data.Marker-based methods can be intrusive because markers are attached to the robot.Source: §2 · p. 3

Event-based Vision

• Event cameras produce asynchronous brightness-change events and offer high dynamic range, low-light performance, and microsecond temporal resolution; this work applies them to robot-robot interaction tasks.Prior event-camera research includes tracking, detection, and face analysis, as well as human detection and pose estimation.Source: §2 · p. 4

Dataset generation

• R2RI renders pairs of four humanoid models—Atlas v5, Unitree G1, iCub, and NAO—in scenes based on human interaction videos.Scenes vary lighting and include indoor and outdoor backgrounds; robot motions are retargeted from human interactions.Source: §3 · pp. 4–5

Human-to-robot motion retargeting

• Human-to-robot retargeting maps SMPL-X human poses to robot joints and optimizes each frame’s joint angles to match mapped joint positions and orientations.The mapping uses each robot’s URDF model, and the per-frame optimization starts from the current robot pose.Source: §3 · p. 5

Rendering

• R2RI provides exocentric stereo cameras and robot-following egocentric cameras, with camera extrinsics exported relative to the world frame.The exocentric camera tracks the midpoint between the robots; the camera setup generates multiple views.Source: §3 · p. 5

Dataset statistics

• R2RI statistics cover synchronized RGB and event streams from egocentric and exocentric views at 120 fps.The example scenes span indoor and outdoor settings and varied interactions.Source: Figure 3 · p. 5, Table 2 · p. 6

Event simulation

• RGB frames are rendered at 120 fps and converted into event streams using ESIM with FILM frame interpolation.The rendering frame rate follows the source video, and the RGB frames are generated at 120 fps to support event simulation.Source: §3 · p. 6

Tasks

• Eight tasks cover robot detection, 2D and 3D pose estimation, 2D and 3D pose forecasting, collision detection, action recognition, and sim-to-real transfer.Tasks use RGB and/or event inputs, with metrics specified by task; sim-to-real evaluates zero-shot transfer from synthetic training data to real videos.Source: §4 · p. 6

Experimental settings

• Four settings use released fixed train, validation, and test splits: standard, challenging lighting, humanoid generalization, and viewpoint generalization.The lighting setting tests Night data at 50%, 25%, and 5% brightness; viewpoint generalization trains on exocentric views and tests egocentric views, and vice versa.Source: §4 · pp. 6–7

Evaluation results

• The paper reports exocentric test results for 2D robot detection and pose estimation, alongside detection and pose results for challenging lighting conditions.The corresponding tables are Tables 3, 4, and 6.Source: Table 3 · p. 7, Table 4 · p. 7, Table 6 · p. 7

Baselines

• Detection and 2D pose baselines include convolutional, YOLO-family, and transformer models, trained for 20 epochs.The 3D pose-estimation baselines are PoseFormerV2, MotionBERT, and MotionAGFormer, supervised with 2D ground-truth keypoints.Source: §4 · p. 7

Detection, Pose Estimation, and Lighting

• YOLO26l leads both modalities on exocentric 2D detection, YOLO26l-pose leads 2D pose estimation, and event models remain consistent as RGB performance drops under severe light degradation.MotionBERT outperforms the other models on 3D pose estimation using 81-frame sequences with stride 27.Source: §5 · p. 7

Humanoid Generalization

• Event inputs consistently outperform RGB for object detection in humanoid generalization, especially on mAP50:95, while both modalities struggle with pose estimation.RGB performs comparably only on domains visually similar to training, such as NAO.Source: Table 7 · p. 8, §5 · p. 8

Viewpoint Generalization

• RGB outperforms events when training exocentrically and testing egocentrically, while the reverse-direction cross-view evaluation shows similar performance.The reported evaluations are across domains for detection and pose.Source: Table 8 · p. 8, Table 9 · p. 8, §5 · p. 8

Pose Forecasting

• Errors rise substantially when the forecasting horizon doubles; the Transformer is most accurate for 2D forecasting at h=30, while constant velocity achieves the best 3D ADE at h=30.Learned temporal models consistently improve endpoint accuracy (FDE), particularly at longer horizons.Source: Table 10 · p. 8, §5 · pp. 8–9

Collision Detection

• Event frames yield higher balanced accuracy and AUROC than RGB with the same ResNet-50 backbone for visual collision detection.With ground-truth 3D poses, Random Forest and a shallow MLP achieve very high F1 and AUROC; visual performance drops under occlusion and appearance variability.Source: Table 11 · p. 9, §5 · p. 9

Action Recognition

• On 20 interaction classes, event-based models outperform RGB counterparts, and ResNet-50 with temporal mean pooling achieves the best Top-1 and Macro-F1.Overall performance is low; VideoMAE provides only limited gains on RGB.Source: Table 11 · p. 9, §5 · p. 9

Sim-to-Real

• Zero-shot transfer from synthetic exocentric RGB training to DHRP real videos yields non-trivial detection but falls substantially below in-domain performance.DINO achieves the best mean AP50 across robots, with RT-DETR as runner-up; performance varies across platforms, and Phoenix remains challenging.Source: §5 · p. 9, Table 12

Dataset

• R2RI is a large-scale synthetic dataset designed specifically for robot-robot interaction, with synchronized RGB, depth, and event data from egocentric and exocentric perspectives.Interactions are generated using a URDF-based human-to-robot mapping algorithm and derived from human interaction data.Source: §6 · pp. 9–10

Construction and Evaluation

• The dataset provides diverse annotated scenarios across robots, viewpoints, and illumination conditions through automatic generation from retargeted human interactions.The paper reports comprehensive comparisons of state-of-the-art methods across multiple perception tasks and RGB and event modalities.Source: §6 · p. 10

Findings

• RGB remains effective under standard conditions, while event-based data is robust under challenging illumination.Source: §6 · p. 10

Limitations

• R2RI is fully synthetic and retains a sim-to-real gap; its interactions cover limited humanoid morphologies and classes, are mostly pairwise, and omit intent and task-success semantics.Source: §6 · p. 10

Code Release

• The authors state that the complete R2RI code will be released upon acceptance.Source: A. Reproducibility · p. 10

Baseline Instructions

• The planned code release includes detailed instructions for training and testing all considered baseline methods.Source: A. Reproducibility · p. 10

Hardware

• All models were trained on a single RTX4090 with 24GB VRAM.Source: A. Reproducibility · p. 10

Gestures and Social Actions

• The list includes social gestures such as waving, pointing, chatting, and thumbs-up.Source: B. List of Interactions · p. 10

Contact Actions

• Physical-contact interactions include hugging, hitting, kicking, pushing, pulling, slapping, and stepping on a foot.Source: B. List of Interactions · p. 10

Other Interactions

• Other listed activities include chasing, rock-paper-scissors, linking arms, bending, imitating, and looking back.Source: B. List of Interactions · p. 10

C. Background 3d Scenes

• Six 3D background scenes comprise four indoor environments and two outdoor environments.Source: C. Background 3d Scenes · p. 10

C.1. Indoor Environments

• Indoor scenes include a historic drawing room, art museum, hospital operating room, and office.Source: §C.1 · p. 10

C.2. Outdoor Environments

• Outdoor scenes are a realistic game-like urban environment and a street in a small town in Viterbo, Italy.Source: §C.2 · p. 10

D. Robot Models

• Four humanoid models were selected for wide adoption and URDF availability: Nao, Unitree G1, Atlas v5, and iCub.Source: D. Robot Models · pp. 10–11

• At minimum brightness, RGB-based models’ mAP drops almost to 0 across models.Source: E. Challenging Lighting Conditions · p. 11
• Event-based models consistently retain overall high mAP across illumination levels, with most models reporting on-par results.Source: E. Challenging Lighting Conditions · p. 11
• The illumination-related pattern holds for both detection and pose estimation.Source: E. Challenging Lighting Conditions · p. 11

F. Qualitatives

• Egocentric and exocentric views vary considerably, and an egocentric view may capture the other robot only partially depending on the action.Source: F. Qualitatives · p. 11
• Movement sequences sampled every 10 frames display detection and 2D pose annotations.Source: Figure B.1 · p. 12
• As visibility falls from full to 5% lightness, RGB frames become nearly unintelligible while event frames preserve clear subject silhouettes.Source: Figure E.1 · p. 13
• Indoor examples show exocentric and both robots’ egocentric views of the same scene.Source: Figure F.1 · p. 14
• Outdoor examples likewise show exocentric and both robots’ egocentric views of the same scene.Source: Figure F.2 · p. 15