GrandTour: A Legged Robotics Dataset in the Wild for Multi-Modal Perception and State Estimation

Introduction

• Legged locomotion and harsh, unstructured environments disrupt sensing, while existing public datasets rarely combine legged motion, varied environments, and multi-modal sensing.These challenges complicate robust state estimation, mapping, scene understanding, and safe navigation.Source: §1 · pp. 1–2
• GrandTour supports benchmarks for LiDAR-inertial, visual-inertial, LiDAR-only, LiDAR-inertial-visual, and legged-specific kinematic-inertial odometry, and evaluates 63 localization methods on six missions.The benchmarks enable systematic evaluation of multi-modal fusion and drift across terrains and challenging environments.Source: §1 · pp. 2–3
• GrandTour has supported learning applications ranging from sim-to-real transfer and physical-parameter learning to navigation foundation models and scene-understanding benchmarks.Its synchronized multi-modal data and accurate ground truth can also support visual-geometry tuning and fine-tuning foundation models for robustness and transfer to real-world legged deployments.Source: §1 · pp. 2–3
• GrandTour provides 49 sequences across indoor facilities, urban outdoors, and natural terrains, with synchronized, calibrated multi-sensor data and high-precision ground truth.Its sensors include multi-LiDAR, multi-camera, depth cameras, IMUs, GNSS, and full proprioception; the paper describes it as the largest open-access legged-robotics dataset to date.Source: §1 · pp. 1, 3
• The open-source release includes tools, documentation, and intermediate and processed outputs such as odometry, deskewed point clouds, terrain maps, point-cloud maps, and occupancy maps.These resources are intended to ease adoption and support targeted development.Source: §1 · p. 3

2.1 Popular robotic state estimation datasets

• Reference datasets established benchmarks across wheeled, aerial, and handheld platforms, but do not capture legged locomotion disturbances, proprioception–exteroception coupling, or unstructured terrain.GrandTour is framed as a legged counterpart with synchronized multi-LiDAR, multi-camera, and IMU sensing, full proprioception, and indoor, urban, industrial, and natural missions.Source: §2.1 · p. 3

2.2 Multi-modal perception datasets

• Multimodal datasets combine complementary sensors to address failures under varied lighting, adverse weather, sparse texture, and self-similar or dynamic scenes.The literature increasingly fuses LiDAR, RGB, and inertial sensing, sometimes complemented by radar or event sensors, across varied environments and platforms.Source: §2.2 · pp. 3–4
• Examples include M2DGR (Yin et al., 2022), FusionPortable/V2 (Jiao et al., 2022; Wei et al., 2025), MulRan (Kim et al., 2020), M3ED (Chaney et al., 2023), MCD (Nguyen et al., 2024b), Oxford Spires (Tao et al., 2025), and SMapper-light (Soares et al., 2025).These efforts span camera, LiDAR, inertial, radar, event, and UWB sensing and diverse platforms and environments.Source: §2.2 · pp. 3–4

2.3 Legged-robot datasets

• Legged-dataset efforts cover deformable and diverse outdoor terrain, multi-robot sessions, degraded-sensing missions, and indoor, outdoor, or underground scenes, though some are not legged-focused or use non-rig camera setups.Examples include TAIL (Yao et al., 2024), TAIL-Plus (Wang et al., 2024b), DiTer (Jeong et al., 2024), DiTer++ (Kim et al., 2024), SubT-MRS (Zhao et al., 2024b), EnvoDat (Nwankwo et al., 2025), and FusionPortableV2 (Wei et al., 2025).Source: §2.3 · p. 4
• GrandTour addresses scarce large-scale legged datasets with more than 49 missions, rich multimodal sensing and proprioception, millimeter-level accuracy, and sub-millisecond synchronization.It spans urban, industrial, and natural settings, including day/night operations and adverse weather, and combines dual RTK-GNSS with survey-grade total-station measurements.Source: §2.3 · p. 4

3.1 Sensor suite

• Boxi combines rotating LiDARs, RGB and stereo depth cameras, seven released IMUs, encoders, and GNSS and total-station pose systems with ANYmal-D sensors.The combined suite weighs about 7.1 kg and draws roughly 120 W; point-foot contacts are onboard-estimator outputs, not direct measurements or ground truth.Source: §3.1 · p. 5

3.2 Temporal and Spatial Calibration

• GrandTour calibration builds on Boxi procedures documented in Frey et al. (2025), while rigidly mounted sensors use known extrinsics and a common timing source for temporal alignment.The compute units connect through a UbiSwitch Ethernet device, with sensors interfacing through USB 3.1, GMSL2, RJ45, and I²C.Source: §3.2 · p. 5, §3.1 · p. 5, Figure 3 · p. 6

LiDARs

• The Livox Mid-360 provides near-field LiDAR at 10 Hz over 0.1–40 m, while the ANYmal-mounted Velodyne VLP-16 operates at 10 Hz over 0.5–100 m.Their listed accuracies are ±0.02 m and ±0.03 m, respectively.Source: §3.1 · p. 5, LiDARs · p. 6

Proprioceptive

• Proprioception includes STIM320, ADIS16475-2, and TDK ICM40609 IMUs, plus 12 ANYmal-D joint encoders providing joint positions, velocities, and torques.The auxiliary STIM320 operates at 500 Hz and the ADIS16475-2 and TDK ICM40609 at 200 Hz; hardware triggering is provided relative to the Livox IMU, and encoder streams are software-triggered relative to the ANYmal IMU.Source: Proprioceptive · p. 6

3.2.1 Time Synchronization

• PTP-enabled devices synchronize to the NovAtel GNSS time grandmaster, while chrony-based NTP aligns heterogeneous compute units; released IMU streams were verified to be aligned within sub-1 ms.LiDARs use IEEE 1588v2 PTP; synchronization accuracy ranges from nanoseconds for PTP to milliseconds for custom software-based kernel timestamping, except the USB-buffer-reliant ZED2i.Source: §3.2.1 · p. 7

3.2.2 Timestamp and Exposure Time of Cameras

• Camera timing uses hardware-triggered, midpoint timestamps for rolling-shutter HDR and PTP-synchronized exposure alignment for CoreResearch cameras, while ZED2i timestamps are unreliable.HDR cameras use a fixed 11 ms exposure; ZED2i’s USB serial buffer prevents reliable timestamps and individual line read-out timing.Source: §3.2.2 · p. 7

3.2.3 Calibration of Boxi

• Camera calibration estimates intrinsics with an April-Grid and pinhole projection, then optimizes inter-camera extrinsics from static observations; IMU intrinsics and camera-relative extrinsics are calibrated separately.Wide-angle fisheye CoreResearch and HDR cameras use Kannala–Brandt distortion, while ZED2i cameras use rad-tan distortion; IMU intrinsics use Allan variance and IMU extrinsics use Kalibr.Source: §3.2.3 · p. 7
• LiDAR-to-camera calibration uses intensity alignment that can exploit partial target observations, and each Boxi LiDAR is jointly calibrated against five CoreResearch cameras using fused camera target poses.Partial pattern observations are especially useful for the narrow-field-of-view Hesai, while Livox’s non-repeating scan pattern provides dense coverage.Source: §3.2.3 · p. 7
• The custom camera-to-prism calibration uses MS60 measurements and static-target detections to estimate a 9-DoF state, achieving 3 mm cross-camera consistency and validated alignment during motion.The procedure positions the camera within 1 m of the target and collects static Boxi poses around it while the total station tracks the prism.Source: §3.2.3 · pp. 7–8

3.2.4 Boxi-to-ANYmal calibration

• The Boxi-to-ANYmal extrinsic transform comes from the CAD assembly and is validated by projecting VLP16 point clouds into camera-bundle frames; LiDAR extrinsics are also checked by map-to-map registration.The payload’s rigid machined-aluminum interface uses precision locating pins and a preloaded bolt pattern.Source: §3.2.4 · p. 8

4 Dataset overview

• The dataset contains 49 missions of 3–7 minutes each, spanning natural and man-made environments, varied weather and lighting, and terrains including sand, snow, and gravel.Missions cover distances up to several hundred meters; Figure 7 summarizes environmental tags for exploration.Source: §4 · p. 8, Figure 7 · p. 12

4.1 Data format and storage

• HuggingFace provides Zarr and JPEG data, downloadable by mission or sensor subset, with calibration metadata and scripts for loading and synchronizing streams.This ROS-independent format is intended to support access with familiar tools beyond the robotics community.Source: §4.1 · p. 8, Figure 8 · p. 13
• ROS users can download separate per-stream bag files, compressed with LZ4, and convert them to CRF-compressed MCAP files compatible with ROS 2 Jazzy and Ubuntu 24.04.Each mission is divided into 34 individual ROS bags, allowing users to select the data they need.Source: §4.1 · p. 9

4.2 Ground Truth Pose Generation

• For outdoor ground truth, CPT7 GNSS/INS data are post-processed with tightly coupled GNSS and inertial measurements; specifications report 0.01–0.02 m position RMS error during outages up to 10 s.The reported specifications also give velocity RMS below 0.015 m s−1 and attitude RMS below 0.01◦.Source: §4.2 · pp. 10–11

4.2.1 Robotic Total Station

• Holistic Fusion combines total-station positions, post-processed Inertial Explorer poses, and IMU data, but reference intervals are restricted to times with total-station measurements or Inertial Explorer poses.For evaluation, the paper recommends selecting the highest-fidelity available reference and cropping estimates to its valid interval; total-station positions are preferred when sparse translation-only evaluation is acceptable.Source: §4.2.1 · p. 12, §4.2.2 · p. 13
• 0.0028 m ATE mean and 0.0034 m RMSE for Holistic Fusion compare with 0.132 m mean and 0.1504 m RMSE for Inertial Explorer tightly coupled.ATE standard deviations are 0.0020 m for Holistic Fusion and 0.0721 m for Inertial Explorer tightly coupled.Source: §4.2.1 · p. 13
• 0.0030 m RTE (∆0.5 m) mean and 0.0037 m RMSE for Holistic Fusion compare with 0.0073 m mean and 0.012 m RMSE for Inertial Explorer tightly coupled.RTE standard deviations are 0.0021 m for Holistic Fusion and 0.0095 m for Inertial Explorer tightly coupled.Source: §4.2.1 · p. 13

5 Data Collection

• Two operators coordinate each mission: a teleoperator navigates ANYmal safely, while a Leica MS60 operator maintains robot tracking and re-acquires it after occlusions.The control prism keeps the ground-truth measurement frame stable and static throughout the mission.Source: Figure 10 · p. 14
• The procedure begins by choosing a safe, diverse route and positioning the total station and reference prism, then verifying sensors and recording the prism position before tracking starts.The prism is measured again at the destination to validate station stability before recording stops and instruments are powered down.Source: §5 · p. 14
• The teleoperator follows behind ANYmal, may intervene for safety, and helps re-establish total-station lock; re-locking can require pauses in the trajectory.Sequences were collected with the reinforcement-learning locomotion controller of Miki et al. (2022a), which contextualizes commanded-velocity tracking and body-state characteristics in perception–action or control-aware analyses.Source: §5 · p. 14, Figure 10 · p. 14

• Online leg odometry uses ANYmal-IMU and ANYmal-JE; its estimator also supplies foot-contact states, which are estimates rather than direct measurements or ground truth.Users can derive contact estimates from joint-torque measurements or use the provided labels for ease of integration.Source: §6 · p. 14
• Post-processing uses leg odometry to generate motion-compensated LiDAR point clouds and provides CPT7-derived INS trajectories and DLIO trajectory data (Chen et al., 2023).The INS trajectories include a real-time solution and a high-precision, tightly coupled solution generated via Inertial Explorer.Source: §6 · p. 14
• Data curation preserves driver-native message formats and filters the robot body from LiDAR point clouds.Source: §6 · pp. 14–15
• Table 5 catalogs the recorded sensor streams for a sequence, while Table 6 lists derived outputs and describes each output.Source: Table 5 · p. 15, Table 6 · p. 15

• GrandTour supports applications in state estimation and localization, perception, and locomotion and navigation.Source: §7 · p. 15

7.1.1 LiDAR Odometry

• Traj-LO achieves the best overall LiDAR-odometry average rank but degrades on ARC-2, while DLO and I2EKF-LO are next overall and lead on selected mission metrics.Traj-LO leads on SPX-2 and EIG-1, SNOW-2 ATE, and CON-4 RTE; I2EKF-LO leads on both ARC-7 metrics, and DLO leads on ARC-2 RTE.Source: §7.1.1 · p. 17, Table 8 · p. 19

7.1.2 LiDAR-Inertial-(Visual) Odometry

• Coco-LIC and FAST-LIVO2 obtain the strongest LIO/LIVO average ranks, but no method dominates across missions, and ARC-2 exposes substantial drift or divergence.The strongest LIVO methods weight LiDAR more heavily than vision, so the advantage of visual input remains ambiguous.Source: §7.1.2 · p. 17, Table 9 · p. 19

7.1.3 Multi-LiDAR-Inertial

• CTE-MLO is the most consistent multi-LiDAR method, with the best average rank and lowest ATE on all reported missions, but gains from multi-LiDAR sensing vary by method and scenario.MA-LIO and RESPLE-MLIO each lead on selected RTE results but degrade on other missions; RESPLE-MLO and FAST-LIO-MULTI fail on ARC-2.Source: §7.1.3 · p. 19, Table 10 · p. 19

7.1.4 Visual-Inertial Odometry

• Voxel-SVIO performs consistently well, but VIO remains sensitive to initialization and illumination, and divergence often persists when methods lack reinitialization or restart mechanisms.SPX-2 dynamic motion can disrupt stationary initialization; dark ARC-7 intervals cause multiple pipelines to diverge or fail, underscoring the importance of exposure control, failure detection, and reinitialization.Source: §7.1.4 · p. 20, Table 11 · p. 20

7.1.5 Kinematic-Inertial Odometry

• The onboard estimator and both MUSE variants achieve the strongest KIO average ranks, while mission-dependent drift and the gap between on-plane and off-plane accuracy highlight contact-estimation challenges.Contact-aware methods use proxy or learned contacts rather than ground-truth labels, and MUSE (Smoother) is offline-only; CAPO is competitive on some absolute-error results, including CON-4.Source: §7.1.5 · pp. 20–21

7.1.6 Ablation: Online-Calibration

• Online calibration produces mixed results rather than a consistent improvement: its effects differ across methods, missions, and error metrics.Calibration parameters may be weakly observable depending on motion excitation and geometric constraints, so online calibration should be treated as scenario-dependent.Source: §7.1.6 · p. 21

7.1 State estimation and localization

• LiDAR-inertial methods generally show stronger accuracy and drift stability than pure LiDAR odometry and especially visual-inertial odometry, while absolute trajectory errors distinguish methods more than relative errors on easier missions.The large-scale CON-4 mission exposes long-term drift, particularly in several VIO pipelines; Coco-LIC and FAST-LIVO2 are among the strongest LIO/LIVO methods, while Traj-LO is a competitive pure-LO baseline in favorable conditions.Source: Table 8 · p. 19, Table 9 · p. 19, Table 10 · p. 19

7.1.7 Summary: State estimation and localization

• Failure behavior varies by method: VIO fails most often on the hardest sequences, while LiDAR methods fail less often but show higher error metrics.Weaker local-odometry baselines tend to drift more, but strong local accuracy does not guarantee low accumulated error on long trajectories; robustness, initialization resilience, and recovery matter alongside per-sequence metrics.Source: §7.1.7 · p. 21
• Performance is highly sensitive to tuning, and configurations often transfer poorly across missions, even for methods described as simple, robust, or adaptive.The analysis identifies a need for methods that adapt to deployment environments.Source: §7.1.7 · p. 21
• Open-source methods build on Fast-LIO2 (Xu et al., 2022), DLIO (Chen et al., 2023), Vins-Mono (Qin et al., 2018), and LOAM (Zhang and Singh, 2017), but legacy complexity can obscure active algorithmic components.This complicates third-party evaluation and motivates reporting configuration choices alongside numerical results.Source: §7.1.7 · p. 21
• Evaluation protocols remain fragmented, and reported centimeter-level trajectory metrics depend on synchronization, alignment, output rate, and reinitialization assumptions.The study uses a common protocol and reports per-mission results to expose these sensitivities.Source: §7.1.7 · p. 21

LiDAR-based 3D mesh generation

• Integrated ImMesh (Lin et al., 2023a) and SLAMesh (Ruan et al., 2023) demonstrate online LiDAR mesh generation for radiance-field convergence, foothold placement, and mission planning.The methods increase environmental awareness and support these downstream applications.Source: §7.2 · p. 22, Figure 13 · p. 23

Multi-Modal Elevation Mapping

• MEM (Miki et al., 2022b; Erni et al., 2023) fuses depth measurements into local or global elevation-map representations; RGB images, semantic classes, and latent features can be added as separate layers.Source: §7.2 · pp. 22–23

WaveMap

• WaveMap (Reijgwart et al., 2023) fuses depth into a wavelet-based, multi-resolution 3D volumetric representation supporting occupancy queries, depth rendering, and high-rate planning.Source: §7.2 · pp. 23–24, Figure 13 · p. 23

Geometric Understanding

• Time-synchronized LiDAR and camera data provide ground-truth metric geometry for training and evaluating monocular depth networks in natural environments, with extension to stereo depth estimation.Source: §7.2 · p. 24

Physical Properties Detection

• Chen et al. (2024a) estimate terrain stiffness and friction from proprioceptive joint data, then train a vision-based decoder to predict those properties from images.Source: Figure 13 · p. 23, §7.2 · p. 24

Dynamic Point Segmentation

• Pre-trained image-segmentation models filter dynamic semantic classes by removing their depth measurements, improving volumetric mapping; generated masks are available on HuggingFace.Source: §7.2 · p. 24

NerfStudio

• Converters provide GrandTour data in NerfStudio-compatible formats for neural scene representation training and studies of sparse and dense depth integration in multimodal neural rendering.The cited format supports training neural scene representations (Tancik et al., 2023).Source: §7.2 · p. 24

NaviTrace

• NaviTrace (Windecker et al., 2025) turns selected GrandTour HDR images into a VQA benchmark for evaluating and fine-tuning vision-language models to generate navigation behavior.Human annotators provide navigation instructions and image-coordinate traces for selected key frames across different robot embodiments.Source: §7.3 · p. 24

MPPI Path Planning

• Recorded reference behavior lets blackbox optimization tune MPPI planner hyperparameters to reduce error between executed and predicted trajectories, producing robust path planning.The planner uses local traversability analysis computed with the MEM framework and a goal position to output safe actions toward the goal.Source: §7.3 · p. 24

Visual Navigation

• Dense geometric information enables MPPI-generated paths to arbitrary goals, expanding navigation-trajectory diversity and supporting state-of-the-art visual navigation with comparatively little training data.GrandTour also supports learning visual navigation costs, avoiding the need for specialized datasets previously collected for testing and benchmarking (Frey et al., 2023); synthetic paths are available on HuggingFace.Source: §7.3 · p. 24

GaussGym

• GaussGym (Escontrela et al., 2025) integrates GrandTour deployment environments into simulation through neural radiance fields and scene reconstruction, enabling fully simulated training of RGB-based visual locomotion policies.This real-to-sim approach enhances simulation capabilities.Source: §7.3 · p. 24

8 Discussion and limitations

• GrandTour currently covers only Switzerland, potentially biasing architectural styles, vegetation, and ambient lighting represented in the dataset.The dataset does not include revisits to the same environments across seasons, weather patterns, or times of day.Source: §8 · p. 24
• The sensor suite lacks radar, FMCW LiDAR, and event cameras.These are identified as emerging modalities not currently included in the dataset.Source: §8 · p. 24
• GrandTour currently provides neither annotated semantic labels nor dynamic-object segmentation masks, which the authors welcome as community-driven additions.Contributors are invited to add annotations on top of the released sequences.Source: §8 · p. 24

Conclusion

• GrandTour contains 49 sequences across varied environments, illumination, and weather, with centimeter-level GNSS poses and millimeter-level, time-synchronized total-station positions.Total-station positions use accurate extrinsic calibration.Source: §9 · pp. 24–25
• The benchmark evaluates 63 state-estimation methods across six missions and reveals mission-dependent robustness limits, initialization sensitivity, tuning effects, and transfer limitations in current kinematic-inertial estimators.Evaluations used consistent protocols in online, real-time configurations, with loop closure disabled.Source: §9 · p. 25
• Post-processed outputs and open-source tools facilitate access and analysis, while synchronized modalities support sensor fusion, depth estimation, real-to-sim transfer, perception, traversability assessment, and navigation.Applications include foundation-model training and self-supervised cross-modal learning; GrandTour also aims to bridge controlled laboratory experiments and real-world deployment.Source: §9 · p. 25

Future work

• Planned expansions include challenging missions, more environments and robot embodiments, Leica RTC-360 ground-truth scans, and sensors such as solid-state and FMCW LiDARs, radar, and event cameras.The stated aim is to support embodiment-agnostic software and autonomy research.Source: §9 · p. 25