PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Accurate End-Effector Localization for UMI-Style Robotic Manipulation Teaching

Junjie Zhang, Deteng Zhang, Zhisong Xu, Bo Sun, Liuyang Li, Yihong Tian, Jie Yin

arXiv:2610.09857v1cs.RO

TL;DR

Close-range manipulation makes accurate and temporally complete end-effector localization important, but existing benchmarks and manipulation datasets provide limited evaluation of that sensing task. MILD supplies real and simulated benchmark data, and AprilVINS combines fisheye visual-inertial estimation with sequence-local tag geometry and guarded state export; its full route reports millimeter-level aligned error with high completion on Insta360 AprilTag4 recordings.

  • Problem

    Localization benchmarks emphasize navigation motions while manipulation datasets prioritize policy learning, leaving manipulation-interface localization comparatively under-evaluated.

    Source: §I · p. 1

  • Method

    MILD provides real and simulation sequences with per-execution end-effector references, while AprilVINS combines fisheye VIO, sequence-local tag priors, and guarded trajectory export.

    Source: §I · p. 1

  • Results

    On Insta360 AprilTag4 recordings, AprilVINS(full) reports millimeter-level SE(3)-aligned TCP-relative APE RMSE with high time completion, while VIO without tag factors remains at centimeter scale.

    Source: §VI · p. 8

  • Takeaways & Limitations

    Replayable teaching trajectories require evaluating accuracy, coverage, and continuity together because low RMSE alone is insufficient.

    Source: §I · p. 2, §VI · p. 8

Abstract

from arXiv · show

Robot demonstration learning requires accurate and temporally complete end-effector localization during close-range manipulation and camera occlusion. Existing SLAM benchmarks emphasize navigation motions, whereas manipulation datasets prioritize policy learning over localization evaluation. We introduce MILD, a Manipulation-Interface Localization Dataset with real-world and simulation sequences. The real-world subset provides 86 sensor sequences from Insta360 X5 and Insight9 across 15 repeated tabletop tasks, calibration assets, and a per-execution robot end-effector reference trajectory. The simulation subset, MILD-Sim, extends task coverage in Isaac Sim for controlled manipulation-replay studies. Benchmarking visual-inertial and fiducial-aided systems on instrumented real-world recordings reveals large differences in both TCP-relative trajectory error and temporal coverage, even under the same nominal task. To support marker-augmented teaching workspaces without a pre-surveyed fiducial map, we present AprilVINS, which combines fisheye visual-inertial estimation with sequence-local AprilTag geometry and separates prior admission from guarded export of the jointly optimized state. On Insta360 AprilTag4 recordings, AprilVINS(full) under a unified protocol with sequence-specific profiles reaches millimeter-level SE(3)-aligned TCP-relative APE RMSE with high time completion and lower reported error than the tested routes under their respective protocols, whereas fisheye VIO without tag factors remains at centimeter scale. Ablations separate accuracy from exportability, and a MILD-Sim replay study provides task-specific tolerance references for interpreting those error magnitudes. Together, MILD and AprilVINS provide a diagnostic benchmarking framework for UMI-style demonstration collection. Code, datasets, and evaluation manifests will be released upon acceptance.

I. INTRODUCTION

Close-range manipulation challenges localization through occlusion, limited visual features, and slow-motion initialization failures, while existing benchmarks and manipulation datasets leave this sensing problem under-evaluated. MILD benchmarks manipulation-interface localization, and AprilVINS adds sequence-local tag geometry with guarded trajectory export.

  • A low-error trajectory fragment is insufficient when the rest of a demonstration is untracked, so evaluation must include accuracy and temporal coverage.Source: §I · p. 1
  • Existing localization benchmarks target aerial, handheld, and mobile-robot motion, while manipulation datasets prioritize policy learning rather than localization evaluation.Source: §I · p. 1
  • MILD pairs repeated tabletop tasks, multi-sensor recordings, scene variants, calibration assets, and per-execution end-effector references with a simulation subset.Source: §I · p. 1
  • AprilVINS estimates sequence-local AprilTag geometry without a surveyed map, admits tag priors into fisheye VIO, and guards export of the optimized trajectory.Source: §I · p. 1
  • The evaluation protocol combines APE RMSE, time completion, and continuity across localization routes, using MILD-Sim replay curves as task-specific tolerance references.Source: §I · p. 2

II. RELATED WORK

Prior work covers visual-inertial estimation, broader localization benchmarks, manipulation datasets, and fiducial-aided systems, but MILD targets their less-evaluated intersection: close-range manipulation-interface localization.

A. Visual–Inertial Localization

  • Visual-inertial systems combine feature tracking, inertial preintegration, and sliding-window optimization, but close-range demonstrations add short paths, occlusion, and end-effector conversion requirements.Source: A. Visual–Inertial Localization · p. 2

B. Localization Benchmarks and Multi-Sensor SLAM

  • Existing localization datasets broaden sensor and degradation coverage, while MILD adds repeated manipulation-workspace variants and executed end-effector references.Source: B. Localization Benchmarks and… · p. 2

C. Manipulation and UMI-Style Datasets

D. Fiducial-Aided Localization

  • Fiducial visibility alone does not ensure replayability; map consistency, synchronization, fisheye projection, and temporal coverage also matter.Source: D. Fiducial-Aided Localization · p. 2

A. Dataset Scope

MILD evaluates end-effector recovery using each task–scene–sensor recording and its own executed TCP reference. Its real-world recordings vary task, sensor, and workspace conditions to distinguish sensing effects from task geometry.

A. Dataset Scope

  • The evaluation unit is a task–scene–sensor recording with calibration and its own executed TCP reference, derived from logged robot motion rather than an external motion-capture system.Source: A. Dataset Scope · p. 2
  • Repeating nominal tasks across scene and sensor variants separates task geometry from sensing conditions, while raw observations and calibration support multiple estimator types.Source: A. Dataset Scope · p. 2

B. Real-World Data

  • The real-world release contains 86 recordings—67 Insta360 X5 and 19 Insight9—across 15 tabletop tasks and uninstrumented, textured, and fiducial-marker workspaces.Source: B. Real-World Data · p. 2
  • Uninstrumented scenes test natural-feature tracking, tablecloth changes texture, and marker layouts test sequence-local map construction without surveyed global poses.Source: B. Real-World Data · p. 2
  • TCP-relative error includes residual calibration, synchronization, and robot-state uncertainty because references come from robot kinematics and encoder feedback.Source: B. Real-World Data · p. 3

C. Simulation Data

MILD-Sim provides controlled manipulation replay episodes, while AprilVINS combines sequence-local tag geometry with a visual-inertial estimate and separates tag-prior admission from state export.

C. Simulation Data

  • MILD-Sim uses Isaac Sim for five manipulation tasks with observable robot, object, and trajectory states; quantitative localization results are from real-world data.Source: C. Simulation Data · p. 3

A. Overview

  • AprilVINS estimates sequence-local tag geometry and inserts admitted tag-pose priors into a fixed-lag sliding window before separately guarding the exported joint state.Source: A. Overview · pp. 3–4

B. State Representation

  • The state representation includes body orientation and position, velocity, IMU biases, and a transform aligning the VIO frame with the tag-map frame.Source: B. State Representation · p. 4

C. Fisheye Tag Observation Model

AprilVINS models AprilTag observations in fisheye images as camera-frame unit-sphere bearings. It predicts corner rays from tag geometry and evaluates their residuals on the tangent plane of the observed ray.

C. Fisheye Tag Observation Model

  • AprilTag observations in fisheye images are modeled as camera-frame unit-sphere bearings obtained by back-projecting and normalizing measured pixels.The pixel back-projection uses the calibrated fisheye inverse projection.Source: C. Fisheye Tag Observation Model · p. 4
  • Predicted camera rays use known tag-corner coordinates, the estimated sequence-local tag pose, normalized projection, and camera-to-body extrinsics.Source: C. Fisheye Tag Observation Model · p. 4
  • Ray residuals are evaluated on the observed ray’s tangent plane, avoiding a pinhole approximation near the image periphery and preserving panoramic-camera consistency.Source: C. Fisheye Tag Observation Model · p. 4

D. Sequence-Local Tag Layout and Pose Prior

AprilVINS estimates tag layout and body poses locally from known tag IDs and physical sizes, using only observations available up to the current frame. The resulting tag-based pose becomes a gated prior for the VIO state and map–VIO alignment.

D. Sequence-Local Tag Layout and Pose Prior

  • Known tag IDs and physical sizes let the fiducial front end estimate a sequence-local layout and framewise body poses from observations available up to the current frame.Source: D. Sequence-Local Tag Layout and Pose… · p. 4
  • The first accepted tag fixes the map gauge, and subsequent optimization uses only the connected observation graph available at that frame, without future frames.Users can place tags without measuring their global poses.Source: D. Sequence-Local Tag Layout and Pose… · p. 4
  • The solved tag-based body pose is used as a gated prior on the VIO state and map–VIO alignment, with translational and rotational confidence encoded separately.Those confidence terms define the tag-prior loss.Source: D. Sequence-Local Tag Layout and Pose… · p. 4

E. Tag-Prior Admission and Guarded Output

AprilVINS admits tag-pose priors into the existing visual–inertial sliding-window objective only after consistency checks, then uses guarded continuation to determine which optimized state can be exported. The reported TCP trajectory is composed from the accepted joint solution, and rejected stages can reduce temporal coverage.

E. Tag-Prior Admission and Guarded Output

  • Tag-pose priors enter the visual–inertial sliding-window objective only when synchronized poses pass geometric, timestamp, and SE(3)-consistency checks.The insertion gate modifies residuals in the existing fixed-lag window rather than adding a second backend optimizer.Source: E. Tag-Prior Admission and Guarded… · p. 4
  • Guarded continuation starts each stage from the previously accepted state and applies a predefined inertial-weight schedule, accepting a state when a stage succeeds.If no stage succeeds, no sample is exported.Source: E. Tag-Prior Admission and Guarded… · p. 4
  • The output pose is composed from the last accepted tag-assisted visual–inertial solution, not exported as a raw marker-only pose.The pipeline applies guarded continuation before TCP conversion.Source: E. Tag-Prior Admission and Guarded… · p. 4, Figure 4 · p. 5
  • Prior rejection protects the window from inconsistent tag poses, while continuation-stage rejection prevents unstable optimized states from being exported; either decision can affect temporal density without necessarily changing aggregate RMSE.The evaluation therefore considers APE RMSE alongside time completion and inter-sample continuity.Source: E. Tag-Prior Admission and Guarded… · p. 5

V. EXPERIMENTS

The experiments compare localization routes across manipulation tasks using accuracy, temporal coverage, and continuity, then examine marker sensitivity, AprilVINS design, practical teaching routes, and replay utility. The protocol treats low error as useful only alongside adequate coverage and continuity.

V. EXPERIMENTS

  • The evaluation asks how localization routes compare across tasks, whether marker count improves results, where AprilVINS gains arise, how it compares with teaching devices, and how errors relate to replay tolerance.The five questions span task-level benchmarking, marker configuration, AprilVINS design, practical routes, and downstream utility.Source: §V · p. 5
  • APE RMSE, time completion, and continuity are measured against the executed TCP reference using SE(3) alignment without scale fitting and a 0.05 s association tolerance.Time completion is the estimated timestamp span clipped to the reference interval, divided by reference duration.Source: §V · p. 5
  • Low RMSE must be read with coverage and continuity because it can reflect only a short, easy segment of the trajectory.Routes with gaps larger than 0.5 s are marked †; AprilVINS(full) uses sequence-specific profiles selected under a unified rule set.Source: §V · p. 5

A. Task-Level Benchmarking

On the tested AprilTag4 recordings, marker-free fisheye VIO remains at centimeter-scale error while AprilVINS(full) reaches millimeter-scale error; route failures and fragmentation also vary across tasks. Increasing marker count alone is not a reliable improvement, and AprilVINS(full) improves across the reported AprilTag2-to-AprilTag4 comparisons.

A. Task-Level Benchmarking

  • Under tested AprilTag4 recordings, marker-free fisheye VIO remains near 30–80 mm RMSE with high completion, while AprilVINS(full) reports 2–10 mm on the same sequences.Several baselines leave tasks without evaluable trajectories or diverge after initialization under close-range fisheye motion (Table III).Source: A. Task-Level Benchmarking · p. 6, Table III · p. 6
  • Fig. 5 maps task difficulty by mean APE RMSE and completion; harder cases occupy the high-error, low-completion region.Its points summarize the three lowest-RMSE evaluable non-AprilVINS routes per task.Source: Figure 5 · p. 6, A. Task-Level Benchmarking · p. 6

B. Marker-Configuration Sensitivity

C. AprilVINS Design Ablations

AprilVINS(full) reports the lowest aligned RMSE among the compared teaching routes with full time completion, although the comparison uses different hardware. Ablations distinguish the accuracy contribution of admitted tag factors from the output gate’s role in trajectory continuity.

C. AprilVINS Design Ablations

  • On the same four-tag tasks, tag-assisted configurations reach millimeter-level aligned RMSE while VIO without tag factors remains at centimeter scale; the output gate separates accuracy from exportability.AprilVINS(full) removes fragmentation markers retained by the ungated route, including gaps larger than 0.15 s even when span completion is high.Source: C. AprilVINS Design Ablations · p. 6, Figure 6 · p. 7

D. Practical Teaching Routes

  • AprilVINS(full) has the lowest aligned RMSE among the three compared routes with full time completion on overlapping tasks, but the cross-device comparison is not controlled.The overlap set omits Box 02 and Pick-and-place 01–03; the table compares Insight9 SDK VIO, PICO Controller, and AprilVINS(full).Source: Table V · p. 7, D. Practical Teaching Routes · p. 7

E. Downstream Replay Study

Controlled MILD-Sim replay perturbations produce task-specific relationships between translational error and success, with tolerance knees ranging from a few millimeters to larger offsets. The study supplies context for interpreting localization errors rather than directly validating estimator outputs.

E. Downstream Replay Study

  • The study adds zero-mean cubic B-spline translation noise to fixed successful episodes while keeping orientations and gripper commands unchanged.Each error bin contains 100 valid trials, and task failures count as unsuccessful replays.Source: E. Downstream Replay Study · pp. 7–8
  • 3 mm for Peg Insertion Side and 19 mm for Ballistic Toss mark the span of the dashed 50% replay-success thresholds across six tasks.Peg Insertion Side reaches 36% success at 3 mm; Cyclic Tabletop Rearrangement reaches 38% at 10 mm.Source: Figure 8 · p. 7, E. Downstream Replay Study · p. 8

VI. CONCLUSION

  • MILD-Sim replay curves provide task-specific tolerance references without replaying the benchmarked trajectories, so they contextualize reported errors rather than validate estimator outputs.The conclusion frames MILD and AprilVINS as a diagnostic framework for UMI-style demonstration collection.Source: §VI · p. 8

Related papers

Loading 2610.09857v1…