PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionDLO: Hybrid Event- and Frame-Based Tracking of Deformable Linear Objects

Annalena Hartmann, Priyamvada Ajithkumar, Patrick Bründl, Jörg Franke

arXiv:2608.22398v1cs.ROcs.CV

TL;DR

Fast, accurate, temporally consistent tracking of deformable linear objects remains difficult because existing approaches trade latency against spatial precision and identity continuity. MotionDLO combines event-based observations, frame-based SAM 3 segmentation, and CPD registration in a hybrid pipeline. Across 54 sequences, it reports 0.54 mm average point-to-curve error with 12 ms event-update inference and stable tracking across tested scenarios.

  • Problem

    Existing DLO tracking approaches leave robust temporally consistent perception under fast motion unresolved, while event-based methods can trade spatial precision for low latency.

  • Method

    MotionDLO combines asynchronous event-derived observations with frame-based zero-shot SAM 3 segmentation and a modified CPD tracker operating at 12 ms update intervals.

  • Results

    0.54 mm average point-to-curve error was achieved over 54 measurement sequences while maintaining 12 ms inference time per event update and eliminating identity switches observed in RT-DLO.

  • Takeaways & Limitations

    The hybrid pipeline supports stable long-horizon DLO tracking across tested dynamic scenarios for robotic manipulation.

  • Takeaways & Limitations

    The current formulation assumes a single actively manipulated DLO and can fail when multiple DLOs move simultaneously or independent event sources contaminate the motion mask.

Abstract

from arXiv · show

Reliably tracking moving deformable linear objects (DLOs) while simultaneously ensuring robustness, accuracy, and temporally consistent state estimation remains a fundamental challenge in robot perception. We introduce MotionDLO, a real-time tracking framework specifically designed to overcome these limitations in temporal continuity and latency. The method exploits the high temporal resolution and sparsity of event-based cameras and combines segmentation with the Coherent Point Drift (CPD) algorithm under the principles of Motion Coherence Theory. This integration enables temporally consistent shape estimation while maintaining a low computational overhead. Existing event-based tracking methods are typically computationally efficient but exhibit reduced accuracy compared to frame-based approaches, or alternatively compromise event sparsity to achieve competitive performance. To resolve this trade-off, we propose a hybrid event- and frame-based tracking architecture that preserves the complementary strengths of both sensing modalities. The event stream ensures high-frequency motion updates, while frame-based information stabilizes spatial accuracy and object identity. We demonstrate that the proposed framework reliably associates DLO instances across video sequences, enabling robust perception for robotic manipulation tasks. Experimental results validate real-time performance at 12 ms update rates and accurate shape tracking with an point-to-curve error as measurement of accuracy of up to 0.43 mm, supporting dynamic path adaptation during manipulation. The source code and demonstration datasets are publicly available.

I. INTRODUCTION

MotionDLO targets temporally consistent DLO tracking by combining asynchronous event observations with frame-based zero-shot segmentation and CPD registration. The hybrid design addresses the gap between low-latency event tracking and spatially accurate, identity-preserving frame-based perception.

  • Motivation: DLO perception supports robotic manipulation in manufacturing, surgery, and agriculture, but flexibility, weak texture, self-occlusion, entanglement, and changing curvature complicate tracking.These properties make reliable detection, tracking, and manipulation difficult across industrial and scientific applications.
  • Proposed Framework: MotionDLO integrates an asynchronous event branch with frame-based SAM 3 segmentation and CPD registration to combine high-frequency updates with spatially accurate, identity-stabilizing observations.Event observations are supplied to CPD within 12 ms, while SAM 3 provides lower-rate polylines for spatial anchoring.
  • Contributions: The framework is presented as computationally efficient and robust under severe motion blur, while temporally coherent correspondence estimation supports stable long-horizon tracking without re-identification failures.The unified formulation leverages complementary event- and frame-based spatiotemporal characteristics.
  • Limitations of Existing Methods: Existing segmentation methods struggle with thin structures because downsampling, restricted kernels, fixed-resolution masks, and limited throughput reduce fidelity or fail to meet closed-loop latency requirements.Reported segmentation systems range from 4–10 fps to 10–13 fps, while SAM 3 is approximately 30 ms per frame on NVIDIA H200.
  • Research Gap: Existing DLO trackers provide temporal continuity or zero-shot segmentation, but RGB-D dependence, small-displacement assumptions, training demands, and segmentation-only pipelines leave fast-motion tracking unresolved.TrackDLO can incur correspondence errors under fast motion, while SAM-based segmentation does not itself resolve temporal continuity.

B. Event-based Tracking

MotionDLO addresses the need for high-rate DLO state estimation while preserving identity and geometric continuity under asynchronous event and lower-rate frame observations. Its formulation targets fast motion regimes where conventional frame-based registration can fail.

  • Motivation: Event-based tracking offers low-latency, sparse observations, but existing approaches often trade spatial precision against event sparsity and rarely address deformable structures.The related-work discussion identifies a persistent accuracy–sparsity trade-off and limited coverage of deformable-object tracking.
  • Operation conditions and constraints: 2000 mm/s motion can produce roughly 40 mm displacement between consecutive frames, exceeding the DLO diameter by a factor of 40.This operating condition illustrates why conventional frame rates can undersample rapidly moving thin objects.
  • Operation conditions and constraints: 30 mm displacement within 50 ms can cause CPD registration failure and identity switches, while TrackDLO is limited to approximately 150 mm/s under its inter-frame-motion constraint.The cited scenario concerns DLOs moved by industrial robots or cobots.
  • Problem formulation: The proposed formulation restricts estimation to two-dimensional event-sensor projections and assumes each DLO is approximately inextensible.Each instance is modeled with an ordered node chain whose total arc length remains approximately constant.
  • Problem formulation: The hybrid problem is to estimate every DLO instance at event-camera update rates while preserving its physical identity across arbitrarily long sequences.The state is represented as ordered control nodes, with identity preservation requiring correspondence to the same physical DLO over time.

B. Overall Pipeline

The overall pipeline combines frame-based zero-shot segmentation with event-based motion observations and a geodesic CPD tracker. Frame-derived polylines provide spatial references, while the unified pipeline supports temporal tracking and RGB-only operation without depth.

  • Overall pipeline: MotionDLO uses a two-branch architecture in which RGB frames initialize and stabilize tracking while event observations provide fast intermediate updates.Branch selection is based on the detected DLO motion state, with frame input for static motion and event input for nonzero motion.
  • RGB-only extension: MotionDLO extends motion-coherence-based tracking to RGB imagery, eliminating the depth dependency of TrackDLO for settings where depth sensing is unreliable or noisy.The RGB frame branch uses high-quality segmentation masks instead of RGB-D point-cloud registration.
  • Frame branch: The frame branch uses multi-prompt SAM 3 segmentation, mask thresholding and de-duplication, mask union, morphological refinement, skeletonization, and ordered polyline fitting.The resulting centerline is converted into a fixed-length sequence of equidistant nodes for CPD-based tracking.
  • Polyline fitting: Junction-aware traversal pairs branches by tangent collinearity, preventing distinct DLOs from being merged at crossing points.Ordered pixel sequences are extracted by traversing the constrained graph from tips through unvisited edges.
  • Polyline fitting: The fitted centerline is smoothed, resampled to equidistant nodes, and represented as a polyline compatible with CPD-based tracking.Chord-length parameterization provides normalized arc positions for the spline fit and final polyline Yt.

D. Event Branch: Dynamic DLO

The event branch converts asynchronous motion changes into a compact geometric observation for CPD tracking. Filtering, polarity-specific timestamping, recentness mapping, and centerline extraction produce ordered points for dynamic DLO updates.

  • Event representation: Moving DLOs generate spatially coherent event activity along their edges, while static backgrounds remain comparatively silent.Event cameras respond asynchronously to local brightness changes, producing sparse observations under fast motion.
  • Event preprocessing: A hardware-accelerated spatio-temporal filter removes isolated events by retaining activations supported by neighboring events within defined spatial and temporal windows.The filtering targets sensor noise and periodic artificial-light flicker.
  • Tracking representation: Geometric event observations preserve an explicit ordered DLO representation that pixel-based optical flow and Gaussian, approximately linear Kalman filtering do not maintain under large deformations.The cited comparison motivates the use of geometric curve tracking for deformable objects.
  • Event representation: The pipeline stores ON and OFF events in separate per-pixel timestamp memories before transforming them into a recentness map based on the Surface of Active Events.An exponential decay constant controls how quickly older activity fades.
  • Event representation: Because only foreground presence is required, the polarity channels are collapsed into one recentness map before geometric extraction.The resulting representation is used to derive a binary DLO mask for subsequent processing.
  • Geometric observation: A PCA-based centerline provides robustness to local event gaps, while a connected skeleton path is used when sufficient connectivity exists.The representation covering the visible DLO extent is selected and uniformly resampled into ordered observation points X.

E. CPD Tracker

The CPD tracker extends TrackDLO with topology-aware regularization, motion warm-start, bounded noise estimation, soft two-pass registration, and visibility-aware correspondence for sparse asynchronous event observations.

  • Core formulation: The tracker builds on TrackDLO’s ordered-chain representation and MCT geodesic kernel, using arc length to preserve DLO topology during deformation.Geodesic distance prevents spatially adjacent but topologically distant segments from coupling when DLOs fold or self-intersect.
  • Core formulation: Chain Laplacian regularization penalizes differences between adjacent deformation vectors, enforcing local consistency along the DLO centerline.This complements the kernel’s global smoothness when event observations are sparse or unevenly distributed.
  • Motion and robustness: A velocity field updated from observed node displacements is propagated through the normalized geodesic kernel to provide a spatially coherent warm start.Geodesic averaging allows nearby regions to provide motion information to nodes with sparse or missing observations.
  • Motion and robustness: Noise variance is initialized from node spacing and constrained by a fit-tied upper cap, preventing runaway correspondence broadening while allowing larger deformation support.The cap uses the mean node-to-observation residual and ε = 20px2 as a margin for fast motion.
  • Registration: Two-pass registration combines a relaxed coarse trust region with soft Gaussian observation weighting, followed by tighter refinement and broader shape regularization.Low-weight observations shift toward the outlier distribution rather than being removed abruptly.
  • Registration: Visibility-aware priors anchor well-supported nodes to the pre-aligned configuration while allowing poorly supported regions to interpolate through the kernel.This prevents the main CPD pass from over-correcting already aligned regions.

F. Fusion Mechanism

The fusion mechanism synchronizes event and frame sensing, uses event activity to detect static periods, and alternates high-rate event tracking with SAM 3 re-initialization for accurate recovery.

  • Synchronization: Hardware triggers synchronize the RGB and event sensors, while a manifest associates each RGB frame with its event-clock timestamp.Cross-modal calibration additionally uses a homography estimated with DLT inside RANSAC, achieving 0.79 px reprojection error.
  • Branch roles: The event branch provides continuous estimates, whereas SAM 3 supplies geometrically complete but infrequent segmentation; neither branch is sufficient alone.The event tracker lacks observations when the DLO is static, while SAM 3 cannot sustain the required real-time temporal resolution.
  • Hibernation and recovery: When active event-mask pixels fall below ρ_thr, the system keeps the last event polyline, reinitializes SAM 3, and uses its segmented polyline to restart event tracking after motion resumes.This hibernation and re-initialization cycle improves results during static intervals and restores event-based tracking on movement.

A. Evaluation Procedure and Metrics

The evaluation jointly measures speed robustness, identity preservation, segmentation and geometric accuracy, and temporal update suitability rather than inference time alone.

  • Evaluation dimensions: The evaluation targets robustness at movement speeds up to 2000 mm/s, preservation of each DLO identity, and segmentation and geometric accuracy.These dimensions are derived from the stated problem requirements.
  • Evaluation dimensions: Real-time suitability requires an estimation interval Δt of 10 ms to support DLO motion at 2000 mm/s.The evaluation uses update steps when frame representations are unavailable.
  • Metrics: Tracking accuracy is quantified with average per-node error averaged over all update steps, together with a point-to-curve metric.The point-to-curve metric accounts for tangential drift that can make index-based correspondence overestimate geometric error.
  • Metrics: Identity switches are tracked by whether the identified DLO number changes, because inference-time comparisons alone can favor methods that lose identity in dense environments.The benchmark compares MotionDLO with TrackDLO, RT-DLO, and SAM 3 under controlled conditions.

B. Experiment Setup and Dataset Generation

The experiments use synchronized multi-camera acquisition, varied DLO types and thicknesses, controlled robot trajectories, and datasets spanning speed and multi-DLO interaction challenges.

  • Experiment setup: The setup compares RGB-D, RGB, and event sensing, with cameras mounted 200 mm from the workspace and aligned parallel to the scene plane.The RGB camera records at 2592 × 1944 pixels and 49 fps, while the RGB-D camera records at 1280 × 720 pixels and 30 fps.
  • Experiment setup: Cross-modal calibration accounts for differing resolutions and non-co-located optical axes by estimating a planar homography between the image planes.Intrinsic calibration is performed independently for both sensors before homography estimation.
  • Dataset design: The benchmark varies manipulation speed, DLO thickness, and multi-DLO interaction across three industrially used object types.The selected objects include plastic tubing, a PROFINET cable, and switchgear cabinet wire.
  • Dataset design: Each dataset contains 1,858 images, 6.2 GB of event data, timestamp assignments, three DLO types, and three speed levels.Sequences are acquired simultaneously with event and frame-based cameras during robotic manipulation.
  • Dataset design: The first dataset contains four overlapping DLOs per scenario and benchmarks hybrid and standalone branches against single-frame methods such as RT-DLO.A second depth-sensor dataset covers all three DLO types because TrackDLO cannot detect the 4 mm third type and only handles single-DLO scenes.

1) Identity Preservation in Multi-DLO Scenes

MotionDLO preserves the identity of a moving DLO in single- and multi-cable scenes by using temporal association and event-based motion selectivity. This supports stable tracking where per-frame segmentation or competing methods lose association under motion and blur.

  • Single-DLO scenes: MotionDLO maintains a continuous, identity-stable estimate in the single-cable sequence, while RT-DLO splits one DLO into several instances.The comparison used representative frames from the same sequences and counted detection and association errors across update timestamps.
  • Multi-DLO scenes: In multi-cable scenes, MotionDLO isolates the manipulated cable because only the moving DLO triggers events, unlike SAM 3 and RT-DLO.SAM 3 captures bundled foreground geometry but lacks temporal cues, while RT-DLO lacks inter-frame association.
  • Multi-DLO scenes: MotionDLO tracks one moving wire by design, so simultaneous manipulation of several wires cannot be distinguished.This scope follows the assumption that one moving wire is the relevant object for robotic manipulation.
  • Comparison under motion: TrackDLO tracking stops after 169 steps when its correspondence estimation fails to converge under larger motion.Its small inter-frame displacement assumption becomes limiting as motion grows, whereas MotionDLO uses an event-based branch for fast updates.

3) Tracking Accuracy across DLO Types and Robot Speeds

MotionDLO is evaluated across DLO types and robot speeds using arc-length paired and point-to-curve errors with timestamp-matched ground truth. It maintains low geometric error and temporal alignment during fast motion, while outperforming TrackDLO at the compared speeds.

  • Across robot speeds: At robot speed 50 %, MotionDLO achieves ¯e = 0.97 mm and ¯eptc = 0.58 mm, while tracking through the phase where TrackDLO diverges.At robot speed 5 %, MotionDLO reports ¯e = 0.91 mm and ¯eptc = 0.56 mm; its accuracy is essentially unchanged between the two speeds.
  • Temporal alignment: 0.00 ms temporal residual is measured for all three DLOs at robot speed 100 %, confirming temporal alignment between predicted and ground-truth centerline states.The evaluation uses asynchronous ground-truth timestamps during the stable-motion phase.
  • Across DLO types: MotionDLO achieves ¯eptc = 0.55 mm on DLO 1, ¯eptc = 0.43 mm on DLO 2, and ¯eptc = 0.78 mm on DLO 3 at robot speed 100 %.The evaluation uses 15 frames per DLO in the stable-motion phase and reports point-to-curve error alongside arc-length paired error.
  • Metric interpretation: Point-to-curve error is 48% lower on DLO 1, 58% lower on DLO 2, and 59% lower on DLO 3 than arc-length paired error.The passage attributes the difference primarily to tangential misalignment of node correspondences rather than normal-direction curve deviation.
  • Metric interpretation: MotionDLO maintains sub-diameter geometric fidelity, with point-to-curve error corresponding to 9.2%, 6.6%, and 19.5% of the diameters for DLOs 1–3.The comparatively larger relative value for DLO 3 is driven primarily by two transient frames involving spurious disconnected mask fragments.

D. Limitations and Failure Modes

MotionDLO combines event-driven updates with frame-based segmentation to achieve temporally coherent DLO tracking, but its current formulation remains bounded by single-instance assumptions, slow correction, and limited geometric representation.

  • Limitations and failure modes: The current formulation assumes one actively manipulated DLO and cannot disambiguate simultaneously moving or frictionally co-moving overlapping instances.Resolving this case would require instance-level event clustering or a multi-instance tracking back end, which the authors leave for future work.
  • Experimental results: 0.54 mm average point-to-curve error was achieved across 54 measurement sequences while maintaining 12 ms inference per event update.The dataset covered 1,858 frames, 6.2 GB of events, three DLO types, and three manipulation speed levels.
  • Experimental results: MotionDLO eliminates identity switches observed in RT-DLO and reduces fast-motion tracking failures relative to TrackDLO.Temporally coherent CPD correspondence estimation supports stable long-horizon tracking across tested scenarios.
  • Deployment scope: The method operates without task-specific training data and is intended to generalize across diverse DLO types without retraining or domain adaptation.The experiments included 4 mm single-core cable to 8 mm pneumatic tubing, with broader diameters addressed by scaling the camera field of view.
  • Limitations and future work: Approximately 1 s SAM 3 inference constrains the frame-based correction rate, motivating lighter segmentation models to increase correction frequency and tighten event-tracker drift bounds.This is identified as a future investigation direction rather than a demonstrated failure of the reported system.
Loading 2608.22398v1…