PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Leaderboard Scores: A Deployment-Focused Protocol for Interpretable Tracking Evaluation in Pedestrian-Centric Environments

Dominik Wojcikiewicz, Diego Paez-Granados

arXiv:2610.01682v1cs.ROcs.CV

TL;DR

Mobile robots among pedestrians need timely, credible trajectories, while aggregate tracking scores and varying detector inputs limit insight into tracker behavior. The paper proposes a shared-detection protocol that evaluates specific tracking behaviors and embedded runtime alongside HOTA. On JRDB, trackers with similar HOTA show different capability profiles, and detector-gap continuation is the largest limitation among the tested properties.

  • Problem

    Aggregate scores provide limited insight into when and how trajectories fail, and varying detector inputs can confound tracker and detector quality in pedestrian-centric settings.

    Source: §I · p. 1

  • Method

    The protocol evaluates tracker behavior under shared detections using interpretable tests for initialization, gaps, recovery, close-neighbor association, and load-dependent runtime, with HOTA as a complementary measure.

    Source: §I · p. 1

  • Results

    On JRDB, trackers with similar HOTA differ across capability profiles, and detector-gap continuation is the largest limitation among tested properties.

    Source: §VI · p. 7, §VII · p. 8

  • Takeaways & Limitations

    The protocol characterizes tracker behavior and deployment suitability in pedestrian-centric environments through shared-input capability profiles and embedded runtime evaluation.

    Source: §I · p. 1

    The study covers one JRDB-trained detector operating point and one pedestrian-centric dataset, and runtime is limited to tracker-step execution on one embedded platform.

    Source: §VI · p. 7

Abstract

from arXiv · show

Mobile robots operating among pedestrians need trajectories that become available quickly, remain spatially credible through missed observations, preserve identity, and fit within an embedded computing budget. Aggregate tracking scores provide limited insight into when and how trajectories fail, while varying detector inputs can confound tracker and detector quality. We present a deployment-focused, tracker-only evaluation protocol that uses shared detections to isolate tracker behavior and directly evaluates initialization, detector-gap continuation, identity recovery, close-neighbor association, and load-dependent tracker-step runtime, while Higher Order Tracking Accuracy (HOTA) is retained as a complementary aggregate measure. We apply the protocol to the JackRabbot Dataset and Benchmark (JRDB) using six open-source trackers and our lightweight Pedestrian Reference Tracker (PedRefTrack), together with a GT-assisted variant that estimates the remaining tracker-side gap under idealized association and motion. Under fixed detections, the non-GT trackers span only 24.26%-29.67% HOTA yet exhibit markedly different capability profiles. After 1.0 s without detector support, no tracker without GT assistance maintains spatially correct, same-identity output in more than half of eligible cases, making missing-observation continuation the dominant limitation among the tested properties. Close-neighbor failures are smaller and increase mainly at the shortest separations. Tracker-step runtime on an NVIDIA Jetson Orin is heavy-tailed and load-sensitive, causing several trackers to fall below the 10 Hz real-time target in crowded frames. The protocol provides a reproducible way to characterize tracker behavior and deployment suitability in pedestrian-centric environments. Code and evaluation scripts are released at https://github.com/SCAI-Lab/tracker_eval.

I. INTRODUCTION

Pedestrian-centric robots need timely, spatially credible trajectories that preserve identity, but aggregate scores and differing detector inputs limit insight into tracker behavior. The paper motivates a shared-detection protocol that evaluates deployment-relevant tracking behavior beyond aggregate accuracy.

I. INTRODUCTION

  • Robots use recent trajectories to estimate pedestrian motion, assess collision risk, and choose socially compliant paths, so tracking must meet temporal, spatial, identity, and computing constraints.Deployment suitability depends on temporal behavior and failure conditions, not only aggregate tracking accuracy.Source: §I · p. 1
  • Roadway-focused 3D MOT evaluation does not characterize close-range pedestrian tracking, where nonlinear motion and crowd occlusions are common.Aggregate scores also provide limited insight into behavior across sequence complexity and deployment situations.Source: §I · p. 1
  • Different detector inputs confound attribution of performance to tracker design, while shared detections are generally not enforced in 3D MOT and robot-centric benchmarks.Shared detections are established in 2D MOT benchmarking.Source: §I · p. 1
  • The proposed tracker-only protocol uses common detections and GT- and detection-defined events to evaluate trackers under identical conditions.It also profiles embedded runtime across input loads and reports interpretable success profiles alongside HOTA.Source: §I · p. 1

A. Tracking requirements and evaluation

  • The protocol complements aggregate metrics with evaluation of the temporal conditions under which trajectories remain usable.MOTA, IDF1, and HOTA summarize detection, association, and localization differently, but do not directly capture all deployment-relevant temporal conditions.Source: A. Tracking requirements and evaluation · p. 2

B. 3D MOT families and representative open-source trackers

The paper situates its tracker comparisons among detection-based 3D MOT designs and pedestrian-centric datasets. These trackers differ in motion prediction, association, and lifecycle management, while robot-centric datasets represent complementary interaction regimes.

B. 3D MOT families and representative open-source trackers

  • The study focuses on online tracking-by-detection methods that consume per-frame 3D boxes, a dominant interface among open-source 3D MOT systems.Joint detection–tracking formulations instead integrate association into the detection model.Source: B. 3D MOT families and representative… · p. 2
  • Detection-based 3D MOT trackers span motion-and-assignment, confidence-aware lifecycle, geometry-aware, probabilistic, and learned-association designs.These families differ primarily in motion prediction, association, and lifecycle management, motivating comparison under common detector inputs.Source: B. 3D MOT families and representative… · p. 2

C. Datasets across tracking domains

  • Autonomous-driving benchmarks primarily represent vehicle-centric road traffic, whereas robot-centric pedestrian navigation features more close encounters and dense human interactions.SiT reports substantially more close-proximity robot–pedestrian encounters than several autonomous-driving datasets.Source: C. Datasets across tracking domains · p. 2
  • JRDB, SiT, THÖR, SCAND, and CrowdBot v2 cover distinct campus, urban, controlled indoor, social-navigation, and dense-crowd regimes.These datasets expose complementary pedestrian interaction settings.Source: C. Datasets across tracking domains · p. 2

III. DEPLOYMENT-FOCUSED TRACKER-ONLY PROTOCOL

The protocol fixes tracker inputs and event eligibility, then pairs HOTA with interpretable tracker profiles. PedRefTrack and its GT-assisted variant provide a reference for tracker behavior and the remaining gap under idealized association and motion.

A. Common inputs and matching

  • Shared detections are matched one-to-one to GT by XY-center distance with a 0.30 m gate, while tracker outputs are separately matched by oriented 3D IoU ≥0.30.Event eligibility, gap duration, and neighbor difficulty depend only on GT and shared detections, so all trackers face the same events.Source: A. Common inputs and matching · p. 3

B. HOTA and PedRefTrack

  • HOTA is the geometric mean of detection and association accuracy at a localization threshold, averaged over thresholds and decomposed into detection, association, and localization measures.The decomposition also reports detection and association recall and precision.Source: B. HOTA and PedRefTrack · p. 3
  • PedRefTrack combines a constant-velocity Kalman filter, score-aware confirmation, two-pass association, and adaptive published coasting based on recent prediction error.Its published horizon is bounded between 0.5 and 2.0 s and can be capped before predicted pedestrian encounters; hidden identity retention lasts longer.Source: B. HOTA and PedRefTrack · p. 3
  • GT-assisted PedRefTrack uses GT identity and displacement for association and motion while retaining shared-detection box geometry and finite lifecycle decisions.It is a diagnostic estimate of performance recoverable with more reliable association and pedestrian motion, not perfect tracking.Source: B. HOTA and PedRefTrack · p. 3

C. Trajectory-level success tests

The protocol defines four trajectory-level tests for usable tracking behavior. Their event eligibility and success criteria distinguish valid output during gaps from identity recovery afterward, while separately measuring initialization and close-neighbor association.

C. Trajectory-level success tests

  • Gap-continuation events require 0.5 s of detector support before a bounded gap, and success requires the pre-gap identity to remain spatially accepted at every evaluated time.Missing or spatially invalid output fails; internal identity retention without a published valid position does not count.Source: C. Trajectory-level success tests · p. 3
  • Post-gap recovery ignores output during the bounded gap and checks whether the first output after detector support returns has the pre-gap identity.This measures identity retention and re-association independently of spatial output during the interruption.Source: C. Trajectory-level success tests · p. 3
  • Close-neighbor association uses nonoverlapping 1.0 s windows after 0.5 s initialization, allowing detector gaps up to 0.2 s and requiring one accepted identity throughout.Failure is stratified by the 10th-percentile GT nearest-neighbor distance over each window, so brief close encounters remain represented.Source: C. Trajectory-level success tests · p. 3
  • Stable initialization is measured by whether a start within delay a of first detector support yields 1.0 s of uninterrupted, spatially accepted output under one identity.The reported curve is the cumulative fraction of trajectories stably established by each delay.Source: C. Trajectory-level success tests · p. 4

A. Data, detections, and evaluation

The evaluation uses a shared JRDB test setup and corrected 3D-IoU evaluator, so tracker results are measured on common detections but are not directly comparable to the official leaderboard.

A. Data, detections, and evaluation

  • The 27-sequence JRDB test split is evaluated in a common world frame using STEAM-LO odometry and shared post-NMS Person-MinkUNet detections with score s ≥0.5.The detector was trained on the JRDB training split, and no tracker uses a GPU.Source: A. Data, detections, and evaluation · p. 4
  • The official JRDB 3D-IoU routine was corrected for vertical overlap and oriented-footprint clipping, and reported results are not directly comparable to the official leaderboard.Source: A. Data, detections, and evaluation · p. 4

B. Compared trackers and runtime platform

The comparison covers six open-source trackers, PedRefTrack, and its GT-assisted variant, with tracker-step runtime measured on an embedded platform against a 10 Hz target.

B. Compared trackers and runtime platform

  • The study compares AB3DMOT, Fast-Poly, SimpleTrack, CBMOT, GNN-PMB, box-only ELPTNet, PedRefTrack, and GT-assisted PedRefTrack.Source: B. Compared trackers and runtime… · p. 4
  • ELPTNet's available implementation did not fully reproduce its paper architecture, so its documented output gate was enabled and prediction through missed detections was bounded.Source: B. Compared trackers and runtime… · p. 4
  • Exposed parameters were tuned to maximize HOTA, configurable output coasting was set to its longest duration before HOTA decreased, and success-rate bands use sequence-resampled 95% bootstrap intervals.Figure 3(b) reports sample counts for gap, nearest-neighbor, and runtime strata.Source: B. Compared trackers and runtime… · p. 4
  • Tracker-step runtime is measured on an NVIDIA Jetson Orin 64 GB in maximum-power mode, with 10 Hz used as a conservative real-time target for common spinning-LiDAR frame rates.Source: B. Compared trackers and runtime… · p. 4

C. GT-derived detector perturbations

The section describes controlled GT-derived detector perturbations and aggregate accuracy variation across JRDB scenes, including tracker-side deficits relative to a GT-assisted reference.

C. GT-derived detector perturbations

  • GT-derived pseudo detections progressively increase dropout and box instability, then combine matched severities so every tracker faces the same input difficulty.Instability includes persistent trajectory-level box-error hypotheses with probabilistic switching and smaller framewise jitter; the caption specifies perturbations to center, orientation, and dimensions.Source: C. GT-derived detector perturbations · p. 4, Figure 2 · p. 4, Table I · p. 4

A. Aggregate accuracy and deployment variation

  • 33.33% HOTA for GT-assisted PedRefTrack compares with 29.67% for PedRefTrack and 27.48% for Fast-Poly; non-GT methods span 24.26–29.67% HOTA.LocA ranges from 65.49–66.15%, and the passage attributes HOTA differences mainly to track-output and association behavior rather than localization.Source: A. Aggregate accuracy and deployment… · p. 4
  • Figure 4 pairs each sequence's HOTA with GT-assisted PedRefTrack to show tracker-side deficits, while its HOTA and frame-runtime ranges vary across sequences.The paired differences remove easy-versus-difficult sequence variation; the varying ranges indicate that tracker choice affects average accuracy and consistency across scenes.Source: A. Aggregate accuracy and deployment… · p. 4, Figure 4 · p. 5
  • Figure 3(b) reports counts supporting the detector-gap, nearest-neighbor, and runtime analyses.Source: Figure 3 · p. 5

B. Trajectory tests expose different failure mechanisms

Trajectory-level tests show distinct patterns in gap continuation, identity recovery, proximity-related continuity, and initialization among trackers evaluated on fixed detections.

B. Trajectory tests expose different failure mechanisms

  • At 1.0 s without detector support, only GT-assisted PedRefTrack maintains same-ID, spatially accepted output in a majority of eligible gaps, at 86.8%; SimpleTrack reaches 45.4%, PedRefTrack 37.0%, and other methods remain below 1%.Gap success reflects both motion prediction and output-lifetime policy.Source: B. Trajectory tests expose different… · p. 5, Figure 5 · p. 6
  • Identity recovery can substantially exceed valid output through gaps: at 0.5 s, Fast-Poly, GNN-PMB, and CBMOT recover pre-gap identity in 78.9%, 59.8%, and 75.1%, despite about 0.2% valid gap output.At 1.5 s, recovery ranges from 41.9% for CBMOT to 69.8% for PedRefTrack among the listed trackers; recovery contrasts reflect whether identity state remains dormant after coasting ends.Source: B. Trajectory tests expose different… · p. 5, Figure 5 · p. 6
  • As nearest-neighbor distance drops from 0.7–0.8 m to 0.3–0.4 m, continuity failure rises by 2.3 percentage points for PedRefTrack and up to 13.5 for GNN-PMB.Close-neighbor failures remain smaller than failures during 1.0 s gaps, with proximity most consequential below about 0.5 m.Source: B. Trajectory tests expose different… · p. 6, Figure 5 · p. 6
  • By 0.5 s, 39.1–56.0% of shared trajectory starts are stably established; at 0.2 s, Fast-Poly establishes 45.0% versus 29.4% for PedRefTrack.By 0.4 s, PedRefTrack reaches 46.5% versus 46.8% for Fast-Poly, with higher detection precision (41.3% versus 39.8%) and longer valid coasting, illustrating an availability–precision trade-off.Source: B. Trajectory tests expose different… · p. 6, Figure 5 · p. 6

C. Runtime depends on detection load

Tracker throughput falls with detection load, with several methods dropping below the 10 Hz target. Under increasingly severe GT-derived dropout and box instability, trackers separate in HOTA deficit, while similar aggregate robustness can mask distinct trajectory behaviors.

C. Runtime depends on detection load

  • At the 75–80 detection bin, PedRefTrack retains about 15.2 FPS, while CBMOT and ELPTNet also stay above 10 Hz at the highest displayed loads.Fast-Poly falls below 10 Hz at roughly 45 detections; AB3DMOT, GNN-PMB, and SimpleTrack do so at substantially lower loads.Source: Figure 6 · p. 7, C. Runtime depends on detection load · p. 7

D. Controlled GT-derived perturbations

  • Under increasing GT-derived corruption, HOTA deficits separate substantially; PedRefTrack and Fast-Poly show the smallest deficit growth.Deficits are compressed under mild corruption; similar aggregate robustness does not imply similar initialization, gap-continuation, or identity-recovery profiles.Source: Figure 7 · p. 7, D. Controlled GT-derived perturbations · p. 7
  • Similar aggregate robustness does not imply similar trajectory behavior: PedRefTrack and Fast-Poly have markedly different initialization, gap-continuation, and identity-recovery profiles.Tracker-step FPS is plotted against input detections, with 10th–90th quantile whiskers and a dashed 10 Hz line; detector inference is excluded.Source: D. Controlled GT-derived perturbations · p. 7, Figure 6 · p. 7

VI. DISCUSSION AND LIMITATIONS

The protocol profiles deployment-relevant tracker behaviors alongside HOTA, and JRDB results show that trackers with similar aggregate scores can differ across capabilities and runtime. Detector-gap continuation is the largest limitation among the tested properties, while the study’s scope is limited to one dataset and detector operating point.

VI. DISCUSSION AND LIMITATIONS

  • Trackers with similar HOTA can reach that aggregate performance through different trade-offs in initialization, missed-observation continuation, identity recovery, and runtime scaling.The protocol profiles these behaviors separately to identify which capabilities remain limiting in a pedestrian-centric environment.Source: §VI · p. 7
  • Detector-gap continuation degrades more severely than close-neighbor association, and several trackers recover identity more often than they maintain valid gap output.The discussion links gap success to motion extrapolation and lifecycle policy, and proposes richer pedestrian-motion models and separating published coasting from hidden identity retention.Source: §VI · p. 7
  • The study covers one JRDB-trained detector operating point and one pedestrian-centric dataset; its GT-derived perturbations do not reproduce the full error structure of a learned detector.The reported profiles also reflect HOTA-tuned configurations under selected matching thresholds and temporal horizons, and runtime excludes the remaining perception and navigation stack; tests are proxies rather than direct navigation measures.Source: §VI · pp. 7–8

VII. CONCLUSION

  • In JRDB, PedRefTrack remains competitive across aggregate accuracy and capability profiles, exceeds 10 Hz at the highest tested loads on one Jetson Orin CPU core, and detector-gap continuation is the largest tested limitation.Post-gap identity recovery remains much stronger, while close-neighbor failures increase mainly at short separations.Source: §VII · p. 8
Loading 2610.01682v1…