Combined Image- and World-Space Tracking in Traffic Scenes

I. INTRODUCTION

• Precise 3D localization and trajectory estimation matter for safe navigation, but most existing vision-based tracking methods operate only in the image domain.The introduction highlights object extent and orientation, especially for objects close to the camera.Source: §I · p. 1
• The method combines 2D detections and 3D proposals in a loosely coupled 2D-3D Kalman filter, with competitive KITTI results and additional evaluation of 3D precision.The filter maintains image- and world-space estimates, allowing distant tracks to continue with more precise close-range information.Source: §I · p. 1

II. RELATED WORK

• Unlike monocular tracking-by-detection methods, this approach performs tracking jointly in 2D and 3D to provide object position and spatial extent for robotics tasks.The related-work passage frames 3D position and extent as desirable for path planning and obstacle avoidance.Source: §II · pp. 1–2
• The method uses category-agnostic proposals from stereo images rather than expensive LIDAR systems, whose object candidates provide boundaries, positions, and shapes but no category information.Prior work using 3D measurements for LIDAR tracking precision motivates testing whether noisy stereo depth can also improve 3D tracking.Source: §II · p. 2

III. METHOD OVERVIEW

• The pipeline combines object detections with stereo-based, class-agnostic 3D proposals that provide object position, physical size, and a point-cloud segmentation mask.The proposals can separate objects from surrounding background or occluders more effectively than points inside a detection box alone.Source: §III · p. 2, Figure 3 · p. 3
• A CRF selects observations by scoring compatibility between their 2D and 3D information and excluding observations that share a detection or proposal.An observation consists of a 2D detection combined with a 3D object proposal.Source: §III · p. 2
• A second CRF selects trajectories from an over-complete set of tracking hypotheses, scoring them for image- and world-space consistency and preventing overlapping pairs.The selection determines which trajectories are reported as results.Source: §III · pp. 2–3
• Kalman filters track hypotheses with a joint 2D-3D state, weakly coupled through projection and back-projection so measurements in either space can help estimate the other.Tracking can use 3D measurements when available and continue without them when they are missing.Source: §III · p. 2

IV. OBSERVATION FUSION MODEL

• Before tracking, observation fusion combines 2D detections with 3D proposals, extending detections with 3D position and size measurements while selecting relevant proposals.The proposals are drawn from a much larger available set.Source: §IV · p. 3

A. Observation Models

• Image-space detections provide 2D bounding-box center, width, and height, along with class information and a detection score.The method uses image-space detectors such as and.Source: A. Observation Models · p. 3
• Stereo point-cloud clustering generates class-agnostic 3D proposals by finding depth clusters that remain stable across multiple scales.The approach searches varying scales because object classes are unknown, and merges overlapping clusters when bounding-box intersection-over-union exceeds 0.9.Source: A. Observation Models · p. 3
• The proposal method modifies by merging projected boxes on the ground plane and adding x- and z-elongated kernels to better localize and represent elongated objects.Source: A. Observation Models · p. 3
• Each 3D proposal carries position, scene-flow-derived velocity, size, and a score.The proposal's position and size estimates complement the class information supplied by detections.Source: A. Observation Models · p. 3

B. Observation Fusion

• The system combines class-bearing detections with 3D proposals that provide localization and object-size estimates, although proposals lack category information.A group of pedestrians can therefore be represented as one proposal or as individual pedestrians.Source: B. Observation Fusion · p. 3
• Gated detection–proposal associations become graph nodes with binary selection labels, and MAP inference in a CRF selects a consistent set.The association score considers proposal size, ground-plane distance, and agreement between projected proposal area and detection-box area.Source: B. Observation Fusion · p. 4
• A pairwise term penalizes overlapping observations and excludes pairs that claim the same proposal or detection.Its overlap component uses the number of shared 3D points in the proposals.Source: B. Observation Fusion · p. 4
• Detections without selected 3D proposals are added as partial observations, helping retain far-away targets beyond proposal coverage.The passage characterizes this selection stage as early fusion; late fusion would cause a combinatorial explosion in the tracker’s state space.Source: B. Observation Fusion · p. 4

V. TRACKING

• The tracking framework creates an over-complete set of hypotheses and selects the most suitable ones.Source: §V · p. 4
• The framework follows the hypothesize-and-select paradigm described as a current state-of-the-art approach for vision-based tracking.The framework is identified as initially proposed in.Source: §V · p. 4
• The tracker filters observations associated with trajectory hypotheses using the 2D-3D Kalman filter.Generating over-complete hypotheses captures multiple possible data associations.Source: §V · p. 4

A. Coupled Filtering of 2D-3D States

• The tracker keeps both image-domain and world-space object estimates, unlike vision-based methods that track only in the image domain.Source: A. Coupled Filtering of 2D-3D States · p. 4
• An Extended Kalman Filter estimates a joint 2D-3D state, coupling bounding-box and ground-plane position estimates through projection and back-projection.The state includes the 2D box and its rate of change, ground-plane position and velocity, and object size; box and ground-plane position use constant-velocity models.Source: A. Coupled Filtering of 2D-3D States · p. 4
• At each timestep, ego-motion corrects the bounding-box footpoint by back-projecting it to world space, applying 3D translation and rotation, then projecting it into the image.Source: A. Coupled Filtering of 2D-3D States · p. 4
• Fused observations update corresponding state quantities, while partial detections also update 3D position by back-projecting the box footpoint and set size to the category mean.The partial-observation update attaches a different measurement variance to 3D position and size.Source: A. Coupled Filtering of 2D-3D States · p. 5

B. Hypothesis Set Generation

• Each hypothesis is built from observations, constrained to the ground plane, and maintains a state estimate over time.It also records frame observations, which may be missing, and a category distribution over cars, pedestrians, and cyclists estimated by forward Bayesian filtering.Source: B. Hypothesis Set Generation · p. 5
• At each timestep, the tracker extends existing hypotheses with observations and starts alternatives at new observations within a temporal window.When detections are close in image space, on the ground plane, and in appearance, it branches a hypothesis across the different detections.Source: B. Hypothesis Set Generation · p. 5
• Hypotheses that leave the camera view are extrapolated for a while, preserving their inlier claims during optimization; duplicates and hypotheses unselected for some time are removed.Source: B. Hypothesis Set Generation · p. 5

C. Hypothesis Selection

• MAP inference in a CRF selects the hypothesis subset with the lowest energy.Each hypothesis has a binary indicator, with 1 meaning it is selected.Source: C. Hypothesis Selection · p. 5
• Observation scores use detection confidence, with past observations downweighted by exponential decay.The current formulation uses only the detection score for an observation.Source: C. Hypothesis Selection · p. 5
• Unary scores combine appearance, ground-plane motion, and image-box projection affinities, weighted by observation distance and category compatibility.The projection affinity is intersection-over-union between predicted and observed 2D bounding boxes.Source: C. Hypothesis Selection · p. 5
• Pairwise CRF terms penalize image-space overlap between hypotheses and observations shared by them.These interactions discourage conflicting hypothesis selections.Source: C. Hypothesis Selection · pp. 5–6
• The selected trajectories are intended to be physically plausible, without overlap in space-time.The selection partitions observations over time, unlike the earlier proposal-to-detection association step.Source: C. Hypothesis Selection · p. 6

Experimental Setup

• KITTI training sequences were split for MOTA-based Hyperopt parameter optimization and validation, using Regionlet detector results.Evaluations use CLEAR MOT metrics and the KITTI training set's public ground truth.Source: §VI · p. 6

Observation Precision

• Across positioning-error cases, the method is one of two best; 3DOP leads car lateral estimates, while DA and 3DOP trade strength across categories.The comparison includes depth and lateral error by distance range against GP-P, DA, and 3DOP.Source: §VI · p. 6, Figure 4 · p. 6

3D Localization Evaluation

• MOTP-3D precision is considerably better for the full method than the detection-only baseline, while MOTP-2D shows barely any difference.The comparison covers cars and pedestrians by distance range; lower MOTP-3D and higher MOTP-2D are better.Source: §VI · p. 6, Figure 5 · p. 7

Ablation Study

• In the 2D ablation evaluation, the full method performs best and each tested component contributes to performance.The variants remove scene flow, 3D proposals, or other pipeline components, or replace the proposals with depth analysis.Source: §VI · p. 6, Table I · p. 7

Exploiting Precise 3D Segmentations

• Accumulated precise 3D segmentations reconstruct more object shape than one frame reveals and separate objects from occluders with less noise.The paper accumulates measurements over tracking time using GCT representation and weighted ICP.Source: §VI · p. 7, Figure 6 · p. 7

KITTI Evaluation Results

• On the official KITTI test set, the method is on par with NOMT-HM for cars and ranks a close second to SCEA for pedestrians.SCEA is the top performer in the pedestrian category.Source: §VI · p. 7

Runtime

• The full pipeline takes 347 ms per frame, compared with 48 ms without observation fusion and the corresponding 3D proposals.Timing used a single-thread Intel I7 CPU and was not optimized; external components were excluded.Source: §VI · p. 7

• The proposed tracking pipeline combines 2D and 3D measurements and shows promising results with a clean, extendable design.Source: §VII · p. 7
• Table II reports KITTI benchmark results for MOTA, MOTP, ID-switches, fragmentations, and tracking-duration categories.Source: Table II · p. 8
• Future work will explore class-agnostic tracking of 3D proposals to identify obstacles beyond cars and pedestrians, including potentially unknown categories.Source: §VII · p. 8