PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Human Trajectory Prediction from RGB-D for Mobile Robots Using Bi-Mamba

Basheer Al-Tawil, Magnus Jung, Ayoub Al-Hamadi

arXiv:2610.33916v1cs.ROcs.LG

TL;DR

Mobile-robot predictors face a train-deploy gap between smoothed training trajectories and noisy live RGB-D estimates. TrajMamba combines a shared sensor-processing pipeline with a lightweight bidirectional Mamba-GRU predictor. It reports the lowest average errors among eight ETH/UCY baselines and is demonstrated on a TIAGo robot.

  • Problem

    Training on clean, smoothed ground truth differs from deployment on noisy pose-detector and depth-sensor estimates, leaving predictors to encounter inputs absent during learning.

    Source: §I · p. 1

  • Method

    TrajMamba shares pose extraction, depth back-projection, and filtering across training and inference, with a compact bidirectional Mamba-GRU model for trajectory prediction.

    Source: §I · p. 1

  • Results

    0.29 m ADE and 0.51 m FDE; TrajMamba attains the lowest errors among eight baselines on ETH/UCY.

    Source: D. Comparison with State of the Art… · p. 6

  • Takeaways & Limitations

    The 262K-parameter model is small enough for real-time deployment, demonstrated on a TIAGo robot with an onboard Jetson Orin.

    Source: §I · p. 1, §V · p. 7

    The system tracks several people but predicts each person independently, leaving social interactions unmodeled.

    Source: §V · p. 7

Abstract

from arXiv · show

Predicting the short-term future trajectory of a person from RGB-D cameras is a key capability for mobile robots navigating in dynamic and human populated environment. The key challenge to deploying learned predictors on real robots is the train-deploy pipeline gap. Training data are often prepared with an idealized, smoothed ground truth, whereas at inference, the model is fed noisy estimates from pose detectors and depth sensors. The model is therefore evaluated on an input distribution it never encountered during training. Most existing models rely on recurrent neural networks (RNNs) or transformers, which either struggle to capture short-term temporal dependencies or suffer from computational cost. We present TrajMamba, a lightweight, bidirectional state space model for short-term human trajectory prediction using an RGB-D camera. The model uses a single shared processing path for both training and deployment, aligning human pose extraction, depth back projection, and smoothing processes. It encodes a short window of persons's hip center velocities with stacked bidirectional Mamba blocks and decodes future velocities with a GRU head. On the ETH/UCY benchmark, TrajMamba reaches an average ADE of 0.29 m FDE of 0.51 m. With 262K parameters, it runs in real time on modest hardware and is deployed on a TIAGo robot with an onboard NVIDIA Jetson Orin.

I. INTRODUCTION

Mobile-robot trajectory predictors face a mismatch between clean, smoothed training data and noisy live RGB-D estimates. TrajMamba addresses this with a shared processing pipeline and a compact model evaluated on ETH/UCY and deployed on a TIAGo robot.

I. INTRODUCTION

  • Training on clean, smoothed ground truth but deploying on noisy pose and depth estimates creates a train-deploy pipeline gap.The live input may also contain jitter and occasional gaps.Source: §I · p. 1
  • TrajMamba uses the same pose extraction, depth back-projection, and Kalman/EMA filtering in training and inference.The shared pipeline is intended to remove the train-deploy gap by design.Source: §I · p. 1
  • The 262K-parameter bidirectional Mamba-GRU model is evaluated on ETH/UCY and deployed in real time on a TIAGo PAL robot with an NVIDIA Jetson Orin.The complete system is released as an open-source ROS2 package with its training pipeline and trained checkpoints.Source: §I · p. 1

II. RELATED WORK

Prior pedestrian predictors commonly use overhead 2D trajectories or precomputed body poses, unlike a mobile robot’s forward-facing RGB-D setting. This work uses pose estimation and depth to predict motion from the camera perspective.

A. Pedestrian Trajectory Prediction

  • Many pedestrian predictors operate in the 2D top-down plane and cannot be directly applied to RGB-D camera-based robot environments.The paper prioritizes real-time prediction rather than sampling and ranking multiple stochastic futures.Source: A. Pedestrian Trajectory Prediction · p. 2

B. 3D Motion Forecasting

  • Some 3D motion forecasters require clean, precomputed skeletons, while a depth-based trajectory method relies on background subtraction that is fragile when the camera moves.TrajMamba instead integrates MediaPipe PoseLandmarker in both training and inference.Source: B. 3D Motion Forecasting · p. 2

C. State Space Models for Sequence Modeling

TrajMamba adapts bidirectional Mamba sequence modeling to RGB-D robot perception. It predicts future motion from normalized hip-center velocities and applies the same sensor-processing logic during training and deployment.

C. State Space Models for Sequence Modeling

  • Mamba uses content-dependent state and input transitions for data-adaptive memory, and bidirectional processing merges forward and backward passes through a learnable gate.The paper adapts this design to RGB-D perception while reducing parameter count and sharing sensor processing across training and deployment.Source: C. State Space Models for Sequence… · p. 2

A. Problem Formulation

  • The model takes first-order differences of 3D hip-center positions as input velocities and recovers predicted positions by cumulative summation.Velocities are standardized per spatial axis using training-set mean and standard deviation.Source: A. Problem Formulation · pp. 2–3

B. Pipeline-Consistent and Pre-processing

  • A single end-to-end module applies the same RGB-D synchronization, pose extraction, 3D back-projection, smoothing, windowing, and normalization logic in training, testing, and deployment.The preprocessing pipeline includes Kalman and EMA smoothing.Source: B. Pipeline-Consistent and… · p. 3, Figure 3 · p. 3

1) RGB-D Frame Association:

The preprocessing steps pair color and depth frames by timestamp and derive hip-center positions from pose landmarks and depth. Those positions are transformed into smoothed, normalized velocity windows for prediction.

1) RGB-D Frame Association:

  • RGB and depth frames are paired using the closest timestamps, with a maximum permitted difference of 20 ms.The threshold is chosen to mirror the live RealSense camera stream.Source: 1) RGB-D Frame Association · p. 3

2) Pose Detection:

  • MediaPipe PoseLandmarker locates the left and right hips, whose normalized coordinates are averaged and scaled by image dimensions to obtain the hip-center pixel.Video mode activates the detector’s timestamp-driven temporal smoothing.Source: 2) Pose Detection · p. 3

3) Depth Sampling and Back-Projection:

The pipeline estimates depth robustly from local image patches, filters noisy 3D coordinates, and constructs normalized temporal windows for prediction.

3) Depth Sampling and Back-Projection:

  • Depth is estimated as the median of valid values in a centered 7 × 7 patch, then converted to meters and back-projected with the pinhole camera model.Zero-valued depth readings are discarded; the TUM scale factor is pixel value / 5000.Source: 3) Depth Sampling and Back-Projection · p. 3

4) Smoothing:

  • A per-axis constant-velocity Kalman filter followed by an EMA filter smooths noisy 3D coordinates.The filters use q=10^-3 and r=10^-2, then EMA coefficient α=0.35.Source: 4) Smoothing · p. 4

C. Sliding-Window Dataset Construction

  • Tracks are downsampled from approximately 30 Hz to 10 Hz and split into overlapping 50-frame windows: 2 seconds observed and 3 seconds predicted.Each window is translated so its first frame is at the origin, emphasizing relative motion.Source: C. Sliding-Window Dataset Construction · p. 4

D. Model Architecture

TrajMamba uses a compact bidirectional Mamba encoder with a GRU decoder, embedding velocity inputs before processing both temporal directions.

D. Model Architecture

  • The model pairs a bidirectional Mamba encoder with a GRU decoder in a compact architecture designed for onboard robotic computation.Source: D. Model Architecture · p. 4

1) Input Embedding:

  • Each 3D velocity vector is linearly projected into d_model dimensions and layer-normalized before encoding.Source: 1) Input Embedding · p. 4

2) Bidirectional Mamba Encoder:

  • Each encoder block processes the sequence forward and backward, fuses the outputs with a learned per-channel gate, and uses input-dependent selective SSM parameters.The diagonal state matrix A is initialized from the HiPPO spectrum and refined during training.Source: 2) Bidirectional Mamba Encoder · p. 4

3) GRU Decoder:

The decoder conditions each future-velocity prediction on a global context derived from the full observed sequence.

3) GRU Decoder:

  • The final encoder state initializes the GRU hidden state and is repeated at each decoding step, conditioning predictions on the full observation.A two-layer MLP with dropout maps GRU outputs to future velocities.Source: 3) GRU Decoder · p. 4
  • Within each Mamba block, the input splits into an SSM-processing branch and a gate branch, whose gated product is projected and added to the residual.Source: Figure 4 · p. 4

E. Training Details

Training combines a robust velocity loss with regularization and training-only augmentation; evaluation uses both a small RGB-D dataset and the ETH/UCY benchmark.

E. Training Details

  • Training minimizes a Huber loss on normalized velocities plus an L1 penalty weighted by 0.1.Optimization uses AdamW and cosine annealing with warm restarts for up to 150 epochs.Source: E. Training Details · p. 4
  • Training-only augmentation scales velocities by ±10% and flips axes; data are split 85/15 for training and validation with seed 42.Source: E. Training Details · p. 4

A. Datasets

  • The RGB-D Bonn dataset contains one person's indoor motions recorded with an Asus Xtion Pro Live, yielding 145 windows split into 124 training and 21 validation windows.Trajectories are extracted from 580 RGB and 580 depth frames, then smoothed and downsampled to 10 Hz.Source: A. Datasets · pp. 4–5
  • The ETH/UCY visualization arranges five scenes by column and samples by row, distinguishing observed history, ground truth, predictions, and sampled prediction spread.Source: Figure 5 · p. 5, A. Datasets · p. 5
  • The ETH/UCY evaluation uses leave-one-scene-out testing with 8 observed and 12 predicted frames, while bypassing RGB-D pose, depth, and filtering stages.Benchmark positions are converted to velocities and standardized; predicted velocities are integrated from the last observed position.Source: A. Datasets · p. 5

B. Evaluation Metrics

ADE and FDE measure trajectory error over the prediction horizon and at its endpoint. Qualitative predictions follow straight paths closely but round off turns; the real-time figure depicts three motion directions.

B. Evaluation Metrics

  • ADE and FDE are computed in meters on integrated positions, using test sequences and the prediction horizon.The passage defines N as the number of test sequences and T_p as the prediction horizon.Source: B. Evaluation Metrics · p. 5
  • Figure 6 depicts real-time predictions for motion right-to-left, left-to-right, and toward the camera.Source: Figure 6 · p. 6
  • Table I compares state-of-the-art methods on ETH/UCY using ADE and FDE in meters.Source: Table I · p. 6

C. Visual Analysis of Trajectory Prediction

  • Turn predictions in ETH and UNIV round off sharp changes, while predictions in Zara1, Zara2, and Hotel closely follow mostly straight ground truth.The analysis attributes the gentler turns to committing to a single trajectory rather than representing several possible futures.Source: C. Visual Analysis of Trajectory… · p. 6

D. Comparison with State of the Art Methods

On ETH/UCY, TrajMamba records the lowest errors across five scenes among eight methods, with 0.29 m ADE and 0.51 m FDE. The complete system is demonstrated on a TIAGo robot, while modeling interactions between people remains future work.

D. Comparison with State of the Art Methods

  • 0.29 m ADE and 0.51 m FDE are the lowest errors across five ETH/UCY scenes among eight methods, including STGAT and Social-STGCNN.Evaluation used the standard leave-one-scene-out protocol with deterministic (K=1) predictions.Source: D. Comparison with State of the Art… · p. 6

E. Real-Time On-Robot Implementation

  • The complete RGB-D prediction pipeline runs on a TIAGo robot with an onboard NVIDIA Jetson Orin and renders observed histories and predicted paths across three motion patterns.The system extracts poses, back-projects depth, filters the estimates, and integrates predicted velocities into a 3 s trajectory.Source: E. Real-Time On-Robot Implementation · p. 7

V. CONCLUSION

  • The system tracks multiple people but predicts each independently; modeling their social interactions is identified as the next step.The authors plan to integrate TrajMamba into a social-aware navigation system.Source: §V · p. 7

Related papers

Loading 2610.33916v1…