Source-linked AI summary
EgoNeMo: Transferable Map of Pedestrian Dynamics via Egocentric LiDAR Scan
Azusa Sawada, Allan Wang, Hideo Saito, Aaron Steinfeld
TL;DR
Traditional MoDs are site-specific and require extensive observations, limiting the availability of motion maps at new locations. This paper trains a continuous MoD estimator across locations using egocentric LiDAR and sampling and frequency-learning strategies for sparse data. Experiments report immediate zero-shot transfer to unseen environments and reliable pedestrian trajectory prediction from one scan.
Problem
Traditional MoDs are site-specific and require extensive observations, while collecting motion data across positions is difficult, particularly in low-traffic environments.
Source: §1 · p. 1Method
The framework learns continuous motion distributions from egocentric LiDAR, using position-balanced sampling, joint frequency-score prediction, and visibility-aware frequency losses.
Source: §1 · p. 2, §3.1 · p. 3, §5 · p. 9Results
Takeaways & Limitations
The results support estimating transferable motion maps directly from 3D geometry for downstream pedestrian trajectory prediction.
Source: §1 · p. 2, §5 · p. 9Outputs in invisible regions are unreliable because they lack geometric cues and sufficient trajectory data for training.
Source: §4.3 · p. 8
Abstract
from arXiv · showhide
This paper proposes a transferable Map of Dynamics (MoD) framework that generalizes to unknown environments using only egocentric 3D LiDAR point clouds to overcome the long-standing limitation of traditional MoD methods. While MoDs are essential for encoding human motion characteristics to enable accurate pedestrian trajectory prediction or safe robot navigation, traditional approaches suffer from site-specificity, requiring exhaustive trajectory accumulation at every new location. Extending recent advances in neural implicit modeling, our framework trains a continuous, LiDAR-based MoD estimator across diverse environments. To mitigate the inherent sparsity and temporal bias of real-world trajectory data, we introduce a position-balanced sampling strategy and a multi-task learning architecture that jointly predicts motion distributions and a spatial frequency score map. The latter is further augmented by visibility-aware losses to compensate for incomplete observation data. Comprehensive experiments demonstrate that our method effectively reconstructs underlying motion maps even in unknown locations from a single instantaneous LiDAR scan, despite highly sparse training data. Finally, we show that our improvements enhance the reliability of downstream trajectory prediction.
1. Introduction
Traditional Maps of Dynamics (MoDs) require extensive site-specific observations, yet pedestrian motion information is important for trajectory prediction and robot navigation. This paper estimates continuous motion distributions from an egocentric LiDAR scan, training across locations to transfer to unknown environments; experiments report improved transferability and reliable downstream motion priors.
1. Introduction
- MoDs represent location-specific motion patterns as probability distributions and support pedestrian trajectory prediction and robot navigation.Semi-Wrapped Gaussian Mixture Models represent joint speed and angle distributions.Source: §1 · p. 1
- Traditional MoDs require extensive observations at each site, and sparse traffic, changing layouts, and limited grid-based representations constrain their deployment in new environments.Sequential updates still need time to accumulate enough trajectories after environmental changes.Source: §1 · pp. 1–2
- The framework learns relationships between motion dynamics and 3D geometry across locations, then estimates continuous motion distributions from a single instantaneous egocentric LiDAR scan.This avoids accumulating historical trajectories at every deployment location.Source: §1 · p. 2
- The paper introduces joint frequency-score prediction, grid sampling, and visibility-guided frequency losses to address sparse, spatially biased, and incomplete trajectory annotations.It also proposes a deterministic downstream algorithm for consistent, efficient trajectory prediction without stochasticity.Source: §1 · p. 2
- Experiments across known and unknown JRDB locations show improved transferability over a baseline MoD extending NeMo-map and BFF, with reliable motion priors for trajectory prediction.The paper presents these results as evidence that MoDs can be estimated directly from 3D geometry for unseen environments.Source: §1 · p. 2
2. Related work
Prior MoD approaches offer either transferable but discrete representations or continuous queries learned from site-specific observations. This work combines continuous motion modeling with transfer from egocentric LiDAR geometry, without relying on top-down images.
2. Related work
- Traditional MoDs include trajectory-, occupancy-, and velocity-based representations; CLiFF-map models multimodal flow with SWGMMs and can predict future trajectories.Grid-based methods can lose information, while later approaches use optimized cells, scene-graph locations, or neural implicit continuous queries.Source: §2 · p. 2
- BFF uses an occupancy-map CNN to infer directional patterns in unseen environments and Bayesian inference to update its geometry-informed prior with on-site observations.Earlier transfer efforts also used synthetic data or occupancy and floor-plan indicators.Source: §2 · p. 2
- BFF remains limited to grid-based discrete transitions, whereas the proposed framework supports continuous spatial queries and motion distributions from instantaneous egocentric LiDAR scans.The text links the LiDAR input to improved transferability and immediate response to environmental changes.Source: §2 · pp. 2–3
- The framework adopts lightweight LiDAR-only encoding of egocentric 3D structure rather than relying on static floor plans, binary occupancy maps, or BEV images.BEV images are not always accessible in general environments.Source: §2 · p. 3
- Prior approaches do not provide continuous, transferable MoDs together: BFF transfers through occupancy maps but uses discrete transitions, while NeMo-map learns continuous fields from on-site observations.The proposed method instead learns a transferable mapping from LiDAR geometry to MoDs across locations.Source: §2 · p. 3
3. Method
EgoNeMo learns relationships between pedestrian motion and LiDAR geometry across multiple environments, then estimates a Map of Dynamics from a single egocentric scan. Its architecture and training procedures address sparse observations while also supporting downstream trajectory prediction.
3. Method
- EgoNeMo learns motion–geometry relationships from paired LiDAR scans and motion samples across locations, then estimates MoDs at deployment from a single egocentric scan.Training and deployment are decoupled: the target environment requires no trajectory observations.Source: §3 · p. 3
3.1. Model architecture
- A PointPillars BEV encoder feeds queried features to an MLP motion head for continuous multimodal SWGMM distributions and a frequency head for spatial preference.Joint frequency-score optimization encourages naturally smooth motion patterns in flat regions.Source: §3.1 · pp. 3–4
3.2.1. Preprocess
- Preprocessing aggregates motion in sequence coordinates after ego-motion cancellation and builds frequency targets from 0.1 m pedestrian-box histograms, normalized and clipped to [0, 1].As datasets scale, the processing can be modified to capture within-sequence changes such as obstacle positions.Source: §3.2.1 · p. 4
3.2.2. Grid Sampling
- Grid sampling selects at most one accumulated motion sample per spatial cell, balancing positions that naive sampling would bias toward slow or stationary pedestrians.The bias arises because such pedestrians can contribute redundant samples at identical coordinates.Source: §3.2.2 · p. 4
3.2.3. Loss functions
- Visibility-mask-based PU and PNi modifications address sparse frequency labels that can misclassify unobserved regions, while SWGMM motion is trained with negative log-likelihood.The visibility-aware losses are evaluated to support robust motion patterns under sparse training data.Source: §3.2.3 · pp. 4–5
3.3. Motion Prediction Algorithm
- For trajectory prediction, deterministic mode-velocity updates reweight SWGMM priors around current velocity, replacing stochastic sampling associated with run variance and jerky paths.The predicted velocity angle mixes the selected mode’s angle with the current angle using a weight kernel.Source: §3.3 · p. 5
4. Experiments
The experiments evaluate transferable motion-map estimation and its use for trajectory prediction across known and unknown locations. The results report stronger transfer to unknown locations, robustness across data-scarcity settings, and practical limitations tied to the LiDAR crop.
4.1. Settings
- Experiments use JRDB with 10 known- and 17 unknown-location test sequences, plus SiT, and report NLL, ADE, and FDE under the stated evaluation protocols.NLL is averaged over 0.5 m grid-cell means within the central 10 m × 10 m region; trajectory metrics use up to seven past observations at 3 fps.Source: §4.1 · pp. 5–6
4.2.1. Benchmarks
- Our method achieves superior NLL performance on JRDB-Unknown and SiT, validating improved zero-shot generalization to unknown locations.Table 1 compares NLL with existing MoD methods; JRDB-Known* denotes three sequences with clear spatial overlap with training sequences.Source: Table 1 · p. 6, §4.2.1 · p. 6
4.2.2. Ablation study
- On unknown locations, grid sampling plus frequency prediction reduces the baseline Motion model’s NLL degradation; PU has the lowest direct-output NLL but inflated angular variance, while PNi has the best average fixed-variance NLL.Motion(GS)-Freq and Motion(GS)-FreqPNi show comparable NLLs on JRDB-Unknown and SiT.Source: §4.2.2 · p. 6, Table 2 · p. 7
- Motion-Freq produces smoother motion maps along flat-ground roads than Motion, while the predicted flow follows ground shape and approaches zero around cafe tables.Figure 4 compares motion outputs on unknown JRDB locations; arrows encode velocity modes.Source: §4.2.2 · p. 6, Figure 4 · p. 7
- The frequency-loss variants produce distinct outputs: PU maps are blocky, whereas PNi assigns higher frequency scores within visible regions than Freq.PNi excludes visible yet unlabeled areas from its loss calculation.Source: §4.2.2 · p. 6, Figure 5 · p. 8
- With fewer locations or shorter training durations, variants combining frequency prediction and grid sampling degrade less than Motion; with fewer locations, PU retains low fixed-variance NLL.The reduced-data evaluation uses the known test locations and the first half of each training sequence.Source: §4.2.2 · p. 6, Table 3 · p. 8
4.2.3. Trajectory Prediction Application
- The deterministic method achieves lower trajectory-prediction error than NeMo, but does not surpass ViTE; transferring MoD information to interaction-aware predictors is future work.The MoD-based predictors focus on environmental conditions and ignore inter-agent interactions.Source: §4.2.3 · pp. 7–8, Table 4 · p. 8
- MoD estimation takes 4.0 ms per LiDAR scan at 100 positions, and trajectory prediction takes around 23.6 ms for K = 3 modes and T = 12.The reported inference speeds were measured on a Quadro RTX 5000 and are described as fast enough for robot operation such as navigation.Source: §4.2.3 · p. 8
5. Conclusion
EgoNeMo estimates continuous motion dynamics in unknown environments from egocentric LiDAR scans, using position-balanced sampling and joint motion-frequency prediction to address sparse, temporally biased data. Its deterministic trajectory predictor selects and adjusts motion modes to maintain temporal consistency.
5. Conclusion
- The framework transfers a continuous LiDAR-based Map of Dynamics to unknown environments, and its experiments report reliable trajectory prediction from a single scan despite highly limited training data.Future work includes RGB-based semantic guidance and integration into trajectory predictors.Source: §5 · pp. 8–9
A. Motion Prediction Algorithm Details
- The predictor encodes the observed trajectory as positions and velocities, maps LiDAR points to BEV features, and queries an SWGMM motion distribution at each position.The SWGMM represents joint speed and angle distributions.Source: A. Motion Prediction Algorithm Details · p. 12
A.1. Common Steps
- At each step, it initializes velocity from an exponentially weighted average of observed frames, then iteratively updates the pedestrian’s position and velocity.The per-step decay rate is exp(−∆t/σt).Source: §A.1 · p. 12
A.2. Mode Reweighting (reweightDistribution)
- The predictor reweights the queried motion prior with a Gaussian transition probability centered on the previous velocity to penalize abrupt or physically unrealistic updates.The paper reports selecting the speed variance using parameter searches on ATC with NeMo-map.Source: §A.2 · p. 12
A.3. Mode Extraction (getVelocityMode)
- It selects the closest sufficiently weighted motion mode and rolls out K deterministic trajectories, reducing the sampling needed for statistically stable forecasts versus stochastic per-step sampling.K = 3 in the evaluation.Source: §A.3 · p. 12
A.4. Velocity Update (predictVelocity)
- The velocity update anchors angular adjustment to the observed heading rather than the previous predicted heading, a strategy reported to mitigate cyclic trajectory artifacts.The observed heading is iteratively updated from the observed trajectory.Source: §A.4 · p. 12
B. Experimental details
Experiments use JRDB for training and testing, with SiT reserved for testing unseen locations. The paper also specifies the model architecture and a modified training and evaluation procedure for Social-transmotion.
B. Experimental details
- JRDB testing covers 10 known-location and 17 unknown-location sequences, while SiT provides additional unseen environments and is not used for training.Five JRDB sequences are kept for a training sanity check.Source: B. Experimental details · p. 13
- The model uses a PointPillars-based BEV encoder, separate motion and frequency heads, and AdamW training on LiDAR scans sampled every 20 frames with random azimuth rotations.The BEV map is 100 × 100 with 64 channels; training runs for 100 epochs.Source: B. Experimental details · p. 13
- Social-transmotion is trained by shuffling the ego agent and evaluated separately for each agent, differing from its official code.Source: B. Experimental details · p. 13
C.1. Reliability at Boundary
The analysis partitions the BEV space into four concentric square zones centered on the sensor and reports that performance generally worsens toward the spatial boundary.
C.1. Reliability at Boundary
- Performance generally degrades toward the boundary across four non-overlapping square zones extending from 0 to 10 m from the sensor origin.The zones are bounded at 2.5 m, 5.0 m, 7.5 m, and 10.0 m on both absolute x and y coordinates.Source: §C.1 · p. 13
C.2. Occlusion or Open-space
The analysis reports no systematic NLL degradation across tested pedestrian configurations at identical locations, but finds outdoor motion-field estimation more challenging than indoor estimation. It also identifies occlusion and limited sensing range as constraints on estimates.
C.2. Occlusion or Open-space
- Estimates behind static obstacles rely on spatial extrapolation and are unreliable in unseen environments; pedestrian occlusions can pass partial geometric information through gaps, though dense occlusion degrades point clouds.Source: §C.2 · p. 13
- At identical locations, NLL shows no systematic degradation across tested pedestrian counts or occlusion levels; joint frequency-map training reduces worst-case NLL, and grid sampling improves heavily occluded frames.The authors attribute structural robustness largely to sequence-level training that aggregates motion samples into shared spatial targets.Source: §C.2 · p. 13
- Worst NLLs occur in low-density sequences, and outdoor motion-field estimation is more challenging than indoors because motion is diffuse and fixed-range crops omit distant geometry.The paper notes that attenuating the MoD prior in highly unconstrained environments can mitigate downstream prediction degradation.Source: §C.2 · pp. 13–14, Figure S2 · p. 14
D. Benchmark details
BFF* is marginally better in known environments, but the 3D LiDAR framework with training strategies outperforms it in unknown environments; on known JRDB sequences, transferable models achieve lower NLL than site-specific NeMo-map.
D.1. Occupancy map comparison
- BFF* extends the discrete BFF baseline to continuous SWGMM outputs from a height-filtered, bird’s-eye-view occupancy grid.The paper uses this continuous 2D-input variant as a comparative baseline because the original BFF and proposed method produce different output types.Source: §D.1 · p. 14
- In unknown environments, the 3D LiDAR framework with training strategies outperforms BFF*, although BFF* is marginally better in known environments.Tab. S4 compares input types; each value is the mean of three trials with standard deviations.Source: §D.1 · p. 14, Table S4 · p. 15
D.2. NeMo-map comparison
- Extreme long-horizon queries beyond the observable spatial crop increasingly rely on zero-padded features, a boundary limitation outside social-navigation scope.Source: §D.2 · p. 15
- The NeMo-map comparison removes temporal dependence from its query encoding to account for temporally clustered training observations per scene.The comparison uses three manually corrected overlapping JRDB train/test pairs.Source: §D.2 · p. 15
- Transferable BFF* and the proposed model achieve substantially lower NLL than NeMo-map on known JRDB sequences, despite all methods supporting queries at unobserved positions.The comparison uses aligned overlapping train/test pairs; only three pairs had clear spatial overlap and were manually corrected.Source: §D.2 · p. 15
D.3. Additional qualitative results
Additional figures visualize motion outputs and known-location evaluation. At 32 seconds, the proposed MoD has slightly lower errors in unseen environments, while ViTE has the lowest overall error across most metrics.
E. Trajectory Prediction
- At T = 96 (32 seconds), the MoD has slightly lower errors in unseen environments, while ViTE has the lowest overall error across most metrics.Unknown sequences have higher errors than known sequences for both methods.Source: E. Trajectory Prediction · p. 15, Table S6 · p. 16
- Figure S3 shows known-location NeMo-map evaluation, with gray dots marking LiDAR points labeled as obstacle classes.Source: Figure S3 · p. 16
D.3. Additional qualitative results
- Figure S4 visualizes predicted frequency scores and motion maps, with color brightness encoding frequency and arrow opacity, length, and hue encoding mixture weight, speed, and heading.The figure extends the motion visualization in Fig. 5.Source: §D.3 · p. 15, Figure S4 · p. 16