Source-linked AI summary
Learning Interaction-Aware Path Corrections from Unlabeled Trajectories: A LiDAR-Based Sim-to-Real GNN Framework for Construction Robots
Yilong Chen, Yong K. Cho
TL;DR
Construction robots need to anticipate workers’ motion, but existing social-navigation methods can depend on hand-crafted rules, annotated demonstrations, or planner-specific policies. This paper learns planner-agnostic path corrections from unlabeled demonstrations using a LiDAR-based GNN and EKF tracking. The reported evaluations show shorter, smoother trajectories and improved minimum TTC relative to baselines, alongside a favorable balance of navigation measures.
Problem
Social-navigation methods can require hand-crafted interaction rules or labeled data, while construction sites provide limited annotations and frame-wise LiDAR detections lack continuous object identities and motion states.
Source: §1 · p. 1, §3.1 · p. 4, §3.4 · p. 5Method
The framework learns residual path corrections from unlabeled demonstrations with a GNN and uses EKF-tracked LiDAR detections to construct online interaction graphs.
Source: §1 · p. 2, §3.4 · p. 5Results
Across the bricklaying trial and Dynablox sequences, the GNN achieved a favorable balance of path efficiency, smoothness, pedestrian clearance, and minimum TTC versus original paths, SFM, and MLP; it produced shorter, smoother trajectories and the highest minimum TTC across the sequences and trial.
Source: §6 · p. 10, §7.1 · p. 11Takeaways & Limitations
The reported results support using constant-velocity residuals as automatically generated targets for learning anticipatory path corrections from unlabeled demonstrations.
Source: §6 · p. 10Real-world validation covered a limited range of construction scenarios on one robot platform, and objective metrics do not capture workers’ perceived comfort, motion naturalness, or social acceptability.
Source: §7.2 · p. 11
Abstract
from arXiv · showhide
Construction robots operating near workers require navigation that can anticipate human motion while remaining robust to dynamic and unstructured site conditions. However, existing social navigation methods depend on hand-crafted interaction rules, labeled demonstrations, or robot-specific control policies, limiting their scalability and transferability in construction environments. We propose a real-time, LiDAR-only framework that learns interaction-aware path corrections from unlabeled demonstrations. Constant-velocity extrapolation generates residual learning targets, while a GNN models relationships among the robot, pedestrians, and goal context. During deployment, an extended Kalman filter tracks LiDAR detections to construct dynamic graphs, and the GNN offsets planner-generated waypoints through a planner-agnostic interface. Within the bricklaying trial, the GNN reduced path irregularity by 39.2\% and increased minimum TTC. Across Dynablox sequences, it increased minimum TTC by 38.3\% while producing shorter, smoother paths than the baselines. Ablation studies supported a 1.0-s prediction interval and confirmed the benefits of target context, reflection augmentation, and feature normalization.
1. Introduction
Construction sites require robots to anticipate task-driven human motion while navigating changing, cluttered spaces with limited onboard sensing. The paper proposes a LiDAR-based GNN framework that learns interaction-aware spatial corrections from unlabeled demonstrations and applies them to planner-generated waypoints.
1. Introduction
- Construction sites are changing, cluttered spaces where workers’ task-driven motion and interactions make safe, efficient navigation challenging with limited onboard sensing and computation.Frame-wise LiDAR detections also lack persistent identities and motion estimates, while camera performance can be affected by illumination, dust, and visual occlusion.Source: §1 · p. 1
- Existing local planners handle geometric obstacles but have limited ability to distinguish pedestrian interaction configurations, while social-force methods rely on manually specified rules and parameters that may not transfer consistently.Learning-based alternatives also commonly require labeled actions, semantic information, or large synthetic datasets, and end-to-end control policies can be planner- or robot-specific [1].Source: §1 · p. 1
- The proposed framework automatically generates residual targets from unlabeled demonstrations, uses a GNN to model agent relationships, and predicts spatial corrections for planner-generated paths.At deployment, an EKF tracks LiDAR detections to provide continuous positions and velocities for online graph construction; the spatial-residual output is planner-agnostic.Source: §1 · p. 2
2. Related Work
Prior work addresses socially aware navigation, graph-based interaction modeling, sim-to-real transfer, and LiDAR tracking, but the paper identifies gaps in transferability and deployable construction-site perception. Its approach is motivated by the need to infer interactions from spatial observations and provide continuous object states without relying on camera data.
2.1. Socially aware robot navigation
- Conventional planners prioritize geometric feasibility, while behavior-cloning and Social Force Model approaches have transfer limits tied to demonstration distributions or scenario- and platform-dependent calibration [10, 24, 7, 5].Socially aware navigation also considers pedestrian comfort, natural interactions, and socially appropriate behavior.Source: §2.1 · p. 2
2.2. Graph neural networks for social navigation
- Existing social-navigation GNN approaches can encode agent interactions, but their reliance on semantically rich annotations and predetermined synthetic layouts constrains applicability where construction interactions are unlabelled [1].Message passing represents pairwise relationships through edge attributes linking nodes that describe agent states.Source: §2.2 · p. 3
2.3. Sim-to-real transfer for robot navigation
- Many sim-to-real navigation methods rely on visual observations, transfer complete control policies, or require extensive system-level adaptation, while studies often target indoor or general navigation rather than construction-site social navigation [28].Navigation transfer faces coupled perception, localization, actuation, and environmental changes, beyond visual or dynamics gaps alone.Source: §2.3 · p. 3
2.4. LiDAR-based dynamic object tracking
- Dynablox and occupancy-grid approaches detect dynamic points but do not provide persistent object identities and motion states, while a tracking alternative uses an additional camera modality [25, 9, 8].The paper motivates a LiDAR tracking layer to associate detections across scans and estimate positions and velocities for graph construction.Source: §2.4 · p. 3
3. Methodology
The methodology learns planner-agnostic interaction-aware spatial corrections from demonstration trajectories and constructs graphs from simulated or LiDAR-tracked agents. It combines robot-centric features, automatically generated residual targets, symmetry-aware training, and EKF tracking for deployment.
3.1. Research Problem
- The research problem is to learn interaction-aware corrections from unlabeled demonstrations and transfer them through domain-invariant graphs and real-world LiDAR tracking.The proposed model predicts spatial residuals rather than planner-specific control commands, and symmetry-aware augmentation is investigated for limited, imperfect data.Source: §3.1 · p. 4
3.2. Training data from human demonstration
- Training uses 36 screened smooth, collision-free demonstrations from a 74-trial simulated dataset spanning crossing, merging, overtaking, and group interactions.Participants controlled a simulated robot through two waypoints while avoiding workers; recorded commands were replayed to compute graph features.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3.1. Domain-invariant scenario-to-graph translation
- Graphs encode agents as nodes with robot-relative position, velocity, and a robot/pedestrian/goal indicator; directed edges encode pairwise displacement, velocity, distance, and orientation.Robot-centric states avoid dependence on global coordinates; the goal node is omitted at inference because LiDAR perception does not provide it directly.Source: §3.3.1 · p. 4
3.3.2. Planner-agnostic interaction-aware path
- Constant-velocity residuals provide automatic learning targets, and candidate samples are retained when a pedestrian is within 2.5 m and the residual-to-path angle exceeds 45°.The residual compares current position with the prior position extrapolated using prior velocity over the prediction interval; the interval was set to 1.0 s.Source: §3.3.2 · p. 5
3.3.3. Symmetry-aware GNN training
- The GNN predicts a two-dimensional path correction, and x-axis reflection augmentation is used to improve generalization across geometrically symmetric layouts.The model uses shared node and edge encoders, message-passing layers, Huber loss, and Adam optimization.Source: §3.3.3 · p. 5
3.4. LiDAR-based dynamic object tracking
- Dynablox detections are associated with EKF tracks to estimate continuous object motion states for graph construction and GNN input.Tracks are updated when detections satisfy spatial and temporal association thresholds; otherwise, a new filter is created and the old one deleted.Source: §3.4 · pp. 5–6, Figure 5 · p. 6
4. Experiments
The experiments pair a construction-specific bricklaying trial with an independent real-world LiDAR benchmark. The bricklaying setup tests robot-worker interactions in a shared corridor, while Dynablox extends evaluation to varied environments and trajectories.
4.1.1. Bricklaying trial
- The 5 × 7 m bricklaying testbed uses a 1.5 m cross-shaped corridor where a LiDAR-equipped quadruped and two brick-carrying workers repeatedly share the space.Workers received no predefined crossing or stopping directions, allowing interactions to arise naturally; onboard detection ran at 10 Hz.Source: §4.1.1 · p. 6, Figure 6 · p. 6
4.1.2. Dynablox dataset
- Dynablox provides eight real-world sequences from a 128-beam Ouster OS0 LiDAR across corridors, ramps, stairs, cluttered spaces, and multilevel structures.Seven sequences were tested; the indoor dataset was excluded because of its short duration, and evaluation used recorded LiDAR rather than closed-loop robot trials.Source: §4.1.2 · p. 6, Figure 7 · p. 6
4.2. Metrics
Navigation is evaluated through complementary measures of efficiency, smoothness, physical safety, and social compliance. The section also defines SFM and interaction-pooling baselines for comparing spatial refinement approaches.
4.2. Metrics
- Four metrics assess navigation: path length for efficiency, path irregularity for smoothness, minimum robot-pedestrian distance for physical safety, and minimum TTC for social compliance.TTC is computed by constant-velocity extrapolation for each robot-pedestrian pair and distinguishes converging motion from close but nonconflicting movement.Source: §4.2 · p. 7
4.3.1. SFM-induced spatial offset
- The SFM baseline offsets DWA waypoints with pedestrian repulsive forces and projects pedestrian positions over an interception interval, but relies on manually calibrated parameters.The paper predicts offsets for every frame, unlike the earlier study’s waypoint-only offsets.Source: §4.3.1 · p. 7
4.3.2. MLP-based interaction pool net
- The MLP pool-net baseline encodes robot-to-agent edges and aggregates them with attention-weighted, permutation-invariant maximum pooling, but lacks node-state encoding and iterative message passing.Comparing it with the GNN tests the value of structured node representations and multi-layer relational message passing.Source: §4.3.2 · p. 7
4.4. Ablation study
The ablations examine the prediction interval and three model components. They are designed to test how target context, reflection augmentation, and feature normalization affect learned interaction corrections.
4.4.1. Ablation on different prediction interval
- The prediction interval sets the physical meaning and magnitude of constant-velocity residual targets, with short horizons vulnerable to noise and long horizons mixing in unrelated motion.The ablation tests whether the interval captures interaction responses without excessive motion-model error.Source: §4.4.1 · p. 7
4.4.2. Ablation on different model variant
- The model-variant ablations test whether target context separates goal-directed motion from social corrections, reflection augmentation improves symmetric-layout generalization, and normalization stabilizes heterogeneous features.The study removes each component in turn to assess its contribution.Source: §4.4.2 · p. 7
5. Results
Across the bricklaying trial and Dynablox, the GNN improves interaction-safety measures and compares favorably with refinement baselines, with trade-offs in travel distance or smoothness. Qualitative results show context-dependent corrections, and ablations support a 1.0 s prediction interval and retaining the full model components.
5.1. Bricklaying trial results; 5.2. Dynablox dataset results
- The GNN reduced bricklaying path irregularity by 39.2% and raised minimum TTC from 0.44 to 0.51 s; on Dynablox, it raised minimum TTC from 1.28 to 1.77 s, a 38.3% improvement.Against refinement baselines, it produced the shortest and least irregular paths in bricklaying and the shortest, smoothest paths with highest minimum TTC on Dynablox.Source: §5.1 · p. 8, §5.2 · p. 8
5.3. Qualitative Analysis of Interaction Patterns
- The GNN corrected crossing paths before pedestrians passed, adjusted gradually during merging, largely preserved the route during overtaking, and deviated for group interactions.The examples indicate corrections varied with pedestrian configuration and relative motion rather than applying uniform repulsion.Source: §5.3 · p. 9, Figure 8 · p. 9
5.4.1. Ablation on different prediction interval
- A 1.0 s prediction interval consistently improved safety metrics over 0.5 s, while 2.0 s produced longer paths and lower minimum TTC on both datasets.The 1.0 s interval was selected for the most consistent balance across efficiency, smoothness, clearance, and collision avoidance, not for the best value on every metric.Source: §5.4.1 · p. 9
5.4.2. Ablation on different model variant
- Removing target context or reflection augmentation reduced minimum TTC, while removing normalization increased bricklaying path irregularity and reduced minimum TTC.The full model did not produce the shortest or smoothest path for every dataset, but it achieved the highest minimum TTC in both evaluations and a more consistent overall balance.Source: §5.4.2 · pp. 9–10
6. Discussion
The 1.0 s prediction horizon provided the most consistent balance across both datasets, while the GNN produced shorter, smoother paths and the highest minimum TTC across the Dynablox sequences and bricklaying trial. LiDAR tracking supplied the dynamic states used for real-time graph-based path refinement.
6. Discussion
- The 1.0 s horizon achieved the most consistent balance across both datasets; 0.5 s favored shorter paths but weakened anticipatory responses, while longer horizons produced increasingly long, irregular paths.Errors accumulated under the constant-velocity assumption at horizons longer than 1.0 s.Source: §6 · p. 10
- The GNN produced shorter and smoother trajectories than SFM and MLP while achieving the highest minimum TTC across the Dynablox sequences and bricklaying trial.Its smaller improvement in minimum pedestrian distance alongside a larger TTC improvement suggests adjustments to response timing and direction rather than simply maximizing clearance.Source: §6 · p. 10
- The target node helped distinguish goal-directed motion from interaction-induced deviations during training, while reflection augmentation and normalization improved transfer to unfamiliar configurations.The target node is omitted during deployment, and the spatial residual output can refine paths from different planners.Source: §6 · p. 10
- The Dynablox-EKF-GNN pipeline converted frame-wise LiDAR detections into continuous object positions and velocities for real-time interaction-aware path refinement.EKF tracking supported dynamic graph construction without camera observations, appearance-based identities, or offline trajectory processing.Source: §6 · p. 11
7. Conclusion
The framework learns interaction-aware path corrections from unlabeled demonstrations and was evaluated on a construction bricklaying trial and the Dynablox dataset. The authors identify limits in scenario and platform coverage, motion-target assumptions, and evaluation of workers’ social experience, with broader evaluations proposed as future work.
7.1. Summary
- The framework combines constant-velocity residual learning targets, a GNN, and EKF-based LiDAR tracking, and achieved a favorable balance of path efficiency, smoothness, pedestrian clearance, and minimum TTC against original paths, SFM, and MLP.Ablations supported a 1.0 s prediction interval and the benefits of the target node, reflection augmentation, and feature normalization.Source: §7.1 · p. 11
7.2. Limitations and future work
- Real-world validation covered a limited range of construction scenarios with one robot platform, and constant-velocity targets may not represent acceleration or sudden directional changes.Objective pedestrian-distance and TTC metrics do not capture perceived comfort, motion naturalness, or social acceptability.Source: §7.2 · p. 11
- Future work will evaluate more diverse tasks, layouts, interactions, and robot platforms, investigate more expressive motion prediction and target generation, and add human-centered experiments.The authors propose combining subjective assessments with objective pedestrian-distance and TTC measures.Source: §7.2 · p. 11
CRediT authorship contribution statement
Yilong Chen and Yong K. Cho contributed to the study across overlapping research and writing roles.
CRediT authorship contribution statement
- Yilong Chen is credited with the original draft, validation, methodology, investigation, formal analysis, data curation, conceptualization, visualization, and software.Source: CRediT authorship contribution statement · p. 11
- Yong K. Cho is credited with conceptualization, data curation, investigation, methodology, project administration, resources, supervision, and writing.Source: CRediT authorship contribution statement · p. 11
- Both authors are credited with conceptualization, data curation, investigation, methodology, and writing and editing.Source: CRediT authorship contribution statement · p. 11
Data Availability
The authors do not have permission to share data.
Data Availability
- The authors do not have permission to share data.Source: Data Availability · p. 11