UniTrackPLA: Unified Panorama-Language-Action Model for Instruction-Guided Navigation and Dynamic Person Tracking

I. INTRODUCTION

• Static navigation targets semantic goals, while person tracking requires adaptation to moving targets; both ground task-relevant entities and translate visual-language observations into motion decisions.This motivates a shared perception-language-action policy.Source: §I · p. 1
• Panoramic systems face a representation gap: perspective-pretrained encoders must retain transferable features while recovering circular spatial structure and temporal evolution.Existing panoramic work has focused mainly on static search, route following, and scene reasoning, leaving dynamic localization across orientations less explored.Source: §I · p. 2
• Fixed-horizon execution may trust stale trajectories as people move or execution errors accumulate; full-chunk execution risks invalid actions, while one-waypoint execution discards most predictions.UniTrackPLA uses lightweight action-conditioned latent future prediction to adapt the execution horizon.Source: §I · p. 2
• OmniTrackNav-Bench combines 5,000 dynamic-following trajectories, 10,000 panoramic language-navigation routes, and 96 real-world routes.The simulated tasks share panoramic observations and a continuous waypoint representation.Source: §I · p. 2
• In closed-loop experiments, UniTrackPLA achieves 35.00% Omni-Tracking success versus 23.50% for reproduced TrackVLA, and Omni-VLN SR/SPL of 19.75%/19.29% versus baseline 13.00%/12.77%.On held-out real-world routes, EP@0.2m improves from 42.92% to 92.08%.Source: §I · p. 2
• UniTrackPLA shares panoramic perception, visual-language representations, and continuous waypoint actions across instruction-guided navigation and dynamic person following.PAE preserves panoramic azimuth and temporal context while reusing perspective-pretrained encoders; WAC predicts action-conditioned future states to guide continuing or replanning.Source: §I · p. 2

A. Embodied Navigation

• Embodied navigation work spans language-guided static goals and dynamic following, including hierarchical, modular, and end-to-end tracking approaches.The cited tracking approaches address target representation, recovery, occlusion, adaptation, and trajectory planning.Source: A. Embodied Navigation · p. 2
• Most tracking methods use limited-FoV forward-facing cameras and assume the target is initially visible in front, leaving side, rear, and panorama-boundary starts underexplored.Dynamic following also requires continuous target identification and motion adaptation, unlike static-goal navigation.Source: A. Embodied Navigation · pp. 2–3

B. Panoramic-Aware Embodied Navigation

• Existing panoramic systems mainly address static search, route following, scene reasoning, or manipulation; dynamic following under arbitrary initial orientations remains less explored.Cross-view localization and identity preservation are particularly underexplored, and navigation and following commonly use separate representations, action spaces, and policies.Source: B. Panoramic-Aware Embodied Navigation · p. 3

III. METHOD

• UniTrackPLA combines PAE, a shared vision-language policy, and WAC to predict waypoint chunks and reuse or replan based on future-state consistency.The policy grounds task instructions in panoramic context; WAC checks predicted future visual consequences of waypoint prefixes during execution.Source: §III · p. 3

A. Panoramic-Aware Encoding

• Direct ERP encoding introduces non-uniform distortion, while independently encoded perspective crops lose spatial arrangement, hindering cross-view association and directional grounding.PAE adapts perspective-pretrained representations to panoramic observations.Source: A. Panoramic-Aware Encoding · p. 4
• PAE projects each panorama into cyclically ordered perspective views and preserves their azimuthal and temporal identities.Fine-grained patch tokens are retained for current views, while historical views use compact global tokens to limit context overhead.Source: A. Panoramic-Aware Encoding · p. 4

B. Unified Vision-Language-Action Policy

• The shared policy uses the same panoramic representation, multimodal backbone, and action space, with task behavior determined by the instruction and visual context.Language identifies a person-following target by appearance and describes routes, landmarks, and destinations in VLN.Source: B. Unified Vision-Language-Action Policy · p. 4
• A frozen Qwen3-0.6B backbone jointly processes panoramic and instruction tokens, while the projector, PAE embeddings, and continuous action head remain trainable.The backbone is frozen to preserve its pretrained semantics.Source: B. Unified Vision-Language-Action Policy · p. 4
• The action head directly predicts a waypoint chunk of length K = 8 instead of generating discretized action tokens autoregressively.Each waypoint is a cumulative planar pose relative to the current robot frame, and the chunk conditions WAC future-feature prediction and consistency checks.Source: B. Unified Vision-Language-Action Policy · p. 4

C. World-Action Consistency

• WAC predicts compact future visual features conditioned on current state and waypoint prefixes, addressing the risk that later waypoints become invalid as targets, environments, or execution states change.It predicts in latent space rather than reconstructing RGB observations.Source: C. World-Action Consistency · p. 4
• The Transformer interleaves action tokens with horizon-specific queries, and its causal mask limits each query to the current state and corresponding action prefix.This prevents information leakage from later waypoints.Source: C. World-Action Consistency · p. 4
• During training, WAC uses expert waypoint chunks and encoded future panoramas; at inference, it receives policy-predicted chunks, while an inverse-action head encourages action-sensitive transitions.The auxiliary objectives update only WAC because policy states, visual states, actions, and targets are detached.Source: C. World-Action Consistency · p. 4

D. Consistency-Guided Closed-Loop Execution

• After each executed waypoint, the controller compares the resulting panorama’s encoded state with WAC’s predicted state; a large discrepancy indicates a change in target, environment, or robot state.The threshold is horizon-specific for prefixes before the final modeled horizon.Source: D. Consistency-Guided Closed-Loop… · pp. 4–5
• If a transition is consistent, the next waypoint runs without another policy query; otherwise, the remaining suffix is discarded and replanning starts from the current robot state.At most four waypoints execute per chunk because WAC verifies only the first four horizons, even if all remain consistent.Source: D. Consistency-Guided Closed-Loop… · p. 5

• Training jointly optimizes waypoint regression, multi-horizon future-state prediction, and inverse-action reconstruction.Source: E. Optimization Objective · p. 5
• The waypoint loss is masked mean-squared error between predicted and expert waypoint chunks in normalized action space, excluding unavailable future waypoints.Source: E. Optimization Objective · p. 5
• WAC is supervised by cosine distance between predicted and target future features; its inverse-action loss reconstructs normalized expert chunks and discourages visually plausible predictions unrelated to conditioned actions.Source: E. Optimization Objective · p. 5
• At inference, future targets and the inverse-action head are removed, while the WAC predictor remains for consistency-guided execution.Source: E. Optimization Objective · p. 5

A. Dataset

• The benchmark combines 5,000 simulated person-following trajectories, 10,000 simulated VLN routes, and 96 verified real-world routes, totaling 919,978 waypoint-supervision instances.The data use 360° ERP observations and a shared continuous waypoint space.Source: A. Dataset · p. 5
• Omni-Tracking evenly divides trajectories between single-target and distractor-rich following, initializes targets at arbitrary azimuths, and generates paths from robot–target position, distance, and heading.The splits have no episode or source-route overlap; validation has 60 episodes and evaluation has 200.Source: A. Dataset · p. 5
• Omni-VLN combines 5,000 R2R-CE and 5,000 RxR-CE training routes, densely resampling paths into panoramic sequences and evaluating on 400 routes across seen and unseen splits.The subsets provide 484,417 navigation decisions.Source: A. Dataset · p. 5
• The real-world dataset covers 9.13 km across 96 verified routes, with 76 routes for training and 20 route-disjoint routes held out for evaluation.The training data also include nested 10-, 50-, and 76-route subsets for data-scaling studies; instructions are generated from ordered observations and landmark evidence.Source: A. Dataset · p. 6

B. Implementation Details

• Each panorama is projected into four overlapping 384 × 384 views with a 180° horizontal field of view, while DINOv3, SigLIP, and Qwen3-0.6B remain frozen.The multimodal projector, PAE, action head, and WAC are optimized.Source: B. Implementation Details · p. 6
• The policy uses up to 31 historical observations to predict eight waypoints, with WAC modeling the first four; waypoint, future-consistency, and inverse-action losses are weighted 10, 0.1, and 0.05, respectively.Training uses per-GPU batch size 8, yielding global batch size 64; hyperparameters are selected on validation sets and fixed for final evaluation.Source: B. Implementation Details · p. 6

C. Quantitative Results

• Omni-VLN SR rises from 13.00% to 19.75% and SPL from 12.77% to 19.29%; tracking SR and FR improve by 11.5 and 6.4 percentage points.TrackVLA terminates after 1.9s and 1.6s in the STF and DRF examples, while UniTrackPLA follows for 6.6s and 6.4s across scene transitions.Source: C. Quantitative Results · p. 6
• The experiments compare panoramic-view configurations on a direction-balanced split and include an ablation study of PAE and WAC.Source: Table IV · p. 7, Table V · p. 7
• With 10 real routes, EP@0.2m rises from 42.92% to 90.00%; using 76 routes achieves 0.038 ADE, 0.069 FDE, and 92.08% EP@0.2m.The 10-route setting also reduces ADE from 0.172 to 0.046 and FDE from 0.288 to 0.085.Source: C. Quantitative Results · p. 7

1) Panoramic View Configuration:

• Four 180° views achieve the highest SR and FR and lowest LR, improving SR by 8.34 percentage points and reducing LR by 10.00 points versus one 360° view.Narrower fields of view provide insufficient overlap, while additional views add redundant information; four views are adopted by default.Source: 1) Panoramic View Configuration · p. 7

2) Effectiveness of PAE and WAC:

• PAE alone raises overall SR from 23.50% to 32.50%, WAC alone reaches 28.50%, and combining them achieves the best overall performance.WAC alone reduces DRF SR from 12.00% to 10.00%, indicating that consistency verification benefits from explicit panoramic directional modeling.Source: 2) Effectiveness of PAE and WAC · p. 7

E. Real-World Deployment Results

• The Go2-W deployment captures 2880×1440 panoramas and runs inference at 6.9 FPS alongside 3.5 Hz closed-loop motion execution.Inference runs on a remote RTX 4090 server, while observation acquisition remains onboard.Source: E. Real-World Deployment Results · p. 7, Figure 5 · p. 7
• When a person moves to the robot’s side, forward-view RGB TrackVLA loses the target, whereas UniTrackPLA maintains tracking with panoramic perception.The comparison is shown in the office.Source: E. Real-World Deployment Results · p. 7, Figure 6 · p. 7
• Real-world supervision lets the robot reach the street lamp, whereas the Sim-only model remains in the dark area; deployments also span outdoor following and indoor navigation and tracking.The experiments include an office, outdoor person following, and confined domestic spaces.Source: Figure 8 · p. 7, Figure 7 · p. 7, E. Real-World Deployment Results · p. 8

V. CONCLUSION

• UniTrackPLA unifies instruction-guided navigation and dynamic person tracking with PAE preserving temporal and azimuthal structure and WAC supporting consistency-guided waypoint reuse and replanning.The model predicts action-conditioned future visual states to guide waypoint reuse and replanning.Source: §V · p. 8
• OmniTrackNav-Bench brings simulated tracking, panoramic navigation, and real-world routes together under shared observation and action spaces.Source: §V · p. 8
• Future work will explore uncertainty-aware replanning, broader embodied tasks, and efficient fully onboard deployment.Source: §V · p. 8