Source-linked AI summary
TITAN: Future Forecast using Action Priors
Srikanth Malla, Behzad Dariush, Chiho Choi
TL;DR
Forecasting trajectories from a moving platform matters for navigation, but the paper identifies a lack of suitable data and prior-action use for this setting. TITAN combines positions, actions, and context with a 700-clip Tokyo dataset; the authors report significant improvement over baselines and state-of-the-art methods.
Problem
The paper studies forecasting scene-agent trajectories from egocentric mobile-platform views, noting that action priors had not been used for this task partly because a suitable dataset was lacking.
Source: §1 · pp. 1–2Method
TITAN combines prior positions, actions, and context to forecast agent trajectories and ego-motion, using a dataset of 700 urban-traffic clips with hierarchical labels and synchronized ego-motion.
Source: §1 · p. 2Results
The authors report significant performance improvement over baselines and state-of-the-art prediction algorithms in comparative experiments.
Source: §6 · p. 8Takeaways & Limitations
AIM identifies objects more relevant to future ego-motion prediction.
Source: §1 · p. 2The method assumes bounding-box detections from past images are supplied by an external module; detection is outside the paper’s scope.
Source: §4 · p. 4
Abstract
from arXiv · showhide
We consider the problem of predicting the future trajectory of scene agents from egocentric views obtained from a moving platform. This problem is important in a variety of domains, particularly for autonomous systems making reactive or strategic decisions in navigation. In an attempt to address this problem, we introduce TITAN (Trajectory Inference using Targeted Action priors Network), a new model that incorporates prior positions, actions, and context to forecast future trajectory of agents and future ego-motion. In the absence of an appropriate dataset for this task, we created the TITAN dataset that consists of 700 labeled video-clips (with odometry) captured from a moving vehicle on highly interactive urban traffic scenes in Tokyo. Our dataset includes 50 labels including vehicle states and actions, pedestrian age groups, and targeted pedestrian action attributes that are organized hierarchically corresponding to atomic, simple/complex-contextual, transportive, and communicative actions. To evaluate our model, we conducted extensive experiments on the TITAN dataset, revealing significant performance improvement against baselines and state-of-the-art algorithms. We also report promising results from our Agent Importance Mechanism (AIM), a module which provides insight into assessment of perceived risk by calculating the relative influence of each agent on the future ego-trajectory. The dataset is available at https://usa.honda-ri.com/titan
1. Introduction
The paper addresses forecasting agents’ future trajectories from a moving platform, a capability relevant to navigation planning and decision-making. It proposes using prior actions alongside positions and context, and introduces the TITAN dataset for this task.
1. Introduction
- Forecasting agents matters for autonomous navigation because it can inform risk assessment and reactive or strategic planning.The paper frames the task around agents observed in egocentric views from a mobile platform.Source: §1 · p. 1
- Action priors had not been used for trajectory forecasting, which the paper links partly to the lack of an appropriate dataset.The proposed setting uses egocentric views from a mobile agent in a road scene.Source: §1 · pp. 1–2
- TITAN combines prior positions, actions, and context to forecast agent trajectories and future ego-motion, with an interaction module and Agent Importance Mechanism (AIM).The interaction module models pairwise behavior using locations and actions; AIM identifies objects more relevant to ego-motion prediction.Source: §1 · p. 2
- The dataset provides 700 Tokyo urban-traffic clips with hierarchical action attributes, participant categories and states, and synchronized ego-motion information.Its pedestrian actions span atomic, contextual, transportive, and communicative categories.Source: §1 · p. 2
2. Related Work
Prior trajectory and action-recognition work provides relevant modeling approaches, while existing driving datasets offer limited coverage of contextual actions. TITAN is distinguished by hierarchically organized labels intended to capture actions in driving scenes.
2.1. Future Trajectory Forecast
- Earlier trajectory methods use motion histories, interaction graphs, environmental context, or LiDAR, while cited egocentric methods do not model road-agent interactions and their influence on ego-future.The paper positions its own approach as explicitly modeling pairwise interactions to identify objects relevant to the target agent.Source: §2.1 · p. 2
2.2. Action Recognition
- Action-recognition research progressed from 2D and two-stream models to 3D convolutional networks, relation modeling, and long-range temporal-dependency methods.The cited examples include I3D [7], 3D ResNet, and Timeception [19].Source: §2.2 · pp. 2–3
2.3. Datasets
- Unlike related driving datasets with limited simple-action labels, TITAN offers 50 labels organized from atomic actions to complex contextual activities observed in driving scenes.TITAN also provides spatio-temporal localization and multiple action labels for each agent.Source: §2.3 · p. 3
3. TITAN Dataset
TITAN was constructed from vehicle-recorded Tokyo traffic videos and annotated at 10 Hz. Its participant and action labels use a hierarchical scheme designed to cover diverse agents, states, and context-rich activities.
3. TITAN Dataset
- The dataset was drawn from 10 hours of 60-FPS Tokyo video and comprises 700 clips, each 10–20 seconds long, annotated at 10 Hz.The recordings include synchronized odometry from an embedded IMU, and the experimental split is 400 training, 200 validation, and 100 test clips.Source: §3 · p. 3
- Existing mobility datasets lack action labels needed for a meaningful quantitative comparison with TITAN, according to the paper.The passage also says a quantitative comparison with action-localization datasets such as AVA is not warranted because AVA lacks ego-centric views from a mobile platform.Source: §3 · pp. 3–4
- Annotations localize participants with bounding boxes and tracking IDs, and cover person and vehicle categories, age groups, vehicle motion and door/trunk states, and five hierarchical person-action sets.The five action sets include 9 atomic, 13 simple-contextual, 7 complex-contextual, 4 transportive, and 4 communicative actions; annotators use “None” where sets 2–5 do not apply.Source: §3 · p. 4
4. Methodology
TITAN combines image, agent-trajectory, and ego-motion inputs to forecast agent trajectories and ego-motion, using action-conditioned interactions and agent-importance weighting. Its modules recognize actions, localize future objects, and use their predicted motion to inform ego-motion prediction.
4. Methodology
- TITAN takes image sequences, agent bounding-box histories, and ego-motion as inputs; past-image detection is assumed to come from an external module.The framework predicts multiple future bounding-box modes and ego-motion, with acceleration and yaw rate as ego-motion quantities.Source: §4 · p. 4
4.1. Action Recognition
- The action detector fine-tunes I3D or 3D ResNet with eight action-set heads and a task-dependent uncertainty loss.Vehicle and pedestrian action sets receive separate supervision.Source: §4.1 · pp. 4–5
4.2. Future Object Localization
- Future object localization conditions pairwise agent interactions on locations and actions, aggregates them with dynamic GRUs, and decodes future boxes using separate Gaussian distributions for centers and dimensions.The decoder predicts boxes from Tobs + 1 to Tpred; centers and dimensions can be sampled independently.Source: §4.2 · p. 5
4.3. Future Ego-motion prediction
- For ego-motion prediction, TITAN weights agent features with AIM using predicted future boxes and action labels, then combines the weighted features with encoded ego-motion history.The ego-motion process is deterministic, while multimodality comes from sampling other agents’ future boxes; AIM weights reflect their influence on the ego-vehicle.Source: §4.3 · pp. 5–6
5. Experiments
Experiments evaluate action recognition, future object localization, and future ego-motion prediction. Interaction and action priors improve trajectory forecasting, while AIM with future-position and action priors gives the best ego-motion prediction performance.
5. Experiments
- Experiments observe one second of past data to predict up to two seconds ahead, using ADE, FDE, and FIOU for localization, mAP for action recognition, and RMSE for ego-motion.FIOU is included because ADE and FDE do not capture the dimensions of the final bounding box.Source: §5 · p. 6
5.1. Action Recognition
- Action recognition compares I3D with InceptionV1 and 3D ResNet with ResNet50, both pretrained on Kinetics-600 and finetuned on TITAN.The models use heads for TITAN’s eight action sets, with per-frame mAP results reported in Table 1.Source: §5.1 · p. 6, Table 1 · p. 6
5.2. Future Object Localization
- Titan AP improves on Titan vanilla, but performs worse than Titan EP; Titan IP outperforms both and Titan EP+AP.Titan vanilla is an encoder-decoder RNN without priors or interactions; Social-GAN and Social-LSTM improve ADE and FDE over Titan vanilla or linear approaches, but are not evaluated on FIOU because they do not predict dimensions.Source: §5.2 · p. 6, Table 2 · p. 7
- Interactions with action information (Titan EP+IP+AP) significantly outperform all other baselines for future object localization.Titan IP alone also outperforms Titan AP, Titan EP, and Titan EP+AP; adding interaction priors to ego priors further improves performance.Source: §5.2 · pp. 6–7, Table 2 · p. 7
5.3. Future Ego-Motion Prediction
- Future-agent predictions alone (Titan FP) do not improve ego-motion performance over Titan vanilla, but adding their action priors improves acceleration and yaw-rate prediction.Const-Vel performs better for yaw-rate prediction, while Const-Acc performs better for acceleration prediction among the constant-motion baselines.Source: §5.3 · pp. 7–8
- AIM with future position and action priors achieves the best performance for ego-motion prediction.Adding future position in AIM decreases error rates; incorporating both future position and action yields the best performance.Source: §5.3 · p. 8
6. Conclusion
The paper presents TITAN, which uses prior positions, actions, and context to forecast scene-agent trajectories and future ego-motion from mobile-platform egocentric views. Experiments on a new dataset show improved performance with action and interaction priors, while AIM identifies agents influential to future ego-motion and offers insight into perceived navigation risk.
6. Conclusion
- TITAN combines prior positions, actions, and context to forecast agent trajectories and ego-motion from egocentric mobile-platform views.The model includes an Agent Importance Mechanism (AIM) to identify agents influential to future ego-motion.Source: §6 · p. 8
- Comparisons against baselines and state-of-the-art prediction algorithms show significant performance improvement when action and interaction priors are incorporated.The evaluation uses a new dataset of over 700 video clips from urban traffic scenes, with diverse action labels.Source: §6 · p. 8
- AIM provides insight into perceived navigation risk by identifying agents influential in predicting future ego-motion.The authors plan to add scene context to capture participant interactions with scene elements or infrastructure.Source: §6 · p. 8
7. Details of the TITAN Dataset
TITAN labels cover vehicle states and actions, pedestrian age groups, and targeted pedestrian actions, organized into categories of differing complexity. The dataset also specifies annotation conventions and minimum bounding-box sizes for localized pedestrians and vehicles.
7. Details of the TITAN Dataset
- TITAN defines 50 labels spanning vehicle states and actions, pedestrian age groups, and targeted pedestrian action attributes organized hierarchically by complexity and composition.Human actions are grouped into five sub-categories; vehicle states use three motion sub-categories and include trunk or door status, annotated only for four-wheeled vehicles.Source: §7 · p. 12
- Annotators were instructed to localize pedestrians and vehicles only when their minimum bounding-box sizes were 70 × 10 pixels and 50 × 10 pixels, respectively, in the image.Source: §7 · p. 12
- Table 4 details the number of labels, instances, and descriptions for each TITAN action set.Example scenarios display four frames per selected road agent, with its action shown below each frame.Source: §7 · p. 12, Table 4 · p. 13
8. Additional Evaluation
Additional evaluation reports qualitative trajectory and agent-importance results, per-class error differences, and improvements associated with multi-task loss. Results also reflect the test split and the interaction model’s current scene-context assumptions.
8.1. Per-Class Quantitative Results
- Some action classes have no test instances because the 700 clips were randomly split, and the authors plan to update TITAN with more clips containing such actions.Kneeling and jumping are given as examples.Source: §8.1 · p. 13
- Getting into or out of a four-wheel vehicle and getting on a two-wheel vehicle have smaller FDE, while entering a building has larger ADE and FDE.The authors relate the vehicle-action results to person–vehicle interactions; entering-building cases lack an interactive object, and they plan to add topology or semantic context.Source: §8.1 · p. 13
8.2. Efficacy of Multi-Task Loss
- Multi-task loss improves atomic and simple contextual pedestrian action recognition and vehicle motion-status recognition, supporting its efficacy for modeling task uncertainty.The comparison is reported in Table 6.Source: §8.2 · p. 13, Table 6 · p. 17
8.3. Additional Qualitative Results
- TITAN EP+IP+AP consistently performs better than the baseline and state-of-the-art methods in the qualitative future-localization comparisons.Figures 12 and 13 show predicted trajectories and comparisons with ground truth.Source: §8.3 · p. 13, Figure 12 · p. 16, Figure 13 · p. 16
- In additional sequences, AIM assigns higher importance to a crossing agent as the ego-vehicle decelerates, and importance rises when a person’s motion shifts from longitudinal to lateral.Figure 14 uses red for high importance and blue for low importance.Source: §8.3 · p. 13, Figure 14 · p. 17
9. Implementation
The implementation trains action recognition, future object localization, and future ego-motion prediction separately, using ground-truth inputs during training and chained module outputs at test time. The section describes each module’s input encoding, prediction horizon, and principal training setup.
9. Implementation
- The three modules are trained separately with ground-truth inputs, while test-time outputs from one module feed later tasks.The framework is trained on a Tesla V100 using PyTorch.Source: §9 · p. 17
9.1. Future Object Localization
- Future object localization encodes 10 observed steps of boxes, actions, ego-motion, and pairwise interactions, then decodes locations for 20 future steps.The implementation uses 512-dimensional hidden states and embeddings; input images are resized to 224 × 224 for the I3D-based action detector.Source: §9.1 · pp. 17–18, Table 7 · p. 18
9.2. Action Recognition
- Action recognition uses Kinetics-600-pretrained I3D and 3D-ResNet weights, with 10-step video inputs and separate loss terms for pedestrians and vehicles.The I3D input is 3 × 10 × 224 × 224, and the pedestrian indicator is 1 for pedestrians and 0 for vehicles.Source: §9.2 · p. 18, Table 8 · p. 18
9.3. Future Ego-Motion Prediction
- Future ego-motion prediction encodes 10 observed motion steps and uses future bounding boxes, current actions, and AIM-weighted agents to predict 20 steps.The implementation uses 128-dimensional hidden states and embeddings, with two-dimensional input and output at each time step.Source: §9.3 · pp. 18–19, Table 9 · p. 18