Source-linked AI summary
Joint Inference of Groups, Events and Human Roles in Aerial Videos
Tianmin Shu, Dan Xie, Brandon Rothrock, Sinisa Todorovic, Song-Chun Zhu
TL;DR
The paper addresses parsing low-resolution aerial videos into people groupings, events, and human roles, a setting where isolated inference is difficult. It uses an ST-AOG with latent spatiotemporal templates, MCMC, and dynamic programming for joint inference, and reports significant improvements over baselines when domain knowledge constrains uncertain inputs.
Problem
The paper studies how to jointly group people, recognize their events, and assign human roles in large-area, low-resolution aerial videos with challenging visual conditions.
Method
The framework uses an ST-AOG whose latent spatiotemporal templates encode event sub-events and n-ary relations among people and objects, with MCMC and dynamic programming for joint inference.
Results
The experiments demonstrate significant performance improvements over baselines when domain knowledge constrains uncertainty in input features.
Takeaways & Limitations
Joint reasoning over groups, events, and roles is supported as a more suitable approach for challenging aerial videos than addressing these tasks in isolation.
Takeaways & Limitations
The model currently infers object functions implicitly from group events and does not explicitly reason about site functional maps or group intentions.
Abstract
from arXiv · showhide
With the advent of drones, aerial video analysis becomes increasingly important; yet, it has received scant attention in the literature. This paper addresses a new problem of parsing low-resolution aerial videos of large spatial areas, in terms of 1) grouping, 2) recognizing events and 3) assigning roles to people engaged in events. We propose a novel framework aimed at conducting joint inference of the above tasks, as reasoning about each in isolation typically fails in our setting. Given noisy tracklets of people and detections of large objects and scene surfaces (e.g., building, grass), we use a spatiotemporal AND-OR graph to drive our joint inference, using Markov Chain Monte Carlo and dynamic programming. We also introduce a new formalism of spatiotemporal templates characterizing latent sub-events. For evaluation, we have collected and released a new aerial videos dataset using a hex-rotor flying over picnic areas rich with group events. Our results demonstrate that we successfully address above inference tasks under challenging conditions.
1. Introduction
The paper targets extremely low-resolution aerial videos where concurrent group events, human roles, and object interactions must be inferred despite camera motion, shadows, and unreliable tracking. It proposes joint inference using spatiotemporal structure and domain knowledge rather than treating grouping, event recognition, and role assignment separately.
- Motivation and objective: Aerial video analysis remains underexplored, especially for large areas containing co-occurring group events viewed top-down under camera motion.The target setting includes parks and picnic areas with complex interactions among people and objects.
- Motivation and objective: The task jointly groups people by event, recognizes each group’s event, and assigns roles to participating people.Examples include Guide and Tourist in Group Tour and Deliverer and Receiver in Exchange Box.
- Scope and challenges: Extremely low resolution, camera motion, and top-view shadows obscure people, portable objects, and event cues.A person is typically only 15×15 pixels, while camera motion can hide objects and shadows complicate background subtraction.
- Scope and challenges: Appearance-based detection and tracking methods often fail in this setting, producing missed foreground tracks, broken tracklets, and switched track identities.These failures motivate departing from appearance-based event recognition and reasoning over uncertain inputs.
- Overview of our approach: The framework registers frames, clusters noisy tracklets into longer trajectories, and combines trajectory and object evidence during inference.Large objects are detected or inferred through scene labeling, while portable objects and people remain difficult to track reliably.
- Overview of our approach: An ST-AOG represents events as temporal sequences of sub-events defined by latent spatiotemporal templates linking human roles and objects.MCMC and dynamic programming jointly infer groups, events, roles, and temporal extents under domain-knowledge constraints.
- Prior work and contributions: The paper contributes a challenging aerial-video setting, a template formalism, efficient joint inference, and a densely annotated dataset.The contributions cover grouping, event and role recognition, temporal localization, and per-frame annotations of trajectories, objects, roles, events, and groups.
2. Representation
The representation models events hierarchically in an ST-AOG, using latent spatiotemporal templates to encode trajectory relations among people and objects. Templates combine unary, pairwise, and n-ary cues extracted from trajectory segments.
- ST-AOG: ST-AOG organizes events, latent sub-events, human roles, small objects, and scene surfaces into a hierarchical representation.Decomposition edges connect events to sub-events and tracklet or object-detection leaves, while lateral edges encode sub-event order.
- ST-AOG: An AND node requires a temporal sequence of sub-events, whereas an OR node represents alternative space-time patterns for an event.For Exchange Box, the sequence includes approaching, transferring the box, and leaving.
- Latent templates: A latent spatiotemporal template represents an n-ary relation pattern among foreground trajectories within a temporal segment.Frequently recurring but not necessarily semantic trajectory patterns are extracted from training videos by unsupervised clustering.
- Template features: Trajectory segments use unary role, speed, and closeness attributes, while pairwise features capture relations and compatibility between two trajectories.Speed is thresholded at 2 pixels/sec and closeness to detected large objects or surfaces at 70 pixels.
- Template features: N-ary relations are represented by an 18-bin velocity histogram combining six orientation bins with three relative-location bins.The polar-coordinate origin is the center of the event in the scene.
- Template learning: Templates are learned by K-means clustering equal-length 2-second trajectory intervals, yielding 27 templates across 11 event categories.The clustered trajectory segments are represented by template exemplars.
3. Formulation and Learning of Templates
The model learns latent templates from clustered training trajectory segments and scores their fit to new foreground trajectories. It uses weighted unary, pairwise, and n-ary relations together with a duration prior for longer assignments.
- Template likelihood: Training videos provide spatiotemporal templates extracted by K-means clustering, which inference seeks in foreground trajectories of a new video.The templates are denoted by ∆L = {La}.
- Template likelihood: Each template weights unary, pairwise, and n-ary trajectory relations through parameters wa.The likelihood is defined for sets of unit-time trajectory segments.
- Learning: Template parameters are learned by maximizing the log-likelihood of clustered training trajectory features.The learning data are the feature representations extracted from the corresponding trajectory clusters.
- Duration modeling: Assignments of templates to longer trajectory intervals combine template likelihood with a log-normal prior over interval length.The prior hyperparameters are estimated by maximum likelihood on training data.
4. Probabilistic Model
The probabilistic model represents each parsed event as a graph explaining a trajectory subset, its temporal sub-events, and role or object assignments. Inference maximizes a posterior whose energy combines event selection, template selection, and template-to-trajectory fit.
- Parse graphs: A parse graph explains an event, its sub-event sequence, and human-role or object labels for a subset of foreground trajectories.The full parsing solution is a set of parse graphs covering trajectory subsets.
- Parse-graph contents: Each graph records the recognized event, its temporal extent, consecutive nonoverlapping template intervals, and trajectory role or object assignments.The event interval is partitioned into template segments whose durations sum to the event duration.
- Posterior objective: Inference seeks the parse-graph set W that maximizes log p(W|G), equivalently minimizing the corresponding energy E(W|G).The optimization is defined over a partition of all foreground trajectories into disjoint groups.
- Posterior objective: The energy combines terms for selecting events, selecting templates, and assigning templates to trajectory segments.Template-assignment terms use the previously defined segment likelihood.
- Parameter estimation: Event and template switching probabilities are estimated from the frequencies of corresponding selections in training data.These probabilities are represented by the ST-AOG OR nodes.
5. Inference
The inference framework jointly partitions trajectories into groups, recognizes events, assigns roles and objects, and detects latent sub-events. MCMC proposes group and role assignments, while dynamic programming selects template sequences and durations.
- Inference partitions foreground trajectories into disjoint groups, assigns event labels and roles or object labels, and detects latent spatiotemporal templates.The first two tasks use separate MCMC processes; template assignment follows after groups, events, and roles are proposed.
- Grouping: The grouping MCMC starts from atomic trajectory clusters and proposes merges with probability 0.7 or splits with probability 0.3, enumerating event labels for each proposal.
- Role and object assignment: Given groups and event labels, a second MCMC randomly changes trajectory role or object labels within each group to generate proposals.
- Latent sub-event detection: Dynamic programming assigns latent templates and optimal durations by partitioning each group’s time interval into 2-second intervals and constructing a directed graph.Graph nodes represent template assignments to intervals, while directed edges represent possible temporal assignments.
- Latent sub-event detection: The optimal latent-sub-event sequence is found through forward belief computation followed by backward traceback of the best path from sink to source.The graph formulation makes template assignment equivalent to selecting an optimal path through candidate transitions.
6. Experiment
The experiments introduce a dedicated aerial-events dataset and evaluate grouping, event recognition, and role assignment under annotated and noisy conditions. The proposed method performs best among compared methods, while noisy inputs and removing latent-sub-event reasoning reduce accuracy.
- Dataset: Existing aerial datasets lack annotations for group events, while ground-based group-activity and role datasets rely on higher-resolution imagery.
- Dataset: The new dataset contains 27 aerial videos totaling 86 minutes, recorded at 60 fps from 25 meters with approximately 15 actors per video.Videos cover two top-down scenes and are registered onto a reference panorama with annotations for people, groups, events, roles, and objects.
- Results: Our method yields the best accuracy with ground-truth bounding boxes and object labels, and remains better than baselines when using noisy tracking and detection results.Accuracy improves when ground-truth objects or grouping are available.
- Results: Removing latent-sub-event reasoning causes a significant accuracy drop, supporting the model’s ability to capture structural variations in group events.
- Evaluation: The evaluation uses three-fold cross-validation and measures grouping, event recognition, and role assignment under unknown scene layouts and varying video coverage.
- Implementation: The implementation registers raw videos, tracks people, detects buildings and cars, and infers other static objects, but does not detect portable objects such as Frisbees and boxes.
7. Conclusion
The paper introduces a detailed aerial-video dataset and a holistic framework for jointly inferring people’s groupings, events, and roles under noisy inputs. Experiments show improvements over baselines using domain knowledge, while the model remains limited in its implicit treatment of object functions and group intentions.
- The paper contributes a new aerial video dataset with detailed annotations and challenges complementary to existing benchmarks.
- Jointly addressing grouping, event recognition, and role assignment is justified because treating these tasks in isolation is very difficult in aerial videos.
- Figure 6 shows that event and role recognition can group people who are far apart, while trajectory and role errors still produce specific failures.
- The framework improves performance over baselines by constraining uncertainty in input features with domain knowledge.
- The model currently infers object functions implicitly from group events and leaves explicit site-function maps and group-intention reasoning for future work.