Source-linked AI summary
Learning where to Attend with Deep Architectures for Image Tracking
Misha Denil, Loris Bazzani, Hugo Larochelle, Nando de Freitas
TL;DR
The paper addresses gaze-based object tracking and recognition under partial information, where extending the full-information policy performs poorly. It proposes a probabilistic model with interacting pathways and Gaussian-Process reward modeling, achieving good performance while exposing tracking-failure limitations.
Problem
Extending the full-information policy to partial information results in poor performance, motivating a different approach.
Method
The paper uses a probabilistic graphical model with two interacting pathways and models the reward surface as a Gaussian Process.
Results
The Gaussian-Process approach gives good performance in the presence of partial information.
Takeaways & Limitations
The experiments demonstrate significant potential for joint classification, tracking, and planning.
Takeaways & Limitations
The current model cannot recover from tracking failure, which can result from choosing a long sequence of poor fixation points.
Abstract
from arXiv · showhide
We discuss an attentional model for simultaneous object tracking and recognition that is driven by gaze data. Motivated by theories of perception, the model consists of two interacting pathways: identity and control, intended to mirror the what and where pathways in neuroscience models. The identity pathway models object appearance and performs classification using deep (factored)-Restricted Boltzmann Machines. At each point in time the observations consist of foveated images, with decaying resolution toward the periphery of the gaze. The control pathway models the location, orientation, scale and speed of the attended object. The posterior distribution of these states is estimated with particle filtering. Deeper in the control pathway, we encounter an attentional mechanism that learns to select gazes so as to minimize tracking uncertainty. Unlike in our previous work, we introduce gaze selection strategies which operate in the presence of partial information and on a continuous action space. We show that a straightforward extension of the existing approach to the partial information setting results in poor performance, and we propose an alternative method based on modeling the reward surface as a Gaussian Process. This approach gives good performance in the presence of partial information and allows us to expand the action space from a small, discrete set of fixation points to a continuous domain.
1 Introduction
The paper presents a gaze-driven system for simultaneous object tracking and recognition, separating identity modeling from control and learning where to attend. Its attentional policy handles partial information and continuous fixation choices by modeling reward with a Gaussian Process.
- Model overview: The system uses interacting identity and control pathways for simultaneous object tracking and recognition from gaze data.The separation is intended to support invariance to transformations such as object location, scale, and orientation.
- Identity pathway: The identity pathway compares appearance observations and classifies the target with a two-hidden-layer deep network built from factored-RBMs and a multi-fixation RBM.The top layer accumulates information from consecutive time steps.
- Control pathway: The control pathway estimates target state and aligns the object template with the scene using localization and fixation modules.The state encodes location, size, speed, and orientation; localization uses a particle filter, while fixation selects points relative to the template.
- Attentional control: The fixation policy learns online from rewards based on target-location uncertainty, using the belief state produced by filtering.The framework treats reward as a function of beliefs, making the problem closer to sequential experimental design than a typical POMDP.
- Policy learning: A straightforward extension of the previous policy to partial information performs poorly, whereas Gaussian-Process reward modeling performs well in that setting.The proposed method also expands fixation choices from a small discrete set to a continuous domain.
- Implementation and scope: The approach uses foveated observations with high resolution near fixation and lower resolution in the periphery, reducing the dimensionality of large images and video.The paper describes the system as simple, modular, suitable for large datasets, and easy to extend.
2 Identity Pathway
The identity pathway models target appearance and performs classification from gaze-selected observations using (factored)-RBMs. Its multi-fixation representation aggregates gaze-relative features across time, while classification decisions are accumulated over time.
- The identity pathway mirrors the ventral pathway by modelling target appearance and performing object classification.
- (Factored)-RBMs model object appearance and classify observations selected by the control module's gaze policy.The RBMs are undirected probabilistic graphical models governed by a Boltzmann distribution over gaze data and hidden features.
- The multi-fixation RBM uses one-hot gaze-location encodings to aggregate first-layer features from consecutive time steps into a higher-level representation.The higher-level hidden representation is used for classification, with gaze locations encoded relative to the multi-fixation sequence.
- A separately trained multinomial logistic regression classifier predicts the class from the aggregated top-layer activation probabilities.Unlike the related multi-fixation RBM, the top hidden layer does not explicitly model the class label.
- Multi-fixation processing increased classification accuracy relative to a single-fixation module, and decisions were accumulated over time to improve class estimates.
- The extra attentional mechanism for choosing fixations to reduce classification uncertainty was left for future work.
3 Control Pathway
The control pathway tracks the target's latent state, normalizes observations, and learns gaze policies that reduce uncertainty in the belief state.
- The control pathway tracks target position, speed, scale, orientation, and related state variables in a nonlinear state-space model.
- Particle filtering approximates the belief state because the posterior recurrence is generally intractable for nonlinear, non-Gaussian models.
- Foveated observations provide higher resolution near the gaze center and lower resolution in the periphery, with appearance features learned using RBM receptive fields.
- The observation likelihood compares RBM hidden-unit features from the current observation with corresponding template-region features.
- The template is static in the current implementation, although the paper notes that it could be adapted over time.
- The gaze policy selects fixation points to maximize instantaneous reward, focusing on observations that reduce filtering-distribution uncertainty.
4 Gaze control
The paper compares discrete gaze-selection strategies with a Gaussian-Process Bayesian optimization method for partial information and continuous fixation spaces.
- Hedge requires rewards for every action, whereas sequential gaze selection makes that assumption unrealistic because the target moves between fixations.
- EXP3 extends Hedge to partial information by using reward feedback only for the selected action, but has substantially worse theoretical guarantees.
- Both Hedge and EXP3 choose among predetermined discrete fixation points, while Gaussian-Process estimation enables a continuous gaze policy.
- The Gaussian Process models a smooth reward surface and supplies predicted reward and uncertainty for balancing exploration against exploitation.
- The method assumes noisy reward observations and uses kernel length scales to control covariance effects across gaze dimensions.
- GP-UCB selects the next fixation by optimizing an acquisition function over the continuous action space.
- Maximum-likelihood hyperparameters enable fast predictions but suffer from data scarcity early in tracking, especially for length-scale parameters.
5 Algorithm
The algorithm combines particle filtering with gaze control for partial-information strategies, updating the belief state and policy from sequential observations.
- Particle filtering approximates the posterior belief over target states when the belief state cannot be computed analytically.
- The partial-information algorithm performs importance sampling for the selected action before updating gaze control.
- Full-information strategies instead perform importance sampling independently for every possible action.
- The gaze-control update can use reward information from each possible action to construct the new strategy in the full-information setting.
1. Initialization
The algorithm initializes the gaze policy before sequential tracking begins.
- The policy π1 is initialized according to the selected control strategy.
2. Importance sampling
The importance-sampling procedure processes particles iteratively, evaluates and normalizes their weights, and receives reward for the chosen action.
- Particles are iterated through prediction, importance-weight evaluation, and importance-weight normalization steps.
- The procedure receives reward for the chosen action during processing.
4. Selection
Particle filtering constructs approximate belief distributions by propagating weighted state trajectories and resampling them into new particles. The selection step concentrates computation on high-weight samples.
- After weighting, resampling discards low-weight samples and replicates high-weight samples to form an unweighted empirical distribution.
- Particle filters combine sequential importance sampling with a selection scheme to obtain N new particles approximately distributed according to the posterior.
- New paths extend previous particle trajectories using a proposal kernel over the next state.
- The proposal distribution factors into the previous-path distribution and a conditional distribution for the next state.
- Because the target posterior cannot itself serve as the proposal, importance weights correct the new particles to obtain consistent estimates.
- Using the transition prior as proposal reduces the importance weights to the likelihood, while richer proposals can use detectors, saliency, optical flow, or approximate filtering.
6 Experiments
The experiments evaluate gaze policies for tracking and classification on synthetic digit videos, multi-target sequences, and real videos. Learned policies improve tracking under occlusion, while Bayesian optimization supports continuous gaze selection and qualitative real-world generalization.
- Three experiments evaluate tracking and classification policies on synthetic digit videos, multi-target sequences, and real videos.
- Synthetic digit tracking: The MNIST task tracks and recognizes a moving digit among static distractors, including partial occlusions and 30% noise.
- Full Information Policies: The learned policy’s tracking error is always below the errors of the deterministic and random policies.
- Full Information Policies: During occlusions, deterministic and random policies usually fail, whereas the learned policy succeeds; tracking improvements are mirrored by classification improvements.
- Full Information Policies: Hedge learns to favor reliable gazes, with its policy converging toward the most reasonable gaze action in the illustrated sequence.
- Partial Information Policies: For partial-information gaze selection, Bayesian optimization compares favorably with Hedge, while EXP3 performs substantially worse.
- Partial Information Policies: Bayesian optimization uses a continuous action space and receives rewards only for selected actions, unlike Hedge’s discretized full-information setting.
- Real videos: On challenging celebrity videos, Bayesian optimization provides qualitative anecdotal evidence of good gaze-selection policies on real-world data.
7 Conclusions and Future Work
The paper proposes a probabilistic model for joint classification, tracking, and planning, with experiments showing potential for learned gaze control. Its partial-information method performs well and expands fixation choices to a continuous domain, but broader deployment remains technically demanding.
- Conclusions: The proposed decision-theoretic probabilistic graphical model jointly addresses classification, tracking, and planning.
- Conclusions: A straightforward extension of the full-information policy performs poorly with partial information, whereas the alternative method performs well and supports continuous fixation points.
- Future Work: Extending the results to long video sequences requires adapting target templates and proposal distributions as objects undergo significant transformations.
- Future Work: Deployment on more complex video sequences requires careful design of proposal and transition distributions, control algorithms, templates, data association, and motion analysis.
- Future Work: The current model cannot recover from tracking failure, although classifier output may help detect and recover from it.
- Future Work: The evaluated gaze-selection methods assume future rewards are independent of past actions, despite poor fixation sequences potentially causing tracking failure.
- Future Work: Future work could incorporate tracking confidence to adjust exploration and exploitation after poor fixation choices.