PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Appearance Models in Visual Object Tracking

Xi Li, Weiming Hu, Chunhua Shen, Zhongfei Zhang, Anthony Dick, Anton van den Hengel

arXiv:1303.4803v1cs.CV

TL;DR

Robust visual object tracking remains difficult because appearance changes, occlusion, camera conditions, and real-time requirements can degrade or even fail tracking. This survey organizes 2D appearance models into visual representation and statistical modeling modules, reviews their categories and benchmark resources, and identifies remaining challenges.

  • Problem

    Robust visual object tracking must handle appearance variations, occlusion, camera limitations, difficult object conditions, and real-time processing requirements.

  • Method

    The survey reviews 2D appearance models through visual representations and statistical modeling schemes, categorizing representations as local or global and models as generative, discriminative, or hybrid.

  • Results

    The survey analyzes existing 2D appearance models from theoretical and practical viewpoints and examines benchmark resources including source code and video datasets.

  • Takeaways & Limitations

    Effective appearance models depend on both robust visual representations and suitable statistical modeling schemes, while balancing tracking robustness against localization accuracy remains challenging.

  • Takeaways & Limitations

    2D appearance models lose 3D information, limiting accurate pose estimation and making them sensitive to occlusion and out-of-plane rotation.

Abstract

from arXiv · show

Visual object tracking is a significant computer vision task which can be applied to many domains such as visual surveillance, human computer interaction, and video compression. In the literature, researchers have proposed a variety of 2D appearance models. To help readers swiftly learn the recent advances in 2D appearance models for visual object tracking, we contribute this survey, which provides a detailed review of the existing 2D appearance models. In particular, this survey takes a module-based architecture that enables readers to easily grasp the key points of visual object tracking. In this survey, we first decompose the problem of appearance modeling into two different processing stages: visual representation and statistical modeling. Then, different 2D appearance models are categorized and discussed with respect to their composition modules. Finally, we address several issues of interest as well as the remaining challenges for future research on this topic. The contributions of this survey are four-fold. First, we review the literature of visual representations according to their feature-construction mechanisms (i.e., local and global). Second, the existing statistical modeling schemes for tracking-by-detection are reviewed according to their model-construction mechanisms: generative, discriminative, and hybrid generative-discriminative. Third, each type of visual representations or statistical modeling techniques is analyzed and discussed from a theoretical or practical viewpoint. Fourth, the existing benchmark resources (e.g., source code and video datasets) are examined in this survey.

1. INTRODUCTION

Visual object tracking estimates a target’s motion state despite diverse appearance and environmental challenges. This survey motivates appearance modeling as a central component of tracking systems and frames key design questions around representations, learning schemes, and motion.

  • Overview of visual object tracking: Visual object tracking estimates a target’s location, orientation, and size in each frame of an image sequence.
  • Overview of visual object tracking: Tracking supports applications including surveillance, traffic monitoring, video compression, and human-computer interaction.
  • Overview of visual object tracking: A typical tracking system contains object initialization, appearance modeling, motion estimation, and object localization modules.
  • Challenges in developing robust appearance models: Robust tracking is challenged by sensor limitations, non-rigid objects, real-time requirements, cross-camera views, and appearance variations.Appearance changes include illumination changes, camera motion, occlusion, noise, deformation, out-of-plane rotation, and pose variation.
  • Challenges in developing robust appearance models: The survey examines which object forms, visual representations, statistical learning schemes, and motion models suit different tracking contexts.The motivating questions include what to track, which representations are robust, which statistical schemes are suitable, and how motion should be modeled.

2. ORGANIZATION OF THIS SURVEY

The survey organizes 2D appearance modeling into visual representation and statistical modeling modules, then reviews related surveys, benchmark resources, representative techniques, and future research issues. Its taxonomy distinguishes local/global representations and generative/discriminative/hybrid tracking-by-detection schemes.

  • Organization of this survey: The visual representation module describes spatio-temporal object appearance using local and global visual information.Local representations encode local statistical information, while global representations reflect image-region characteristics such as color histograms.
  • Organization of this survey: The statistical modeling module reviews tracking-by-detection schemes classified as generative, discriminative, and hybrid generative-discriminative.
  • Organization of this survey: The survey examines source codes and video datasets to facilitate tracking experiments.
  • Main differences from other related surveys: Compared with broader related surveys, this work focuses intensively on 2D appearance modeling and provides more detailed appearance-model analysis.It uses a general module-based architecture rather than organizing tracking primarily by point, kernel, and silhouette categories.
  • Contributions of this survey: The survey reviews visual representations by feature-construction mechanism and statistical models by model-construction mechanism.It also discusses properties of representation and learning techniques and examines benchmark resources including source codes and databases.

3. VISUAL REPRESENTATION

The survey organizes visual representations into global and local mechanisms for encoding object appearance, covering pixel, motion, histogram, covariance, wavelet, contour, and interest-point features. It also describes trade-offs involving spatial structure, robustness, computational cost, and sensitivity to noise or illumination.

  • Global representations encode object appearance through raw pixels, optical flow, histograms, covariance matrices, wavelet filtering, or active contours.
  • Raw pixel representation: Raw pixels provide simple, efficient tracking features, but pixel information alone is insufficient for robust tracking and is often enriched with edge or texture cues.
  • Histogram representation: Histogram representations include single-cue and multi-cue variants that model color, spatial-color, spatial-texture, or shape-texture information.
  • Covariance representation: Covariance representations compare image-region features using affine-invariant or Log-Euclidean Riemannian metrics, with the latter mapping covariance matrices into a Euclidean vector space.
  • Histogram representation: Single-cue histograms efficiently capture feature distributions but lose spatial structure, while shape-texture histograms improve robustness to illumination and pose variation.

4. STATISTICAL MODELING FOR TRACKING-BY-DETECTION

For tracking-by-detection, the survey classifies statistical appearance models as generative, discriminative, or hybrid generative-discriminative according to how models are constructed. It reviews their mechanisms and notes practical limitations involving model specification, component selection, fusion, and flexibility.

  • Tracking-by-detection dynamically performs statistical modeling to support object detection through generative, discriminative, or hybrid model-construction mechanisms.
  • Generative appearance models: Generative models fit data from the object class and use online updates to incrementally learn foreground representations while reducing background influence.
  • Discriminative appearance models: Discriminative models formulate tracking as binary classification, maximize object–non-object separability, and seek informative features through online learning.
  • Hybrid generative-discriminative appearance models: Hybrid models fuse generative and discriminative information, but heuristic fusion cannot guarantee improvement and may introduce extra constraints and parameters.
  • Mixture generative appearance models: Mixture generative models adaptively learn components to capture spatio-temporal appearance diversity, including WSL mixture and Gaussian mixture models.
  • Mixture generative appearance models: Gaussian mixture models approximate object-appearance density with multiple Gaussian distributions, but selecting the correct component number k is difficult and often relies on heuristic criteria.

4.2. Kernel-based generative appearance models (KGAMs)

KGAMs use kernel density estimation for visual representation and mean shift for object localization. The survey distinguishes six branches according to kernel construction or mode-seeking mechanisms.

  • KGAMs combine kernel density estimation with mean shift to localize objects.The survey illustrates this mode-seeking process in Fig. 11.
  • Color-driven KGAMs: Color-driven KGAMs represent appearance with spatially smooth isotropic color histograms and use Bhattacharyya-based mean shift localization.These models can ignore edge and shape information, which may reduce robustness.
  • Shape-integration KGAMs: Shape-integration KGAMs model appearance in joint color-shape space using spatially normalized kernels for color and object boundaries.This design captures spatio-temporal appearance properties through color and boundary cues.
  • Scale-aware KGAMs: Scale-aware KGAMs capture multi-scale appearance distributions using Difference-of-Gaussian mean shift features or maximum-likelihood optimization.These approaches address kernel scale selection through local-maxima detection or optimization in joint spatial-color space.
  • Non-symmetric KGAMs: Non-symmetric KGAMs adapt kernel scale and orientation to reduce estimation bias for complicated density functions.Unlike symmetric mean shift, asymmetric mean shift jointly estimates image coordinates and kernel properties.
  • Sequential-kernel-learning KGAMs: Sequential kernel density approximation reduces the computational and memory burden of batch kernel density estimation by learning density representations sequentially.SKDE also propagates density modes over time for real-time tracking.

4.3. Subspace learning-based generative appearance models (SLGAMs)

SLGAMs model target appearance through learned subspaces and basis templates, covering linear, nonlinear, sparse, autoregressive, and multi-subspace approaches. These models also include shape-appearance formulations such as active appearance models.

  • A target can be represented as a linear combination of basis templates, with coefficients specifying its position in an underlying appearance subspace.SLGAMs focus on learning these subspaces and associated templates through subspace analysis.
  • Conventional subspace models: Conventional subspace models divide into linear and nonlinear subspace models.Linear models are further distinguished by lower-order vector representations and higher-order matrix- or tensor-based representations.
  • Linear subspace models: Incremental PCA and robust incremental subspace learning improve the efficiency and robustness of linear subspace updates for online tracking.Higher-order matrix- or tensor-based models address the small-sample-size problem affecting vector-based representations.
  • Nonlinear subspace models: Nonlinear subspace models use techniques such as Local Linear Embedding and kernel PCA when appearance samples lie on nonlinear manifolds.These methods aim to capture the underlying geometric information of target samples.
  • Unconventional subspace models: Unconventional subspace models include sparse or non-sparse representation, autoregressive modeling, and multi-subspace learning.Examples include ℓ1-regularized representation, compressive sensing, temporal silhouette models, and multiple visual-cue subspaces.
  • Active appearance models: Active appearance models jointly represent shape and appearance as linear combinations of bases and fit them by minimizing a warped-image cost function.A piecewise affine warp transforms pixels from the base shape into the active appearance model.

4.4. Boosting-based discriminative appearance models

Boosting-based discriminative appearance models select features and learn object/non-object classifiers through self-learning or co-learning strategies. The survey covers adaptations for dynamic appearance, noise, drift, localization ambiguity, and computational constraints.

  • BDAMs use discriminative learning to classify object versus non-object regions and are categorized into self-learning and co-learning models.Co-learning models exploit multiple information sources, whereas self-learning models use a single source.
  • Self-learning single-instance BDAMs: Conventional BDAMs evaluate candidate features discriminatively and select top-ranked features for tracking.Online boosting variants support feature selection and classifier updates during tracking.
  • Self-learning single-instance BDAMs: Dynamic ensemble BDAMs construct and update weak-classifier sets to adapt to time-varying object appearance.Other boosting variants use noise-insensitive loss functions or embed feature selection within particle filtering.
  • Transfer learning-based BDAMs: Transfer-learning BDAMs separate auxiliary samples from earlier frames and target samples from the current frame to address drift.This design responds to violations of the assumption that samples from adjacent frames have similar distributions.
  • Co-learning BDAMs: Semi-supervised and co-learning BDAMs incorporate prior information or multiple visual cues into classifier updating.Co-learning can exchange information between sources such as color and texture.
  • Multi-instance BDAMs: Multi-instance BDAMs represent an object with image-patch bags to handle ambiguity in object localization.Their assumptions can be violated when positively labelled instances are not all truly positive.
  • BDAMs require a large local feature pool, which lowers computational speed, and their local-feature focus can yield local optima.MIL-based tracking may also lose accuracy when patches do not precisely capture object appearance.

4.5. SVM-based discriminative appearance models (SDAMs)

SDAMs use SVM-based discriminative learning to separate foreground from background, with self-learning and co-learning variants. The survey emphasizes their generalization strength alongside challenges in obtaining reliable labels and adapting kernel information.

  • SDAMs learn margin-based SVM classifiers that maximize inter-class separability and retain informative support vectors.This provides strong discriminative power for object/non-object classification.
  • SDAMs are organized into self-learning and co-learning models according to their learning mechanisms.Kernel selection and efficient kernel computation are important design concerns.
  • Self-learning SDAMs: Self-learning SDAMs construct SVM classifiers without using unlabeled data or multiple information sources.Offline models may require substantial prior training data, limiting extension to general object tracking.
  • Co-learning SDAMs: Co-learning SDAMs use semi-supervised or multi-kernel learning to incorporate multiple information sources into SVM classifiers.Multi-kernel learning can combine kernels derived from features such as color and spatial-pyramid visual words.
  • Reliable labeled samples are difficult to obtain under noise, occlusion, and illumination changes, making heuristic sample collection unstable.Structured SVM and ranking SVM address this issue through structural relationships or relative ranking information.

4.6. Randomized learning-based discriminative appearance models (RLDAMs)

Randomized learning-based discriminative appearance models build diverse classifier ensembles through random input and feature selection, offering computational and extensibility advantages. Visual tracking applies these techniques through online random forests, random naive Bayes classifiers, and MIForests.

  • Randomized learning constructs diverse classifier ensembles by randomly selecting inputs and features.The survey contrasts this approach with boosting and SVM in terms of computational efficiency and ease of extension to multi-class learning.
  • RLDAMs include online random forests, random naive Bayes classifiers, and MIForests for visual object tracking.
  • Online random naive Bayes tracking is motivated by the classifiers’ low computational and memory costs.

4.7. Discriminant analysis-based discriminative appearance models (DADAMs)

DADAMs learn decision hyperplanes that separate object from background classes, using conventional or graph-driven discriminant analysis. The survey covers uni-modal, multi-modal, graph embedding, and graph transductive variants, while noting distributional and computational limitations.

  • Overview: DADAMs seek a decision hyperplane separating object and background classes through supervised subspace learning.Conventional methods operate in vector spaces, whereas graph-driven methods use graphs for supervised subspace learning.
  • Conventional DADAMs: Uni-modal DADAMs assume the object data follow a single Gaussian distribution, with examples including IFLDA, LDA, incremental 2DLDA, and discriminant metric learning.Some formulations also assume approximately Gaussian object and background distributions with equal covariance.
  • Graph-driven DADAMs: Graph embedding DADAMs embed high-dimensional samples into discriminative low-dimensional spaces for object/non-object classification.They can model irregular, multi-modal background distributions while retaining a discriminative representation.
  • Graph-driven DADAMs: Graph transductive DADAMs use topological relations among labeled and unlabeled samples to find decision hyperplanes for candidate likelihood evaluation.Figure 17 contrasts a conventionally supervised hyperplane with one further adjusted using unlabeled samples.
  • Conventional DADAMs: Multi-modal discriminant analysis uses data clustering to address poor performance when object and background classes have multi-modal distributions.
  • Limitations: Graph-based DADAMs may be impractical for tracking because graph learning requires retaining many labeled and unlabeled samples.

4.8. Codebook learning-based discriminative appearance models (CLDAMs)

CLDAMs construct foreground and background codebooks to capture dynamic appearance information and discriminative cues. Their flexibility is limited by scene-specific training requirements and the difficulty of selecting a codebook size.

  • CLDAMs construct foreground and background codebooks to adaptively capture dynamic appearance information.One approach uses separate RGB- and LBP-based image-patch codebooks to improve robustness to occlusion, scaling, and rotation.
  • CLDAMs can incorporate discriminative information from both background samples and other object instances.
  • Limitations: Universal codebooks are difficult to construct across scenes or objects, requiring different training samples and reducing practical flexibility.Determining the codebook size is also difficult in practice.

4.9. Hybrid generative-discriminative appearance models (HGDAMs)

HGDAMs combine generative and discriminative appearance models because the two approaches have complementary advantages and disadvantages. The survey organizes these combinations by fusion layer and mechanism, including score fusion, intermediate-level integration, sequential combination, and interleaving combination.

  • Overview: HGDAMs aim to combine the complementary benefits of generative and discriminative models in visual object tracking.
  • Single-layer combination: Single-layer HGDAMs fuse generative and discriminative models at the same layer through confidence-score or intermediate-level combination.
  • Single-layer combination: Decision-level combination fuses generative and discriminative confidence scores, with performance associated with the balance between purely generative and purely discriminative models.
  • Single-layer combination: Intermediate-level combination jointly uses low-level features and high-level confidence scores from generative and discriminative models.One cited model imposes negative-data, bottom-up pair-wise, and adaptation-dynamics constraints to address adaptation drift.
  • Multi-layer combination: Multi-layer HGDAMs combine generative and discriminative information through sequential or interleaving mechanisms.Sequential methods pass one model’s decision output to the other, while interleaving methods let each model guide the other’s learning.

5. BENCHMARK RESOURCES FOR VISUAL OBJECT TRACKING

The survey describes qualitative and quantitative evaluation of tracking algorithms, emphasizing ground truth and publicly available benchmark resources. Quantitative evaluation uses positional errors or overlap measures, while point-based annotation can simplify ground-truth collection.

  • Benchmark resources: Benchmark evaluation requires a shared test video dataset, ground truth, and implementations of competing tracking algorithms.The survey notes that major public resources are listed in Table VI.
  • Evaluation methods: Qualitative evaluation judges whether tracked target regions cover true object regions while containing fewer non-object pixels.This assessment is based on human intuitive perception.
  • Benchmark resources: Table VII compares visual representations by whether they cope with occlusions, illumination changes, and shape deformations.The table uses √ and × to indicate whether each representation can or cannot handle these conditions.
  • Benchmark resources: Table VIII provides a qualitative comparison of representative statistical-modeling-based appearance models.
  • Evaluation methods: Quantitative evaluation compares tracked regions with ground truth using corner positional errors or bounding-box and ellipse overlap ratios.The overlap ratio is defined using tracked area At and ground-truth area Ag.
  • Ground-truth annotation: Ground-truth annotation with bounding boxes or ellipses is difficult and time-consuming, motivating point-based annotations for simpler or more accurate evaluation.Researchers record object centers for efficiency or mark several points within object regions for accuracy.

6. CONCLUSION AND FUTURE DIRECTIONS

The survey organizes 2D appearance models into visual representations and statistical modeling schemes, then identifies unresolved challenges involving robustness, accuracy, efficiency, dimensionality, saliency, and cross-camera or low-frame-rate tracking.

  • Survey organization: The survey reviews visual representations and statistical modeling schemes as two closely related modules of 2D appearance models.Visual representations describe spatio-temporal object appearance, while statistical models capture generative or discriminative information from object regions.
  • Open challenges: Existing appearance models cannot simultaneously guarantee tracking robustness and tracking accuracy.More features and geometric constraints can improve localization but reduce generalization, whereas relaxed constraints increase localization ambiguity.
  • Open challenges: Simple visual features are computationally efficient but less discriminative, whereas robust features are more expensive and require sophisticated parameter settings.The survey identifies balancing simplicity and robustness as an important tracking problem.
  • Open challenges: 2D appearance models are efficient and simple to implement but lose 3D pose information, while 3D models improve pose characterization at higher computational cost.Transferring 2D tracking results into effective 3D tracking initialization remains unsolved.
  • Open challenges: Salient region features can remain stable during tracking, but unreliable saliency detection under noise or drastic illumination variation causes feature mismatches across frames.The survey calls for more intelligent vision modeling to address this dependence on saliency detection.
  • Open challenges: Tracking across adjacent camera scenes remains unresolved because appearance models often handle the same scene from different viewpoints but cannot independently track the same object across scenes.Large camera networks therefore require effective transfer of target information between camera sub-networks.
  • Open challenges: Low-frame-rate video makes tracking difficult because abrupt object motion demands strong online generalization and adaptation to appearance variation.Efficient visual and statistical modeling is identified as important for real-time applications on constrained devices.
Loading 1303.4803v1…