PaperlayerView exact revision on arXiv

Source-linked AI summary

The Evolution of First Person Vision Methods: A Survey

Alejandro Betancourt, Pietro Morerio, Carlo S. Regazzoni, Matthias Rauterberg

arXiv:1409.1484v3cs.CV

TL;DR

FPV research needs methods for analyzing videos produced by increasingly commercial wearable devices, including under real-time constraints. This survey synthesizes the field’s evolution from 1997 to 2014 through a hierarchical review of objectives, features, algorithms, datasets, challenges, and opportunities. It reports heterogeneous strategies, growing developments, and unresolved requirements for adaptive and resource-efficient systems.

  • Problem

    Wearable FPV devices are expanding applications and demand for analysis, but the literature contains heterogeneous methods and unresolved challenges involving features, learning, and deployment.

  • Method

    The paper surveys FPV video-analysis methods using a hierarchical structure organized by objectives, subtasks, features, algorithms, datasets, and temporal evolution.

  • Results

    The survey identifies a large number of developments in FPV over the last 20 years and summarizes major achievements and upcoming research directions.

  • Takeaways & Limitations

    FPV methods remain heterogeneous, while future research must address adaptive operation and efficient real-time processing for wearable devices.

  • Takeaways & Limitations

    Existing methods generally operate on previously recorded videos rather than continuously learning and adapting in a closed loop.

Abstract

from arXiv · show

The emergence of new wearable technologies such as action cameras and smart-glasses has increased the interest of computer vision scientists in the First Person perspective. Nowadays, this field is attracting attention and investments of companies aiming to develop commercial devices with First Person Vision recording capabilities. Due to this interest, an increasing demand of methods to process these videos, possibly in real-time, is expected. Current approaches present a particular combinations of different image features and quantitative methods to accomplish specific objectives like object detection, activity recognition, user machine interaction and so on. This paper summarizes the evolution of the state of the art in First Person Vision video analysis between 1997 and 2014, highlighting, among others, most commonly used features, methods, challenges and opportunities within the field.

1 INTRODUCTION

FPV video analysis grew from late-1990s academic work into an increasingly commercial wearable-technology field. This survey organizes its evolution, methods, challenges, and opportunities through 2014.

  • Technological emergence: Portable head-mounted cameras became common among sportsmen, extending earlier research devices into commercial first-person recording.The paper links this technological trend to academic results from the late 1990s and growing interest in recording daily activities.
  • Terminology: FPV terminology remains unsettled, with First Person Vision, Egocentric Vision, and Ego-vision all used in the literature.The paper describes First Person Vision as arguably the most commonly used term.
  • Commercialization: Companies including Google and Microsoft pursued smart-glasses and augmented-reality wearables, with Google also announcing Project Glass.Project Glass demonstrated FPV recording and head-up-display information delivery.
  • Applications and challenges: Smart-glasses could support applications in military strategy, enterprise services, tourism, surveillance, medicine, and driving assistance.The paper also identifies privacy, battery life, and algorithmic challenges accompanying this expansion.
  • Survey scope: The survey reviews FPV state-of-the-art and temporal evolution from 1997 to 2014 using reference tables, timelines, and objective-specific summaries.Its proposed categories reflect the authors’ schematic perception of a heterogeneous and rapidly changing literature.

2 FIRST PERSON VISION (FPV) VIDEO ANALYSIS

FPV analysis combines wearable hardware, image and motion features, learning methods, and hierarchical objectives, while confronting camera motion, environmental variability, and resource constraints. Its advantages include scene-centered recording, but practical deployment remains bounded by computation, battery, and privacy-related concerns.

  • Devices: Smart-glasses combine sensors, processing, and head-up displays for real-time interaction, whereas action cameras support sports and lifelogging applications.Smart-glasses remain immature because battery life, price, and target market challenges are unresolved.
  • Advantages: FPV can record the scene’s most relevant parts through wearable devices, reducing reliance on complex controlled multi-camera systems.The perspective also produces datasets with substantial variability in environments and scene configurations.
  • Challenges: Moving cameras complicate foreground-background separation and make calibration impossible, motivating scale-, rotation-, and translation-invariant features.Camera motion is a defining FPV characteristic that affects low-level feature selection.
  • Challenges: Uncontrolled illumination requires robust methods, and the paper notes that shape descriptors may be preferred over color-based features.Examples include sunny tourism scenes, nighttime driving, and indoor kitchen activities.
  • Challenges: Efficient video-processing strategies must balance battery life, processing power, and communication limits; cloud processing additionally requires improved compression and protocols.The paper identifies real-time operation as a motivation while noting that both hardware and software still need development.
  • Analytical structure: The survey organizes analysis hierarchically from raw video to six general objectives, fifteen subtasks, image features, and quantitative or computational methods.This structure connects methods to their final objectives and the features used.

2.1 Objectives

The survey organizes FPV video analysis into six objectives and fifteen subtasks, linking them to features and computational methods. Object recognition and tracking is the most explored objective and supports advanced tasks, while activity recognition, interaction, summarization, and mapping address broader uses and challenges.

  • Objective hierarchy: Table 2 reviews 117 articles organized into six objectives, with additional categories for hardware, software, surveys, and conceptual work.The six objectives are object recognition and tracking, activity recognition, user-machine interaction, video summarization and retrieval, environment mapping, and interaction detection.
  • Object recognition and tracking: Object recognition and tracking is the most explored objective and commonly serves as a starting point for activity recognition and other advanced tasks.The survey also identifies it as a foundation for video summarization, retrieval, and environment mapping.
  • Object recognition and tracking: FPV object analysis must handle unknown object number, type, scale, viewpoint, frequent occlusion, and difficulty constructing background-foreground models.These constraints arise from uncontrolled videos and mobile wearable cameras.
  • Object recognition and tracking: Model-driven hand methods infer detailed posture but may require substantial computation, controlled environments, or additional sensors.RGB-D cameras are also used to improve hand detection and tracking in realistic scenarios.
  • User-machine interaction: User-machine interaction includes intentional hand commands and unintentional commands inferred from activities, recognized objects, or scenes.Examples include gesture-based control and automatically presenting information while the user cooks or examines an item.
  • Video summarization and retrieval: Video summarization and retrieval selects important frames or subsequences using importance functions based on objects, temporal relations, similarity, or motion.Motion features can segment videos according to the user’s activity, while storage and visualization improvements can reduce device computation and privacy concerns.
  • Environment mapping: Environment mapping constructs 2D or 3D representations of surroundings, primarily physical maps, while attention points and attraction fields represent emerging intangible mapping.Three-dimensional mapping remains complex because of calibration restrictions, with multi-camera depth inference offering partial relief.

2.2 Subtasks

The survey organizes FPV analysis around 15 co-dependent subtasks, emphasizing how outputs from smaller tasks feed broader objectives. Hand detection, scene and object identification, and gaze estimation are especially interconnected across objectives.

  • 2.2 Subtasks: The proposed hierarchy contains 15 subtasks linked to objectives through a many-to-many relationship.A subtask may serve multiple objectives, while one objective may require several subtasks.
  • 2.2 Subtasks: Hand detection supports advanced objectives including object recognition and user-machine interaction.It can also provide cues for activity recognition.
  • 2.2 Subtasks: Global scene identification and object identification are important subtasks for activity recognition.The survey highlights a tight relationship between activity recognition and object recognition.
  • 2.2 Subtasks: Gaze estimation is used across multiple objectives, supporting the use of eye-trackers with FPV videos.This reflects the survey’s observation that eye-tracking is a recent and useful trend in FPV analysis.
  • 2.2 Subtasks: Background subtraction is less prominent than in fixed-camera scenarios because first-person backgrounds are highly unstable.The survey contrasts this reduced role with the stability typically available in fixed-camera settings.

2.3 Video and image features

FPV feature extraction spans pixel, frame, and temporal levels, with feature choice shaped by dynamic scenes and computational constraints. Color histograms and motion features are common, while superpixels and shape features address efficiency and robustness.

  • 2.3 Video and image features: FPV feature extraction progresses from pixel-level color channels to frame-level indicators such as saliency, texture, superpixels, and gradients.These frame features can support subtasks including object recognition and scene identification.
  • 2.3 Video and image features: Motion features add dynamic information through optical flow, geometrical frame transformations, or temporal aggregation.They are often computationally expensive and therefore commonly used when videos are processed after activities finish.
  • 2.3 Video and image features: 60% lower computational complexity was reported for a method that initializes current-frame superpixels using information from the previous frame.The method uses temporal continuity in superpixels to reduce computation.
  • 2.3 Video and image features: Color histograms are the most commonly used feature across subtasks, despite their dependence on illumination changes.Motion features are especially used for activity recognition and video summarization, whereas object recognition mainly uses frame features such as FPD and color histograms.
  • 2.3 Video and image features: Color features can reduce computational complexity for hand detection and segmentation, while HOG can filter erroneous measurements under large illumination changes.The survey reports that a suitable color space is important and that shape features improve classification robustness.

2.4 Methods and algorithms

FPV methods combine machine learning, probabilistic models, clustering, and feature encoding to map extracted features to objectives. The survey emphasizes trade-offs among interpretability, adaptability, training requirements, and computational cost.

  • 2.4 Methods and algorithms: Quantitative methods use selected features as inputs to produce objective outputs, with method choice influencing result quality and computational demands.The survey organizes these methods by their use for objectives and subtasks.
  • 2.4 Methods and algorithms: Classifiers are the most popular FPV tool, with SVMs especially common for assigning categories to feature arrays.Applications range from scene recognition to activity recognition based on detected objects.
  • 2.4 Methods and algorithms: 90% classification rate and 93% true-positive and true-negative rates were achieved by the HOG-SVM combination for hand detection.The comparison included HOG, GIST, and Color features with SVM, Random Forest, and Random Threes classifiers.
  • 2.4 Methods and algorithms: Feature encoding methods such as Bag-of-Words help control the size of large feature vectors created by combining multiple features.BoW can also incorporate human intuition, including object presence, attention level, and object role.
  • 2.4 Methods and algorithms: Unsupervised learning requires less human interaction and adapts more easily to new objects or uncontrolled environments, but its results require human interpretation.Clustering algorithms such as k-means are the most common unsupervised methods in FPV.
  • 2.4 Methods and algorithms: Supervised methods are interpretable and operate on labeled inputs, but require more training and depend more heavily on the training data.Weakly supervised learning is presented as a middle ground that can reduce large training-data requirements and handle unexpected data.

3 PUBLIC DATASETS

Public FPV datasets have expanded in scale, sensing, camera placement, objectives, and environmental realism. The survey also reports evidence that camera position affects feature discriminativeness in FPV and TPV classification.

  • 3 PUBLIC DATASETS: The first public FPV dataset contained 600 frames recorded in a controlled office with a shoulder-mounted camera during interaction with five objects.A later dataset included two people interacting with 42 object instances.
  • 3 PUBLIC DATASETS: 80.9% classification accuracy was obtained using blur, illumination changes, and optical flow as input features across FPV and TPV datasets.The study also found that camera position substantially affected classification rates.
  • 3 PUBLIC DATASETS: More stable cameras produce less blur and motion, reducing the discriminative power of blur, illumination-change, and optical-flow features.The survey identifies the definition of FPV for hand-held first-person recordings as an open issue.
  • 3 PUBLIC DATASETS: Dataset cameras evolved from shoulder-based to head-based placements alongside smart-glasses and action-camera adoption.Dataset objectives also expanded from low-level object recognition to social interaction and user-machine interaction.
  • 3 PUBLIC DATASETS: Recent datasets include less controlled environments and YouTube sequences recorded with GoPro cameras to test method robustness.Some datasets also combine multiple sensors, including eye tracking, RGBD cameras, GPS, and other sensing modalities.

4 CONCLUSION AND FUTURE RESEARCH

The survey finds substantial FPV development while identifying unresolved commercialization, real-time, adaptability, personalization, and multi-device challenges. It outlines future research toward self-adaptive, personalized, cooperative, and battery-efficient systems.

  • The survey summarizes FPV methods across academic and commercial developments, highlighting achievements and emerging research directions.It organizes the field’s evolution and hierarchical structure over the past 20 years.
  • Privacy, legal regulation, and user control over recording remain important issues before wearable FPV technology can be properly commercialized.These concerns vary with country and with whether users avoid recording private places or activities.
  • Existing methods generally process previously recorded videos rather than continuously learning and adapting in closed-loop operation.The survey identifies cognitive approaches as a possible direction for developing self-adaptive devices.
  • Personalized training from the device owner may help address user-specific variation, including changes in skin-color models for hand detection.The survey also notes that color features have been especially suitable in this field.
  • The survey primarily covers tasks involving one user and one wearable device, leaving cooperative devices as an opportunity for applications such as mapping, military activities, games, and sports.Multi-device systems could expand the range of supported applications.
  • Real-time FPV analysis still requires hardware and software advances that reduce computation without draining device batteries.The survey calls for improved processing units and lighter, faster, better-optimized methods.
Loading 1409.1484v3…