PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Storytelling from Visual Lifelogging: An Overview

Marc Bolaños, Mariella Dimiccoli, Petia Radeva

arXiv:1507.06120v5cs.CV

TL;DR

Automatically telling stories from egocentric photo streams remains challenging, despite their potential for long-term life observation. This review synthesizes visual lifelogging methods and concludes that photo-camera algorithms and richer datasets are important research needs.

  • Problem

    Automatically building stories from egocentric photo streams is challenging because camera motion and varied image content undermine visual analysis.

    Source: §I · pp. 1–2

  • Method

    The paper reviews methods for acquiring, organizing, summarizing, and browsing visual lifelog data, then organizes research around questions about interaction, location, time, and activity.

    Source: §I · p. 2, §V · p. 10

  • Results

    The review identifies a need for photo-camera algorithms, especially for social interaction, activity, and context recognition, alongside richer lifelogging datasets.

    Source: A. Egocentric Vision Datasets · p. 9, §V · p. 10

  • Takeaways & Limitations

    For photographic lifelog streams, semantic-oriented summarization and richer representations may better address event meaning and limited motion cues.

    Source: D. Egocentric Summarization · p. 4, C. Temporal Segmentation · p. 4

Abstract

from arXiv · show

Visual lifelogging consists of acquiring images that capture the daily experiences of the user by wearing a camera over a long period of time. The pictures taken offer considerable potential for knowledge mining concerning how people live their lives, hence, they open up new opportunities for many potential applications in fields including healthcare, security, leisure and the quantified self. However, automatically building a story from a huge collection of unstructured egocentric data presents major challenges. This paper provides a thorough review of advances made so far in egocentric data analysis, and in view of the current state of the art, indicates new lines of research to move us towards storytelling from visual lifelogging.

I. INTRODUCTION

Visual lifelogging uses wearable cameras to passively record everyday experiences from a first-person perspective, with long-term photo streams offering potential for understanding behavior. This review examines egocentric analysis through the goal of storytelling, emphasizing how methods and resources apply to low-temporal-resolution photo data.

I. INTRODUCTION

  • Photo cameras capture 2–3 frames per minute for long-term lifelogs, while video cameras record 25–60 frames per second for specific moments and detailed activity analysis.The review contrasts long-term behavioral inference from photo streams with the in-depth analysis of particular activities possible with video.Source: §I · p. 1
  • Abrupt changes, occlusions, artefacts, non-informative images, and up to 2,000 images daily make visual lifelog analysis and relevant-content retrieval difficult.Free camera motion and low temporal resolution also make temporal-coherence and motion-estimation methods unreliable.Source: §I · p. 1
  • The authors identified 274 visual-lifelogging papers through November 2015, and publications increased by over 50% in 2014 alone.Their search found relatively little work analyzing photo-camera streams, apart from a temporary increase associated with SenseCam’s popularity from 2007 to 2010.Source: §I · p. 2
  • The review assesses work on both photo and video cameras, considering which methods need reformulation for low-temporal-resolution storytelling.The paper frames its review around how far automated storytelling from egocentric photo streams remains.Source: §I · p. 2
  • The literature is organized around interaction, location, time, and activity questions, with methods assessed for low-temporal-resolution applicability and datasets, code, and open problems reviewed.The paper also surveys methods for acquiring, organizing, summarizing, and browsing unstructured collections, and identifies research directions toward storytelling.Source: §I · p. 2

II. VISUAL LIFELOGGING ACQUISITION, SEGMENTATION AND SUMMARIZATION

The section reviews methods for acquiring, structuring, summarizing, and browsing visual lifelogging data. Its remarks highlight camera-placement trade-offs and the need to clean and organize images around meaningful events.

II. VISUAL LIFELOGGING ACQUISITION, SEGMENTATION AND SUMMARIZATION

  • The survey reviews work on acquiring, organizing, summarizing, and browsing large collections of unstructured visual lifelogging data.Table I summarizes the reviewed papers in these areas.Source: §II · p. 2, Table I

A. Data Acquisition

  • Chest placement is usually preferred for long-term lifelogging because it is socially acceptable and captures the handling space, though its field of view is narrow.Shoulder placement offers a compromise among field of view, access to handling regions, motion sensitivity, and social acceptability.Source: §II-E · p. 3

B. Informative Image Detection

  • Non-intentional lifelogging images require cleaning to remove artefacts and uninformative scenes before structuring, analysis, or summarization.The reviewed definitions describe informative images as intentional or as showing people or objects at reasonable quality without artefacts.Source: B. Informative Image Detection · p. 3

II-C Temporal Segmentation

  • Desired event segmentation groups lifelogging images by their main event, considering the activities, objects, or people involved.Source: Figure 4 · p. 3

C. Temporal Segmentation

Temporal segmentation organizes lifelogging streams into event or environment segments, using either scene characteristics or the wearer’s activity. For summarization, the review favors semantic approaches and finer-grained event segmentation suited to photo streams.

C. Temporal Segmentation

  • Scene-based segmentation ranges from low-level visual features and sensor metadata to CNN image features combined with temporal graph-cut and clustering methods.Reviewed approaches include time-constrained K-means, Gaussian mixture models, and graph cuts with agglomerative clustering and concept drift.Source: C. Temporal Segmentation · p. 3
  • Activity-based segmentation commonly uses optical flow, while integral motion targets long-term ego-activities; one approach removes head-rotation variation to retain forward motion.A CNN-based activity method trained on one user can be retrained for new users, but requires several thousand newly labelled images for each new user.Source: C. Temporal Segmentation · p. 3, Figure 5 · p. 4
  • Photo streams need richer representations than motion alone, time-dependent segmentation, and potentially semantic cues to bring machine segmentation closer to human judgments.Motion features are more applicable to video than to photo streams.Source: C. Temporal Segmentation · pp. 3–4

D. Egocentric Summarization

  • Summarization condenses a sequence into representative frames or segments, as either a static storyboard or a shorter dynamic skim.For lifelogs, it should support visualization, indexing, and browsing of autobiographical events with minimal semantic loss.Source: D. Egocentric Summarization · p. 4
  • Storyboard methods select keyframes from coherent groups, while later approaches add semantic signals such as important people, objects, narrative coherence, topics, relevance, and diversity.Some methods also filter non-informative images before ranking event keyframes.Source: D. Egocentric Summarization · p. 4
  • Semantic-oriented summarization is considered most suitable for lifelogging; photo-camera methods need keyframe selection adapted to narrative sub-shots or finer event segments.Fine-grained segmentation should create enough events for sub-shot selection.Source: D. Egocentric Summarization · pp. 4–5

E. Content-Based Search and Retrieval

The section reviews content-based search and retrieval methods for browsing personal image databases, alongside open challenges in representing lifelogging content. It also situates visual-lifelogging analysis around questions of interaction, place, time, and activity.

E. Content-Based Search and Retrieval

  • A content-based browsing system combines time attributes, visual and audio features, clustering, and user query images, but recall, simple visual descriptors, and unavailable audio remain issues.Another approach matches new sequences to previously acquired alignment sequences using dynamic time warping.Source: E. Content-Based Search and Retrieval · p. 5
  • Semantic representations include concept combinations and sparse image graphs; local-density clustering suits highly redundant lifelogging data better than global clustering.Source: E. Content-Based Search and Retrieval · p. 5
  • Open retrieval challenges include using recognized actions, people, objects, and environments to form multi-level, multimodal descriptions, while photo data has greater consecutive-image variability than video.Such descriptions could approach text-level detail and allow high retrieval accuracy.Source: E. Content-Based Search and Retrieval · p. 5

III. VISUAL LIFELOGGING ANALYSIS

  • The review organizes visual-lifelogging analysis around interaction, location, event timing, and what the camera wearer is doing.Table II lists the reviewed analysis-related papers.Source: §III · p. 5, Table II

A. Interacting? How?: Social Interactions

The review treats social interactions as important cues for understanding behavior and event memorability, and describes F-formations as a model for analyzing them. Egocentric detection relies on people tracking, head orientation, and 3D location, but photographic sequences pose particular challenges.

A. Interacting? How?: Social Interactions

  • F-formations model social interaction through people’s spatial and orientational relationships around a shared space,.Social interactions are also associated with event memorability, making their detection potentially useful for keyframe extraction or event-importance estimates.Source: A. Interacting? How?: Social… · p. 5
  • Detecting F-formations requires estimating head pose and people’s 3D locations to infer attention, proximity, and interaction.Different circular, vis-à-vis, L-shaped, and side-by-side arrangements are illustrated in Fig. 6.Source: A. Interacting? How?: Social… · p. 6, Figure 6 · p. 6
  • For photo sequences, frequent abrupt view changes make people tracking difficult; one approach groups forward and backward face correspondences into person-specific tracklet bags,,.Most other methods described rely on temporal coherence and were designed for video sequences.Source: A. Interacting? How?: Social… · p. 6
  • Social-interaction methods need to adapt the significance of distance and orientation to context, while camera-wearing position also changes those cues.The review points to body movements as potentially less time-sensitive cues for photographic cameras.Source: A. Interacting? How?: Social… · p. 6

B. Where? Scene Understanding

Scene understanding aims to identify objects, people, and environments around the wearer. The review covers object recognition and discovery, spatial localization, and scene parsing, emphasizing that lifelog settings differ from conventional datasets and camera views.

B. Where? Scene Understanding

  • Lifelog scene understanding seeks cues from objects, people, and environments to determine what techniques can best interpret the wearer’s surroundings.The review covers object recognition, spatial localization, scene parsing, and scene recognition.Source: B. Where? Scene Understanding · p. 6

1) Object Recognition and Object Discovery:

  • Object recognition identifies known categories, while object discovery detects and reveals objects the algorithm may not have seen before.Lifelog objects are often occluded and vary in appearance; early supervised recognition work found temporal consistency in visual properties and concepts.Source: 1) Object Recognition and Object… · p. 6
  • Active labeling uses clustering to select informative examples for user labeling, while hand-held object methods separate moving foreground objects from more static backgrounds,,,.Object-discovery methods cluster correlated segments, merge parts that co-occur, or combine objectness detection and CNN features with clustering and knowledge reuse,, [16].Source: 1) Object Recognition and Object… · p. 7

2) Spatial Localisation:

  • FOV localization matches wearable-camera images to reference imagery, while 3D scene models can recover information from previously seen scenes [13],.The reference data can include Google Street View or fixed-camera video, and the localization system can handle multiple wearable-camera users.Source: 2) Spatial Localisation · p. 7
  • Scene parsing assigns object classes at pixel level, but the reviewed work was neither designed nor tested specifically on egocentric or lifelog datasets.Conventional scene-parsing datasets emphasize natural and urban outdoor scenes, unlike lifelogs’ largely indoor routine settings; lifelog methods would benefit from context and knowledge reuse.Source: 2) Spatial Localisation · p. 7
  • High-level scene description remains incomplete; lifelog methods need to handle blur, saturation, and occlusion, and scene parsing has no reported egocentric application in this review.The review identifies Fully Convolutional Networks as a promising approach and notes that GPS or visual localization could improve environmental understanding; scene recognition could also inform event segmentation.Source: 2) Spatial Localisation · pp. 7–8

1) Object Recognition and Object Discovery:

  • Object-recognition work relies on head-mounted views, whereas practical applications often use less obtrusive neck-hung cameras; video-frame and motion-dependent algorithms are also unsuitable for low-temporal-resolution photographic cameras.Neck-hung cameras may not always show what the wearer is doing.Source: 2) Spatial Localisation · p. 8

C. When? Time-Based Localisation

The review describes timestamps as a basis for ordering lifelog images and locating them in time, while noting that time alone may not retrieve memories. It then surveys activity recognition, whose methods and target categories depend on the application and available camera signals.

C. When? Time-Based Localisation

  • Camera timestamps establish long-term placement, image order, and short-term temporal distance in photographic lifelogs.They have been used for event segmentation and event classification, but a psychological study found that time and date alone did not enable memory retrieval,, [23],.Source: C. When? Time-Based Localisation · p. 8

D. What? Action Recognition

  • Activity categories depend on the application, ranging from eating for diet monitoring to dressing, coffee-making, and cooking for dementia-related diagnosis.The review also names sports, housework, television, work or study, and eating or drinking as relevant categories across applications.Source: D. What? Action Recognition · p. 8
  • Action-recognition representations commonly use body movement or action-related objects, with scene context introduced more recently; the appropriate representation depends on the actions being classified.Fig. 10 illustrates first-person images of sports activities.Source: D. What? Action Recognition · p. 8, Figure 10 · p. 8

1) Body movement-based methods:

  • Video-based ego-motion methods classify activities from motion patterns, including cumulative displacement that isolates consistent forward motion from head rotation.These approaches depend on motion features being available at the camera’s temporal resolution; other work used modified motion features in a Bayesian clustering model.Source: 1) Body movement-based methods · p. 8

2) Object-hand interaction-based methods:

The review surveys hand-object, attention-based, and other activity-recognition approaches. Their applicability depends on what the camera captures: motion, manipulation, and gaze cues can be unavailable or unreliable with photographic or freely worn cameras.

2) Object-hand interaction-based methods:

3) Attention-based methods:

  • Attention-based methods identify attended objects without requiring manipulation, but generally require head-, eyeglass-, or ear-mounted cameras that can follow gaze,.Source: 3) Attention-based methods · p. 9

4) Other approaches:

  • Other approaches use local and global motion and visual-word features for activities not captured by movement, manipulation, or gaze; dense trajectories reached about 80% accuracy on the LENa dataset.The review also describes temporal pooling of CNN frame features, information sharing across tasks, and activity methods combining object and scene understanding,,.Source: 4) Other approaches · p. 9
  • Body-movement methods are not directly applicable to photo streams, while attention and hand-object methods can be difficult with necklace- or clothing-mounted cameras.The review reports no published activity-recognition work on freely worn cameras and identifies wearer motion state as a requirement for robust recognition.Source: 4) Other approaches · p. 9

A. Egocentric Vision Datasets

The review catalogs public egocentric datasets and their annotations, while identifying a shortage of datasets suited to long-term visual lifelogging and storytelling evaluation.

A. Egocentric Vision Datasets

  • Table III summarizes public datasets by name and reference, description, annotation types, and acquisition camera.Source: A. Egocentric Vision Datasets · p. 9
  • Only EDUB [16] and AIHS among the public egocentric datasets use photographic cameras, limiting their suitability for testing visual lifelogging algorithms.Most other datasets use video cameras, which make analysis of long periods difficult.Source: A. Egocentric Vision Datasets · p. 9
  • The review identifies a lack of well-established, widely accepted egocentric datasets and calls for richer datasets with detailed annotations.Source: A. Egocentric Vision Datasets · p. 9

B. Egocentric Vision Software

The review presents egocentric-vision software repositories and frames source-code availability as important for reproducibility and quantitative comparison.

B. Egocentric Vision Software

  • Publishing source code is crucial for reproducibility and quantitative comparisons across datasets.Source: B. Egocentric Vision Software · p. 9
  • The review lists relevant public repositories for object recognition and discovery, activity recognition, event segmentation, keyframe-based summarization, and informative-image detection.Source: B. Egocentric Vision Software · p. 10

V. CONCLUSIONS AND FUTURE DIRECTIONS

The review surveys visual-lifelog analysis from a storytelling perspective and identifies research directions for adapting analysis to photographic lifelogs. It highlights richer datasets, semantic approaches, and text descriptions as areas for further work.

V. CONCLUSIONS AND FUTURE DIRECTIONS

  • The survey reviews acquisition, organization, summarization, and browsing methods, then assesses methods for answering who, how, where, when, and what questions in lifelog stories.It also reviews available datasets and source code.Source: §V · p. 10
  • More photo-camera algorithms are needed, particularly for social-interaction analysis, activity recognition, and context recognition.Source: §V · p. 10
  • The review recommends community cooperation on richer lifelogging datasets to support algorithm validation and competition.Source: §V · p. 10
  • Semantic information and ontologies are identified as promising resources for preserving story semantics and addressing varied lifelogging settings.Source: §V · p. 10
  • The review proposes transfer learning and temporal coherence of concepts as research directions, while noting CNN computational costs would limit analysis to post-processing.Source: §V · p. 10
  • Text description generation from images is identified as an unexplored storytelling direction; cited examples include and.Source: §V · p. 10
  • The authors suggest that image-to-text methods could provide human-like descriptions of scenes or events and richer context for people with dementia.Source: §V · p. 10

VINST - Visual Diaries [4]

The supplied passages describe varied egocentric datasets, including interaction, activity, and hand-related data, with differing settings and annotations.

VINST - Visual Diaries [4]

  • The UCI Activities of Daily Living Dataset contains one million frames of 18 indoor activities performed by 20 subjects, with 42 object classes annotated.Source: VINST - Visual Diaries [4] · p. 14
  • EGO-GROUP provides 10 videos of social-group scenarios recorded in settings including a laboratory, coffee break, conference room, and outdoors.Source: VINST - Visual Diaries [4] · p. 14
  • The NUS First-person Interaction Dataset contains 260 videos spanning 16 classes across first- and third-person perspectives.Its classes include human-human, human-object-human, and human-object interactions.Source: VINST - Visual Diaries [4] · p. 14
  • CMU-MMAC combines cooking videos from 18 subjects with audio, motion-capture, and IMU data.Source: VINST - Visual Diaries [4] · p. 14
Loading 1507.06120v5…