PaperlayerView exact revision on arXiv

Source-linked AI summary

Object Referring in Videos with Language and Human Gaze

Arun Balajee Vasudevan, Dengxin Dai, Luc Van Gool

arXiv:1801.01582v2cs.CV

TL;DR

Object referring methods largely target static images, which lack motion and spatial-temporal cues, while speakers’ gaze can also help identify a referent. The paper presents a stereo-video dataset annotated with descriptions and gaze, and a model integrating appearance, motion, gaze, and context; experiments report that it uses these cues effectively and consistently outperforms image-based methods.

  • Problem

    Object referring has largely focused on static images, which lack many motion and spatial-temporal cues used in descriptions of dynamic scenes.

    Source: §1 · p. 1

  • Method

    The paper introduces a stereo-video dataset with descriptions and gaze recordings and a Temporal-Spatial Context Recurrent ConvNet integrating appearance, motion, gaze, and context.

    Source: §1 · p. 2

  • Results

    The model effectively utilizes motion, gaze, and spatio-temporal cues from stereo videos and consistently outperforms image-based object referring methods.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Video object referring can draw on motion and spatial-temporal cues that static images fall short of providing.

    Source: §1 · p. 1

    The method assumes the target frame is known in advance, although real applications need to infer it from speech and the visual scene.

    Source: §3 · p. 3

Abstract

from arXiv · show

We investigate the problem of object referring (OR) i.e. to localize a target object in a visual scene coming with a language description. Humans perceive the world more as continued video snippets than as static images, and describe objects not only by their appearance, but also by their spatio-temporal context and motion features. Humans also gaze at the object when they issue a referring expression. Existing works for OR mostly focus on static images only, which fall short in providing many such cues. This paper addresses OR in videos with language and human gaze. To that end, we present a new video dataset for OR, with 30, 000 objects over 5, 000 stereo video sequences annotated for their descriptions and gaze. We further propose a novel network model for OR in videos, by integrating appearance, motion, gaze, and spatio-temporal context into one network. Experimental results show that our method effectively utilizes motion cues, human gaze, and spatio-temporal context. Our method outperforms previousOR methods. For dataset and code, please refer https://people.ee.ethz.ch/~arunv/ORGaze.html.

1. Introduction

Object referring in videos addresses cues that static images lack, including motion, spatial-temporal context, and speaker gaze. The paper contributes a stereo-video dataset with descriptions and gaze recordings, plus a model integrating these cues with appearance.

1. Introduction

  • Static-image object referring misses motion and spatial-temporal cues used to describe dynamic scenes, motivating object referring in videos.Examples include a car turning left and a boy running under a tree.Source: §1 · p. 1
  • Speakers’ gaze can help co-observers localize a referred object, so the paper combines gaze with language and video cues.The system uses appearance, motion, and depth features alongside expression and gaze.Source: §1 · p. 1, Figure 1 · p. 1
  • The dataset contains 30,000 objects in 5,000 stereo video sequences, with descriptions guided by Gricean Maxims and gaze recordings for annotated objects.Descriptions are intended to be truthful, informative, relevant, and brief so co-observers can identify targets easily and unambiguously.Source: §1 · p. 2
  • The Temporal-Spatial Context Recurrent ConvNet integrates appearance, motion, gaze, and spatio-temporal context for video object referring.The paper describes the approach as learning from these modalities together in one network.Source: §1 · p. 2

2. Related Work

Related work connects language-based object referring to broader language-vision tasks and to methods for embedding or conditioning visual and textual information. The paper extends image-based work to stereo video and gaze, alongside prior datasets and gaze applications.

2. Related Work

  • The paper identifies referring expression generation and language-based object detection as especially relevant areas of language-visual research.These topics share fundamental techniques with image captioning and visual question answering.Source: §2 · p. 2
  • Multimodal embedding learns shared text-visual representations, while conditional LSTMs generate language conditioned on visual input.The paper notes that its model conditions LSTMs on motion, depth, and gaze as well as images.Source: §2 · p. 2
  • Object referring includes referring expressions [57] [34], phrase localization, phrase grounding [44] [40], retrieval [23], and segmentation [22].Prior approaches learn vision-language embeddings [15] [25] or model global or local context [34] [23] [36] [57].Source: §2 · p. 2
  • Existing large-scale referring-expression datasets use static images, whereas this paper presents stereo-video data encouraging descriptions of 3D configuration and motion when useful.The paper notes that image datasets omit motion and 3D spatial configuration.Source: §2 · p. 2
  • Prior gaze research includes object localization and related vision tasks; this paper records gaze through an AMT web interface for video object localization.The authors say the interface was inspired by Krafka et al..Source: §2 · p. 2

3. Approach

The approach casts video object referring as scoring candidate boxes against a referring expression using visual and gaze cues. Its recurrent architecture combines local and global information, with proposals and features drawn from stereo video, motion, depth, and gaze.

3. Approach

  • The model scores candidate boxes using appearance, motion, depth, gaze, spatial configuration, and global temporal-spatial context, then selects the highest-scoring candidate.The task is evaluated at the frame where the expression is issued, which the method assumes is known in advance.Source: §3 · p. 3

3.1. Network Architecture

  • Local and global visual-language LSTMs predict expression words from concatenated language and visual features, and candidate scores derive from the expression’s generative probability.The architecture extends a static-image model to incorporate stereo-video information and human gaze.Source: §3.1 · pp. 3–4

3.2. Object Proposals

  • Language-based Object Proposals provide candidate boxes, which the more complex referring model then verifies.The proposal method is selected because it produces expression-aware candidates.Source: §3.2 · p. 4

3.3. Feature Encoding

  • The feature pipeline encodes local and global appearance, depth, and optical flow, alongside gaze estimates and each box’s spatial configuration.Local features may use the current frame alone or multiple preceding frames for longer-range temporal interaction; gaze estimates are pooled over each object’s box.Source: §3.3 · p. 4

4. Dataset Annotation

The authors build a video object-referring dataset from Cityscapes stereo sequences, adding language descriptions, object boxes, and gaze recordings. The annotation protocol uses the final video frame for referring expressions and records workers’ gaze while they view each annotated object.

4. Dataset Annotation

  • The new dataset addresses the lack of prior datasets for learning and evaluating temporal, spatial-context, and gaze information.The authors note that collecting all modalities simultaneously in realistic human-to-robot interactions would be labor-intensive and infeasible to crowdsource.Source: §4 · pp. 4–5
  • Annotators watched each video and labeled objects in its 30th frame to ensure sufficient temporal context.The videos are two seconds long and comprise 30 frames.Source: §4 · p. 5
  • The authors annotated 30,000 objects in 5,000 Cityscapes videos, collecting object bounding boxes, class names, and referring expressions; they also recorded human gaze for objects already annotated with expressions and boxes.Source: §4 · p. 5
  • Referring expressions averaged 15.59 words, and 53.99% of annotations referred to cars.The dataset includes 20 object classes and averages 4.2 referring expressions per image.Source: §4 · p. 5
  • Workers recorded gaze in a separate task by watching each video and gazing at each previously annotated object while their faces were recorded.When face detection failed, gaze recordings were repeated for the affected videos.Source: §4 · pp. 5–6

5. Experiments

The experiments evaluate multimodal object referring on Cityscapes, comparing proposal and feature choices before testing gaze and temporal cues. Depth, motion, and gaze improve reported performance, with the four-modality model producing the largest stated gain.

5.1. Implementation Details

  • The system extracts RGB, HHA depth, optical-flow, and gaze features, tracking objects from annotated boxes on Cityscapes’ final frame.Source: §5.1 · p. 6

5.2. Evaluation

  • 8.367% improvement: combining appearance, depth, motion, and gaze yields the best reported OR results.Source: Table 1 · p. 6, §5.2 · p. 7
  • Depth and motion together improve Acc@1 by 5.105% over NLOR under LOP; depth alone improves it by 2.743% and motion by 3.855%.Source: Table 1 · p. 6, §5.2 · p. 7
  • Longer flow tracks do not significantly improve accuracy; the authors cite last-frame descriptions and Cityscapes’ two-second clips as possible constraints.Source: Table 2 · p. 6, §5.2 · p. 7
  • Evaluation uses 80% of Cityscapes videos for training and 20% for evaluation; Acc@1 counts top-scoring candidates with IoU above 0.5 as true detections.Source: §5.2 · pp. 6–7
  • Gaze raises performance by 3.773% for Ours (I), 1.748% for Ours (I,O), and 3.262% for Ours (I,D,O) under max pooling.Source: Table 3 · p. 7, §5.2 · p. 8

6. Conclusions

The paper presents a video object-referring dataset and an end-to-end Temporal-Spatial Context Recurrent ConvNet combining appearance, motion, depth, gaze, and spatio-temporal context. The model effectively uses these cues and consistently outperforms image-based OR methods; evaluation in real human-to-robot communication remains future work.

6. Conclusions

  • The model effectively uses motion, gaze, and stereo-video spatio-temporal context, consistently outperforming image-based OR methods.Source: §6 · p. 8
  • The dataset contains 30,000 objects annotated across 5,000 video sequences.The paper also reports gaze recordings for all annotated objects.Source: §6 · p. 8
  • The end-to-end Temporal-Spatial Context Recurrent ConvNet integrates appearance, motion, depth, human gaze, and spatio-temporal context.Source: §6 · p. 8
  • Evaluation in a real human-to-robot communication system, especially of the multiple modalities’ contribution, is left for future work.Source: §6 · p. 8

Related papers

Loading 1801.01582v2…