PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder

Jingkuan Song, Hanwang Zhang, Xiangpeng Li, Lianli Gao, Meng Wang, Richang Hong

arXiv:1802.02305v1cs.CV

TL;DR

Existing video hashing can lose temporal order through frame pooling and separated binarization. SSVH addresses this with an end-to-end hierarchical binary autoencoder and neighborhood structure, reporting superior unsupervised retrieval performance on FCVID and YFCC.

  • Problem

    Existing video hashing inadequately captures temporal order because frame pooling and separated binarization can cause information loss.

  • Method

    SSVH uses Binary LSTM within a hierarchical binary autoencoder to generate codes while reconstructing visual content and neighborhood structure.

  • Results

    SSVH achieves superior performance compared with other unsupervised video hashing methods on FCVID and YFCC.

  • Takeaways & Limitations

    The framework provides an unsupervised binary video-hashing approach that explicitly exploits long-term temporal information.

  • Takeaways & Limitations

    The model relies on pre-trained VGGNet frame features that ignore consecutiveness and temporal information.

Abstract

from arXiv · show

Existing video hash functions are built on three isolated stages: frame pooling, relaxed learning, and binarization, which have not adequately explored the temporal order of video frames in a joint binary optimization model, resulting in severe information loss. In this paper, we propose a novel unsupervised video hashing framework dubbed Self-Supervised Video Hashing (SSVH), that is able to capture the temporal nature of videos in an end-to-end learning-to-hash fashion. We specifically address two central problems: 1) how to design an encoder-decoder architecture to generate binary codes for videos; and 2) how to equip the binary codes with the ability of accurate video retrieval. We design a hierarchical binary autoencoder to model the temporal dependencies in videos with multiple granularities, and embed the videos into binary codes with less computations than the stacked architecture. Then, we encourage the binary codes to simultaneously reconstruct the visual content and neighborhood structure of the videos. Experiments on two real-world datasets (FCVID and YFCC) show that our SSVH method can significantly outperform the state-of-the-art methods and achieve the currently best performance on the task of unsupervised video retrieval.

I. INTRODUCTION

Large-scale content-based video retrieval needs compact codes that preserve temporal information, but existing approaches often pool frames or rely on costly stacked architectures. SSVH addresses this with an end-to-end hierarchical binary autoencoder and neighborhood structure.

  • Motivation: Massive online video collections make large-scale content-based video retrieval an urgent need.The passage cites rapid growth in video uploads and views as motivation.
  • Motivation: Frame pooling discards temporal order, causing information loss and suboptimal binary video codes.The loss particularly affects nuanced dynamics such as long-term event evolution.
  • Motivation: RNN-based hashing captures sequential information but commonly requires human labels that are costly to obtain at scale.The paper therefore targets unsupervised temporal-aware hashing.
  • Proposed framework: SSVH jointly encodes temporal and visual information with a hierarchical binary autoencoder and neighborhood structure.The framework is designed to address temporal modeling, binary optimization, and retrieval representation together.
  • Proposed framework: Binary LSTM provides the temporal-aware hash-function building block and directly tackles binary optimization without relaxation.This is presented as a core methodological contribution.
  • Results: Experiments on FCVID and YFCC report superior performance over other unsupervised video hashing methods.The datasets contain 91,223 and 700,882 videos, respectively.

A. Hashing

Hashing maps high-dimensional data to compact binary codes for efficient retrieval, while video hashing must additionally preserve temporal structure. SSVH combines hierarchical recurrent encoding, reconstruction, and neighborhood-based learning in an unsupervised framework.

  • Hashing: Hashing transforms high-dimensional features into compact binary codes supporting efficient storage and Hamming-space search.Binary codes enable fast bitwise operations for approximate nearest-neighbor retrieval.
  • Hashing: Supervised video hashing uses labels or pairwise labels, whereas unsupervised methods exploit data distribution or manifold structure.The cited supervised methods include VHDT, HashNet, and DPSH; ITQ is an unsupervised example.
  • Video hashing: Prior video hashing methods variously ignore temporal order, require labels, produce frame-level codes, or use non-binary quantization codes.SSVH is contrasted as unsupervised, binary, and explicitly temporal.
  • Video content analysis: LSTM-based encoder-decoder models process sequential video information and can exploit longer-range structure through hierarchical recurrence.Hierarchical recurrence reduces the length of the input information flow.
  • SSVH formulation: SSVH is formulated with a hierarchical binary autoencoder, neighborhood structure, and an optimization method for learning video hash codes.The method section introduces the problem definition before these components.

1) Hierarchical Binary Encoder:

The hierarchical binary encoder converts frame features into compact binary video codes while modeling temporal structure at multiple granularities. Its sparse inter-layer connections shorten information flow and reduce computation relative to stacking.

  • Encoder structure: The encoder uses a two-layer hierarchical recurrent structure combining vanilla LSTM with Binary LSTM.The first layer extracts higher-level frame features, which feed the binary layer.
  • Binary encoding: The Binary LSTM embeds higher-level real-valued video features into a binary code.The encoder output is b ∈ {−1, 1}^L.
  • Hierarchical connections: Unlike stacked LSTM, hierarchical BLSTM connects only selected first-layer outputs to the second layer according to a stride.For stride l, the output at time step l×i feeds the i-th second-layer step.
  • Hierarchical connections: Sparse hierarchical connections compose consecutive frames at a higher level while reducing the length of input information flow.The paper reports significantly reduced computation operations compared with stacking.

2) Forward Hierarchical Binary Decoder:

The forward hierarchical binary decoder reconstructs frame features in temporal order from the binary video code. It uses hierarchical LSTM decoding with sparse inter-layer connections followed by linear reconstruction.

  • Forward reconstruction: The forward decoder reconstructs input frame features in forward temporal order using the binary code b.Its outputs are reconstructed features v̄1, v̄2, ..., v̄M.
  • Decoder structure: The decoder contains two LSTM layers arranged hierarchically rather than fully connected across all time steps.The binary code initializes the first-layer hidden state.
  • Hierarchical decoding: With stride l, first-layer outputs are supplied to the second-layer input at selected i×l-th time steps.Intermediate steps receive zero input under the stated reconstruction rule.
  • Reconstruction loss: Linear reconstruction transforms decoder LSTM outputs into reconstructed frame features.The forward decoder loss is defined using Euclidean distance between original and reconstructed features.

3) Backward Hierarchical Binary Decoder:

The backward hierarchical binary decoder reconstructs frame-level features in reverse temporal order and supports global video-level reconstruction through mean-pooled features.

  • The backward decoder reconstructs frame-level features in reverse order, from ˜vM through ˜v1.Its first LSTM layer processes hidden states backward, while the second layer produces reconstructed features at corresponding reverse-time positions.
  • Linear reconstruction layers obtain the reconstructed frame features from the decoder LSTM outputs.The decoder uses a weight matrix and bias to produce the reconstructed features.
  • A global hierarchical decoder reconstructs video-level features formed by mean-pooling all frame-level features.The global reconstruction uses two basic LSTMs and adds a video-level reconstruction objective alongside forward and backward reconstruction.

5) Hierarchical Binary Auto-Encoder Loss Function:

The loss combines forward, backward, and global reconstruction with a neighborhood objective so binary codes preserve both video content and relational structure.

  • The hierarchical binary auto-encoder uses forward, backward, and global reconstruction losses.These three components jointly reconstruct frame-level and video-level information.
  • Neighborhood preservation encourages similar videos to obtain close binary codes and dissimilar videos to obtain different codes.The paper motivates this objective because reconstruction alone does not ensure accurate retrieval.
  • The neighborhood structure is constructed before training from VGG frame features, mean-pooled video representations, and cosine similarity.The preprocessing identifies neighboring videos using video-level representations.
  • The method expands neighborhood information by intersecting the indexes of highly relevant videos before selecting additional neighbors.A small K1 may preserve too little information, whereas a large K1 reduces retrieved-neighbor accuracy.
  • The neighborhood structure is fixed after preprocessing because updating it during training is time-consuming and yields no significant performance improvement.The authors therefore calculate the structure from extracted features and keep it fixed.
  • The final objective combines reconstruction loss and pairwise neighborhood loss, with λ balancing their contributions.The binary codes are related to the pre-binarization encoder states when imposing neighborhood-preserving regularization.

D. Optimization Method

SSVH obtains binary codes with a hierarchical binary encoder, reconstructs video features through three decoder modes, and uses an approximated sign function for trainable optimization.

  • The hierarchical binary encoder produces binary codes that reconstruct video frame features in forward, backward, and global modes.The encoder and the three decoder parameter groups are optimized through backpropagation.
  • The reconstruction loss is formulated from the decoder outputs and optimized jointly with the model parameters.Backpropagation updates the encoder, forward decoder, backward decoder, and global decoder parameters.
  • Training is NP-hard because binary hash-code optimization has a combinatorial search space.The method follows SSTH to address this optimization difficulty.
  • SSVH replaces the sign function with an approximated function p(h) during optimization.The approximation enables gradient-based training of the binary encoder.
  • The backpropagation rule passes gradients through the approximated sign function only when neural responses lie between -1 and +1.The paper also notes that tanh(h) can serve as an alternative approximation.

E. Comparison with SSTH

Compared with SSTH’s stacked recurrent structure, SSVH uses hierarchical recurrence to shorten computation paths and encode temporal information at multiple granularities.

  • SSVH replaces SSTH’s stacked LSTMs with a hierarchical recurrent structure that shortens the path from input to output.The design retains nonlinear modeling while improving the efficiency–effectiveness trade-off described by the authors.
  • The hierarchical architecture extracts video information at different time scales and learns hash codes with multiple granularities.
  • SSVH reconstructs videos with forward and backward hierarchical decoders plus a global hierarchical binary decoder.The global decoder is intended to preserve appearance information through global reconstruction.
  • SSVH additionally introduces a neighborhood structure to improve the representation ability of binary codes.

IV. EXPERIMENTS

Experiments evaluate SSVH on FCVID and YFCC using standard retrieval metrics and Hamming ranking. The implementation uses sampled frame features, SGD, and dataset-specific neighborhood construction.

  • Efficiency: The model trains at approximately 32 videos per second on a single NVIDIA TITAN X GPU.Experiments are implemented with Theano.
  • Datasets: SSVH is evaluated on FCVID and YFCC, two large-scale video datasets used for unsupervised video retrieval.FCVID contains 91,223 annotated videos, while the collected YFCC data includes 700,882 videos.
  • Implementation: Each video is represented by 24 equally spaced frames whose 4096-dimensional features are extracted with VGG.The hierarchical autoencoder uses stride 2, and the second decoder layer has 12 units.
  • Implementation: Training uses stochastic gradient descent with mini-batches of 256 and regularization parameters η = 0.2 and λ = 0.001.The neighborhood parameters are set to K1 = 20 and K2 = 10.

D. Components and Baseline Methods

The component study compares hierarchical reconstruction variants, neighborhood modeling, and unsupervised baselines on FCVID. Results indicate that the combined SSVH objectives improve retrieval while hierarchical structure reduces computation relative to stacked LSTMs.

  • Components: The study compares forward-backward reconstruction, global reconstruction, neighborhood structure, and their full SSVH combination.SSVH combines Lossf, Lossb, Lossg, and NeighborLoss.
  • Baseline Methods: SSVH is compared with ITQ, Submod, MFH, and DH, alongside SSTH as a stacked recurrent baseline.The baselines include both traditional and deep unsupervised hashing methods.
  • Results: The hierarchical forward-backward reconstruction variant outperforms SSTH while requiring less computational cost.Adding global reconstruction improves mAP by about 1% at different K values and stabilizes training.
  • Results: SSVH achieves the best performance among the tested component combinations, while human-label neighborhoods outperform the unsupervised neighborhood structure.This indicates that each included component contributes to the combined result and that more accurate neighbor similarity could improve performance.

2) Trade off between Neighbor Loss and Reconstruction Loss:

The experiments examine the balance between neighborhood and reconstruction losses, neighborhood sizes, activation functions, and code lengths. Performance depends on combining objectives and choosing neighborhood parameters appropriate to the data.

  • Loss Trade-off: The best performance is obtained at λ = 10^-3, whereas using only neighborhood loss or only reconstruction loss is inferior.The study evaluates λ values from 0 through 1 and reports mAP at top-20, top-40, top-60, top-80, and top-100 retrieval.
  • Neighborhood Parameters: Neighborhood parameters K1 and K2 both set to 10 produce the best FCVID performance, while values around 50 produce the worst.Small values preserve insufficient neighboring information, whereas large values merge similar video categories.
  • Activation Functions: The approximated sign activation p(h) performs slightly better than tanh(h) on FCVID at top-20 retrieval.Both activation functions achieve satisfactory mAP.

5) Cross-dataset evaluation comparison:

SSVH shows strong retrieval performance across FCVID and YFCC, while cross-dataset results reveal sensitivity to training scale and domain shift. Qualitative examples further indicate gains in temporal discrimination alongside persistent failures on visually similar food-related actions.

  • Cross-dataset comparison: Cross-dataset evaluation shows performance drops when training on FCVID and testing on YFCC, whereas training on YFCC improves mAP when testing on FCVID.The authors associate these differences with training-set scale and possible domain shift.
  • FCVID comparison: SSVH achieves the best performance at all tested bit lengths on FCVID, outperforming SSTH by 9.3%–9.6% in mAP@K at 256 bits.The reported gains are 9.6%, 9.3%, 9.6%, 9.4%, and 9.4% for K=20,40,60,80,100, respectively.
  • YFCC comparison: On YFCC, SSVH consistently outperforms the compared hashing baselines and improves mAP by 13% on average over SSTH at 256 bits.The reported advantage is more pronounced for short codes, while the gap becomes marginal for some longer-code metrics.
  • Qualitative results: Qualitative retrieval indicates that SSVH better captures temporal information than SSTH, especially for concepts involving human actions.Both methods can retrieve categories that depend mainly on visual appearance, such as Tornado and Patio.
  • Qualitative results: Both methods fail to distinguish some food-related actions, including Making Hotdog and Pantry, from visually similar videos such as nail painting.These examples mark a concrete limitation of the qualitative retrieval results.
  • Limitations: The conclusion identifies pre-trained VGGNet frame features as a limitation because they ignore frame consecutiveness and temporal information.The authors propose adding motion features and fusing multiple features in future work.
Loading 1802.02305v1…