PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional RNN: an Enhanced Model for Extracting Features from Sequential Data

Gil Keren, Björn Schuller

arXiv:1602.05875v3stat.MLcs.CL

TL;DR

Traditional convolutional feature extraction may be too simple for sequential windows because it does not use their internal temporal structure. The paper feeds each window frame by frame into recurrent layers, obtaining improved classification results over traditional convolutional layers on two audio classification tasks.

  • Problem

    Traditional convolutional layers extract patch features with an affine function followed by a non-linearity, which may be too simple and overlook the temporal structure inside sequential windows.

  • Method

    CRNN layers feed each window frame by frame into a recurrent layer and use hidden states, outputs, or cell states to compute the window features.

  • Results

    CRNN models improved classification results over traditional convolutional layers on two audio classification tasks.

  • Takeaways & Limitations

    Exploiting temporal structure within sequential windows and allowing more feature-generation computation can improve audio classification results in the evaluated tasks.

Abstract

from arXiv · show

Traditional convolutional layers extract features from patches of data by applying a non-linearity on an affine function of the input. We propose a model that enhances this feature extraction process for the case of sequential data, by feeding patches of the data into a recurrent neural network and using the outputs or hidden states of the recurrent units to compute the extracted features. By doing so, we exploit the fact that a window containing a few frames of the sequential data is a sequence itself and this additional structure might encapsulate valuable information. In addition, we allow for more steps of computation in the feature extraction process, which is potentially beneficial as an affine function followed by a non-linearity can result in too simple features. Using our convolutional recurrent layers we obtain an improvement in performance in two audio classification tasks, compared to traditional convolutional layers. Tensorflow code for the convolutional recurrent layers is publicly available in https://github.com/cruvadom/Convolutional-RNN.

I. INTRODUCTION

The paper proposes CRNN layers for sequential data, exploiting temporal structure within local windows to create richer features than traditional convolutional layers. It evaluates variants on audio classification tasks.

  • I. INTRODUCTION: Traditional convolutional layers may produce overly simple features because they apply a non-linearity to an affine function of each data patch.The paper motivates additional computation as a way to obtain more complicated nonlinear feature mappings.
  • I. INTRODUCTION: CRNN layers feed each sequential window frame by frame into a recurrent layer, using recurrent outputs or hidden states to compute features.This exploits the fact that a short window of consecutive frames is itself a sequence.
  • I. INTRODUCTION: CRNN feature extraction can use temporal information within windows and more computation than traditional convolutional feature extraction.The proposed comparison is framed as potentially producing better features than standard convolutional layers.
  • I. INTRODUCTION: Experiments compare proposed CRNN variants with models using traditional convolutional layers on multiple audio classification tasks.The stated objective is to assess whether the proposed models improve classification rates.

II. RELATED WORK

Related work has used multilayer or recurrent networks to extract features from data patches, but these approaches do not fully exploit the sequential structure within patches. The paper distinguishes CRNNs by processing window frames as a sequence.

  • II. RELATED WORK: Prior work replaced convolutional affine functions with multilayer feedforward networks, increasing feature-extraction complexity without recurrent processing.That approach was reported for object-classification datasets.
  • II. RELATED WORK: Another approach used a recurrent layer to process image patches, but repeatedly fed the same patch and did not exploit a patch’s sequential nature.The paper also notes that this repeated processing can be computationally expensive.
  • II. RELATED WORK: A further method scanned image grids with recurrent networks, yet fed each patch as one piece and lacked sequential patch processing.It also did not provide overlapping patches for accounting for small translations.

III. CONVOLUTIONAL LONG SHORT TERM MEMORY

The paper defines a general layer that extracts pooled temporally local features by applying a feature-extraction function to overlapping or shifted windows from variable-length sequences.

  • A. Extracting local features: Each example is a variable-length sequence of l frames, with k features per frame, represented as a k × l array.The frame representation can be generalized to multidimensional data by flattening each frame.
  • A. Extracting local features: The method forms windows of r1 consecutive frames, shifts successive windows by r2 frames, and applies f to each window.Each window has size k × r1.
  • A. Extracting local features: The feature-extraction function produces n features describing every window, which can then be pooled across the temporal sequence of windows.The passage introduces this as a general structure covering temporally local feature extraction.

B. Long Short Term Memory networks and Bidirectional Long Short Term Memory networks

The paper introduces recurrent building blocks for sequential feature extraction, including simple RNNs, LSTMs, and bidirectional LSTMs. These units produce hidden states and outputs that can represent sequence information.

  • B. Long Short Term Memory networks and Bidirectional Long Short Term Memory networks: A simple RNN maps an input sequence to corresponding sequences of hidden states and outputs using recurrent computations.The recurrent formulation uses weight matrices and biases, with the logistic sigmoid specified for the activation.
  • B. Long Short Term Memory networks and Bidirectional Long Short Term Memory networks: LSTMs add gating mechanisms to recurrent networks to better model long-term dependencies in sequential data.The paper identifies LSTM as a special kind of recurrent neural network.
  • B. Long Short Term Memory networks and Bidirectional Long Short Term Memory networks: Figure 1 contrasts feature extraction from a time window using a traditional convolutional layer versus a CRNN layer.The figure provides the visual comparison of the two extraction procedures.
  • B. Long Short Term Memory networks and Bidirectional Long Short Term Memory networks: Bidirectional LSTMs process inputs in both standard and reversed order, combining past and future information at each time step.The two LSTM directions produce separate hidden and cell states that are combined for the layer output.

C. Convolutional layers and Convolutional Recurrent layers

Traditional convolutional layers compute window features with affine transformations and nonlinearities, whereas CRNN layers process each window as a short sequence with recurrent units. CRNN variants can use hidden states, outputs, or cell states, and Extended CLSTM additionally assigns frame-specific recurrent weights.

  • Traditional convolutional layers: A standard convolutional layer extracts each window feature by applying a nonlinearity to an affine function defined by weights and biases.The operation multiplies the data patch element-wise by a weight matrix, sums the elements, adds a bias, and applies the nonlinearity.
  • Convolutional recurrent layers: CRNN interprets a k×r1 window as r1 frames, feeds them sequentially into an RNN, and represents the window with a length-n feature vector.The recurrent layer produces hidden states of size n × 1 for each frame; the feature vector is computed from these states.
  • Convolutional recurrent layers: The same recurrent layer parameters are reused across windows, preserving convolutional sharing while allowing features to be computed from recurrent hidden states, outputs, or cell states.Cell states are available for CLSTM or BLSTM variants, and outputs can be used when the hidden dimension is unconstrained.
  • Extended CLSTM: Extended CLSTM uses separate copies of four LSTM weight matrices for each frame position so the layer can exploit position within the window.Standard LSTM uses the same matrices for all frames, whereas Extended CLSTM introduces position-specific matrices.

A. Experiments on emotion classification: FAU-Aibo corpus

The emotion-classification experiments compare a baseline LSTM with models preceded by standard convolutional, CLSTM, or Extended CLSTM layers on the FAU Aibo corpus. The proposed models outperform both the baseline and traditional convolutional layers on the reported task.

  • Dataset and setup: The FAU Aibo corpus contains spontaneous German child speech with emotion labels, and the experiments use the challenge’s train and test subsets.The corpus contains about 9.2 hours of speech, with 9,959 training examples and 8,257 test examples.
  • Dataset and setup: The experiments use 26-dimensional log mel filter-banks as input features, computed with 25 ms windows and a 10 ms shift.The study distinguishes these input representations from features extracted by convolutional layers.
  • Models: The baseline is a 256-unit LSTM followed by a 400-unit ReLU dense layer and a softmax classifier whose final predictions average the last four time steps.The class with the highest averaged probability is selected as the utterance prediction.
  • Models: The convolutional variants prepend two convolutional layers with five-frame windows and two-frame shifts, using standard, CLSTM, or Extended CLSTM feature extraction.For CLSTM and Extended CLSTM, the window output is based on cell states.
  • Results: The proposed models outperform the baseline, traditional convolutional layers, and the challenge baseline on emotion classification.Performance is reported using Unweighted Average Recall, the official benchmark measure for the task.

B. Experiments on age and gender classification: aGender corpus

The age- and gender-classification experiments use the aGender corpus and compare standard convolutional, CLSTM, and CBLSTM layers preceding bidirectional recurrent baselines. CBLSTM improves over the comparison models with both log mel filter-banks and eGeMAPS inputs, while CLSTM often also outperforms the baseline and standard convolutional layer.

  • Dataset and setup: The aGender corpus contains predefined and free-speech utterances labeled by four age groups and three gender classes.The challenge training data contains 32,527 utterances from 471 speakers across 23.43 hours of speech.
  • Dataset and setup: The experiments evaluate 26-dimensional log mel filter-banks and 25-dimensional eGeMAPS acoustic features as two input representations.eGeMAPS includes low-level descriptors such as pitch, jitter, formant characteristics, shimmer, loudness, and spectral slope.
  • Models: The baseline uses a bidirectional LSTM with forward and backward layers of dimension 256, followed by a 400-dimensional ReLU dense layer.The convolutional models prepend one standard convolutional, CLSTM, or CBLSTM layer with settings matching the emotion experiments.
  • Results: CBLSTM improves over the baseline, traditional convolutional, and CLSTM layers for both input feature sets on the age- and gender-classification task.The comparison uses test-set Unweighted Average Recall, the official benchmark measure.
  • Results: CLSTM outperforms the baseline and standard convolutional layer in most cases, and also exceeds the challenge baseline in the reported comparisons.This pattern is reported alongside the stronger CBLSTM result.

C. Comparing Different CLSTM architectures

The study compares six CLSTM feature-extraction variants across emotion, age, and gender classification tasks, finding no single implementation consistently superior.

  • Six CLSTM variants combined hidden or cell states with max, mean, or last-vector aggregation for empirical comparison.The experiments used 26-dimensional log mel filter-banks and matched network architectures and training procedures across tasks.
  • Across four tasks, cell-state implementations averaged better than hidden-state implementations, although one task favored hidden states.
  • The results did not identify one consistently best state sequence or aggregation operation, with only a slight preference for max pooling.

V. DISCUSSION

The discussion attributes CBLSTM’s advantage on age and gender classification to bidirectional temporal context and the competing demands placed on CLSTM cell states. It also reports that eGeMAPS outperformed low-complexity log mel filter-banks.

  • CBLSTM outperformed CLSTM for speaker age and gender classification, potentially because each time step can use both past and future window information.
  • CLSTM cell states must simultaneously preserve information for the next recurrent step and provide features to later layers, potentially creating conflicting requirements.
  • Longer windows may particularly benefit CRNNs because they contain more temporal structure for feature extraction.
  • eGeMAPS performed better than low-complexity log mel filter-banks in the reported experiments.

VI. CONCLUSION

The conclusion presents CRNNs as recurrent feature extractors for sequential windows, exploiting temporal structure and additional computation. Across tested audio settings, the models improved classification results over traditional convolutional layers.

  • CRNN models improved classification results over traditional convolutional layers with the same number of extracted features.The evaluated variants were CLSTM, Extended CLSTM, and CBLSTM.
  • CRNNs feed each window frame by frame into a recurrent layer, then use recurrent hidden states, outputs, or LSTM cell states to compute window features.
  • Improvements were observed with both low-complexity log mel filter-banks and higher-level eGeMAPS audio features.
  • The CRNN features require further evaluation and comparison with features from standard convolutional layers.
Loading 1602.05875v3…