PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Multiscale Recurrent Neural Networks

Junyoung Chung, Sungjin Ahn, Yoshua Bengio

arXiv:1609.01704v7cs.LG

TL;DR

Recurrent networks have lacked empirical evidence that they can discover latent hierarchical structure while modeling temporal dependencies, especially when boundaries and segment lengths vary. The paper proposes HM-RNN, which learns boundary detectors and uses adaptive UPDATE, COPY, and FLUSH operations across layers. It reports state-of-the-art or comparable language-modeling results and better handwriting generation than a standard recurrent baseline.

  • Problem

    RNNs lack sufficient evidence of discovering latent hierarchical temporal structure, while fixed or explicit timescales do not fit variable-length segments without reliable boundary information.

  • Method

    HM-RNN learns binary boundary detectors and selects UPDATE, COPY, or FLUSH operations to encode temporal dependencies at adaptive hierarchical timescales.

  • Results

    HM-RNN achieves state-of-the-art Text8 performance, comparable state-of-the-art results on Penn Treebank and Hutter Prize Wikipedia, and outperforms standard recurrent models on handwriting generation.

  • Takeaways & Limitations

    The experiments provide evidence that a multiscale recurrent architecture can discover intrinsic hierarchical structure from sequences without explicit boundary information.

Abstract

from arXiv · show

Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural networks, which can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that our proposed multiscale architecture can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence modelling.

1 INTRODUCTION

The paper addresses the challenge of learning hierarchical and temporal representations when temporal data contains variable-length abstractions and boundaries are unavailable. It proposes HM-RNN, which adaptively discovers hierarchical timescales and shows strong results across language and handwriting tasks.

  • Motivation: Deep models benefit from hierarchical representations, but temporal data requires representations that also capture dependencies across different timescales.High-level abstractions change slowly, while low-level features change quickly with local timing.
  • Motivation: Fixed timescales are poorly suited to non-stationary sequences whose hierarchical segments have variable lengths.Words and other abstractions may differ in length, so fixed update frequencies can misalign with natural boundaries.
  • Contribution: HM-RNN learns hierarchical multiscale structure without explicit boundaries by using boundary detectors and adaptive layer operations.Its UPDATE, COPY, and FLUSH operations select sparse updates, state copying, or segment completion and transfer.
  • Contribution: The model discovers fine timescales in low-level layers and coarse timescales in high-level layers.This organization is learned from temporal data rather than imposed through fixed update rates.
  • Empirical evaluation: HM-RNN achieves state-of-the-art results on Text8, comparable state-of-the-art results on Penn Treebank and Hutter Prize Wikipedia, and outperforms standard RNNs on handwriting generation.The evaluation covers character-level language modelling and handwriting sequence generation.

2 RELATED WORK

Prior multiscale and hierarchical RNNs improve temporal modeling through structured update schedules, but typically rely on fixed timescales, explicit boundaries, or segment labels. HM-RNN instead discovers hierarchical structure directly from sequences.

  • Multiscale RNNs: Early multiscale RNNs stacked layers with decreasing update frequencies to improve computational and learning efficiency.These approaches motivated organizing temporal dependencies hierarchically across layers.
  • Multiscale RNNs: LSTM units operate at different timescales, but those timescales are not organized hierarchically and long-term information can gradually dilute.The cited discussion contrasts LSTM’s soft multiscale updates with the paper’s hierarchical organization.
  • Multiscale RNNs: Clockwork RNN assigns hard, fixed timescales to modules, with module i updating every 2^(i−1) time steps.This improves computational efficiency but does not dynamically match variable-length segments.
  • Hierarchical RNNs: Hierarchical RNNs have modeled characters, words, phrases, and dialogue utterances when boundary information is explicitly available.Examples use tokenization or other provided hierarchical boundaries to coordinate layers.
  • Segmentation: Other hierarchical RNNs discover segmentation by optimizing against explicit hierarchical segment labels, unlike HM-RNN’s label-free discovery.The distinction is whether segment labels are supplied during optimization.

3 HIERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS

HM-RNN extends stacked recurrent models with learned boundary detectors that coordinate hierarchical information flow across layers. Its discrete UPDATE, COPY, and FLUSH operations enable adaptive timescales while training uses a straight-through estimator and slope annealing.

  • 3.1 MOTIVATION: A hierarchical RNN can pass character summaries to word-level and phrase-level layers when segment boundaries are provided.The upper layer remains unchanged while the lower layer processes characters within a word.
  • 3.1 MOTIVATION: Hierarchical multiscale structure can save computation, shorten gradient paths, and enable layer-wise capacity control.Upper layers update less frequently, while lower layers handle short-term dependencies at higher frequency.
  • 3.1 MOTIVATION: Fixed update frequencies fail when hierarchical segments have different lengths, motivating boundary discovery from raw sequences.The paper highlights variable-length words and unavailable or unreliable boundary information.
  • 3.2 THE PROPOSED MODEL: Each layer uses a binary boundary detector to identify when its current abstraction-level segment ends.Detected boundaries send a summarized segment representation to the upper layer.
  • 3.2 THE PROPOSED MODEL: At each time step, boundary states determine whether a layer performs UPDATE, COPY, or FLUSH.UPDATE changes the summary representation, COPY preserves states, and FLUSH ejects the current state upward before resetting.
  • 3.2 THE PROPOSED MODEL: HM-LSTM implements the model using recurrent hidden and cell states, with updates conditioned on boundary states across adjacent layers.The formulation uses LSTM-style state transitions and combines lower-layer, same-layer, and top-down information.
  • 3.3 COMPUTING GRADIENT OF BOUNDARY DETECTOR: The straight-through estimator trains the discrete boundary decisions by using a differentiable hard sigmoid during backpropagation.Slope annealing gradually makes the hard sigmoid closer to the forward-pass step function, reducing estimator bias.

4 EXPERIMENTS

The HM-LSTM is evaluated on character-level language modeling and handwriting sequence generation, achieving strong benchmark performance while learning interpretable multiscale boundaries and reducing recurrent updates.

  • Evaluation setup: The experiments cover Penn Treebank, Text8, Hutter Prize Wikipedia, and IAM-OnDB handwriting sequences, using BPC for text and average test log-likelihood for handwriting.The language-modeling setup uses three HM-LSTM layers and evaluates next-character prediction; handwriting inputs contain pen coordinates and pen-tip indicators.
  • Character-level language modeling: 1.24 test BPC is achieved on Penn Treebank with the HM-LSTM using step boundaries and slope annealing.The step-function hard boundary decision outperforms sampling and soft boundary variants, while slope annealing further improves test BPC.
  • Character-level language modeling: 1.29 test BPC on Text8 is reported as state of the art, while 1.32 on Hutter Prize Wikipedia ties the state-of-the-art neural result.The Hutter Prize result remains behind specialized compression systems such as PAQ8hp12 and decomp8.
  • Learned hierarchical structure: Boundary detectors learn word-like and higher-level segmentations without explicit boundary information, including first-layer boundaries near spaces and second-layer semantic or syntactic segments.The visualizations also show within-word boundaries and higher-layer segmentation patterns such as “consumers may” and “want to move their telephones a”.
  • Update mechanism and interpretability: The three-layer example performs 335 total updates instead of 810 for a standard RNN, a 60% reduction, while update frequency decreases with layer depth.The first layer performs 270 updates, the second 56, and the third 9; COPY operations leave hidden-state norms unchanged within segments.
  • Handwriting sequence generation: The HM-LSTM outperforms the standard LSTM on IAM-OnDB, with slope annealing improving test log-likelihood to 1167.The model predicts real-valued handwriting outputs using a mixture density network.

5 CONCLUSION

The HM-RNN captures latent hierarchical structure through learned boundary detectors and novel multiscale update operations. Across language modelling and handwriting generation, the model achieves strong performance while discovering sequence structure without explicit boundaries.

  • HM-RNN uses COPY, UPDATE, and FLUSH operations controlled by learned binary boundary detectors to capture latent hierarchical structure.Each detector learns segment boundaries at its layer through a novel update rule.
  • On character-level language modelling, HM-LSTM achieves state-of-the-art results on Text8 and comparable results on Penn Treebank and Hutter Prize Wikipedia.
  • On handwriting sequence generation, HM-LSTM outperforms the standard LSTM.
  • The results and analysis suggest that HM-RNN discovers latent hierarchical structure and learns efficient hierarchical multiscale representations associated with better generalization performance.
Loading 1609.01704v7…