PaperlayerView exact revision on arXiv

Source-linked AI summary

Nonlinear Systems Identification Using Deep Dynamic Neural Networks

Olalekan Ogunmolu, Xuejun Gu, Steve Jiang, Nicholas Gans

arXiv:1610.01439v1cs.NE

TL;DR

The paper investigates whether deep neural networks can model complex dynamical systems from input-output data. It evaluates neural-network models on sequential system-identification datasets, including a soft-robot actuator and a DaISy MIMO dataset. The authors conclude that deep architectures are adaptable for identifying nonlinear dynamical systems.

  • Problem

    The paper asks whether deep neural networks can identify complex nonlinear dynamical systems from input-output data.

    Source: §I · p. 1

  • Method

    The authors evaluate feed-forward and recurrent neural-network Hammerstein models, including RNN, LSTM, and GRU variants, on sequential identification datasets.

    Source: §I · p. 1, 1) Soft-Robot · p. 6, D. LSTM Model Architecture · p. 6

  • Results

    The authors report that deep network architectures are adaptable for identifying nonlinear dynamical systems, evaluating a soft-robot actuator dataset and a DaISy MIMO dataset.

    Source: §VIII · p. 8

  • Takeaways & Limitations

    The paper concludes that deep networks can model complicated input-output relationships without hand-coded features.

    Source: §VIII · p. 8

    Real-time identification and control of the proposed soft-robot system remain future work.

    Source: §VIII · p. 8

Abstract

from arXiv · show

Neural networks are known to be effective function approximators. Recently, deep neural networks have proven to be very effective in pattern recognition, classification tasks and human-level control to model highly nonlinear realworld systems. This paper investigates the effectiveness of deep neural networks in the modeling of dynamical systems with complex behavior. Three deep neural network structures are trained on sequential data, and we investigate the effectiveness of these networks in modeling associated characteristics of the underlying dynamical systems. We carry out similar evaluations on select publicly available system identification datasets. We demonstrate that deep neural networks are effective model estimators from input-output data

I. INTRODUCTION

The paper applies deep neural networks to nonlinear system identification, motivated in part by soft-robot motion alignment during cancer radiotherapy. It introduces neural-network Hammerstein models and reviews how neural networks represent and approximate input-output relationships.

I. INTRODUCTION

  • The introduction distinguishes multilayer, recurrent, and convolutional networks, noting their prior uses in nonlinear systems, sequential tasks, and pattern recognition, respectively.Source: §I · p. 1
  • The work is motivated by identifying and controlling soft robots for head-and-neck motion alignment during cancer radiotherapy.The authors propose self-organizing networks connected in a deep-network structure to develop internal structures suited to system-identification and control learning.Source: §I · p. 1
  • The study presents neural-network Hammerstein models for complex nonlinear systems and evaluates them on SISO and MIMO datasets.It frames the work as extending neural-network approaches from pattern recognition to recursive identification, parameter estimation, and control.Source: §I · p. 1
  • The paper investigates multilayer, simple recurrent, and LSTM neural networks on selected SISO and MIMO system-identification datasets from DaISy.Source: §I · p. 1

II. PRELIMINARIES AND BASIC CONCEPTS

  • Dynamic-system identification seeks a function that approximates the system operator from input-output data; the text cites universal approximation on bounded compact input spaces.A neural network with enough nodes can approximate a continuous function to an acceptable bounded error.Source: §II · p. 2

III. RELATED WORK

Related work describes neural-network identification and control methods alongside assumptions and training challenges. It also discusses self-organizing networks as a way to learn task-appropriate internal structure.

III. RELATED WORK

  • Earlier work used static, dynamic feed-forward, and recurrent neural networks for nonlinear-system identification and control, with assumptions about stability and system properties noted as a drawback.The cited work also reported neural-network effectiveness for multivariable, higher-order complex dynamical systems.Source: §III · p. 2
  • A Hammerstein model combining static feed-forward and dynamic recurrent networks reportedly converged faster than random initialization by avoiding saturated sigmoid layers.Source: §III · p. 2
  • Recurrent-network training can be difficult because weight initialization is problem-specific, temporal gradients can vanish or grow, and hidden-layer saturation increases training time.Source: §III · p. 2
  • Self-organizing neural networks can learn adaptive rules and develop internal structures suited to a nonlinear parameter-estimation task.The cited account contrasts this with hand-coded features for recognition systems.Source: §III · p. 2

IV. SUPERVISED LEARNING WITH NEURAL NETWORKS

This section describes supervised neural-network training as adjusting weights to make predicted outputs approximate observed outputs. It presents loss minimization and gradient-based updates, including stochastic mini-batch estimates.

IV. SUPERVISED LEARNING WITH NEURAL NETWORKS

  • The network topology learns weight-adjustment rules from finite input-output data so predicted outputs approximate desired outputs within a specified error.The passage notes that deep networks can identify model structure despite data noise, while noise may trap optimization in local minima during backpropagation.Source: §IV · p. 2
  • Feed-forward network parameters are chosen to minimize a global prediction loss, typically mean-squared error for regression system-identification tasks.Source: §IV · p. 3
  • Gradient descent updates parameters using loss gradients; stochastic gradient descent estimates those gradients from randomly sampled training examples or mini-batches to reduce training time.The passage identifies backpropagation for feed-forward networks and backpropagation through time for recurrent networks.Source: §IV · p. 3

V. LEARNING WITH DEEP DYNAMIC NNS

The paper considers feed-forward and recurrent architectures for sequential system-identification data. Multilayer networks pass signals forward, while recurrent networks use hidden-layer feedback and sequence-based training.

V. LEARNING WITH DEEP DYNAMIC NNS

  • The paper considers feed-forward multilayer networks, simple recurrent networks, LSTMs, and gated recurrent units for sequential datasets with temporal input correlations.Source: §V · p. 3

A. Multilayer Networks

  • A multilayer network sends signals forward from input through hidden layers to output, and its weights and biases parameterize the trained system.Training compares estimated outputs with true outputs, back-propagates errors, and evaluates generalization on held-out testing data.Source: A. Multilayer Networks · p. 3

B. Recurrent Neural Networks

  • A recurrent network adds hidden-layer self-feedback, allowing signals to flow asynchronously and modeling nonlinear dynamics through recurrent state updates.Its cumulative time-step loss is trained using backpropagation through time or real-time recurrent learning.Source: B. Recurrent Neural Networks · p. 3

C. Long Short Term Memory (LSTM) Cells

The section explains why ordinary recurrent networks struggle to retain long-term context and describes LSTM gates that regulate memory and output. A fast LSTM variant omits peephole connections to reduce model complexity.

C. Long Short Term Memory (LSTM) Cells

  • RNN gradients can become intractable when modeling long-term context, limiting their ability to retain information beyond recent inputs.Backpropagated errors may grow or vanish over time.Source: C. Long Short Term Memory (LSTM) Cells · p. 3
  • LSTM cells use input, forget, and output gates to regulate cell state and output, while peephole connections support accurate learning of timings.The forget gate resets state; peephole connections link the cell to its gates.Source: C. Long Short Term Memory (LSTM) Cells · p. 4

D. Fast LSTM

E. Gated Recurrent Units (GRU)

The GRU is presented as a simpler alternative to the LSTM, using an encoder-decoder arrangement to summarize an input sequence and generate outputs. Its hidden state and reset gate support sequence processing, while activation functions shape the outputs.

E. Gated Recurrent Units (GRU)

  • A GRU uses two RNNs in an encoder-decoder arrangement: one summarizes the source sequence in a fixed-length vector, and the other generates a variable-length target sequence.The pair is jointly trained to maximize the conditional probability of the target given the input.Source: E. Gated Recurrent Units (GRU) · p. 4
  • The GRU hidden state summarizes the input sequence, and a reset gate allows it to forget previous information.Source: E. Gated Recurrent Units (GRU) · p. 4
  • The GRU decoder predicts each output from its hidden state, the previous output, and the sequence summary, using activation functions such as sigmoid or soft-max.Source: E. Gated Recurrent Units (GRU) · p. 4

VI. EXPERIMENTS

The experiments train models on a soft-robot actuator dataset and selected DaISy datasets. For the soft robot, the study maps valve current to head pitch and uses task-informed network structures, with dropout considered for larger models.

A. Data and Baseline Systems

  • The soft-robot experiment records 10,070 input-output samples and divides them 60:40 between training and testing.The actuator moves a mannequin head; current is the input and measured head height is the output.Source: A. Data and Baseline Systems · pp. 4–5
  • The glass-furnace DaISy dataset has three inputs—two heating signals and one cooling signal—and six temperature-sensor outputs.Source: A. Data and Baseline Systems · p. 5

1) Soft-Robot:

  • For the soft-robot task, the model maps valve current through a six-neuron hidden layer and ReLU to mocap output, using current-pitch data.Dropout is considered for larger recurrent and dynamic feedforward Hammerstein models because their parameter counts could lead to overfitting.Source: 1) Soft-Robot · p. 5, A. Data and Baseline Systems · p. 5

2) GlassFurnace:

This section describes a Hammerstein approach for the soft-robot system, combining recurrent nonlinear input-to-state modeling with linear state-to-output dynamics. It also reports training-data fit values for a six-hidden-layer MLP.

2) GlassFurnace:

  • The GlassFurnace model uses three input linear layers and reshapes its output layer to six outputs.Source: 2) GlassFurnace · p. 5
  • The six-hidden-layer MLP’s estimation-data fit is 99.8% for SISO and 87.5% for SIMO.Source: Figure 4 · p. 5

1) Soft-Robot:

  • The Hammerstein model places a recurrent nonlinear element before a multilayer network that models linear dynamics from system states to output.The nonlinear block integrates input history and transforms inputs into system states; backpropagation through five time steps was sufficient in the tested models.Source: 1) Soft-Robot · p. 5
  • The recurrent-neural-network and feedforward multilayer Hammerstein model trains faster and reaches the desired mean-square-error minimum more quickly than the earlier forward network.Source: 1) Soft-Robot · p. 6

2) Glassfurnace Data:

The Glassfurnace model is reshaped to accommodate its data, while the LSTM architecture is described as a three-module Hammerstein model. On the soft-robot dataset, the LSTM handles input delay better, with slightly longer training time than the MLP or RNN-MLP.

2) Glassfurnace Data:

  • For Glassfurnace, the model reshapes its input and output layers to accommodate the dataset widths.Its training performance is shown in Fig. 8.Source: 2) Glassfurnace Data · p. 6

D. LSTM Model Architecture

  • The soft-robot Hammerstein model uses three nonlinear LSTM modules with dropout outputs, followed by a fully connected linear dynamic module.The model is fed the head-motion measurements from the motion-capture system.Source: D. LSTM Model Architecture · p. 6
  • The soft-robot LSTM handles input delay better, but trains slightly longer than MLP or RNN-MLP architectures.The passage attributes its delay handling to modeling long-term dependencies and adapting parameters to temporal evolution.Source: Figure 6 · p. 6, D. LSTM Model Architecture · p. 6

E. Fast LSTM Architecture

Across the two datasets, the models show good fit and generally acceptable mean-square losses on noisy data. The results also describe a speed–capacity tradeoff and a FastLSTM simplification that retains approximately the same convergence with fewer parameters and less training time.

F. Gated Recurrent Units Structure

E. Fast LSTM Architecture

VII. RESULTS AND ANALYSES

  • The models show good fit and generally acceptable mean-square losses on the noisy datasets, without dataset preprocessing or batch normalization.The tables report performance for the soft-robot and Glassfurnace models.Source: §VII · p. 7, Table I · p. 7, Table II · p. 7
  • The multilayer network trains fastest, reaching a decent mean-square error in 422 seconds, but its forward-only architecture limits approximation capacity.RNN-MLP Hammerstein models may be more sluggish; the authors hypothesize they better capture input-output delay and long-range mappings, with that assumption left for future verification.Source: §VII · pp. 7–8

VIII. CONCLUSIONS

The paper concludes that supervised deep networks can identify nonlinear dynamical systems that are difficult to model with hand-coded features. It also identifies real-time identification and control of the soft-robot radiotherapy alignment system as future work.

VIII. CONCLUSIONS

  • Supervised deep network architectures identify nonlinear dynamical systems that are otherwise complicated to model using hand-coded features.With suitable hyperparameter selection, model choice, and weight tuning, the paper reports that complex hand-coding of classical identification features can be discarded.Source: §VIII · p. 8
  • Real-time identification and control of the soft-robot motion-alignment system for head-and-neck cancer radiotherapy remain future work.The authors also name other complex nonlinear phenomena as future applications.Source: §VIII · p. 8

Related papers

Loading 1610.01439v1…