PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Long Short-Term Memory Recurrent Neural Networks Using Ant Colony Optimization to Predict Turbine Engine Vibration

AbdElRahman ElSaid, Travis Desell, Fatima El Jamiy, James Higgins, Brandon Wild

arXiv:1710.03753v1cs.NEcs.AI

TL;DR

Accurately predicting excessive engine vibration matters, but recurrent neural networks face training difficulties from vanishing and exploding gradients. This paper applies ant colony optimization to LSTM cell connections, improving 10-second-ahead prediction error and reducing network weights.

  • Problem

    Predicting excessive engine vibration matters to aviation, while vanishing and exploding gradients impede recurrent neural network training and learning over long intervals.

    Source: §1 · pp. 1–2

  • Method

    The authors use ant colony optimization to optimize connections in LSTM cell gates, seeking network structures with improved predictive ability and fewer connections.

    Source: §6.4 · p. 24, §7 · p. 26

  • Results

    1.35% improvement reduced 10-second-ahead prediction error from 5.51% to 4.17%; top 30 evolved networks reduced weights by 42% to 45% versus fully connected Architecture I.

    Source: §6.4 · p. 24

  • Takeaways & Limitations

    ACO-optimized LSTM gates can reduce the connections required while improving the network’s predictive ability.

    Source: §7 · p. 26

    The study used one connection mesh across all LSTM gates and time-steps; future work proposes varying those meshes and optimizing them simultaneously.

    Source: §7 · p. 30

Abstract

from arXiv · show

This article expands on research that has been done to develop a recurrent neural network (RNN) capable of predicting aircraft engine vibrations using long short-term memory (LSTM) neurons. LSTM RNNs can provide a more generalizable and robust method for prediction over analytical calculations of engine vibration, as analytical calculations must be solved iteratively based on specific empirical engine parameters, making this approach ungeneralizable across multiple engines. In initial work, multiple LSTM RNN architectures were proposed, evaluated and compared. This research improves the performance of the most effective LSTM network design proposed in the previous work by using a promising neuroevolution method based on ant colony optimization (ACO) to develop and enhance the LSTM cell structure of the network. A parallelized version of the ACO neuroevolution algorithm has been developed and the evolved LSTM RNNs were compared to the previously used fixed topology. The evolved networks were trained on a large database of flight data records obtained from an airline containing flights that suffered from excessive vibration. Results were obtained using MPI (Message Passing Interface) on a high performance computing (HPC) cluster, evolving 1000 different LSTM cell structures using 168 cores over 4 days. The new evolved LSTM cells showed an improvement of 1.35%, reducing prediction error from 5.51% to 4.17% when predicting excessive engine vibrations 10 seconds in the future, while at the same time dramatically reducing the number of weights from 21,170 to 11,810.

1. Introduction

Aircraft engine vibration is difficult to predict because many interacting parameters vary across engines and operating conditions, and recurrent models must address training challenges. This study builds on earlier LSTM vibration-prediction architectures by using ant colony optimization to evolve their cell structure.

1. Introduction

  • Engine vibration involves many interacting parameters over unknown time spans, making its causes difficult to analyze; recurrent networks retain information from prior time steps.The introduction also identifies vanishing and exploding gradients as obstacles to training recurrent networks, while LSTM gates can prevent or lessen them.Source: §1 · pp. 1–2

1.2. Ant Colony Optimization

  • The study applies ant colony optimization to evolve LSTM cell structure, addressing the open question of how to choose neural-network structure.ACO uses artificial ants that explore paths and reinforce promising ones with pheromone; prior work applied the approach to neural-network training.Source: §1 · p. 2, §1.2 · pp. 2–3

1.1. Previous work

  • Earlier work compared three LSTM RNN architectures trained on flight records from a regional airline’s flights with excessive vibration.Those networks predicted vibration 1, 5, 10, and 20 seconds ahead, with respective mean absolute errors of 2.84%, 3.3%, 5.51%, and 10.19%.Source: §1.1 · p. 2

1.3. Study’s Contribution

  • The evolved LSTM architecture improved 10-second-ahead vibration prediction by 1.35%, reducing error from 5.51% to 4.17%, while cutting weights from 21,170 to 11,810.The study reports accuracy increasing from 94.49% to 95.83% based on mean absolute error calculations.Source: §1.3 · p. 3

2. Related Work

The related work frames engine vibration as a safety and maintenance concern and surveys prediction approaches ranging from statistical methods to recurrent networks. It also reviews prior aviation applications of neural networks, LSTMs, and evolutionary topology optimization.

2.1. Turbine Engine Vibration

  • Excessive engine vibration can risk engine damage, while maintenance programs combine scheduled checks and preventive replacement; flight-history monitoring is another approach to forecasting vibration.The paper notes that statistical and holistic prediction methods are not exact, so reasonable safety factors should be considered.Source: §2.1 · p. 3

2.2. Time Series Data Prediction

  • Prediction methods include qualitative judgment and quantitative time-series approaches; statistical models may outperform neural networks, but the paper considers them and hybrids beyond scope for this expected nonlinear problem.The time-series discussion identifies univariate and multivariate quantitative prediction and names AR, MA, ARMA, ARIMA, SARIMA, and VAR models.Source: §2.2 · pp. 3–4

2.3. Time Series Prediction in Aviation

  • Prior aviation work reported 84% successful faulty-engine classifications using healthy-engine training data, while another study forecast vibration 2.5 hours ahead after collecting half an hour of flight data.The latter study also reported ground-based prediction two flights ahead and did not specify how much data was needed for sound prediction.Source: §2.3 · p. 4

2.3.1. RNN for Predicting Flight Parameters

  • Recurrent networks use internal memory to process sequential inputs, and earlier ACO-evolved networks predicted airspeed, altitude, and pitch with reported improvements over prior published results.The passage reports improvements of 63%, 97%, and 120%, respectively; the cited work followed earlier weight optimization with evolutionary algorithms and then ACO-based structure evolution.Source: §2.3.1 · pp. 4–5

2.3.2. LSTM RNN

  • LSTM gates address recurrent gradient problems, and prior work describes adaptive forget gates that let cells learn when to reset; LSTMs have also been used in diverse prediction tasks.The related work also notes that LSTMs do not outperform time-window approaches on some simpler time-series tasks.Source: §2.3.2 · p. 5

2.4. Evolutionary Optimization Methods

  • NEAT evolves network topology and weights through genetic algorithms, but prior work identifies limitations for evolving LSTM cells in time-series prediction.NEAT uses historical markings for recombination and speciation to protect innovation as networks grow from small initial structures.Source: §2.4 · p. 5

3. Methodology

The methodology selects flight-record parameters and develops LSTM architectures for predicting future engine vibration. It describes the cell gates, forward-propagation equations, and three network architectures.

3. Methodology

  • ACO is used to evolve LSTM cell structure while allowing backpropagation to optimize connection weights.The approach targets the cell topology and retains gradient-based weight training.Source: §3 · p. 5

3.1. Experimental Data

  • Flight data contain 76 aircraft-recorded parameters, including engine vibration.The data-processing phase then seeks parameters that contribute most to vibration.Source: §3.1 · p. 5

3.1.1. Data Correlation Parameter Selection

  • Cross-correlation ranks parameters using the area under normalized correlation plots, but its selected inputs produced noisy predictions.The authors note that slowly changing flight-configuration parameters could appear highly correlated with vibration.Source: §3.1.1 · p. 6

3.1.2. Aerodynamics/Turbo-machinery Parameter Selection

  • Expert-selected aerodynamic and turbo-machinery inputs produced encouraging results, while parameters that added noise were excluded.The final input set comprises 15 parameters, including flight conditions, engine speeds, oil measures, and vibration.Source: §3.1.2 · pp. 6–7

3.2. Recurrent Neural Network Design

  • The three LSTM architectures use 15 selected flight-record parameters, cell memory, and gates controlling information flow.Each cell receives flight data or output from a lower layer, along with the previous cell's output; the gates regulate input, memory, and output.Source: §3.2 · pp. 7–8

3.2.1. LSTM RNN Forward Propagation Equations

  • Forward propagation computes gate outputs from current inputs and previous outputs using learned weight matrices and activation functions.The supplied equations define gate outputs and the associated weights; the sigmoid function is also specified.Source: §3.2.1 · pp. 8, 11

3.3. LSTM RNN Architectures

  • The three architectures differ in design, with their weight dimensions and total weight counts reported in Tables 1 and 2.The architecture descriptions distinguish network structures while retaining the shared LSTM cell design.Source: §3.3 · p. 11

Architecture I

Architecture I processes ten time steps through two LSTM levels before producing a vibration prediction. Its 21,170 connections make its structure comparatively large.

Architecture I

  • 10 time-series inputs—the current instant and previous nine—feed Architecture I's first level.Each of the 15 parameters is represented at each time step, with a bias node.Source: Architecture I · p. 11
  • The second level reduces each cell's 16 input nodes to one, yielding ten outputs for a final prediction neuron.The first and second levels are treated as hidden layers.Source: Architecture I · p. 11
  • Architecture I has 21,170 connections, with Figure 8 showing cell connectivity and Figure 9 detailing one time step.Ten instances of Figure 8 are connected as specified in Figure 9.Source: Architecture I · p. 11

Architecture II

Architecture II follows Architecture I but omits its third level and averages the second-level outputs to produce the network output.

Architecture II

  • Architecture II is nearly identical to Architecture I, except that it removes the third level and averages second-level outputs.The averaging replaces Architecture I's final prediction level.Source: Architecture II · p. 11
  • The architecture's weight dimensions and total weight count are reported in Tables 1 and 2.The supplied passage does not give the table values.Source: Architecture II · p. 11

Architecture III

Architecture III uses a deeper LSTM network with 20 time-series inputs and a final neuron for vibration prediction. Its forward path combines cell outputs into a scalar network output.

Architecture III

  • 20 time-series inputs—the current instant and previous 19—feed Architecture III's first level, followed by a second LSTM level.The second level repeats the first-level procedure to support more abstract decision making.Source: Architecture III · p. 11
  • A third level reduces each cell's 16 inputs to one, producing 20 outputs that feed a final prediction neuron.The third level is treated as a hidden layer before the network's final level.Source: Architecture III · p. 11

3.4. Forward Propagation

  • At each time step, cell inputs and previous outputs are processed by the cell, input, forget, and output gates.The forget gate controls how much prior cell memory passes, while the output gate regulates the cell's current output.Source: §3.4 · p. 15
  • The cell combines its input and prior memory into a new memory, then sends its gated output to the next cell and upper level.At Level 2, ten one-dimensional cell outputs form a vector that a final neuron reduces to a scalar prediction.Source: §3.4 · p. 15

4. Evolving LSTM RNN Cells using Ant Colony Optimization

The section uses ant colony optimization (ACO) to select LSTM cell connections and evolve network topology. It also describes a distributed training workflow that evaluates candidate networks and rewards paths from the best-performing designs.

4. Evolving LSTM RNN Cells using Ant Colony Optimization

  • Each M1 cell has eight 16x16 input gates, while each M2 cell has eight 8x1 input gates.For both cell types, four gates receive input from the previous cell in the same layer and four from the time series or lower layer.Source: §4 · p. 16
  • Ants select paths through a fully connected gate, and the unique paths they choose form the evolved cell topology.The same ACO-generated topology is applied to each of the eight input gates.Source: §4 · p. 16
  • The generated mesh is shared across the Main, Input, Forget, and Output gates and across time steps, with different connection sets used for transitions with and without data reduction.The paths connect inputs and previous-cell outputs to hidden neurons; reaching a hidden node also includes its connection to the output node.Source: §4 · p. 16

4.1. Distributed ACO Optimization

  • Training each LSTM took approximately 8.5–9 hours, so the ACO process was parallelized with MPI for Python to use high-performance computing resources.Each evolved topology requires its own training run.Source: §4.1 · pp. 16–17
  • An asynchronous master-worker approach load-balances training by assigning workers new networks when they finish previous ones, without waiting for other workers.The master can generate a network from the population currently available.Source: §4.1 · p. 17
  • Workers train candidate networks on flight data records and return test error and cell paths; the master rewards paths from a new population-best network by increasing their pheromones 15%, up to a threshold of 20.Paths from networks that do not outperform the population best are not penalized.Source: §4.1 · p. 17

5. Implementation

The implementation uses Theano for neural-network training and MPI on a high-performance computing cluster for the ACO algorithm. GPU training was slower than CPU training, and sending the full input dataset to the GPU did not resolve the overhead.

5.1. Programming Language

  • Theano was selected for compiling functions to C and CUDA, reducing backpropagation update overhead, computing gradients, and using GPUs.The authors note that automatic gradient computation avoids manually deriving and debugging gradients, which is particularly challenging for LSTM neurons.Source: §5.1 · p. 17

5.2. Data Processing

  • Flight data parameters were normalized between 0 and 1, and sigmoid activation performed significantly better than ArcTan on the data.Source: §5.2 · p. 17

5.3. Machine Specifications

  • The algorithm ran on the University of North Dakota HPC cluster, and 575 epochs were used to match the previous LSTM training setup.The cluster had 31 nodes with 8 cores each, for 248 cores total.Source: §5.3 · p. 17

5.4. Using GPUs for LSTM RNN Training

  • The implementation concatenates gate weight matrices so their outputs can be extracted from one dot product and weights transferred between CPU and GPU as one data structure.Source: §5.4 · pp. 17, 19
  • The ACO path-generation procedure probabilistically selects input and hidden paths using pheromone values, then increases pheromones on paths from population-best networks.The algorithm caps pheromone values at a user-specified maximum.Source: §5.4 · pp. 18–19
  • For Architecture I, one learning iteration took more than twenty minutes on the GPU and slightly more than two minutes on the CPU.The GPU version was slower despite measures to manage GPU threads, blocks, grids, and CPU–GPU data transfer.Source: §5.4 · p. 19
  • Sending the full input dataset to the GPU did not improve performance; the authors concluded that the matrices were too small to overcome transfer overhead and called for further study.Source: §5.4 · p. 19

6. Results

Among the fixed designs, Architecture I had the best overall vibration-prediction accuracy; ACO optimization of its LSTM cell structure improved 10-second-ahead prediction and reduced network weights.

6.1. Previous Training Results

  • Mean squared error was used to train the original three architectures, and Table 3 reports their training results.The authors state that training results are directly proportional to testing results.Source: §6.1 · p. 19, Table 3 · p. 20

6.2. Cost Function

  • Mean squared error served as the training cost for predictions 1, 5, 10, and 20 seconds ahead, with cost curves shown on a logarithmic scale.The authors chose it because it provides a smoother optimization surface for backpropagation than mean average error.Source: §6.2 · p. 20, Figure 11 · p. 20

6.3. Previous Architecture Results

  • Mean squared error measured training error, while mean absolute error measured final accuracy and equaled percentage error because parameters were normalized between 0 and 1.Figures condense test-flight predictions on shared plots; each flight ends when vibration reaches the normalized critical value of 1.Source: §6.3 · pp. 20–21, Table 4 · p. 21, Table 5 · p. 21

6.3.1. Results of Architecture I

  • Architecture I had the best overall accuracy; predicting one second ahead improved mean absolute error by about 0.5% compared with predicting five seconds ahead.Its higher vibration peaks were predicted more accurately than lower peaks.Source: §6.3.1 · p. 21, Table 4 · p. 21

6.3.2. Results of Architecture II

  • Architecture II was the least successful at vibration prediction and performed weakly at peaks compared with the other architectures.The authors suggest averaging second-layer outputs may explain its weaker performance.Source: §6.3.2 · p. 21, Table 4 · p. 21

6.3.3. Results of Architecture III

  • Despite being deeper and using 20 seconds of past memory, Architecture III was less accurate than the simpler Architecture I.Doubling its training epochs brought no significant improvement; at 20 seconds, its cost curve was unsmooth and some predictions substantially overshot actual vibration.Source: §6.3.3 · pp. 21, 24, Figure 13 · p. 23

6.4. ACO Results

  • ACO-evolved Architecture I improved 10-second-ahead prediction error from 5.51% to 4.17% (1.35% improvement); the top 30 evolved networks reduced weights by 42% to 45%.Optimization changed the M1 gate meshes while M2 retained its original topology.Source: §6.4 · p. 24, Table 6 · p. 26

7. Discussion and Future Work

ACO gate optimization reduced connections while improving predictive ability, though the best sparse matrix retained all selected inputs. Future work proposes using connection patterns to investigate vibration contributors and expanding optimization to gate-specific meshes, network structure, and ACO hyperparameters.

7. Discussion and Future Work

  • ACO gate optimization reduced connections while improving predictive ability; the best sparse matrix retained every input, which the authors interpret as each selected parameter contributing positively.The authors also suggest that additional connections can make the LSTM RNN harder to train and reduce predictive ability.Source: §7 · p. 26
  • Connection counts and weight magnitudes may help identify vibration contributors, and adding flight parameters could help the algorithm determine which inputs contribute most.The authors also propose examining inputs connected to individual hidden neurons to assess combined parameter effects.Source: §7 · p. 26
  • Future work would vary the connection mesh across the four LSTM gates and time steps, rather than reusing one mesh throughout.The present work generated one connection mesh and used it in all gates at all time steps.Source: §7 · p. 30
  • Future work would optimize network-structure connections alongside within-cell connections, potentially enabling non-adjacent links and removal of unused cells or layers.The proposed extensions are described as having potential, not as demonstrated results.Source: §7 · p. 30
  • ACO hyperparameter tuning, including ant counts and pheromone updates, is proposed to improve convergence speed.Suggested changes include reducing pheromones on lower-fitness paths and periodically decreasing pheromone levels.Source: §7 · p. 30
Loading 1710.03753v1…