PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges

Lei Lei, Yue Tan, Kan Zheng, Shiwen Liu, Kuan Zhang, Xuemin, Shen

arXiv:1907.09059v3cs.LGstat.ML

TL;DR

DRL for AIoT faces challenges from incomplete perception, delayed control, multi-agent coordination, and resource constraints. This paper surveys DRL methods and applications, proposes a general AIoT model, and identifies open research issues for autonomous IoT systems.

  • Problem

    AIoT control involves incomplete perception, delayed actions that can violate the Markov property, and complex multi-agent coordination challenges.

  • Method

    The paper provides a DRL tutorial, proposes a general AIoT DRL model based on IoT’s three-layer structure, and classifies applications by their states, actions, rewards, and methods.

  • Results

    The paper presents a structured survey of DRL applications in AIoT and identifies challenges and open issues for future research.

  • Takeaways & Limitations

    The proposed model organizes existing AIoT DRL research across application categories and provides a framework for analyzing future autonomous-control problems.

  • Takeaways & Limitations

    DRL training can require tens of thousands or millions of time steps, making computation and energy consumption problematic for resource-constrained IoT devices.

Abstract

from arXiv · show

The Internet of Things (IoT) extends the Internet connectivity into billions of IoT devices around the world, where the IoT devices collect and share information to reflect status of the physical world. The Autonomous Control System (ACS), on the other hand, performs control functions on the physical systems without external intervention over an extended period of time. The integration of IoT and ACS results in a new concept - autonomous IoT (AIoT). The sensors collect information on the system status, based on which the intelligent agents in the IoT devices as well as the Edge/Fog/Cloud servers make control decisions for the actuators to react. In order to achieve autonomy, a promising method is for the intelligent agents to leverage the techniques in the field of artificial intelligence, especially reinforcement learning (RL) and deep reinforcement learning (DRL) for decision making. In this paper, we first provide a tutorial of DRL, and then propose a general model for the applications of RL/DRL in AIoT. Next, a comprehensive survey of the state-of-art research on DRL for AIoT is presented, where the existing works are classified and summarized under the umbrella of the proposed general DRL model. Finally, the challenges and open issues for future research are identified.

I. INTRODUCTION

The paper frames AIoT as IoT integrated with autonomous control, using DRL agents to map sensed states to actions across physical, network, and computing layers. It contributes a DRL tutorial, a general AIoT model, a survey of applications, and discussion of challenges and future research.

  • Motivation: AIoT combines IoT sensing and connectivity with autonomous control so intelligent agents can process system status and actuate physical responses.The AIoT environment may include IoT devices, wireless networks, edge/fog servers, and cloud servers.
  • Motivation: DRL addresses the difficulty of learning policies from high-dimensional sensory data by combining reinforcement learning with deep learning.The paper presents DRL as a solution for agents that must represent environmental state and learn control policies.
  • Contributions: It provides a unified tutorial of DRL methods, classifying basic algorithms as value-based or policy-gradient and introducing POMDP-based and multi-agent DRL.Policy-gradient methods are further organized by policy type, policy evaluation, and parameter-update technique.
  • Contributions: The paper proposes a general AIoT DRL model that divides the environment into perception, network, and application layers and defines their states, actions, and rewards.The model also relates logical agents to physical locations and supports separated or integrated actuator and resource control.
  • Contributions: The survey classifies AIoT DRL applications by model type, DRL elements and algorithms, agent location, and centralized or distributed implementation.It compares existing models with the proposed general model to derive insights for future work.
  • Challenges: The paper identifies challenges including incomplete perception and delayed control, alongside broader open issues in applying DRL to autonomous AIoT control.It positions these challenges as directions for future research in an emerging field.

A. Basic DRL Algorithms

This section surveys basic and advanced DRL algorithms, covering value-based methods, policy gradients, actor-critic methods, and techniques that stabilize DQN training. It also introduces DDQN to reduce Q-function overestimation.

  • DRL algorithms are organized into value-based methods, policy gradients, Monte-Carlo actor-critic, deterministic policy gradients, and advanced DRL algorithms.
  • Value-Based Methods: DQN uses a neural network that maps a state to approximated Q-functions for every available action.The algorithm initializes network parameters and derives target values from the Bellman equation.
  • Value-Based Methods: DQN minimizes a regression loss through stochastic gradient descent, with α specifying the learning rate.The regression loss evaluates how well neural networks approximate Q-functions or value functions.
  • Value-Based Methods: Freezing target-network parameters for a time period makes DQN training more stable and controllable.The online network is updated each iteration and periodically shares its parameters with the target network, reducing divergence risk and instability from rapid propagation.
  • Value-Based Methods: Experience replay stores each interaction and trains on the stored dataset, removing observation correlations and smoothing changes in data distribution.It broadens updates across the state-action space and permits larger parameter updates.
  • Advanced DRL Algorithms: DDQN uses two parameter sets to derive target values, addressing DQN's tendency to overestimate actions when one target Q-function selects and evaluates them.

Y DDQN

The paper organizes DRL methods around value-function and policy approximation, then distinguishes stochastic, deterministic, Monte Carlo, and actor-critic approaches. It explains their update mechanisms, architectures, and principal trade-offs, including limitations of value-based and Monte Carlo methods.

  • Value-based methods: DDQN selects actions with the online network and evaluates them with the target network, reducing Q-value overestimation and improving stability.DDQN retains the rest of the DQN algorithm while combining double Q-learning with separate selection and evaluation networks.
  • Value-based methods: Value-based DRL methods approximate value or Q-functions, but cannot handle large or continuous action spaces or optimal stochastic policies.Their deterministic-policy restriction makes most value-based algorithms, including DQN, off-policy.
  • Policy gradient methods: Policy gradient methods directly approximate a policy πθ(s) from state inputs and update its parameters toward higher expected reward.Stochastic policy gradients use trajectory returns, while deterministic policy gradients model the policy as a deterministic decision.
  • Monte Carlo policy gradient: Monte Carlo policy gradients estimate trajectory returns by simulation and multiply each return by ∇θ log π(at|st; θ) for policy updates.A baseline b(st) independent of the action reduces policy-gradient variance, but these on-policy methods remain sample intensive.
  • Actor-critic methods: Actor-critic methods use an actor network for policy decisions and a critic network for value estimation, combining policy-gradient and value-based mechanisms.For stochastic policy gradients, the critic estimates the baseline and advantage-related update quantities; actor and critic use separate learning rates.
  • Actor-critic methods: Actor-critic methods require fewer samples than Monte Carlo methods and support continuous actions and stochastic policies, but recursive value estimates can make them unstable.The critic improves value-function estimates while the actor updates the policy to seek higher reward.

1) POMDP-based DRL:

POMDP-based DRL addresses environments where agents cannot directly observe states by using observations and history to guide policy learning. The surveyed methods include recurrent, belief-state, and memory-based approaches.

  • POMDP formulation: POMDPs extend MDPs with observation spaces and observation models when agents cannot directly observe system states.
  • POMDP formulation: Agents exploit observation histories to reduce uncertainty about the underlying state.
  • Recurrent methods: DRQN adds recurrence to DQN through an LSTM layer, integrating information across time for partial observability.
  • Recurrent methods: RPG methods learn policies that map observation histories to actions rather than directly mapping states to actions.
  • Memory and belief methods: MERLIN uses a memory-based predictor and policy to produce action distributions from entire past observation sequences.
  • Memory and belief methods: DBQN maps estimated belief states to actions with DQN, while replay, target networks, and adaptive learning stabilize updates.

2) Multi-Agent DRL:

Multi-agent DRL extends reinforcement learning to interacting agents that may cooperate or compete, but simultaneous policy changes create non-stationarity and instability. The surveyed solutions stabilize value learning or coordinate policy-gradient agents through centralized training and decentralized execution.

  • Multi-agent formulation: Multi-agent systems contain autonomous interacting agents sharing an environment, with joint actions, transition dynamics, and agent-specific rewards.
  • Multi-agent formulation: Fully collaborative games share rewards, fully competitive games use rewards summing to zero, and mixed games are neither.
  • Challenges: Changing policies among simultaneously learning agents can alter another agent’s optimum, affecting convergence and causing instability.
  • Multi-agent value-based methods: MAIS and MAF stabilize DQN experience replay using importance sampling or replay-memory fingerprints.
  • Multi-agent value-based methods: A coordinated DQN method factorizes the global Q-function into local subproblems and uses max-plus coordination for joint-action optimization.
  • Multi-agent policy-gradient methods: Centralized-training methods address high policy-gradient variance by combining centralized critics with decentralized actors.
  • AIoT context: AIoT models span perception, network, and application layers, with states, actions, and rewards defined separately or jointly across layers.

B. Network Layer

The AIoT model organizes reinforcement-learning environments across perception, network, and application layers, then integrates their states, actions, rewards, and agents. The paper uses this model to structure its DRL application review and formulation procedure.

  • Network layer: Network-layer models use communication states, resource-control actions, and performance rewards such as delay, error probability, and power consumption.
  • Application layer: Application-layer models use computing states, resource-control actions, and rewards involving resource utilization and offloading delay.
  • Integration of three layers: Integrated AIoT states aggregate physical, network, and computation states, while integrated actions aggregate actuator, communication, and computing controls.
  • Integration of three layers: The integrated reward normally optimizes physical-system performance as a function of network and computing performance: raIoT = rphy(rnet, rcomp).
  • Agent placement: Agents may reside in IoT devices, wireless APs, or edge/fog/cloud servers, and their physical location can differ from their logical layer.
  • Cross-layer coordination: Cross-layer environments require agents to share information and jointly optimize policies, such as exchanging delay information or optimization objectives.
  • Application review: The review classifies prior work by AIoT layer, system model, DRL elements, algorithms, and implementation choices including agent placement and centralized or distributed execution.
  • DRL formulation procedure: The proposed procedure moves from a system model to a DRL model, its state, action, and reward elements, and finally an appropriate DRL algorithm.

A. AIoT Network Layer - IoT Communication Networks

IoT communication networks must autonomously manage scarce wireless resources while meeting latency and reliability requirements. The reviewed WSN studies apply DRL to topology, relay selection, cognitive transmission, and sensing coverage problems.

  • Network motivation: IoT networks range from short-range LANs to long-range WANs and must manage scarce radio resources for massive IoT data.
  • Wireless sensor networks: WSNs contain sensor, relay, and gateway nodes organized through star, tree, or mesh topologies.
  • Wireless sensor networks: Mesh connectivity can improve throughput but activating more relay nodes increases energy consumption, motivating transmission-range control.
  • Wireless sensor networks: A cooperative-communication MDP uses channel state and broadcasting energy as state information, selects a relay with DQN-type methods, and rewards energy consumption and information gain.
  • Cognitive networks: DRL is applied to cognitive IoT transmission, where cognitive nodes sense network conditions and make transmission decisions in crowded spectrum.
  • Sensing coverage: Multi-agent sensing coverage assigns each sensor an agent, with actions controlling power modes and rewards combining coverage gain with energy consumption.
  • Sensing coverage: DRLT accelerates exploration for mobile robotic sensor placement and outperforms RRT and RRLR in finding near-optimal sampling locations.

2) Wireless Sensor and Actuator Networks:

DRL is applied across wireless sensor and actuator networks to address scheduling, routing, mobility, NB-IoT coverage, and energy-harvesting control. Existing studies commonly use channel, queue, topology, and energy states to optimize communication or actuator outcomes.

  • Scheduling: DRL formulates sensor-channel scheduling as an MDP for remote state estimation in volatile wireless networks.The scheduling problem allocates wireless channels to sensors and is solved using DQN.
  • Routing: Routing agents use topology information and optimize decisions against expected network lifetime and average network latency.The state includes hop-count and energy-source weights, while actions change to a neighbor state or retain the current state.
  • Mobility control: DQN-based node mobility control targets connectivity, a requirement for collaborative actuator tasks in WSANs.Moving nodes can improve connectivity, coverage, energy consumption, and accuracy; one study focuses specifically on connectivity.
  • NB-IoT: RL-based dynamic spectrum access reduces NB-IoT transmission repetitions while improving coverage and reducing device energy consumption.The reward is 0 or 1 depending on whether the selected channel is occupied or vacant.
  • Energy harvesting: Energy-harvesting studies model stochastic energy management with MDPs, POMDPs, LSTM-based DQN, DDQN, and DDPG.Objectives include minimizing weighted packet loss under delay constraints, accounting for sum rate, and optimizing net bit rate in continuous action spaces.
  • Comparison and insights: Most surveyed communication-network studies use star topologies, DQN-family algorithms, and centralized implementations, leaving mesh, continuous-action, and resource-aware distributed designs open.Centralized control can incur increasing complexity and overhead, while distributed implementations must balance performance against device constraints.

B. AIoT Application Layer - IoT Edge/Fog/Cloud Computing Systems

DRL supports IoT edge/fog/cloud decisions about task offloading, resource allocation, and system organization. The surveyed models span layered architectures and centralized, distributed, or semi-distributed control, with scalability and device constraints shaping open directions.

  • System model: Edge/fog/cloud systems use DRL to decide task offloading and allocate communication and computation resources.These decisions determine whether tasks execute locally or on edge, fog, or cloud servers.
  • System architecture: Architectures are classified as two-layer or three-layer, with two-layer systems further divided into one-to-multiple, multiple-to-one, and multiple-to-multiple forms.The layers represent offloading and offloaded components, with a middle layer added in three-layer systems.
  • Multiple-to-multiple architecture: Multiple-to-multiple systems require routing and load balancing alongside offloading and communication or computation resource allocation.Reported objectives include reducing delay, balancing load, and managing power consumption across collaborating edge servers.
  • Task arrival model: Dynamic task arrivals create sequential decision problems because current offloading decisions affect future tasks and long-term system performance.Tasks may arrive stochastically or deterministically and remain buffered when immediate processing is unavailable.
  • Implementation: Centralized control becomes difficult at scale because the DRL state space grows exponentially with device count, increasing computation complexity and communication overhead.Fully distributed learning also faces mutual-exclusion, training-time, processing, and energy constraints on IoT devices.
  • Implementation: A semi-distributed design decomposes global state into local states, uses neural networks to approximate value functions, and lets edge servers and devices cooperate on actions.It optimizes offloading and user scheduling in an NB-IoT edge-computing system for delay and energy consumption.

2) Caching:

The paper surveys DRL applications from caching and edge computing through autonomous robots and cloud robotics. Across these settings, models encode environment, kinematics, manipulation, resource, and task-completion information according to the control problem.

  • Caching: DRL caching policies place or replace edge content to balance communication cost against data freshness loss.Caching can also be jointly optimized with offloading using an actor-critic algorithm.
  • Comparison and insights: The surveyed edge/fog/cloud literature mainly uses two-layer one-to-multiple or multiple-to-one architectures, while more complex architectures remain less studied.DQN-family methods are most frequent, whereas actor-critic methods appear particularly in continuous-action settings.
  • Applications: Autonomous robot applications include mobile behavior control, robotic manipulation, multi-robot coordination, and cloud robotics.The surveyed work covers path planning, navigation, compound manipulation, collaborative tasks, and cloud-supported robotics.
  • DRL model: Robot-control states commonly include camera-based environmental observations and kinematic states, with manipulation states added for manipulation tasks.Actions represent movement or manipulation controls, and rewards often reflect successful task completion.
  • Multi-robot systems: Multi-robot methods use multi-agent reinforcement learning to coordinate shared observations, roles, and joint task-space behavior under environmental uncertainty.Collaborative methods can interleave online execution and relearning to accommodate uncertainty and improve performance.
  • Cloud robotics: Cloud robotics uses DRL for offloading and resource allocation, including autonomous decisions about request acceptance and computing-resource quantities.One reported scheme reduces human participation in planning and improves overall utility through online learning.

5) Comparison and Insights:

Across autonomous-vehicle research, DRL models represent traffic environments and vehicle kinematics, choose motion controls, and reward safety or other driving objectives. Studies cover single-vehicle and cooperative multi-vehicle settings, with evaluation relying mainly on simulation.

  • System model: Autonomous-driving studies formulate single- and multi-vehicle control as DRL decision problems involving nearby traffic, road facilities, obstacles, and vehicle kinematics.Environmental information may come from onboard sensing or vehicle-to-vehicle and vehicle-to-Internet communication.
  • DRL model: Vehicle actions include discrete or continuous velocity control and discrete or continuous direction control such as lane changes, turns, or steering angles.The action representation depends on whether the controller selects categories or specific control values.
  • Reward design: Driving rewards commonly reflect collision avoidance or safety, while other studies incorporate desired velocity, travel distance, average traffic velocity, fuel consumption, or power consumption.Some work combines reinforcement learning with formal safety verification to restrict actions to safe choices.
  • Multi-vehicle systems: Cooperative multi-vehicle problems use multi-agent methods for coordinated movement planning and traffic-signal control.One example coordinates two cars to avoid collisions before returning to their intended paths.
  • Evaluation: Autonomous-driving methods are evaluated mostly in simulated traffic environments, although some studies use real traffic datasets.Simulators provide experimental datasets, while real datasets are described as better reflecting actual traffic conditions.
  • DRL algorithm: Actor-critic models estimate Q-values as critics that guide policy-based action selection in autonomous-driving examples.The state combines driving-environment information with the vehicle’s kinematic state.

4) Comparison and Insights:

The surveyed smart-vehicle and smart-grid studies vary in system scale, DRL formulation, and implementation. Smart-vehicle work commonly distinguishes single-vehicle from multi-vehicle settings, while smart-grid research models uncertainty using historical observations and energy-management objectives.

  • Smart Vehicles: Smart-vehicle problems may involve one vehicle or multiple vehicles; multi-vehicle settings require multi-agent DRL.Autonomous driving often uses deep Q-Learning or policy-gradient methods for single-vehicle problems, whereas vehicular networking and edge/fog/cloud computing usually involve multiple vehicles.
  • Smart Vehicles: Vehicle DRL states combine kinematic and environmental features, while actions represent velocity or direction and rewards typically target safety, smoothness, efficiency, or environmental benefits.Existing studies rarely optimize all listed driving criteria simultaneously.
  • Smart Vehicles: Discrete state and action spaces commonly use deep Q-Learning, whereas continuous spaces commonly use actor-critic methods such as A3C or DQN.The surveyed algorithm choice follows the representation of the state and action spaces.
  • Smart Grid: Smart-grid energy-management studies address energy storage management, demand response, and energy trading under renewable-generation and demand uncertainty.Historical data and DRL are used to learn energy-management policies, and related models are classified by system-model considerations.
  • Smart Grid: Microgrid energy-storage studies learn charging and discharging policies using states such as battery SoC, energy generation, and demand, with rewards based on balance or transaction cost.Other formulations add diesel-generator dispatch and energy trading to the action space and use ADP, RNN, or actor-critic methods.
  • Smart Grid: Because renewable generation and load demand are uncertain, existing microgrid studies often formulate POMDPs from historical time series plus observable states such as battery SoC.The surveyed smart-grid work relies on datasets containing historical prices, loads, and renewable generation to simulate real-time systems.

3) Energy Trading:

Energy trading studies model interactions among producers, brokers, consumers, and microgrids, while related smart-grid research often uses distributed multi-agent formulations. The survey identifies underexplored system-model and implementation choices, especially beyond direct demand response and centralized control.

  • System Model: Energy-trading scenarios include producer, broker, and consumer entities with wholesale and retail market interactions.Consumers’ demand can influence wholesale prices, while brokers purchase energy wholesale and supply consumers through retail markets.
  • Related Work: Existing studies range from isolated-microgrid consumer models to producer-consumer and multi-microgrid trading systems using multiple agents.Agents may represent generation, storage, consumption, bidding, brokers, or consumers, depending on the model.
  • Insights: The survey reports that direct demand response dominates current research, while price-driven demand response needs more detailed study.Energy-trading studies mainly use producer-broker-consumer or producer-consumer models; consumer-broker and producer-broker models are less common.
  • DRL Model: For energy storage management, battery SoC is commonly part of the state; partial observability adds historical features and POMDP formulations, while energy trading often uses distributed multi-agent models.These choices align the DRL model with uncertainty and with the roles of multiple trading agents.
  • DRL Algorithm: Q-Learning and DQN are widely used because smart-grid action spaces are often discrete and relatively small, while actor-critic methods support stochastic policies and continuous actions.Classical neural-network architectures can be used to improve DRL performance.
  • Implementation: Most energy-storage and demand-response implementations are centralized, leaving cooperation among demand-response devices as an open area for further study.The survey contrasts centralized energy-management implementations with the need for more distributed cooperation.
  • Cross-Domain Insights: Across AIoT domains, existing work usually concentrates on a single architectural layer rather than jointly controlling perception, communication, computation, and physical systems.Smart-grid studies mainly focus on the perception layer, while communication-layer information is often assumed readily available despite sensor and transmission requirements.
  • Open Issues: Joint-layer modeling remains constrained by communication effects: transmission delay and reliability can affect information delivery and vehicle-control effectiveness.The paper identifies integrated control across AIoT layers as a future direction.

V. CHALLENGES, OPEN ISSUES, AND FUTURE RESEARCH DIRECTIONS

The paper identifies open AIoT challenges arising from incomplete perception, delayed control, multi-agent coordination, heterogeneous capabilities, and coupled resource and actuator decisions. It concludes that DRL models and algorithms must be adapted to complex, application-specific settings.

  • Incomplete Perception Problem: Incomplete sensing and information loss can make AIoT states partially observable, invalidating a standard MDP formulation.POMDP approaches introduce belief states but require model knowledge and create an information-gathering versus exploitation dilemma.
  • Incomplete Perception Problem: Belief-state POMDP methods require model knowledge that may be unsuitable for many complicated AIoT scenarios.Finding an optimal policy also requires additional environment interactions to resolve uncertainty.
  • Delayed Control Problem: Real AIoT systems face control delay between state measurement and action execution because sensing, transmission, computation, and actuation take time.The executed action may be an action generated τ time steps earlier, so the next state depends on a previously determined action.
  • Delayed Control Problem: Existing delayed-control methods mainly address constant delay, whereas stochastic or resource-dependent delay remains an open issue.Delay may depend on communication and computation resource-control actions.
  • Multi-Agent Systems: Distributed and semi-distributed multi-agent systems require efficient collaboration, fair competition, and joint rewards for coupled agents.Heterogeneous multi-agent systems remain underexplored because differing capabilities complicate modeling other agents.
  • Joint Resource and Actuator Control Problem: AIoT research must jointly optimize resource-control and actuator-control actions because network and computation performance affect physical-system control.Most existing studies optimize either infrastructure performance or physical performance under ideal infrastructure conditions.
  • Complex DRL Problems: Multi-layer DRL problems require careful feature selection and discount-factor tuning to balance generalization, overfitting, convergence stability, and distant rewards.The paper notes that too many features complicate learning, while high or low discount factors create different training problems.
  • Conclusion: The paper surveys DRL methods, proposes a three-layer AIoT model, classifies applications, summarizes model components, and identifies challenges and open issues.Its scope covers the model, applications, and challenges of DRL in AIoT systems.

APPENDIX

The appendix introduces RL through its agent-environment formulation and core MDP components, then explains value functions, Bellman equations, and iterative learning. It also distinguishes exploratory policies and on-policy versus off-policy evaluation.

  • RL Fundamentals: RL models an agent interacting with a stochastic environment whose states, actions, rewards, transitions, and policy define the decision process.The agent selects actions from an action space and observes their consequences in the environment.
  • RL Fundamentals: An RL policy maps states to actions to optimize the expected reward from an initial state over a trajectory.The trajectory may use total, discounted total, or average reward, with terminal time T potentially infinite.
  • Value Functions: The optimal policy selects the action with the maximum Q-function value, with V*(s) = max_a Q*(s, a).The Q-function represents expected reward for taking an action and then following a policy.
  • Bellman Equations: Bellman optimality equations relate current and next-state value or Q-functions by combining immediate reward with future expected reward.These equations underpin bootstrap methods such as Q-Learning and temporal-difference learning.
  • Learning Process: RL learning iteratively alternates policy prediction and policy evaluation until value or Q-functions converge.Soft policies such as ϵ-greedy support exploration instead of always selecting the current greedy action.

B. Building Blocks of DRL - Deep Learning

Deep learning uses neural networks to learn mappings from data, with architectures ranging from feedforward networks to recurrent networks that incorporate input history. Training minimizes a loss function through iterative parameter updates, while step-size selection affects optimization efficiency.

  • Deep learning uses artificial neural networks to learn autonomously from large amounts of data for regression and classification tasks.Regression predicts continuous values, whereas classification predicts finite categorical outputs.
  • Feedforward neural networks contain input, hidden, and output layers whose neurons use weights, biases, and nonlinear activation functions.Networks with two or more hidden layers are called deep neural networks.
  • Recurrent neural networks process input sequences by combining current data with prior hidden-state outputs through internal memory.This allows the network to learn from historical information, unlike feedforward networks that consider only current input.
  • Training evaluates the network output against target data with a loss function and updates parameters iteratively using gradient descent over mini-batches.The process repeats with successive mini-batches until convergence.
  • Natural gradients use the Fisher information metric to determine descent direction and replace the ordinary gradient update.The simple gradient is easy to derive, but its step size can be inefficient when chosen poorly.
Loading 1907.09059v3…