Source-linked AI summary
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
Stefan Elfwing, Eiji Uchibe, Kenji Doya
TL;DR
The paper addresses whether new activation functions and a traditional on-policy reinforcement-learning design can compete with DQN-based methods. It proposes SiLU and dSiLU with TD(λ) or Sarsa(λ), eligibility traces, and annealed softmax selection, and reports state-of-the-art Tetris results plus higher Atari scores than DQN and double DQN. The authors conclude that this simpler approach can be competitive without experience replay or a separate target network.
Problem
The study asks whether neural-network activation functions and traditional on-policy learning can provide competitive reinforcement-learning performance relative to DQN-based approaches.
Method
The paper combines SiLU and dSiLU activations with TD(λ) or Sarsa(λ), eligibility traces, and softmax action selection with annealed temperature.
Results
The agents achieve new state-of-the-art results in stochastic SZ-Tetris and 10×10 Tetris and outperform DQN and double DQN on mean and median Atari 2600 normalized scores.
Takeaways & Limitations
On-policy learning with eligibility traces and softmax selection can be competitive with DQN without experience replay or a separate target network.
Takeaways & Limitations
The method is not feasible for standard 20-row Tetris because of prohibitively long learning time, and stochastic SZ-Tetris results are not directly comparable with standard-Tetris methods.
Abstract
from arXiv · showhide
In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro's TD-Gammon achieved near top-level human performance in backgammon, the deep reinforcement learning algorithm DQN achieved human-level performance in many Atari 2600 games. The purpose of this study is twofold. First, we propose two activation functions for neural network function approximation in reinforcement learning: the sigmoid-weighted linear unit (SiLU) and its derivative function (dSiLU). The activation of the SiLU is computed by the sigmoid function multiplied by its input. Second, we suggest that the more traditional approach of using on-policy learning with eligibility traces, instead of experience replay, and softmax action selection with simple annealing can be competitive with DQN, without the need for a separate target network. We validate our proposed approach by, first, achieving new state-of-the-art results in both stochastic SZ-Tetris and Tetris with a small 10$\times$10 board, using TD($λ$) learning and shallow dSiLU network agents, and, then, by outperforming DQN in the Atari 2600 domain by using a deep Sarsa($λ$) agent with SiLU and dSiLU hidden units.
1 Introduction
The study proposes new activation functions and a traditional on-policy reinforcement-learning approach, then evaluates them across Tetris and Atari 2600 tasks. The reported results include state-of-the-art Tetris performance and improvements over DQN-based baselines.
- Contributions: The study proposes SiLU and dSiLU activation functions for neural-network function approximation in reinforcement learning.SiLU multiplies the input by its sigmoid; dSiLU is its derivative.
- Contributions: The approach uses on-policy learning with eligibility traces and softmax action selection instead of experience replay, without a separate target network.The authors suggest this traditional approach can be competitive with DQN.
- Evaluation: 20%: the best dSiLU agent improves the average state-of-the-art score in stochastic SZ-Tetris.A deep SiLU-dSiLU agent also outperforms the previous state-of-the-art average final score in stochastic SZ-Tetris.
- Evaluation: 232% and 161%: the deep SiLU-dSiLU agent improves mean DQN normalized scores over DQN and double DQN, respectively, across 12 Atari 2600 games.The games were unbiasedly selected.
2 Method
The method combines TD(λ) and Sarsa(λ) with neural-network function approximators using SiLU and dSiLU units, and selects actions through annealed softmax probabilities. The paper also describes the units' functional forms and their relation to existing activation functions.
- Learning algorithms: TD(λ) estimates the state-value function, while Sarsa(λ) is used for action-value learning with neural-network approximators.The parameter updates use gradients of the approximated value functions and eligibility traces.
- Activation functions: SiLU activations are computed by multiplying each hidden-unit input by its sigmoid.For large-magnitude inputs, SiLU approximately resembles ReLU but is non-monotonic and has a minimum near −0.28.
- Activation functions: dSiLU activations are computed as the derivative of SiLU and resemble steeper, overshooting sigmoid functions.The dSiLU reaches approximately 1.1 and −0.1 at inputs near ±2.4.
- Action selection: Softmax action selection uses a Boltzmann distribution, with temperature controlling exploration and exploitation.The temperature is decreased after each episode using hyperbolic annealing.
3 Experiments
The experiments evaluate shallow and deep activation-function variants across stochastic SZ-Tetris, 10×10 Tetris, and Atari 2600 games, using neural-network agents trained with reinforcement-learning methods. dSiLU and SiLU-dSiLU agents achieve strong results, including new state-of-the-art scores in the tested Tetris settings and higher normalized Atari performance than prior DQN variants.
- 3 Experiments: Stochastic SZ-Tetris provides a faster benchmark by restricting pieces to S- and Z-shaped tetrominoes on a 10×20 board.Its likely episode length is a few thousand pieces, requiring at least five orders of magnitude less computation than standard Tetris.
- 3 Experiments: The study compares shallow agents using SiLU, ReLU, dSiLU, and sigmoid hidden units with TD(λ), softmax action selection, and hand-coded state features.The shallow networks use one hidden layer with 50 hidden units and a linear output layer.
- 3 Experiments: The deep experiments use raw board configurations and compare SiLU-SiLU, ReLU-ReLU, and SiLU-dSiLU convolutional networks trained with TD(λ) and softmax action selection.Each deep-network type was trained for 200,000 episodes across five separate runs.
- 3.1 SZ-Tetris: 263 points was the dSiLU shallow agent’s final average score in stochastic SZ-Tetris, improving the previous best performance by 43 points or 20%.Its best single-run score was 320 points, improving the previous best by 25 points or 8%; the deep SiLU-dSiLU agent reached an average final score of 229 points versus the previous 220-point state of the art.
- 3.2 10×10 Tetris: 4,900 points was the dSiLU agent’s average score over the final 10,000 episodes and five runs in 10×10 Tetris, establishing a new state of the art.The best individual run reached 5,300 points, also exceeding the previous best of 5,000 points; the agent used features similar to the original Bertsekas features.
- 3.3 Atari 2600 games: 332% was the SiLU-dSiLU agent’s mean DQN-normalized best mean score across the Atari evaluation, compared with 127% for double DQN.It achieved the highest best mean score in 6 of 12 games and performed much worse than the other agents in Breakout.
4 Analysis
The analysis examines whether TD(λ) and Sarsa(λ) accurately estimate discounted returns and how softmax compares with ε-greedy action selection. Value estimates generally tracked returns, while ε-greedy exploration substantially reduced scores in the evaluated games.
- Value estimation: TD(λ) and Sarsa(λ) were investigated for their ability to estimate discounted returns in reinforcement-learning episodes.The analysis motivates this evaluation because these methods avoid the max operator used in DQN targets.
- Value estimation: Figure 6 compares learned V(s_t)-values with discounted returns across short, medium-long, and long SZ-Tetris episodes, including a fit over 1,000 episodes.The right panel reports the normalized sum of differences and its best linear fit.
- Value estimation: −0.43: the linear fit indicated a small underestimation for an average-length SZ-Tetris episode of 866 time steps.Value estimates overestimated returns for short episodes and underestimated them for long episodes, especially in their middle portions.
- Value estimation: 0.59 and −0.23: normalized sums of differences for representative Asterix and Asteroids episodes, where learned action values matched discounted returns closely overall.The corresponding absolute differences were 1.05 and 1.28; both episodes lasted the full 4,500 time steps.
- Action selection: 4% to 10%: ε-greedy selection with ε = 0.05 produced only this fraction of softmax scores in SZ-Tetris, Asterix, and Asteroids.Even ε = 0.001 reduced mean scores by 30% and 20% in Asteroids and SZ-Tetris, respectively, compared with softmax selection.
5 Conclusions
The study reports that SiLU and dSiLU agents achieved state-of-the-art results across Tetris tasks, while a deep Sarsa(λ) agent outperformed DQN and double DQN on Atari 2600. It also finds that traditional on-policy learning with eligibility traces and softmax selection can be competitive with DQN.
- The dSiLU network agent achieved new state-of-the-art results in stochastic SZ-Tetris and 10×10 Tetris.
- A deep Sarsa(λ) agent with SiLUs in convolutional layers and dSiLUs in fully connected hidden layers outperformed DQN and double DQN on Atari 2600.Performance was measured using mean and median DQN normalized scores.
- A traditional on-policy approach using eligibility traces and softmax selection can be competitive with DQN.The authors describe this approach as leaving room for further improvements, including a target network, dueling architecture, and asynchronous learning.