Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks

Annie Wong, Jacob de Nobel, Thomas Bäck, Aske Plaat, Anna V. Kononova

Although deep reinforcement learning methods can learn effective policies for challenging problems, the underlying algorithms are complex, and training times are often long. This study investigates how several state-of-the-art versions of Evolution Strategies perform compared to gradient-based deep reinforcement learning methods. We use Evolution Strategies to optimise the weights of a neural network via neuroevolution, performing direct policy search. We benchmark deep and linear policy networks for gradient-based methods, including Proximal Policy optimisation, Soft Actor-Critic, and Deep Q-Learning. These are evaluated against linear Evolution Strategies and Augmented Random Search, and additionally against scalable ES for deeper policies, such as LM-MA-ES. Our results reveal that Evolution Strategies can find effective linear policies for many reinforcement learning benchmark tasks, unlike deep reinforcement learning methods that can only find successful policies using much larger networks, suggesting that current benchmarks are easier to solve than previously assumed. Linear Evolution Strategies also outperform scalable ES variants trained on larger neural networks. Interestingly, Evolution Strategies, which do not use any gradient information, also achieve results comparable to gradient-based deep reinforcement learning algorithms for higher-complexity tasks. Furthermore, by directly accessing the memory state of the game, Evolution Strategies can find successful policies in Atari that outperform the policies found by Deep Q-Learning. Evolution Strategies also outperform Augmented Random Search in most benchmarks, demonstrating superior sample efficiency and robustness in training linear policy networks.