Removing PER From Rainbow DQN Set a New Snake AI World Record

Short summary

Prioritized Experience Replay (PER), considered one of Rainbow DQN's two most critical components on Atari, actually degrades performance on the Snake game when combined with distributional RL (C51). The author identified and corrected a beta_anneal_steps misconfiguration in parallel training that was suppressing PER's impact during early training. C51 without PER achieves a world record of 153 versus 134 with full Rainbow, demonstrating that component importance is task-dependent.

• •PER degrades Snake performance despite being Rainbow's second-most important component on Atari
• •Beta annealing misconfiguration in parallel training masked PER's true impact until corrected
• •C51 without PER (153) far outperforms full Rainbow with PER (134), showing task-dependency

Generated with AI, which can make mistakes.