DQN Explained: Replacing Q-Tables with Neural Networks in JAX

Short summary

Day 5 of a public RL learning series explains Deep Q-Networks (DQN) by replacing a Q-table lookup with a neural network forward pass. The article covers the two key problems DeepMind solved — correlated samples via experience replay and moving targets via a target network — with code snippets in raw JAX. The Bellman equation and epsilon-greedy strategy remain unchanged; only the source of Q-values shifts from a table to a network.

• •DQN replaces Q-table lookup with neural_network(state) returning Q-values per action — Bellman equation stays the same
• •Experience replay breaks correlation by sampling random batches from a transition buffer instead of sequential frames
• •Target network provides stable Bellman targets by keeping a separate copy of weights updated every 100 steps

Generated with AI, which can make mistakes.