SkewAdam: A tiered optimizer that cuts MoE state memory by 97% (fits a 6.7B MoE on a 40GB GPU) [R]

Short summary

SkewAdam is a new tiered optimizer that cuts optimizer state memory for Mixture-of-Experts models by 97.4%, from 50.6 GB to 1.29 GB. It allocates precision based on parameter behavior: full state for backbone, factored 2nd moment for experts, and exact 2nd moment for routers. This enables a 6.7B MoE to train on a single 40GB GPU without sacrificing convergence or router stability.

• •SkewAdam reduces MoE optimizer state memory by 97.4% (50.6 GB → 1.29 GB)
• •Tiered allocation: backbone gets momentum + factored 2nd moment, experts get factored 2nd moment only, router gets exact 2nd moment
• •A 6.7B MoE fits on a single 40GB GPU with no convergence or router stability loss

Generated with AI, which can make mistakes.