The SDXL VAE overflow that decoded black images in fp16

Short summary

SDXL's VAE decoder overflows in fp16 when post-convolution activations exceed 65504, producing black images. Photoroom debugged this 1-in-600 production bug and tested four solutions: full fp32 costs +210% latency (unusable), VAE-only fp32 upcast costs +18%, bf16 VAE costs +6%, and rescaled weights cost zero latency. Recommendation depends on GPU: bf16 on Ampere hardware; rescaled weights on older GPUs.

• •SDXL VAE overflows in fp16 precision when activations exceed 65504, producing black renders
• •Photoroom debugged to root cause and benchmarked four solutions with measured latency tradeoffs
• •bf16 VAE (+6% latency) recommended for Ampere GPUs; rescaled weights (zero cost) for older hardware

Generated with AI, which can make mistakes.