Channels-last memory format cut our conv backbone latency 22%

Short summary

Photoroom reduced inference latency by 22% on A100s by switching their U-Net segmentation model to PyTorch's channels-last memory format with just four lines of code. The optimization exploits how NVIDIA tensor cores prefer NHWC byte ordering, eliminating per-layer transposes. The gain is hardware and model dependent, strongest for convolution-heavy architectures running in reduced precision like float16.

• •22% latency reduction via channels-last memory format (NHWC instead of NCHW)
• •Four-line code change with no accuracy loss; straightforward to revert if unhelpful
• •Hardware dependent: best on tensor-core GPUs with float16; convolution-heavy models see largest gains

Generated with AI, which can make mistakes.