Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions

Short summary

Researchers introduce HyenaND, a subquadratic operator that applies input-dependent global convolutions directly on native multi-dimensional data geometry, avoiding the rasterization required by recurrent alternatives. The CUDA implementation nSubQ fuses FFT-convolution for wall-clock speedups. Hybrid configurations interleaving HyenaND and attention layers outperform both pure attention and recurrence-based hybrids across genomics, vision, medical imaging, and PDE tasks.

• •HyenaND is a subquadratic, global, input-dependent operator for native multi-dimensional data
• •CUDA implementation nSubQ fuses FFT-convolution for O(L log L) wall-clock speedups
• •Hybrid HyenaND+attention configs outperform pure attention and recurrence baselines across multiple domains

Generated with AI, which can make mistakes.