Back to feed
arXiv cs.CL
arXiv cs.CL
7/22/2026
Convolution for Large Language Models

Convolution for Large Language Models

Short summary

This study tests whether lightweight depthwise convolutions can add local inductive bias to Transformer-based LLMs without increasing model size. The best results come from applying convolution to projected queries, keys, and values before attention in Qwen3 blocks. A residual depthwise convolution with kernel size 3 improves average accuracy across seven benchmarks while adding less than 0.01% parameters.

  • •Depthwise convolutions on QKV projections before attention improve LLM accuracy
  • •Best config: residual depthwise convolution with kernel size 3, no extra norm or activation
  • •Adds <0.01% parameters while improving average accuracy on seven downstream benchmarks

Generated with AI, which can make mistakes.

#research-breakthrough
Read full article at arXiv cs.CL

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment
arXiv CS.AI
Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models
arXiv cs.LG
ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modelin
arXiv cs.CL
Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression
arXiv cs.LG