The Local AI Developer (Part 2): A Reality Check After Real-World Testing

Short summary

After two months of daily use, a local AI developer stack revealed significant practical challenges: 4-5 tokens/second throughput, frequent OOM crashes from underestimated RAM needs, and agents running in circles due to misconfigured context windows. Solutions include using smaller specialized models, experimenting with different runtimes optimized for specific hardware, and practicing strict context hygiene. The key takeaway is that local AI development is a distinct discipline requiring systems-engineering thinking around memory, context, and model-task matching.

• •Local LLMs run at 4-5 tokens/sec on laptops — slower than human reading speed, degrading developer experience
• •RAM is consumed by context windows, sessions, and system prompts beyond just model weights, causing frequent OOM crashes
• •Solutions: smaller specialized models, hardware-optimized runtimes, and strict context window management

Generated with AI, which can make mistakes.