I Trained a 6.4M-Parameter Transformer From Scratch to Talk About Recipes

Short summary

A developer trained a 6.4M-parameter decoder-only transformer from scratch in PyTorch to power a recipe intelligence app called Rasaveda, avoiding all external API dependencies. The model was pre-trained on WikiText-2 then fine-tuned on recipe data, completing both stages in 40 minutes on a single Colab T4. The post details architecture choices, training metrics, and practical rationale for sizing a model to a narrow task.

• •6.4M-parameter transformer trained from scratch in PyTorch on a single Colab T4 in 40 minutes
• •Two-stage training: pre-training on WikiText-2 then fine-tuning on 2,139 recipe examples
• •Model runs inference on CPU inside FastAPI with no external API dependencies

Generated with AI, which can make mistakes.