Notes on the systems I build, with the figures executed at build time.
Week 1 of the residency was word2vec from scratch. The trick that makes it practical is a shortcut around a 200,000-way softmax, and you can watch it work on a corpus small enough to train in under a minute.
Classifying an image needs one label. Reading a number off one needs a sequence, of a length the model has to decide for itself. That is what a decoder is for, and the alignment it learns is not something anybody specifies.
Week 4 was image captioning the way LLaVA does it: take a vision model and a language model that have never met, freeze them both, and train only a small translator in between.
The second half of week 1 was predicting Hacker News scores from a post’s title and its metadata. The interesting part turned out to be a question about architecture: where in the model should the two kinds of input meet?
Scaling a container to zero is free until someone knocks on the door. A one-line queueing model says how often that actually happens — and where to spend the money to stop it.
Week 5 was fine-tuning Whisper on a single clip, because one example makes visible what a held-out score hides. It also makes visible what you pay for it.
Week 3 was a Vision Transformer built from scratch. The whole idea fits in one move at the very start, and you can see it working on a model small enough to train while this page loads.
Week 2 was a two-tower retrieval system. Building it was straightforward. The part worth writing down is how convincingly the training loss lied about whether it was working.