Rosh Beed

Words get their meaning from the company they keep

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

The first week of the residency was word2vec: build the thing from scratch, no pretrained anything, and see whether the vectors that come out behave the way Mikolov’s papers say they do.

I’m not claiming anything new here. The 2013 papers [1, 2] are thirteen years old and the results are well known. The point of building it was to understand why the training objective is shaped the way it is, which is not obvious from reading about it.

The full run trained on text8 — 17 million words of Wikipedia — and scored at or above published word2vec numbers on the same corpus:

benchmark	untrained	skip-gram	CBOW	published word2vec
WordSim-353 (ρ)	0.028	0.726	0.730	0.68
SimLex-999 (ρ)	0.008	0.297	0.290	0.30
Google analogies	0.000	0.464	0.408	0.38

The untrained column matters more than it looks. A score means nothing until you know what the same model reports before it has learned anything, and I got that habit from a later project where a model that looked trained turned out not to be.

This post rebuilds the idea at a size you can watch: a fiftieth of the corpus, 64 dimensions, trained here while the page builds.

The idea: you never say what a word means

Nobody labels the training data. The model is only ever shown a word and one of its neighbours, and asked whether that pairing is real.

That’s the distributional hypothesis, and it’s an old idea in linguistics: words that turn up in the same contexts tend to mean similar things. Coffee and tea get poured, spilled and brewed. The model never learns that either is a drink. It learns that they keep the same company, which turns out to be close enough to be useful.

So the training signal is: here is the word king, here is the word throne which appeared four words away — score that pair high. And here are five words drawn at random from the corpus — score those low.

The problem the paper is actually solving

The obvious way to train this is a softmax over the whole vocabulary: given the centre word, produce a probability for every word in the corpus being its neighbour, and push up the probability of the one that really was.

That needs a normalising sum over every word in the vocabulary, on every training pair. With 200,000 words and tens of millions of pairs per epoch, you are doing 200,000 dot products to learn one thing.

Negative sampling replaces that question with a much cheaper one. Instead of which of these 200,000 words is the neighbour, ask is this specific pair real or did I make it up — once for the true neighbour, and k more times for randomly drawn words. With k = 5 that is six dot products instead of 200,000.

Show the code

The emotion is a token, not a head

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

The other half of week 5: recognise the emotion in a clip of speech.

The obvious way to build this is a classifier. Take the encoder you already have, mean-pool its output, put a Linear(dim, n_emotions) on top, train with cross-entropy. It works, it’s three lines, and it’s what most people reach for.

I built it the other way, because of something about how Whisper already works.

Whisper already talks about itself in tokens

They aren’t a separate mechanism. They are ordinary vocabulary entries, predicted by the same softmax, trained by the same cross-entropy, sampled the same way.

So the question is whether emotion is a different kind of thing from language and task, or the same kind of thing. And it’s hard to argue it’s different. Adding , and the rest to the vocabulary lets the model say what it heard through machinery it already has.

Show the code

Teaching a model to know when it has finished

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

The second half of week 3. The classifier looks at one digit and picks one of ten labels. This one looks at three digits side by side and has to read the number.

That sounds like a small step and it is a different problem. A classifier produces a fixed-size answer. Reading produces an ordered sequence of unknown length — and something has to decide where it ends.

Everything below is standard encoder–decoder transformer, from the original attention paper [1]. The reason to build it rather than read about it is that three separate mechanisms have to be in place before a single digit comes out right, and they’re easy to nod along to and hard to get correct.

The three things a decoder adds

A start token. The decoder generates one position at a time, each conditioned on what came before. At the first step there is no “before”, so you feed it a token that means begin.

A causal mask. During training the decoder sees the whole target sequence at once, for speed. But position 2 must not be allowed to look at position 3, or the model learns to read the answer it is being asked to predict, and then produces nothing useful at inference when the future genuinely isn’t there. The mask enforces that: each position attends only to itself and everything to its left.

Cross-attention. The decoder needs to look at the image. Self-attention lets the output positions look at each other; cross-attention lets each output position query the encoder’s patch tokens and pull in what it needs.

And then an end token, so the model can say it’s done. That’s what makes the output length the model’s decision rather than a hyperparameter.

Show the code

Freeze both ends and train the middle

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

Week 4: image captioning. Look at a picture, write a sentence about it.

The obvious approach is to train one model on image–caption pairs. The approach that actually gets used is stranger and much cheaper, and it’s what LLaVA [1] and PaLiGemma popularised:

Take a vision encoder that already understands images. Freeze it.
Take a language model that already writes English. Freeze it.
Train a small adapter that turns the vision model’s output into something the language model will accept as input.

Nothing in either big model ever receives a gradient. The project version used CLIP on one side and Qwen3-0.6B on the other, and only ~1.58M adapter parameters were trainable.

The bet is that the two models already contain everything needed, and what’s missing is a translation between two representation spaces that were never trained together.

Why that bet is reasonable

A vision encoder’s output is a vector that means something — but it means something in the vision model’s coordinate system, which was shaped by a completely different training run.

The language model reads token embeddings, which live in their own coordinate system with their own geometry. It has no reason to interpret a CLIP vector as anything at all.

The adapter’s job is not to understand images or to write English. It’s to land the image’s vector somewhere in the language model’s embedding space that makes the language model say the right thing. That’s a much smaller job, which is why a small matrix can do it.

Building both ends from scratch, small

To show the recipe honestly I need two models that genuinely have never met. So I train both here, from scratch, on separate tasks, then freeze them.

The vision model learns to classify MNIST digits. Its hidden layer becomes the “image embedding”.

The language model is a character-level transformer trained on exactly one sentence pattern — the digit is — and nothing else. It knows the template and it knows the ten words. It has never seen an image.

Show the code

A good title doesn’t add upvotes, it multiplies them

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

The second project of week 1: given a Hacker News post — its title, where it links, when it was submitted and who by — predict how many upvotes it will get.

872,554 real posts, split by time so the model is always predicting a future it hasn’t seen. The served model’s median error is 10.1 upvotes against 16.2 for guessing the average every time.

But the number isn’t what made this project interesting. The interesting part was a design question I didn’t expect to have to think about: a post is two different kinds of thing at once — a piece of text, and a handful of numbers and categories. Where in the model should they meet?

Early and late fusion

The multimodal literature has names for the two ends of this. In Snoek et al.’s 2005 survey of video retrieval [1], which is where the terms come from:

Early fusion glues the inputs together first and runs one model over the whole thing. The model can learn anything that depends on both — but it also has to learn the text and the metadata in the same layers, entangled from the start.

Late fusion gives each input its own model, reduces each to a score, and combines the scores at the end. Clean, modular, and you can retrain one branch without touching the other.

Snoek’s objection to late fusion is the one that matters: by the time the two branches meet, each has been squeezed into a single number, so any correlation between the two in feature space is already gone.

They’re the same architecture with a switch

The thing that made this tractable for me was realising early and late fusion aren’t two designs. Late fusion is early fusion with some weights deleted.

Take one hidden layer over all the inputs concatenated. Early fusion lets every hidden unit see every input. Late fusion splits the units into groups and lets each group see only its own inputs — which is the same layer with the off-diagonal blocks zeroed.

Same inputs, same width, same depth, same head. The only difference is which weights are allowed to be non-zero.

Show the code

I asked a sweep how many dimensions I needed, and it refused to answer

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

When I trained word2vec from scratch, one of the settings I had to pick was how many numbers each word vector gets. 100? 300? The papers use 300 and so does nearly everyone, but I wanted to know what it was buying.

So I did what seemed obvious: put embedding size into the hyperparameter sweep along with everything else. 48 runs, twelve hours, Bayesian search over architecture, embedding size, window, negatives, subsampling and learning rate, ranked on a composite of three word-similarity benchmarks.

It found a good configuration. It could not answer my question, and the reason is worth more than the answer was.

Show the code

Where the inputs meet, and when it stops mattering

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

A follow-on from the Hacker News post, going properly into the architecture question rather than mentioning it.

The setup: a model whose inputs come in groups — a title, a timestamp, a domain, an author. Early fusion concatenates them and runs one network over the lot. Late fusion gives each group its own network, reduces each to a score, and adds the scores. The literature [1] treats these as two designs with different strengths.

Every number below is recomputed from the raw per-seed measurements when this page builds, including the significance tests. Nothing here is a figure I saved.

Show the code

The cold-start tax on scaling to zero

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

I’m rebuilding this site as the front door to nine machine learning demos. Each one is its own container image, each one scales to zero, and each one therefore pays a cold start when it wakes up. On a multi-gigabyte image that is ten to thirty seconds of somebody staring at a spinner.

The instinct is to keep everything warm. That instinct is expensive, and mostly wrong. Before paying for it, it’s worth asking a narrower question: what fraction of requests actually hit a cold container?

The model

Take one endpoint with a single warm worker and an idle timeout — the platform keeps the container alive for after the last request, then reclaims it. Assume requests arrive as a Poisson process with rate .

A request finds the container cold exactly when the gap since the previous request exceeded the idle timeout. For Poisson arrivals those gaps are exponential, so:

That’s the whole model. It is crude — it ignores concurrency, platform-side reclamation before the timeout, and the fact that traffic to a portfolio site is nothing like Poisson — but it gets the shape right, and the shape is what decides the architecture.

What it looks like

The traffic axis is logarithmic. A portfolio site lives at the left-hand end of it.

Show the code

Fine-tuning on one example, and knowing when to stop

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

Week 5: speech. The project was fine-tuning Whisper [1] on exactly one clip — a recording where the base model mishears my name — and watching a handful of gradient steps correct it.

One example rather than a dataset, on purpose. Fine-tuning is normally reported as a number moving on a held-out set, which is honest and tells you nothing about what actually happened. With one example you can watch the mechanism.

Two things become visible that way. The first is that it works, quickly. The second is what it costs, which is the part I hadn’t appreciated.

First, how audio becomes something a transformer can read

A transformer needs a sequence of vectors. A waveform is a very long list of amplitudes — 16,000 numbers a second — with the useful structure spread across frequencies rather than sitting in the samples.

The standard answer is a log-mel spectrogram, and it’s three steps:

Chop the waveform into short overlapping frames and take the Fourier transform of each. Now you have how much energy sits at each frequency, over time.
Squash the frequency axis onto the mel scale, which spaces bands the way hearing does — fine detail low down, coarser high up. 201 frequency bins become 80 mel bands.
Take the log, because loudness is perceived multiplicatively.

What comes out is a picture: 80 rows by however many frames. That’s what the model sees. Here it is on the actual clip from the project, computed from scratch.

Show the code

An image is not a sequence, so you make it one

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

Week 3: take the transformer, which was designed for text, and point it at images. No convolutions.

The awkward part is that a transformer eats a sequence of tokens, and an image isn’t one. It’s a grid of pixels with no natural order and far too many of them — 28×28 is 784 pixels, and attention costs grow with the square of the sequence length. On a real photograph you’d be computing attention over hundreds of thousands of positions.

The Vision Transformer paper [1] solves this in one move, in the first layer, and everything after it is the ordinary transformer with nothing changed: cut the image into fixed-size squares and treat each square as a word.

Show the code

The loss went to zero and the model had learned nothing

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

Week 2: build the thing behind semantic search. Given a query, find the passage in a corpus that answers it.

The obvious approach is to take the query and a candidate passage, feed both into one model, and let it score the pair. That works well and is completely impractical, because you have to run it once for every passage in the corpus, for every query.

The dual encoder gets around it by refusing to let the query and the document meet. Two separate towers, one for each, both producing a vector in the same space. A query is relevant to a passage if their vectors are close.

That constraint is the whole point: because the document tower never sees the query, you can run it before any query exists. Embed the entire corpus once, offline, put the vectors in an index, and at query time you embed one short string and do a nearest-neighbour lookup.

Show the code

What the KL penalty is actually holding back

Rosh Beed — Fri, 18 Sep 2026 00:00:00 GMT

Week 6, the last one: reinforcement learning from human feedback. The loop that turned GPT-3 into InstructGPT [1], built from scratch — generation, KL penalty, reward, advantage estimation, and the clipped policy and value updates, one small module each.

The reason RL shows up here at all is that you can’t write down a loss for “a good summary”. What you can do is show people two summaries and ask which they prefer, train a model to predict those preferences, and then optimise the language model against that.

Which introduces the problem the whole method is organised around: you are now optimising a learned approximation of what you want. Push hard enough on any learned reward and you stop finding good outputs and start finding its mistakes.

A language, and a reward model with a weakness

To watch that happen I need something that has grammar, and a reward model that is slightly wrong about what’s good.

The language: eight tokens, where each one usually follows the one before it — token 4 tends to be followed by 5, then 6, and so on. A small bigram model trained on samples of it is the reference policy, standing in for the supervised model you start RLHF from.

The reward model: it likes token 3. That’s it. Think of it as a preference model that has correctly noticed people enjoy a particular thing, and has no opinion about anything else — which is roughly how real reward models fail.

Show the code