Fine-tuning on one example, and knowing when to stop

Week 5 was fine-tuning Whisper on a single clip, because one example makes visible what a held-out score hides. It also makes visible what you pay for it.

Week 5: speech. The project was fine-tuning Whisper [1] on exactly one clip — a recording where the base model mishears my name — and watching a handful of gradient steps correct it.

One example rather than a dataset, on purpose. Fine-tuning is normally reported as a number moving on a held-out set, which is honest and tells you nothing about what actually happened. With one example you can watch the mechanism.

Two things become visible that way. The first is that it works, quickly. The second is what it costs, which is the part I hadn’t appreciated.

First, how audio becomes something a transformer can read

A transformer needs a sequence of vectors. A waveform is a very long list of amplitudes — 16,000 numbers a second — with the useful structure spread across frequencies rather than sitting in the samples.

The standard answer is a log-mel spectrogram, and it’s three steps:

1. Chop the waveform into short overlapping frames and take the Fourier transform of each. Now you have how much energy sits at each frequency, over time.
2. Squash the frequency axis onto the mel scale, which spaces bands the way hearing does — fine detail low down, coarser high up. 201 frequency bins become 80 mel bands.
3. Take the log, because loudness is perceived multiplicatively.

What comes out is a picture: 80 rows by however many frames. That’s what the model sees. Here it is on the actual clip from the project, computed from scratch.

import sys

sys.path.insert(0, "..")
import numpy as np
from huggingface_hub import hf_hub_download

REVISION = "f705fed08827ff6c36e3b5329495c943a5e544e8"
clip = np.load(hf_hub_download("roshbeed/ai-residency-blog-data", "audio/clip-16k.npz",
                               repo_type="dataset", revision=REVISION))
waveform, rate = clip["waveform"], int(clip["sample_rate"])

N_FFT, HOP, N_MELS = 400, 160, 80


def stft(x, n_fft, hop):
    window = np.hanning(n_fft + 1)[:-1]
    frames = 1 + (len(x) - n_fft) // hop
    return np.stack([np.fft.rfft(x[i * hop:i * hop + n_fft] * window)
                     for i in range(frames)], axis=1)


def mel_filterbank(rate, n_fft, n_mels):
    """Triangular filters, evenly spaced on the mel scale."""
    to_mel = lambda f: 2595 * np.log10(1 + f / 700)
    to_hz = lambda m: 700 * (10 ** (m / 2595) - 1)

    edges = to_hz(np.linspace(to_mel(0), to_mel(rate / 2), n_mels + 2))
    bins = np.floor((n_fft + 1) * edges / rate).astype(int)

    bank = np.zeros((n_mels, n_fft // 2 + 1))
    for m in range(1, n_mels + 1):
        left, centre, right = bins[m - 1], bins[m], bins[m + 1]
        if centre > left:
            bank[m - 1, left:centre] = (np.arange(left, centre) - left) / (centre - left)
        if right > centre:
            bank[m - 1, centre:right] = (right - np.arange(centre, right)) / (right - centre)
    return bank


power = np.abs(stft(waveform, N_FFT, HOP)) ** 2
log_mel = np.log10(np.maximum(mel_filterbank(rate, N_FFT, N_MELS) @ power, 1e-10))

print(f"{len(waveform):,} samples at {rate} Hz = {len(waveform) / rate:.2f} seconds")
print(f"becomes a {log_mel.shape[0]} x {log_mel.shape[1]} picture")

16,982 samples at 16000 Hz = 1.06 seconds
becomes a 80 x 104 picture

import matplotlib.pyplot as plt
from _style import COLOURS, MUTED, style_axes

fig, (top, bottom) = plt.subplots(2, 1, figsize=(7.4, 4.4),
                                  gridspec_kw={"height_ratios": [1, 2]})

top.plot(np.arange(len(waveform)) / rate, waveform, color=COLOURS[0], linewidth=0.5)
top.set_xlim(0, len(waveform) / rate)
style_axes(top, ylabel="amplitude", grid=None)
top.set_xticks([])

bottom.imshow(log_mel, aspect="auto", origin="lower", cmap="magma",
              extent=(0, len(waveform) / rate, 0, N_MELS))
style_axes(bottom, "Seconds", "Mel band", grid=None)

fig.tight_layout()

Figure 1: One second of speech, before and after. The model never sees the waveform on top; it reads the picture underneath.

A speech model small enough to watch

Whisper is an encoder–decoder transformer over exactly that picture — the same shape as the multi-digit reader from week 3, with a spectrogram in place of an image.

I can’t run Whisper while this page builds, so here’s the same architecture at a size that trains in seconds, on a language I can synthesise: words of three letters, where each letter is a tone at its own frequency. Speaking a word means playing its tones in order; transcribing it means reading them back.

import torch
import torch.nn as nn

RATE, TONE_SECONDS, NOISE = 8000, 0.08, 0.15
ALPHABET = "abcdefgh"
TONES = {c: 300 * 1.28 ** i for i, c in enumerate(ALPHABET)}

rng = np.random.default_rng(0)
WORDS = sorted({"".join(rng.choice(list(ALPHABET), 3)) for _ in range(60)})[:40]


def speak(word, seed, shift=1.0):
    """Play each letter's tone in turn, with a little jitter and noise."""
    g = np.random.default_rng(seed)
    parts = []
    for c in word:
        t = np.arange(int(RATE * TONE_SECONDS)) / RATE
        f = TONES[c] * shift * (1 + 0.02 * g.normal())
        parts.append((np.sin(2 * np.pi * f * t) + 0.4 * np.sin(4 * np.pi * f * t))
                     * np.hanning(len(t)))
    x = np.concatenate(parts)
    return (x + NOISE * g.normal(size=len(x))).astype(np.float32)


SMALL_FFT, SMALL_HOP, SMALL_MELS = 256, 64, 32
BANK = mel_filterbank(RATE, SMALL_FFT, SMALL_MELS)


def to_picture(x):
    power = np.abs(stft(x, SMALL_FFT, SMALL_HOP)) ** 2
    return np.log10(np.maximum(BANK @ power, 1e-10)).T.astype(np.float32)


def build(repeats, seed0):
    pictures, labels = [], []
    for index, word in enumerate(WORDS):
        for k in range(repeats):
            pictures.append(to_picture(speak(word, seed0 + index * 1000 + k)))
            labels.append(index)
    return torch.from_numpy(np.stack(pictures)), torch.tensor(labels)


X, Y = build(12, 0)
X_test, Y_test = build(4, 500_000)
print(f"{len(WORDS)} words, {len(X)} training clips of shape {tuple(X.shape[1:])}")

40 words, 480 training clips of shape (27, 32)

LETTERS = sorted(set("".join(WORDS)))
START, END = len(LETTERS), len(LETTERS) + 1
VOCAB = len(LETTERS) + 2
DIM, FRAMES = 64, X.shape[1]

TARGET = torch.tensor([[START] + [LETTERS.index(c) for c in w] + [END] for w in WORDS])


class Transcriber(nn.Module):
    """An encoder over the spectrogram, a decoder that spells the word."""

    def __init__(self):
        super().__init__()
        self.input = nn.Linear(SMALL_MELS, DIM)
        self.audio_positions = nn.Parameter(torch.randn(1, FRAMES, DIM) * 0.02)
        self.encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(DIM, 4, 4 * DIM, batch_first=True,
                                       norm_first=True, dropout=0.0), 2)

        self.embed = nn.Embedding(VOCAB, DIM)
        self.text_positions = nn.Parameter(torch.randn(1, 5, DIM) * 0.02)
        self.decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(DIM, 4, 4 * DIM, batch_first=True,
                                       norm_first=True, dropout=0.0), 2)
        self.out = nn.Linear(DIM, VOCAB)

    def forward(self, picture, tokens):
        memory = self.encoder(self.input(picture) + self.audio_positions)
        h = self.embed(tokens) + self.text_positions[:, :tokens.shape[1]]
        mask = nn.Transformer.generate_square_subsequent_mask(tokens.shape[1])
        return self.out(self.decoder(h, memory, tgt_mask=mask))


torch.manual_seed(0)
model = Transcriber()
optimiser = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
generator = torch.Generator().manual_seed(0)


@torch.no_grad()
def transcribe(pictures):
    tokens = torch.full((len(pictures), 1), START)
    for _ in range(3):
        tokens = torch.cat([tokens, model(pictures, tokens)[:, -1].argmax(-1, keepdim=True)], 1)
    return tokens[:, 1:]


def spell(ids):
    return "".join(LETTERS[i] for i in ids if i < len(LETTERS))


def accuracy():
    return (transcribe(X_test) == TARGET[Y_test][:, 1:4]).all(1).float().mean().item()

from _arch import diagram

diagram(Transcriber(), input_shape=((1, FRAMES, SMALL_MELS), (1, 4)), style="flow",
        input_dtype=(torch.float32, torch.long))

Figure 2: Whisper’s shape at 1/1000th the size: a spectrogram into the encoder, the text so far into the decoder, one letter out.

for epoch in range(20):
    perm = torch.randperm(len(X), generator=generator)
    for i in range(0, len(perm) - 64, 64):
        b = perm[i:i + 64]
        target = TARGET[Y[b]]
        loss = nn.functional.cross_entropy(model(X[b], target[:, :-1]).reshape(-1, VOCAB),
                                           target[:, 1:].reshape(-1))
        optimiser.zero_grad()
        loss.backward()
        optimiser.step()

print(f"transcribes {accuracy():.1%} of held-out clips exactly right")

transcribes 99.4% of held-out clips exactly right

Now an accent it has never heard

The base model has only ever heard these tones at their standard frequencies. Say a word with everything shifted up 10% — a different voice, the same word — and it mishears.

WORD, SHIFT = "acd", 1.10
one_clip = torch.from_numpy(to_picture(speak(WORD, 999, SHIFT))).unsqueeze(0)
one_target = TARGET[WORDS.index(WORD)].unsqueeze(0)

print(f"the word is {WORD!r}")
print(f"the model hears {spell(transcribe(one_clip)[0].tolist())!r}")
print(f"and it is otherwise fine: {accuracy():.1%} on the clean test clips")

the word is 'acd'
the model hears 'adc'
and it is otherwise fine: 99.4% on the clean test clips

This is the situation the project was in: a model that works, and one specific thing it gets wrong. So fine-tune it on that one clip and nothing else, and watch both numbers at once.

fine_tune = torch.optim.AdamW(model.parameters(), lr=1e-4)

print(f"{'step':>5} {'loss':>10} {'hears':>8} {'clean accuracy':>16}")
print(f"{0:>5} {'-':>10} {spell(transcribe(one_clip)[0].tolist()):>8} {accuracy():>15.1%}")

steps, losses, clean = [], [], []
for step in range(1, 13):
    loss = nn.functional.cross_entropy(
        model(one_clip, one_target[:, :-1]).reshape(-1, VOCAB), one_target[:, 1:].reshape(-1))
    fine_tune.zero_grad()
    loss.backward()
    fine_tune.step()

    steps.append(step)
    losses.append(loss.item())
    clean.append(accuracy())
    if step % 2 == 0:
        print(f"{step:>5} {loss.item():>10.5f} {spell(transcribe(one_clip)[0].tolist()):>8} "
              f"{clean[-1]:>15.1%}")

 step       loss    hears   clean accuracy
    0          -      adc           99.4%
    2    0.25263      acd           99.4%
    4    0.01363      acd           97.5%
    6    0.00430      acd           96.9%
    8    0.01027      acd           94.4%
   10    0.01837      acd           93.1%
   12    0.01824      acd           92.5%

from _style import figure

fig, ax = figure(height=4.0)
ax.plot(steps, losses, color=COLOURS[1], linewidth=2)
style_axes(ax, "Fine-tuning step on the single clip", "Loss on that clip")
ax.annotate("loss on the one clip", xy=(steps[-1], losses[-1]), xytext=(-6, 14),
            textcoords="offset points", ha="right", fontsize=9, color=COLOURS[1])

right = ax.twinx()
right.plot(steps, clean, color=COLOURS[0], linewidth=2)
right.set_ylabel("Accuracy on the other words", color=MUTED, fontsize=9)
right.tick_params(colors=MUTED, labelsize=9, length=0)
right.spines["top"].set_visible(False)
right.annotate("everything else", xy=(steps[-1], clean[-1]), xytext=(-6, -16),
               textcoords="offset points", ha="right", fontsize=9, color=COLOURS[0])

fig.tight_layout()

Figure 3: The fix lands almost immediately. Everything after that is the model memorising one clip at the expense of the other forty words.

What that shows

The fix is cheap. Two gradient steps and the model hears the word correctly. That’s the appealing part, and it’s real — on the actual project a handful of steps on one clip corrected a name the base model consistently got wrong.

The loss goes to nearly zero, and that is not good news. One example, a loss of 0.004 — the model has memorised a single clip. Memorisation is precisely what you’re asking for here, and it’s also the thing that goes wrong when you scale this up without noticing.

You pay for it somewhere you weren’t looking. Accuracy on the other 39 words drops steadily from step 4 onward, and by step 12 it has given up several points to keep improving on a clip it already got right. Nothing in the fine-tuning loop reports that. You have to go and measure it.

So the interesting question isn’t whether fine-tuning works. It’s when to stop, and the only way to answer it is to keep evaluating the thing you’re not training on.

One Whisper-specific trap

Worth writing down because it cost me time and produced no error at all.

Whisper’s tokenizer starts a transcript with a <|startoftranscript|> token. It is tempting to pass the tokenizer’s output straight through as the training labels. Don’t: the model prepends decoder_start_token_id itself when it shifts the labels right, and that token is start-of-transcript. Pass the tokenizer’s output unchanged and every position is off by one.

Nothing raises. The loss goes down. The model is simply learning a different task than the one you meant.

The full project is on GitHub.

---

[1] Radford et al. Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023.

Built:      2026-09-18 02:23:57 UTC
Python:     3.13.15
matplotlib: 3.11.2
numpy:      2.5.3
torch:      2.14.0+cpu