The emotion is a token, not a head

Whisper’s decoder already emits special tokens for language and task. Week 5’s second project asked whether emotion could just be another one — and what that choice costs when the decoder is small.

The other half of week 5: recognise the emotion in a clip of speech.

The obvious way to build this is a classifier. Take the encoder you already have, mean-pool its output, put a Linear(dim, n_emotions) on top, train with cross-entropy. It works, it’s three lines, and it’s what most people reach for.

I built it the other way, because of something about how Whisper already works.

Whisper already talks about itself in tokens

Whisper’s decoder doesn’t only emit words. Its output starts with control tokens drawn from the same vocabulary as everything else — <|en|> for the language, <|transcribe|> or <|translate|> for the task, timestamps if you ask for them.

They aren’t a separate mechanism. They are ordinary vocabulary entries, predicted by the same softmax, trained by the same cross-entropy, sampled the same way.

So the question is whether emotion is a different kind of thing from language and task, or the same kind of thing. And it’s hard to argue it’s different. Adding <happy>, <sad> and the rest to the vocabulary lets the model say what it heard through machinery it already has.

import sys

sys.path.insert(0, "..")
import matplotlib.pyplot as plt
from _style import COLOURS, MUTED

fig, axes = plt.subplots(1, 2, figsize=(8.4, 3.4))


def box(ax, x, y, w, h, label, colour, text="white", size=9):
    ax.add_patch(plt.Rectangle((x, y), w, h, facecolor=colour, edgecolor="none"))
    ax.text(x + w / 2, y + h / 2, label, ha="center", va="center",
            color=text, fontsize=size)


def arrow(ax, x1, y1, x2, y2):
    ax.annotate("", xy=(x2, y2), xytext=(x1, y1),
                arrowprops=dict(arrowstyle="-|>", color="#aeb6bf", linewidth=1.3))


for ax, title in zip(axes, ("a classifier head", "a token in the vocabulary")):
    box(ax, 0.05, 0.42, 0.22, 0.18, "encoder", COLOURS[0])
    ax.set_title(title, fontsize=10, color=MUTED, pad=10)
    ax.set_xlim(0, 1); ax.set_ylim(0, 1); ax.axis("off")

# left: two outputs, two losses
arrow(axes[0], 0.27, 0.56, 0.40, 0.72)
arrow(axes[0], 0.27, 0.46, 0.40, 0.30)
box(axes[0], 0.40, 0.64, 0.22, 0.16, "decoder", COLOURS[2])
box(axes[0], 0.40, 0.22, 0.22, 0.16, "Linear", COLOURS[3])
axes[0].text(0.66, 0.72, "a c d", fontsize=10, color=MUTED, va="center")
axes[0].text(0.66, 0.30, "shaky", fontsize=10, color=MUTED, va="center")
axes[0].text(0.5, 0.06, "two losses", fontsize=9, color=MUTED, ha="center")

# right: one output, one loss
arrow(axes[1], 0.27, 0.51, 0.40, 0.51)
box(axes[1], 0.40, 0.43, 0.22, 0.16, "decoder", COLOURS[2])
arrow(axes[1], 0.62, 0.51, 0.70, 0.51)
axes[1].text(0.72, 0.51, "<shaky>  a c d", fontsize=10, color=MUTED, va="center")
axes[1].text(0.5, 0.06, "one loss", fontsize=9, color=MUTED, ha="center")

fig.tight_layout()

Figure 1: The same encoder, two ways to get an emotion out of it. On the left the emotion leaves through a separate head trained by its own loss; on the right it is the first thing the decoder says.

The appeal isn’t elegance for its own sake. Three concrete things follow:

Adding a label is a vocabulary entry, not an architecture change. A new emotion is one more row in an embedding table. With a head it’s a new output dimension, a reshaped weight matrix, and a checkpoint that no longer loads.

One loss instead of two. No weighting term to tune between a classification loss and a transcription loss.

The emotion conditions what comes after it. It’s emitted first, so every subsequent token is generated with it in context. A head produces its answer off to the side, where the transcription can’t see it.

That last one is the real argument, and it’s also the one that costs something.

Both, measured

Same synthetic speech as the fine-tuning post: three-letter words where each letter is a tone. Emotion is an acoustic property on top — bright raises the pitch, shaky adds a tremolo, calm is neither.

Same encoder, same decoder size, same data. The only difference is where the emotion comes out.

import numpy as np
import torch
import torch.nn as nn

RATE, TONE_SECONDS, NOISE = 8000, 0.08, 0.15
ALPHABET = "abcdefgh"
TONES = {c: 300 * 1.28 ** i for i, c in enumerate(ALPHABET)}
EMOTIONS = ["calm", "bright", "shaky"]

rng = np.random.default_rng(0)
WORDS = sorted({"".join(rng.choice(list(ALPHABET), 3)) for _ in range(60)})[:40]


def speak(word, seed, emotion):
    g = np.random.default_rng(seed)
    parts = []
    for c in word:
        t = np.arange(int(RATE * TONE_SECONDS)) / RATE
        f = TONES[c] * (1 + 0.02 * g.normal())
        if emotion == 1:                       # bright: everything a bit higher
            f *= 1.18
        tone = np.sin(2 * np.pi * f * t) + 0.4 * np.sin(4 * np.pi * f * t)
        if emotion == 2:                       # shaky: amplitude wobbles at 22 Hz
            tone = tone * (1 + 0.6 * np.sin(2 * np.pi * 22 * t))
        parts.append(tone * np.hanning(len(t)))
    x = np.concatenate(parts)
    return (x + NOISE * g.normal(size=len(x))).astype(np.float32)

N_FFT, HOP, N_MELS = 256, 64, 32


def stft(x, n_fft, hop):
    window = np.hanning(n_fft + 1)[:-1]
    frames = 1 + (len(x) - n_fft) // hop
    return np.stack([np.fft.rfft(x[i * hop:i * hop + n_fft] * window)
                     for i in range(frames)], axis=1)


def mel_filterbank(rate, n_fft, n_mels):
    to_mel = lambda f: 2595 * np.log10(1 + f / 700)
    to_hz = lambda m: 700 * (10 ** (m / 2595) - 1)
    edges = to_hz(np.linspace(to_mel(0), to_mel(rate / 2), n_mels + 2))
    bins = np.floor((n_fft + 1) * edges / rate).astype(int)
    bank = np.zeros((n_mels, n_fft // 2 + 1))
    for m in range(1, n_mels + 1):
        left, centre, right = bins[m - 1], bins[m], bins[m + 1]
        if centre > left:
            bank[m - 1, left:centre] = (np.arange(left, centre) - left) / (centre - left)
        if right > centre:
            bank[m - 1, centre:right] = (right - np.arange(centre, right)) / (right - centre)
    return bank


BANK = mel_filterbank(RATE, N_FFT, N_MELS)


def to_picture(x):
    return np.log10(np.maximum(BANK @ np.abs(stft(x, N_FFT, HOP)) ** 2, 1e-10)).T.astype(np.float32)


def build(repeats, seed0):
    pictures, words, emotions = [], [], []
    for index, word in enumerate(WORDS):
        for k in range(repeats):
            emotion = (index * 7 + k) % 3
            pictures.append(to_picture(speak(word, seed0 + index * 1000 + k, emotion)))
            words.append(index)
            emotions.append(emotion)
    return torch.from_numpy(np.stack(pictures)), torch.tensor(words), torch.tensor(emotions)


X, W, E = build(12, 0)
X_test, W_test, E_test = build(4, 500_000)
print(f"{len(X)} clips, {len(WORDS)} words x {len(EMOTIONS)} emotions")
print(f"guessing the emotion: {1 / len(EMOTIONS):.3f}")

480 clips, 40 words x 3 emotions
guessing the emotion: 0.333

LETTERS = sorted(set("".join(WORDS)))
EMOTION_TOKENS = [len(LETTERS) + i for i in range(3)]   # the extra vocabulary entries
START, END = len(LETTERS) + 3, len(LETTERS) + 4
VOCAB = len(LETTERS) + 5
DIM, FRAMES = 64, X.shape[1]

# <start> <emotion> l l l <end> — the emotion slot is filled per clip
TARGET = torch.stack([torch.tensor([START, 0] + [LETTERS.index(c) for c in w] + [END])
                      for w in WORDS])


class Model(nn.Module):
    """One encoder, one decoder, and a classifier head that only the head
    variant ever trains."""

    def __init__(self):
        super().__init__()
        self.input = nn.Linear(N_MELS, DIM)
        self.audio_positions = nn.Parameter(torch.randn(1, FRAMES, DIM) * 0.02)
        self.encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(DIM, 4, 4 * DIM, batch_first=True,
                                       norm_first=True, dropout=0.0), 2)
        self.embed = nn.Embedding(VOCAB, DIM)
        self.text_positions = nn.Parameter(torch.randn(1, 6, DIM) * 0.02)
        self.decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(DIM, 4, 4 * DIM, batch_first=True,
                                       norm_first=True, dropout=0.0), 2)
        self.out = nn.Linear(DIM, VOCAB)
        self.classifier = nn.Linear(DIM, 3)

    def forward(self, picture, tokens):
        memory = self.encoder(self.input(picture) + self.audio_positions)
        h = self.embed(tokens) + self.text_positions[:, :tokens.shape[1]]
        mask = nn.Transformer.generate_square_subsequent_mask(tokens.shape[1])
        return self.out(self.decoder(h, memory, tgt_mask=mask)), self.classifier(memory.mean(1))

from _arch import diagram

diagram(Model(), input_shape=((1, X.shape[1], N_MELS), (1, 5)), style="flow",
        input_dtype=(torch.float32, torch.long))

Figure 2: The shared model. Both designs are this; only the loss differs, and whether the emotion leaves through the decoder or through the small head hanging off the encoder.

def train(design, epochs=45):
    torch.manual_seed(0)
    model = Model()
    optimiser = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
    generator = torch.Generator().manual_seed(0)

    for _ in range(epochs):
        perm = torch.randperm(len(X), generator=generator)
        for i in range(0, len(perm) - 64, 64):
            b = perm[i:i + 64]
            target = TARGET[W[b]].clone()
            target[:, 1] = torch.tensor(EMOTION_TOKENS)[E[b]]

            if design == "token":
                logits, _ = model(X[b], target[:, :-1])
                loss = nn.functional.cross_entropy(logits.reshape(-1, VOCAB),
                                                   target[:, 1:].reshape(-1))
            else:
                text = torch.cat([target[:, :1], target[:, 2:]], dim=1)  # emotion removed
                logits, predicted = model(X[b], text[:, :-1])
                loss = (nn.functional.cross_entropy(logits.reshape(-1, VOCAB),
                                                    text[:, 1:].reshape(-1))
                        + nn.functional.cross_entropy(predicted, E[b]))

            optimiser.zero_grad()
            loss.backward()
            optimiser.step()
    return model


@torch.no_grad()
def evaluate(model, design):
    tokens = torch.full((len(X_test), 1), START)
    steps = 4 if design == "token" else 3
    for _ in range(steps):
        tokens = torch.cat([tokens, model(X_test, tokens)[0][:, -1].argmax(-1, keepdim=True)], 1)

    if design == "token":
        emitted = tokens[:, 1]
        emotion = torch.tensor([EMOTION_TOKENS.index(i) if i in EMOTION_TOKENS else -1
                                for i in emitted.tolist()])
        word = (tokens[:, 2:5] == TARGET[W_test][:, 2:5]).all(1)
    else:
        emotion = model(X_test, tokens[:, :1])[1].argmax(1)
        word = (tokens[:, 1:4] == TARGET[W_test][:, 2:5]).all(1)

    return (emotion == E_test).float().mean().item(), word.float().mean().item()


print(f"{'design':>26} {'emotion':>9} {'word':>9}")
results = {}
for design, name in (("head", "a classifier head"), ("token", "a token in the vocabulary")):
    results[design] = evaluate(train(design), design)
    print(f"{name:>26} {results[design][0]:>9.3f} {results[design][1]:>9.3f}")

                    design   emotion      word

         a classifier head     0.788     0.969

 a token in the vocabulary     0.800     0.944

What that says

On the emotion itself, the two designs are a tie. Which is the answer I’d expect: the information is in the encoder either way, and both designs are reading the same encoder. Where you attach the readout doesn’t change what there is to read.

The token design transcribes worse here, and that’s the real trade. A two-layer decoder now has to produce the emotion and spell the word, and it doesn’t have the capacity to do both as well as a decoder that only spells.

That cost is a small-model effect. Whisper’s decoder is much larger and already emits several control tokens before it writes anything, so one more costs it nothing noticeable. But it would have been easy to run this comparison at one size, see the token design lose on transcription, and conclude the design is worse — when what I’d actually measured is that my decoder was too small.

The architectural argument survives the accuracy result. Adding a seventh emotion to the token design is a row in an embedding table. In the head design it’s a new output dimension and a checkpoint that no longer loads. And only the token design lets the transcription condition on what the model decided about the emotion, because only there is the emotion part of the same sequence.

The full project is on GitHub.

Built:      2026-09-18 02:17:12 UTC
Python:     3.13.15
matplotlib: 3.11.2
numpy:      2.5.3
torch:      2.14.0+cpu