Where the inputs meet, and when it stops mattering

Early fusion versus late fusion is usually argued as a choice between two architectures. It is one architecture with a dial, and once you sweep the dial most of the difference turns out to be about width.

A follow-on from the Hacker News post, going properly into the architecture question rather than mentioning it.

The setup: a model whose inputs come in groups — a title, a timestamp, a domain, an author. Early fusion concatenates them and runs one network over the lot. Late fusion gives each group its own network, reduces each to a score, and adds the scores. The literature [1] treats these as two designs with different strengths.

Every number below is recomputed from the raw per-seed measurements when this page builds, including the significance tests. Nothing here is a figure I saved.

import json
import sys

sys.path.insert(0, "..")
import numpy as np
from _style import COLOURS, MUTED, figure, style_axes

synthetic = json.load(open("synthetic-alpha.json"))
counts = json.load(open("pairings-counts.json"))
logged = json.load(open("pairings-log.json"))
ranked = json.load(open("pairings-rank.json"))
replicate = json.load(open("pairings-replicate.json"))
robust_128 = json.load(open("robustness-128.json"))
robust_512 = json.load(open("robustness-512.json"))

print(f"synthetic sweep: {len(synthetic)} interaction strengths x "
      f"{len(next(iter(synthetic.values())))} model variants")
print(f"Hacker News: {len(counts)} pairings of input groups, 10 seeds each,")
print(f"  measured on raw counts, on log1p(counts) and on rank")

synthetic sweep: 12 interaction strengths x 12 model variants
Hacker News: 11 pairings of input groups, 10 seeds each,
  measured on raw counts, on log1p(counts) and on rank

First: they are not two architectures

Take one hidden layer over all the inputs concatenated. Early fusion lets every hidden unit read every input. Late fusion splits the units into per-group blocks and lets each block read only its own group — the same layer with the off-diagonal blocks held at zero.

So “early” and “late” are the two ends of one dial: which layer do the groups first meet at? Everything before that layer is block-diagonal, everything from it on is shared. Fusing at layer 1 is early fusion. Fusing at the last layer is late fusion. And now you can ask a better question than which is better — how long can you delay fusion before it costs you?

A control where the answer is known

Before trusting any of this on real data, the measurement needs to be checked on a target whose interaction strength I set myself:

\[y = g + h + \alpha \cdot g h\]

At \(\alpha = 0\) the two groups contribute independently and late fusion is exactly the right model. As \(\alpha\) rises, a model that can only add should fall behind.

This is a real sweep — twelve interaction strengths, three architectures, four widths, ten seeds each.

alphas = sorted(synthetic, key=float)
widths = [64, 128, 256, 512]

fig, ax = figure(height=4.2)
for width, colour in zip(widths, COLOURS):
    gap = [synthetic[a][f"early@{width}"]["median"] - synthetic[a][f"late-snoek@{width}"]["median"]
           for a in alphas]
    ax.plot([float(a) for a in alphas], gap, color=colour, linewidth=2, marker="o", markersize=3)
    ax.annotate(f"{width} units", xy=(float(alphas[-1]), gap[-1]), xytext=(6, 0),
                textcoords="offset points", fontsize=9, color=colour, va="center")

ax.axhline(0, color=MUTED, linewidth=0.8)
ax.set_xlim(0, float(alphas[-1]) * 1.18)
style_axes(ax, "Interaction strength in the target (alpha)", "How far late fusion falls behind")
fig.tight_layout()

Figure 1: Snoek late fusion against early fusion on a target with a known interaction, at four widths. The penalty grows with the interaction and barely moves with width.

widest, narrowest = "512", "64"
worst = alphas[-1]
print(f"at alpha={worst}:")
for width in widths:
    gap = (synthetic[worst][f"early@{width}"]["median"]
           - synthetic[worst][f"late-snoek@{width}"]["median"])
    print(f"  {width:>4} units: late fusion is {gap:.3f} behind")

zero_gap = max(abs(synthetic["0.0"][f"early@{w}"]["median"]
                   - synthetic["0.0"][f"late-snoek@{w}"]["median"]) for w in widths)
print(f"\nat alpha=0, the largest gap at any width is {zero_gap:.4f}")

at alpha=8.0:
    64 units: late fusion is 0.118 behind
   128 units: late fusion is 0.112 behind
   256 units: late fusion is 0.114 behind
   512 units: late fusion is 0.108 behind

at alpha=0, the largest gap at any width is 0.0008

Two things to take from that.

The measurement works. When an interaction exists, this comparison finds it, and when one doesn’t, the two architectures are within 0.001 of each other at every width. So a null result later is a null result, not a broken harness.

And width barely rescues it. Eight times the hidden units recovers about 9% of the gap. That is worth stating plainly because it is the opposite of what happens on the real data below: when an interaction genuinely exists in the target, an additive model cannot buy its way out with capacity. It is not underfitting. It is the wrong shape.

Now the real data

The same comparison on real Hacker News upvote counts, for every pairing of input groups. A model that cannot represent an interaction — a tower each, reduced to one score, then summed — against one that can.

The control that decides whether this means anything is capacity. A joint model has more parameters, so it can win for reasons that have nothing to do with interaction. So the additive model here gets towers three times wider: still structurally unable to represent an interaction, and now with more parameters than its rival. Anything the joint model wins by is the interaction, not the budget.

def permutation_test(a, b, trials=20_000, seed=0):
    """How often would shuffling the labels produce a difference this large?"""
    a, b = np.asarray(a), np.asarray(b)
    observed = np.median(b) - np.median(a)
    pool = np.concatenate([a, b])
    rng = np.random.default_rng(seed)

    hits = 0
    for _ in range(trials):
        rng.shuffle(pool)
        if abs(np.median(pool[len(a):]) - np.median(pool[:len(a)])) >= abs(observed):
            hits += 1
    return observed, (hits + 1) / (trials + 1)


def interaction(dataset, pairing):
    return permutation_test(dataset[pairing]["additive-matched"], dataset[pairing]["joint"])


rows = sorted(((name, *interaction(counts, name)) for name in counts), key=lambda r: -r[1])

print(f"{'does the first group depend on the second?':<42} {'effect':>9} {'p':>8}")
for name, effect, p in rows:
    print(f"{name:<42} {effect:>+9.4f} {p:>8.3f}{'*' if p < 0.05 else ' '}")

does the first group depend on the second?    effect        p
title x when you post                        +0.0381    0.000*
title x all metadata                         +0.0220    0.001*
title x title shape                          +0.0090    0.007*
title x where it links                       +0.0081    0.139 
title x the author                           +0.0020    0.628 
where x author                               +0.0013    0.307 
author x title shape                         -0.0008    0.721 
when x title shape                           -0.0014    0.619 
where x title shape                          -0.0062    0.065 
when x author                                -0.0067    0.066 
when x where                                 -0.0117    0.023*

names = [r[0] for r in rows][::-1]
effects = [r[1] for r in rows][::-1]
significant = [r[2] < 0.05 for r in rows][::-1]

fig, ax = figure(height=4.6)
ax.barh(range(len(names)), effects,
        color=[COLOURS[0] if s else "#c3ccd6" for s in significant], height=0.66)
ax.axvline(0, color=MUTED, linewidth=0.9)
ax.set_yticks(range(len(names)), names, fontsize=9)
style_axes(ax, "Interaction, over a wider additive model", grid="x")
fig.tight_layout()

Figure 2: Each pairing’s interaction against the capacity-matched additive control. Filled bars survive a permutation test at p < 0.05.

The title and the timing is the largest effect in the table by a distance, and it is what decided the architecture this service actually serves. The mechanism is the one in the other post: a good title doesn’t add a fixed number of upvotes, it multiplies whatever the timing was going to give you, and a model that scores the two separately and sums can only add.

Notice how much of the rest of the table is noise. Two-thirds of the pairings are indistinguishable from zero and several point the wrong way. This is one specific pair of inputs interacting, not a general property of the task.

The part that nearly fooled me

Additivity is not a property of data. It is a property of the units you measure the data in.

Run the identical comparison — same models, same seeds, same everything — against log1p(score) instead of the raw count, and against rank instead of either.

targets = [("raw counts", counts), ("log1p(counts)", logged), ("rank", ranked)]

fig, ax = figure(height=4.2)
for x, (label, dataset) in enumerate(targets):
    values = [np.median(dataset[n]["joint"]) - np.median(dataset[n]["additive-matched"])
              for n in dataset]
    ax.scatter([x] * len(values), values, s=36, color=COLOURS[0], alpha=0.65, zorder=3)

highlight = "title x when you post"
ax.plot(range(3), [np.median(d[highlight]["joint"]) - np.median(d[highlight]["additive-matched"])
                   for _, d in targets], color=COLOURS[1], linewidth=2, zorder=4)
ax.annotate(highlight, xy=(0, np.median(counts[highlight]["joint"])
                           - np.median(counts[highlight]["additive-matched"])),
            xytext=(10, 0), textcoords="offset points", fontsize=9,
            color=COLOURS[1], va="center")

ax.axhline(0, color=MUTED, linewidth=0.9)
ax.set_xlim(-0.4, 2.6)
ax.set_xticks(range(3), [t[0] for t in targets])
style_axes(ax, "What the model is asked to predict", "Interaction")
fig.tight_layout()

Figure 3: The same eleven comparisons in three coordinate systems. The interaction is a property of the raw counts and survives neither transformation.

for label, dataset in targets:
    print(f"{label}:")
    for name in dataset:
        effect, p = interaction(dataset, name)
        if p < 0.05:
            print(f"    {name:<26} {effect:>+8.4f}   p={p:.3f}")
    print()

raw counts:
    title x when you post       +0.0381   p=0.000
    title x title shape         +0.0090   p=0.007
    title x all metadata        +0.0220   p=0.001
    when x where                -0.0117   p=0.023

log1p(counts):
    title x the author          -0.0041   p=0.019
    when x author               -0.0077   p=0.001
    author x title shape        +0.0055   p=0.001

rank:
    title x where it links      +0.0035   p=0.013
    title x title shape         +0.0042   p=0.007
    where x title shape         +0.0014   p=0.035
    author x title shape        +0.0083   p=0.001

The big one is gone. title x when you post goes from +0.0381 to roughly zero under both transformations, and the cells that remain significant are different ones, an order of magnitude smaller.

A logarithm turns multiplication into addition, so an additive model fitted on log1p can represent precisely the thing it could not represent on counts. Rank keeps every comparison between posts and discards all the magnitudes, and the interaction goes with them.

So this interaction lives in how large the counts get, not in which post beats which. A service that ranked posts would not need early fusion at all. This one predicts counts, so it does.

There is one exception, and it’s the interesting one. author x title shape survives every coordinate system — not significant on counts, +0.0055 on log1p, +0.0083 on rank. That one is about ordering rather than magnitude: a known name’s Show HN: really does land differently from a stranger’s, and it changes who beats whom rather than by how much.

I find this the most useful thing in the project, because it generalises well past fusion. “Is there an interaction in my data” is not a well-formed question until you have said what scale you are measuring on.

It also replicates, which matters given how many times I had to withdraw a reading of this comparison.

print(f"{'pairing':<26} {'first run':>10} {'replication':>12}")
for name in ("title x when you post", "title x all metadata", "title x title shape"):
    first = np.median(counts[name]["joint"]) - np.median(counts[name]["additive-matched"])
    again = np.median(replicate[name]["joint"]) - np.median(replicate[name]["additive-matched"])
    print(f"{name:<26} {first:>+10.4f} {again:>+12.4f}")

pairing                     first run  replication
title x when you post         +0.0381      +0.0334
title x all metadata          +0.0220      +0.0222
title x title shape           +0.0090      +0.0134

Late fusion’s actual advantage

The surveys don’t mainly claim late fusion is more accurate. They claim it’s more robust: when one input goes missing or bad, it can lean on the surviving branch, where early fusion has entangled everything in its first layer.

That is true, and it’s measurable. Train all three architectures on clean data, then damage one input only at inference.

damage = ["0.0", "0.25", "0.5", "0.75", "1.0"]

fig, ax = figure(height=3.8)
for data, colour, label in ((robust_128, COLOURS[0], "128 hidden units"),
                            (robust_512, COLOURS[1], "512 hidden units")):
    advantage = [np.median(data[f"missing-title@{d}"]["late-snoek"])
                 - np.median(data[f"missing-title@{d}"]["early"]) for d in damage]
    ax.plot([float(d) for d in damage], advantage, color=colour, linewidth=2, marker="o",
            markersize=4)
    ax.annotate(label, xy=(1.0, advantage[-1]), xytext=(-6, 8), textcoords="offset points",
                ha="right", fontsize=9, color=colour)

ax.axhline(0, color=MUTED, linewidth=0.8)
style_axes(ax, "Share of the title removed at inference", "Late fusion's advantage")
fig.tight_layout()

Figure 4: Late fusion’s advantage when an input is damaged, at two widths. It is real at 128 hidden units and much smaller at 512.

for data, label in ((robust_128, "128 units"), (robust_512, "512 units")):
    clean = (np.median(data["missing-title@0.0"]["late-snoek"])
             - np.median(data["missing-title@0.0"]["early"]))
    broken = (np.median(data["missing-title@1.0"]["late-snoek"])
              - np.median(data["missing-title@1.0"]["early"]))
    print(f"{label}: late fusion pays {-clean:+.3f} when nothing is wrong "
          f"to gain {broken:+.3f} when the title is gone")

128 units: late fusion pays +0.012 when nothing is wrong to gain +0.015 when the title is gone
512 units: late fusion pays +0.010 when nothing is wrong to gain +0.006 when the title is gone

So the insurance is real, and it is also a small-model effect. At 128 hidden units late fusion gives up a little on clean data to gain a lot when the title disappears. At 512 it gives up about the same and gains far less, because early fusion has enough capacity to learn to cope on its own.

What I’d actually conclude

Four separate measurements — the fusion-point sweep, the width sweep, the pairing table and this robustness probe — all point the same way, and it isn’t the conclusion I expected to write:

The architectural distinction mostly matters when something else is constrained. Given enough width, early fusion matches late fusion’s robustness and late fusion matches early fusion’s accuracy. The gap between them is largest exactly where the model is too small, and it closes as the model grows.

Which means the honest version of “early versus late fusion” is not a ranking. It’s a question about your particular data — is there an interaction to model? — and your particular budget. Quote a fusion result without the width it was measured at and you have not said anything.

The measurements, and the service they came from, are on GitHub.

---

[1] Snoek, Worring, Smeulders. Early versus Late Fusion in Semantic Video Analysis. ACM Multimedia 2005.

Built:      2026-09-18 02:23:47 UTC
Python:     3.13.15
matplotlib: 3.11.2
numpy:      2.5.3
torch:      2.14.0+cpu