Early vs Late Fusion

Early and late fusion are usually argued as two architectures. They are one architecture with a dial, and most of the difference is about width.

A follow-on from the Hacker News post, going properly into the architecture question rather than mentioning it.

The setup is a model whose inputs come in groups: a title, a timestamp, a domain, an author. Early fusion concatenates them and runs one network over the lot. Late fusion gives each group its own network, reduces each to a score, and adds the scores. The literature [1] treats these as two designs with different strengths.

Two bits of vocabulary first, since they appear throughout.

• Spearman is a rank correlation. It asks whether two orderings agree and ignores the actual values.
• A permutation test checks whether a difference is real. Shuffle the labels between the two groups a few thousand times, and see how often chance alone produces a gap this large.

Every number below is recomputed from the raw per-seed measurements when this page builds, including those tests. Nothing here is a figure I saved.

import json
import sys

sys.path.insert(0, "..")
import numpy as np
from _style import COLOURS, MUTED, figure, style_axes

from huggingface_hub import hf_hub_download

REVISION = "f7f04575591e5dfb4996534911c0c55e993f3899"


def results(name):
    """The raw per-seed measurements, pinned to a dataset commit."""
    return json.load(open(hf_hub_download("roshbeed/ai-residency-blog-data",
                                          f"fusion/{name}.json",
                                          repo_type="dataset", revision=REVISION)))


synthetic = results("synthetic-alpha")
counts = results("pairings-counts")
logged = results("pairings-log")
ranked = results("pairings-rank")
replicate = results("pairings-replicate")
robust_128 = results("robustness-128")
robust_512 = results("robustness-512")

print(f"synthetic sweep: {len(synthetic)} interaction strengths x "
      f"{len(next(iter(synthetic.values())))} model variants")
print(f"Hacker News: {len(counts)} pairings of input groups, 10 seeds each,")
print(f"  measured on raw counts, on log1p(counts) and on rank")

synthetic sweep: 12 interaction strengths x 12 model variants
Hacker News: 11 pairings of input groups, 10 seeds each,
  measured on raw counts, on log1p(counts) and on rank

One Architecture, Not Two

Take one hidden layer over all the inputs concatenated. Early fusion lets every hidden unit read every input. Late fusion splits the units into per-group blocks, so each block reads only its own group. It is the same layer with the off-diagonal blocks held at zero.

So early and late are two ends of one dial: which layer do the groups first meet at? Everything before it is block-diagonal, everything from it on is shared. Fusing at layer 1 is early fusion, fusing at the last layer is late fusion, and the question stops being which is better. It becomes: how long can you delay fusion before it costs you?

I swept that across 13 layer shapes at every valid fusion point, ten seeds each, 490 runs. Ten rather than three because late fusion’s three-seed spread was 0.106 Spearman against early fusion’s 0.026, wide enough to swallow the effect being measured.

Delaying always costs something, and almost all of that cost is how wide the isolated layer is rather than the delay itself. At depth three, holding everything else fixed and changing only the width:

shape	first-layer width	fuse at 1	fuse at 2	cost of one private layer
640,512,384	640	0.249	0.254	+0.005
320,256,192	320	0.251	0.242	−0.008
256,256,256	256	0.243	0.191	−0.053
64,64,64	64	0.247	0.186	−0.061

At 640 units a private layer per group is free. At 64 it costs 0.061, more than the entire early-versus-late gap. The reason is the allocation rule. Each group’s share of a layer is proportional to how many columns it brings. The author’s history gets round(w × 3/329) units: six at width 640, and one at width 64.

A narrow block-diagonal layer does not delay fusion so much as strangle four of the five inputs before they reach it.

A Control With a Known Answer

Before trusting any of this on real data, the measurement needs checking. So here is a target whose interaction strength I set myself:

\[y = g + h + \alpha \cdot g h\]

At \(\alpha = 0\) the two groups contribute independently and late fusion is exactly the right model. As \(\alpha\) rises, a model that can only add should fall behind.

This is a real sweep: twelve interaction strengths, three architectures, four widths, ten seeds each.

alphas = sorted(synthetic, key=float)
widths = [64, 128, 256, 512]

fig, ax = figure(height=4.2)
for width, colour in zip(widths, COLOURS):
    gap = [synthetic[a][f"early@{width}"]["median"] - synthetic[a][f"late-snoek@{width}"]["median"]
           for a in alphas]
    ax.plot([float(a) for a in alphas], gap, color=colour, linewidth=2, marker="o", markersize=3)
    ax.annotate(f"{width} units", xy=(float(alphas[-1]), gap[-1]), xytext=(6, 0),
                textcoords="offset points", fontsize=9, color=colour, va="center")

ax.axhline(0, color=MUTED, linewidth=0.8)
ax.set_xlim(0, float(alphas[-1]) * 1.18)
style_axes(ax, "Interaction strength in the target (alpha)", "How far late fusion falls behind")
fig.tight_layout()

Figure 1: Snoek late fusion against early fusion on a target with a known interaction, at four widths. The penalty grows with the interaction and barely moves with width.

The gap at the strongest interaction, at each width:

widest, narrowest = "512", "64"
worst = alphas[-1]
print(f"at alpha={worst}:")
for width in widths:
    gap = (synthetic[worst][f"early@{width}"]["median"]
           - synthetic[worst][f"late-snoek@{width}"]["median"])
    print(f"  {width:>4} units: late fusion is {gap:.3f} behind")

zero_gap = max(abs(synthetic["0.0"][f"early@{w}"]["median"]
                   - synthetic["0.0"][f"late-snoek@{w}"]["median"]) for w in widths)
print(f"\nat alpha=0, the largest gap at any width is {zero_gap:.4f}")

at alpha=8.0:
    64 units: late fusion is 0.118 behind
   128 units: late fusion is 0.112 behind
   256 units: late fusion is 0.114 behind
   512 units: late fusion is 0.108 behind

at alpha=0, the largest gap at any width is 0.0008

Two things to take from that.

The measurement works. When an interaction exists this comparison finds it, and when one doesn’t the two architectures land within 0.001 of each other at every width. So a null result later is a null result rather than a broken harness.

Width barely rescues it. Eight times the hidden units recovers about 9% of the gap, the opposite of what the width sweep showed on real data. The difference matters. When an interaction really exists in the target, an additive model cannot buy its way out with capacity. It’s not underfitting. It’s the wrong shape.

The Real Data

The same comparison on real Hacker News upvote counts, for every pairing of input groups. One model cannot represent an interaction: a tower each, reduced to one score, then summed. The other can.

The control that decides whether this means anything is capacity. A joint model has more parameters, so it can win for reasons that have nothing to do with interaction. So the additive model here gets towers three times wider: still structurally unable to represent an interaction, and now with more parameters than its rival. Anything the joint model wins by is the interaction, not the budget.

def permutation_test(a, b, trials=20_000, seed=0):
    """How often would shuffling the labels produce a difference this large?"""
    a, b = np.asarray(a), np.asarray(b)
    observed = np.median(b) - np.median(a)
    pool = np.concatenate([a, b])
    rng = np.random.default_rng(seed)

    hits = 0
    for _ in range(trials):
        rng.shuffle(pool)
        if abs(np.median(pool[len(a):]) - np.median(pool[:len(a)])) >= abs(observed):
            hits += 1
    return observed, (hits + 1) / (trials + 1)


def interaction(dataset, pairing):
    return permutation_test(dataset[pairing]["additive-matched"], dataset[pairing]["joint"])


rows = sorted(((name, *interaction(counts, name)) for name in counts), key=lambda r: -r[1])

print(f"{'does the first group depend on the second?':<42} {'effect':>9} {'p':>8}")
for name, effect, p in rows:
    print(f"{name:<42} {effect:>+9.4f} {p:>8.3f}{'*' if p < 0.05 else ' '}")

does the first group depend on the second?    effect        p
title x when you post                        +0.0381    0.000*
title x all metadata                         +0.0220    0.001*
title x title shape                          +0.0090    0.007*
title x where it links                       +0.0081    0.139 
title x the author                           +0.0020    0.628 
where x author                               +0.0013    0.307 
author x title shape                         -0.0008    0.721 
when x title shape                           -0.0014    0.619 
where x title shape                          -0.0062    0.065 
when x author                                -0.0067    0.066 
when x where                                 -0.0117    0.023*

Drawn as a chart, sorted by effect:

names = [r[0] for r in rows][::-1]
effects = [r[1] for r in rows][::-1]
significant = [r[2] < 0.05 for r in rows][::-1]

fig, ax = figure(height=4.6)
ax.barh(range(len(names)), effects,
        color=[COLOURS[0] if s else "#c3ccd6" for s in significant], height=0.66)
ax.axvline(0, color=MUTED, linewidth=0.9)
ax.set_yticks(range(len(names)), names, fontsize=9)
style_axes(ax, "Interaction, over a wider additive model", grid="x")
fig.tight_layout()

Figure 2: Each pairing’s interaction against the capacity-matched additive control. Filled bars survive a permutation test at p < 0.05.

The title and the timing is the largest effect in the table by a distance, and it decided the architecture this service serves. The mechanism is the one in the other post. A good title doesn’t add a fixed number of upvotes. It multiplies whatever the timing was going to give you, and a model that scores the two separately and sums them can only add.

Notice how much of the rest of the table is noise. Two-thirds of the pairings are indistinguishable from zero and several point the wrong way. This is one specific pair of inputs interacting, not a general property of the task.

Units Matter

Additivity is not a property of data. It is a property of the units you measure the data in.

So I ran the identical comparison three times. Same models, same seeds, same everything. Once against the raw count, once against log1p(score), once against rank.

targets = [("raw counts", counts), ("log1p(counts)", logged), ("rank", ranked)]

fig, ax = figure(height=4.2)
for x, (label, dataset) in enumerate(targets):
    values = [np.median(dataset[n]["joint"]) - np.median(dataset[n]["additive-matched"])
              for n in dataset]
    ax.scatter([x] * len(values), values, s=36, color=COLOURS[0], alpha=0.65, zorder=3)

highlight = "title x when you post"
ax.plot(range(3), [np.median(d[highlight]["joint"]) - np.median(d[highlight]["additive-matched"])
                   for _, d in targets], color=COLOURS[1], linewidth=2, zorder=4)
ax.annotate(highlight, xy=(0, np.median(counts[highlight]["joint"])
                           - np.median(counts[highlight]["additive-matched"])),
            xytext=(10, 0), textcoords="offset points", fontsize=9,
            color=COLOURS[1], va="center")

ax.axhline(0, color=MUTED, linewidth=0.9)
ax.set_xlim(-0.4, 2.6)
ax.set_xticks(range(3), [t[0] for t in targets])
style_axes(ax, "What the model is asked to predict", "Interaction")
fig.tight_layout()

Figure 3: The same eleven comparisons in three coordinate systems. The interaction is a property of the raw counts and survives neither transformation.

Listing which cells survive a permutation test in each coordinate system makes the difference concrete:

for label, dataset in targets:
    print(f"{label}:")
    for name in dataset:
        effect, p = interaction(dataset, name)
        if p < 0.05:
            print(f"    {name:<26} {effect:>+8.4f}   p={p:.3f}")
    print()

raw counts:
    title x when you post       +0.0381   p=0.000
    title x title shape         +0.0090   p=0.007
    title x all metadata        +0.0220   p=0.001
    when x where                -0.0117   p=0.023

log1p(counts):
    title x the author          -0.0041   p=0.019
    when x author               -0.0077   p=0.001
    author x title shape        +0.0055   p=0.001

rank:
    title x where it links      +0.0035   p=0.013
    title x title shape         +0.0042   p=0.007
    where x title shape         +0.0014   p=0.035
    author x title shape        +0.0083   p=0.001

The big one is gone. title x when you post goes from +0.0381 to roughly zero under both transformations, and the cells that remain significant are different ones, an order of magnitude smaller.

A logarithm turns multiplication into addition, so an additive model fitted on log1p can represent precisely the thing it couldn’t represent on counts. Rank keeps every comparison between posts and discards all the magnitudes, and the interaction goes with them.

So this interaction lives in how large the counts get, not in which post beats which. A service that ranked posts wouldn’t need early fusion at all. This one predicts counts, so it does.

There’s one exception, and it’s the interesting one. author x title shape survives every coordinate system. Not significant on counts, +0.0055 on log1p, +0.0083 on rank.

That one is about ordering rather than magnitude. A known name’s Show HN: really does land differently from a stranger’s. It changes who beats whom, not by how much.

I find this the most useful thing in the project, because it generalises well past fusion. “Is there an interaction in my data” isn’t a well-formed question until you have said what scale you’re measuring on.

It also replicates, which matters given how many times I had to withdraw a reading of this comparison.

print(f"{'pairing':<26} {'first run':>10} {'replication':>12}")
for name in ("title x when you post", "title x all metadata", "title x title shape"):
    first = np.median(counts[name]["joint"]) - np.median(counts[name]["additive-matched"])
    again = np.median(replicate[name]["joint"]) - np.median(replicate[name]["additive-matched"])
    print(f"{name:<26} {first:>+10.4f} {again:>+12.4f}")

pairing                     first run  replication
title x when you post         +0.0381      +0.0334
title x all metadata          +0.0220      +0.0222
title x title shape           +0.0090      +0.0134

Late Fusion’s Advantage

The surveys don’t mainly claim late fusion is more accurate. They claim it’s more robust: when one input goes missing or bad, it can lean on the surviving branch, where early fusion has entangled everything in its first layer.

They are right, and it is measurable. Train all three architectures on clean data, then damage one input only at inference.

damage = ["0.0", "0.25", "0.5", "0.75", "1.0"]

fig, ax = figure(height=3.8)
for data, colour, label in ((robust_128, COLOURS[0], "128 hidden units"),
                            (robust_512, COLOURS[1], "512 hidden units")):
    advantage = [np.median(data[f"missing-title@{d}"]["late-snoek"])
                 - np.median(data[f"missing-title@{d}"]["early"]) for d in damage]
    ax.plot([float(d) for d in damage], advantage, color=colour, linewidth=2, marker="o",
            markersize=4)
    ax.annotate(label, xy=(1.0, advantage[-1]), xytext=(-6, 8), textcoords="offset points",
                ha="right", fontsize=9, color=colour)

ax.axhline(0, color=MUTED, linewidth=0.8)
style_axes(ax, "Share of the title removed at inference", "Late fusion's advantage")
fig.tight_layout()

Figure 4: Late fusion’s advantage when an input is damaged, at two widths. It is real at 128 hidden units and much smaller at 512.

Putting numbers on the two ends of that:

for data, label in ((robust_128, "128 units"), (robust_512, "512 units")):
    clean = (np.median(data["missing-title@0.0"]["late-snoek"])
             - np.median(data["missing-title@0.0"]["early"]))
    broken = (np.median(data["missing-title@1.0"]["late-snoek"])
              - np.median(data["missing-title@1.0"]["early"]))
    print(f"{label}: late fusion pays {-clean:+.3f} when nothing is wrong "
          f"to gain {broken:+.3f} when the title is gone")

128 units: late fusion pays +0.012 when nothing is wrong to gain +0.015 when the title is gone
512 units: late fusion pays +0.010 when nothing is wrong to gain +0.006 when the title is gone

So the insurance is real. It’s also a small-model effect.

At 128 hidden units late fusion gives up 0.012 on clean data and gains 0.015 when the title disappears, so the cover roughly pays for itself. At 512 it gives up about the same and gains 0.006. Early fusion now has enough capacity to cope on its own.

Conclusion

Four separate measurements point the same way: the fusion-point sweep, the width sweep, the pairing table and the robustness probe. It is not the conclusion I expected to write.

The architectural distinction mostly matters when something else is constrained.

• Given enough width, early fusion matches late fusion’s robustness
• Given enough width, late fusion matches early fusion’s accuracy
• The gap is widest where the model is too small, and closes as it grows
• An interaction that exists in one coordinate system may not exist in another

So “early versus late fusion” is not a ranking. It is a question about your data and your budget. Is there an interaction to model, and can you afford the width to model it? Quote a fusion result without the width it was measured at and you have not said anything.

The measurements, and the service they came from, are on GitHub.

---

[1] Snoek, Worring, Smeulders. Early versus Late Fusion in Semantic Video Analysis. ACM Multimedia 2005.