Engram.

Retrieval simulated as computation

Language modelling runs two workloads through one mechanism. Compositional reasoning — multi-step inference, arithmetic, code — genuinely needs deep, input-dependent computation. Static recall — named entities, idioms, formulaic collocations — does not. It needs a table.

Transformers only have the first mechanism, so they emulate the second. The canonical demonstration comes from PatchScope: decode the residual stream at the token Wales layer by layer and watch six blocks of attention and FFN assemble a single entity that a lookup would have returned in one step.

The wasted depth is the target. Engram's thesis is that if the model can address that entity directly, the freed layers become available for work that actually requires them — which is why the paper's largest gains land on BBH (+5.0) and ARC-Challenge (+3.7), not on the trivia benchmarks you would expect a memory module to move.

Conditional computation, conditional memory

MoE is conditional computation: a router reads the hidden state and activates a sparse subset of experts. Engram is conditional memory: a hash of the input tokens addresses a sparse subset of embedding rows. Both raise total parameters without raising per-token FLOPs. They differ in the one property that determines where the parameters are allowed to live.

The third row is the load-bearing one. Because an Engram address is a pure function of input_ids, the table can sit off-accelerator and still arrive on time — the property Section 11 turns into a 100B-parameter layer in host memory at a 2.8% throughput cost.

From token IDs to 16 table addresses

Everything upstream of the embedding lookup is integer arithmetic on input_ids — no weights, no gradients, no data dependence on the hidden state. Four stages: canonicalise, window, mix, reduce.

normalizers.Sequence([
  NFKC(),                      # compatibility composition
  NFD(), StripAccents(),        # decompose, drop diacritics
  Lowercase(),
  Replace(Regex(r"[ \t\r\n]+"), " "),   # collapse whitespace runs
  Replace(Regex(r"^ $"), SENTINEL),     # protect a lone space …
  Strip(), Replace(SENTINEL, " "),      # … from Strip()
])
# byte-fallback tokens (decode to U+FFFD) bypass this and key on
# their raw token string, so they stay distinct.

offsets = cumsum([0, M₂,₁, M₂,₂, …, M₃,₇])
rows    = embedding[ z + offsets ]   # [B,L,16,d_head]
e_t     = rows.flatten(-2)         # [B,L,d_mem]

Gating a static prior against live context

The retrieved vector e_t is context-free by construction. It is the same 1280 numbers whenever those three canonical tokens co-occur — correct for “Alexander the Great”, wrong when the n-gram is a hash collision or a polysemous fragment. Engram therefore never adds e_t to the residual stream directly. It scores it first.

The addressing, running

Below is the real arithmetic — the DeepSeek-V3 tokenizer's IDs, the canonical map P, the seed-0 multipliers, and the prime moduli computed by the repository's own prime search. Pick a token to see the 16 addresses its suffix n-grams resolve to.

Verified against the reference implementation. Multipliers come from np.random.default_rng(0 + 10007·ℓ); the mix is computed in signed 64-bit (BigInt here, int64 there) and never overflows because each m_k is bounded by (2⁶³−1)/|V′|/2. The paper places Engram at layers 2 and 15; the shipped demo file uses layer_ids = [1, 15], and the moduli shown here follow the paper.

The U-curve: pure MoE is the wrong allocation

Hold P_tot and P_act fixed — same parameter count, same FLOPs per token — and sweep ρ, moving capacity from routed experts to embedding slots. If memory were merely a cheaper substitute for compute, loss would rise monotonically as ρ falls. It does not.

Iso-parameter, iso-FLOPs, 262B tokens

Engram-27B is MoE-27B with 17 routed experts removed and the freed parameters spent on a 5.7B embedding module — ρ = 74.3%. Identical backbone, identical data in identical order, identical activated parameters. The only difference is where the sparse capacity sits.

Table 1. All four models trained on the same 262B tokens. The Δ column compares the two iso-parameter, iso-FLOPs 26.7B models. Engram-40B keeps the activated budget fixed and only grows the table; it does not dominate on every task, which the authors attribute to under-training — the loss gap was still widening at the end of the run.

Attention, relieved of local work

Attention heads that spend themselves binding adjacent subwords into entities are not available for retrieval across 32k tokens. If Engram absorbs the local dependencies, the prediction is that long-range behaviour improves — even though Engram itself has a receptive field of three tokens.

The evaluation is unusually careful about a confound the authors flag themselves: long-context ability tracks general model quality, so comparing at equal steps conflates architecture with base capability. They therefore include an iso-loss checkpoint — Engram at 46k steps has the same pre-training loss as MoE at 50k — and give all variants the identical YaRN extension run (32,768 tokens, 5,000 steps, 30B tokens).

Table 2. S / MK / MV / MQ — single, multi-key, multi-value and multi-query needle-in-a-haystack. VT — multi-hop variable tracking. CWE / FWE — common and frequent word extraction. QA — long-document question answering.

The 82%-compute row is the sharper result: an Engram checkpoint stopped 9,000 steps early, at visibly worse pre-training loss, already matches the fully-trained MoE on LongPPL and beats it on RULER. Two honest caveats: single-needle accuracy is marginally lower (99.3 vs 100.0 — a saturated metric), and CWE stays broken for both models at 4.5 vs 5.9. Whatever common-word extraction needs, neither architecture has it.

Evidence that Engram buys depth

The claim from §01 — that lookups return sequential depth to the backbone — is testable with interpretability tools rather than benchmarks. Three independent probes agree.

What the gate actually fires on

Because α_t is a single scalar, it can be read straight off the model. Engram operates on suffix n-grams, so a high value at token t means the phrase ending at t was recognised — the gate marks pattern completion, not pattern onset.

Figure 7 — schematic reproduction. The sentences are the paper's; the highlighted spans are the ones it reports the gate activating on. Per-token α values are published only as a heatmap image, so the intensities here are illustrative of the described pattern rather than measured. Engram-27B computes eight gates per token (4 mHC branches × 2 layers); the paper visualises the branch most correlated with semantic matching.

Storage decoupled from compute

This is the property that makes the architecture interesting to build on rather than merely to benchmark. An MoE router cannot tell you which expert a token needs until the hidden state reaches that layer, so expert weights must already be in HBM. An Engram address is a function of input_ids alone — it is computable before the first block runs.

What it is not, and what is unresolved

Engram is frequently described as giving a model “a second brain” or fixing “LLM amnesia.” It does neither. Being precise about the boundaries is the fastest way to understand the mechanism.