Dissecting HOPE: Do Self-Modifying Memories Actually Self-Modify?

Download full PDF →

HOPE (Higher-Order Parametric Experience) extends the Titans memory architecture with three mechanisms: a multi-timescale contextual memory (CMS) whose levels update at different frequencies, a self-modifying memory that changes its own update rule at inference time, and surprise gating that steers computation toward tokens that are hard to predict. The aggregate performance in the original paper is strong, but how much each of the three mechanisms actually contributes is hidden behind that combined number. So I dissected HOPE — peeling off components one at a time, looking inside the internal dynamics, and varying the scale.

The result leaned heavily to one side. The only thing that produced a performance advantage was self-modification, and its importance widened to 4.5x as the model grew. The multi-timescale CMS gave no benefit at any scale, and its fast and slow levels learned nearly the same representation (cosine similarity ≈ 0.77). Surprise gating was effectively not working: per-token losses sat far above any reasonable threshold (mean 66.7, minimum 9.3), so the gate simply stayed open for more than 99.9% of tokens. And even self-modification, the one mechanism doing any work, looked less like the continual adaptation the theory promises and more like an initialization shift that settles once, early in training.

Start with the background. The Titans architecture attaches a neural long-term memory module to attention. As an input sequence arrives, the memory M is updated by gradient descent on an associative-recall objective (M ← M − η∇ₘℒ(M; k, v)), and a surprise metric Sₜ = ‖ℒₜ‖ decides whether each token triggers a memory update. HOPE adds three axes on top of this. The CMS updates a fast level every p_fast tokens and a slow level every p_slow tokens, then combines the outputs through learned gating. Self-modification replaces the fixed projection matrices for key, value, query, and learning rate with meta-learned networks (m_k, m_v, m_q, m_η, m_α, m_memory) that generate projections conditioned on a teaching signal; the parameters produced this way form a fast state that can diverge from the meta parameters during inference. Finally, the surprise threshold is left as a hyperparameter that controls how selective memory updates are. The original paper reports only the combined performance of all three, which leaves an obvious question: which mechanism actually contributes to performance, and does it work the way its theoretical motivation says it should?

I ran the experiments at two scales. The small model is dim=256, 6 layers, about 19M parameters; the scaled model is dim=512, 8 layers, about 73M parameters. Every model was trained for 2,000 steps on a 512-token context with AdamW (lr=3e-4, batch size 8), and the CMS update periods were set to p_fast=8 and p_slow=32. At each scale I trained a Full HOPE with all three mechanisms on, one with only CMS removed, one with only self-modification removed, one with surprise turned off (threshold 0), and alongside them a standard Transformer and the original Titans. At the scaled setting I also added a parameter-matched Transformer (dim=768, 81.2M) to control for HOPE's larger parameter count. The data was OpenWebText with a GPT-2 BPE tokenizer (vocabulary 50,257), and I saved a checkpoint every 200 steps to extract inter-level similarity in the CMS, the drift of self-modification parameters from initialization, the per-token surprise distribution, and the divergence between meta parameters and the fast state.

Here is what peeling off the components one at a time produced.

Table 1: Component ablation results. Δ is the loss difference from Full HOPE (positive = worse). Best result per scale in bold.

Three patterns come out of this. Removing CMS consistently improved performance — by 0.015 at small scale and 0.158 at scaled. The multi-timescale hierarchy contributes nothing to learning while piling on parameters and computation. Removing self-modification, in contrast, made performance collapse — by +0.316 at small scale and +1.412 at scaled, a 4.5x jump in importance as the model grew. And while the Transformer baseline beat every HOPE variant at small scale (-0.336), HOPE reverses the outcome at scaled (+0.593); even a parameter-matched Transformer with 81.2M parameters, more than HOPE's 73.1M, could not close the gap (+0.700).

Why CMS gives no benefit becomes clear once you look inside it. The layer comes from complementary learning systems theory, in which a fast-learning system and a slow-learning system capture different kinds of information — but that separation never actually happened.

Figure 1: CMS dynamics across training. (a) The parameter norms of the fast and slow levels stay almost identical. (b) Inter-checkpoint drift is similar for both levels. (c) The cosine similarity between the two levels' outputs averages 0.77, showing high redundancy. (d) Per-layer norms are uniform across all 6 layers.

Throughout training the L2 norms of the fast and slow levels stayed within 0.2% of each other (~26.15 for both), and their inter-checkpoint drift was nearly the same (~29.2–29.3). Even with a fourfold difference in update frequency (p_fast=8 vs. p_slow=32), the cosine similarity between the two levels' outputs averaged 0.77, converging to similar representations with no sign of differentiating. Per layer, the norms were identical across all 6 layers, so there was no layer-specific specialization in the memory hierarchy. When the two levels perform the same computation redundantly and only add parameters to optimize, it is no surprise that removing CMS improves performance.

Self-modification is a little more subtle. This mechanism generates fast-state parameters from meta-learned networks.

Figure 2: Self-modification dynamics. (a) L2 drift of the self-modification parameters from initialization over training. Large projection matrices (m_k, m_v, m_memory) show a drift of ≈13 that stabilizes immediately; small components (m_η, m_α) drift ≈0.8–5. (b) The relative drift between the meta parameters and the fast state is ≈1.0 across all components, confirming that self-modification occurs.

I tracked 84 self-modification parameters across 6 layers and 6 memory components. The large weight matrices drifted considerably from their Kaiming initialization (≈13), while the smaller components moved about ≈0.8. The problem is that almost all of this drift happens within the first 200 training steps and then stays roughly constant afterward. To check whether self-modification really occurs at inference time, I measured the relative drift between the meta parameters and the fast state, ‖θ_fast − θ_meta‖ / ‖θ_meta‖, and got ≈1.0 for every component. Self-modification clearly does happen — the fast state is meaningfully different from the meta parameters. But the temporal dynamics tell a different story: the meta parameters quickly find, early on, a configuration that produces useful fast states, and from then on the two move together at a fixed offset. This is closer to a learned reparameterization than to the dynamic self-modification HOPE advertises.

Surprise gating was not even switched on.

Figure 3: Surprise gating analysis. (a) The distribution of per-token loss values — mostly concentrated between 50 and 100 (median 67.0). (b) Gate hit rate vs. surprise threshold: even at a threshold of 20, 99.8% of tokens pass the gate.

At this training scale the per-token cross-entropy loss is clustered in a high range. The mean is 66.7, the standard deviation 16.5, the minimum is 9.3, and the 10th percentile is 45.0. In other words, even the loss of the easiest tokens sits well above any reasonable surprise threshold. As a result, up to a threshold of 5.0, 100% of tokens pass the gate; at 10, 99.97%; and at 20, still 99.8%. Meaningful filtering only begins around a threshold of 50 (83.6% pass), and at that level what gets filtered out are the easiest tokens the model has already learned to some degree.

Table 2: Surprise threshold sensitivity (small scale). Every threshold passes more than 99.9% of tokens, so the performance differences are negligible.

The surprise gate is effectively a no-op. For this mechanism to matter, you would need either a much larger, well-trained model where some tokens genuinely have near-zero loss, or a relative rather than absolute surprise metric. In its current form, every token is a surprising token.

The most striking thing is how component importance changes with scale.

Figure 4: Component importance across scales. (a) Small-scale ablation (dim=256, 6L). (b) Scaled ablation (dim=512, 8L). (c) Scale effect comparison: self-modification's importance grows 4.5x, and the scale reversal between the Transformer and HOPE shifts from -0.34 to +0.59.

Self-modification scales superlinearly. The cost of removing it was +0.32 at small scale and +1.41 at scaled — a 4.5x increase. This is probably because as model capacity grows, more parameters benefit from adaptive reparameterization. CMS, on the other hand, sees its penalty grow from +0.015 to +0.158, so the redundancy problem worsens with scale. And the Transformer–HOPE reversal ultimately comes down to self-modification too, since the No CMS variant — self-modification left in, CMS taken out — was the best at both scales.

Figure 5: Scaled training curves (dim=512, 8L). (a) The overall convergence pattern. The No Self-Mod variant converges noticeably slower and reaches a higher loss. (b) In the final 500 steps the separation between variants is clear.

The parameter-matched comparison confirms this again. An 81.2M-parameter Transformer, larger than HOPE's 73.1M, reached a loss of 7.936 — worse than HOPE's 7.237. Growing the Transformer's capacity does not reproduce the benefit of self-modification. HOPE's advantage comes from the mechanism, not the parameter count.

HOPE's effective architecture is far simpler than it is presented to be: a single self-modifying memory module, with no CMS hierarchy and no surprise gating. The No CMS variant — keeping only self-modification and (the no-op) surprise — gave the best performance at both scales. CMS seems to fail because the gradient signal passing through the two levels is too similar (both receive the same loss), so unless diversity is explicitly enforced it is easier to learn one good representation twice than to differentiate them. Self-modification is better understood as a learned reparameterization than as dynamic online adaptation. It is closer to a hypernetwork in which a generator network learns a compressed representation of the target weights, and its performance benefit may come from the implicit regularization and parameter sharing this produces rather than from adaptation during inference.

The limitations are clear. These experiments ran up to 73M parameters, 2,000 steps, on a single GPU. The conclusions about CMS and surprise could change at larger scales: per-token losses might drop low enough to switch surprise gating on, and longer training might let the CMS levels differentiate. Only the scaling trend of self-modification says one thing unambiguously — its importance only grows with scale. In practical terms this means CMS and surprise gating can be stripped out of HOPE with no loss of performance, and more broadly, it is a reminder that in complex neural networks the interactions between mechanisms can differ sharply from what the theory expects.

@article{kim2026dissectinghope,
  title={Dissecting HOPE: Do Self-Modifying Memories Actually Self-Modify?},
  author={Kim, Junghun},
  year={2026},
  url={https://www.multi-turn.ai/blog/dissecting-hope}
}