Credit vs actually deleting the click

What we did

• Froze the seed-42 keep-the-pad checkpoint. No training — scoring only.
• Walked all 15,697 MIND-small validation impressions in order.
• At each impression, split the score into one signed credit per past click.
• Then, for every click, replayed the session from scratch without it and re-scored. About 49 clicks per session, so ~49 replays per impression.
• Compared claimed credit against the real score change, over 22.2 million click-candidate pairs. Roughly 7 hours on one GPU.

The maths

The claim side is the same exact split used in the top-k run:

The truth side is a counterfactual. Remove click t, rebuild the notepad from the remaining clicks, re-score:

• Why credit and effect are not the same number. Each click writes to the notepad with a delta rule — it corrects whatever error the memory currently has. Remove an early click and every later write does something different. So credit_t is "what click t contributed to the score we served", while Δ_t is "what the score would have been". Both are valid; they answer different questions.
• Correlation. Pearson r between credit_t(c) and Δ_t(c) across all pairs.
• Calibration. Least squares Δ ≈ slope · credit + b. A slope of 1 would mean credit is a drop-in deletion forecast.
• Top-1 agreement. How often argmax_t |credit_t(c)| equals argmax_t |Δ_t(c)| — does the most-blamed click really matter most?
• Comprehensiveness gap. mean |Δ| of the top-credit click − mean |Δ| of a random click.

Mean score change when one click is deleted

Full results · seed 42 · validation

• The negative half of the notepad is the faithful half. What the user skipped tracks its own deletion effect far better (r = 0.66, slope 0.45, top-1 43%) than what they clicked.
• The positive-memory gap is negative, and that is not a bug. Only about 12% of events carry positive feedback, so almost every positive click is influential. Picking one at random is already a strong baseline — it beats the top-credit pick here.
• Exactness held everywhere: max residual 4.6e−6, mean 1.3e−6.
• Average absolute gap between claimed credit and real effect: 0.071 per impression.

Why this strengthens the explainability claim

• The credits survive a real causal test. Not a proxy, not a surrogate model — we deleted the click, replayed the session, and measured. The top-blamed click moves the score 12× more than a random one.
• Two independent runs agree. The batch top-k deletion run found the same ~12× separation at k=3. Different code path, different statistic, same conclusion.
• The decomposition is exact, and stays exact at scale. 15,697 impressions, no residual above 4.6e−6. Attention weights have no equivalent property to check.
• We can say where the explanation is most trustworthy. The negative memory is measurably more faithful than the positive. An attention map offers no such breakdown, because it has no signed components to separate.
• It tells us how to phrase a user-facing reason. "This was influenced by these articles" is supported. "Remove this and the score drops by X" is not.

What this run does not show

• Credit is not a deletion forecast. Slope 0.181 and R² 0.233 — one unit of claimed credit corresponds to about 0.18 units of real effect. Read credits as contributions to the score served, not as "what if".
• Top-1 agreement is weak at 18%. The single most-blamed click is usually not the single most-influential one. The ranking is informative in aggregate; the exact winner often is not.
• Not a ranking-quality result, and not a head-to-head against SASRec attention. Neither was run.
• The suppression slice came out identical to the all-clicks slice, so it is omitted above — the mask selected nearly every candidate and measured nothing extra.
• One seed, one dataset, validation split only. 40 impressions had no prior clicks and were skipped.