Does our SASRec actually work?
Every baseline we compare against in this project is our own rewrite of someone else's model, not their original code. Before spending hundreds of GPU-hours training the rest of that list, we checked the cheapest, most-cited one first: does our SASRec rewrite actually work? The original SASRec paper tests on a different setup than we normally use — a different way of splitting the data, a different way of picking wrong answers to rank against, a longer viewing history. We rebuilt that exact setup, left our SASRec code untouched, and ran it three times. We also tried shrinking our version down to the paper's exact size, to see whether being bigger than the paper was secretly propping up the score.
It reproduces, comfortably. Test HR@10 0.827 vs the paper's own 0.704, NDCG@10 0.631 vs 0.426 — both above the published numbers and in line with a well-known independent PyTorch copy of the same model. Shrinking to the paper's exact size made things slightly worse, not better, so size isn't propping the score up.
HR@10 · test set
NDCG@10 · test set
Our SASRec · test set by seed
| Run | HR@10 | NDCG@10 | Epochs to converge |
|---|---|---|---|
| Seed 42 | 0.8283 | 0.6288 | 35 |
| Seed 43 | 0.8267 | 0.6276 | 27 |
| Seed 44 | 0.8263 | 0.6354 | 59 |
| Mean ± std | 0.8271 ± 0.0011 | 0.6306 ± 0.0042 | — |
| Paper-size diagnostic (seed 42) | 0.8166 | 0.6153 | 35 |
Isolated suite on damlr-w04, kept separate from every other lab in this repo (experiments/isolated/sasrec_repro_lab/). The shared MovieLens loader and SASRec model code were not touched — this experiment writes its own copy of the paper's leave-one-out split, all-interactions-as-positive labeling, and 99-sampled-negative evaluation, then reuses the existing SASRec model exactly as configured in every other table in this repo. "Original paper" is Kang & McAuley, ICDM 2018, Table 2. "Known PyTorch copy" is the midpoint of two runs reported by a widely-used independent reimplementation (pmixer/SASRec.pytorch) of the same official command. "Paper-size diagnostic" reruns our code with the paper's exact embedding size and head count instead of this repo's defaults — one seed, reported alongside the main numbers, never in place of them. Training batch size was raised from the paper's 128 to 1024 partway through this run for GPU-utilization reasons; the evaluation protocol itself was not changed. Full deviation-by-deviation notes: experiments/isolated/sasrec_repro_lab/FINDINGS.md.