Split games,
not positions.

Our count-based baseline currently predicts the next human move in 22.4% of held-out positions. Suppose that next month a shiny new neural model reports 78% validation accuracy. Champagne? Not yet — because there is a boring way to reach 78% without learning any more chess: let the evaluation quietly reuse games the model already studied.

That failure mode is called data leakage, and it is not a beginner’s mistake — leaky evaluations regularly slip into published research. For us it is also the single most expensive mistake available, because every later experiment chooses between models by comparing validation numbers. If those numbers reward memorization, we will spend the tournament’s 100 MB and one exaFLOP optimizing for the wrong thing, and discover it on match night.

The plan: commit to a prediction, see the exact mechanism of the leak with rows you can point at, watch it inflate a score in a simulator, and then approve the split our real code uses.

What a validation score is supposed to measure

Training accuracy answers an easy question: “how well did the model fit the games it studied?” With enough capacity, memorizing wins that contest without any understanding. The question we actually care about is different: “how will it move in a game nobody has played yet?” — because that is literally what the tournament is. A validation set is a stand-in for that future: games we possess but the model has never touched. The score it produces is trustworthy only while “never touched” stays literally true.

One game becomes many training rows

Language models don’t train on games; they train on (context → next move) rows. Here is a six-ply Ruy Lopez opening and every row it generates:

Notice the redundancy: R6’s context contains R2, R3, R4, and R5 in their entirety. These six rows are not six independent facts about chess — they are one game photographed from six angles.

Hold out one row, leak it through its neighbours

Say the shuffle sends R4 to validation and the other five rows to training. The exam question is: “after e4 e5 Nf3, what came next?” The answer is Nc6. Now look at what the model studied. Training row R5 begins e4 e5 Nf3 Nc6… — the exam’s answer is printed, verbatim, inside a study row. So is R6. A model with spare capacity doesn’t need to understand knight development; it can simply remember “that game where Nc6 followed Nf3” and recite it back.

To be fair: a different game that reached e4 e5 Nf3 and continued Nc6 would also “give away” this answer — but that is legitimate generalization, a pattern learned across many players. The leak is that this exact game answers its own exam. And the distinction we care about — pattern versus memory — is precisely the distinction the leaky split erases.

Randomness is not independence

The shuffle in the leaky protocol is genuinely random, and that is the trap: randomness controls which rows cross the boundary, not whether related rows may cross it. The fix is choosing the right split unit — the smallest thing that must stay whole. Rows from one game share information, so the unit is the game. The same reasoning appears everywhere in machine learning: split by patient in medical imaging, by speaker in audio, by day in market data. Whenever examples share a source, the source is the unit.

Three lines that keep the evidence honest

Our baseline decides each game’s fate with one tiny function:

• site is the game’s Lichess URL — a stable identity that never changes, no matter how the file is sorted or re-downloaded.
• blake2b(...) turns seed + identity into a huge, effectively random — but perfectly deterministic — number.
• % 100 drops that number into one of a hundred buckets; < validation_percent sends, say, buckets 0–9 to validation. That is a 10% split.

Four properties fall out of those three lines: the whole game lands on one side (assignment depends only on identity), the split is reproducible on any machine in any row order, no chronology bias can sneak in, and changing the seed deals a completely fresh split without touching the data. Read the real function — pinned to the exact commit, it is these same three lines.

What honesty costs

A game-level split scores lower — that is the point, it stopped grading memory — and its estimate wobbles more when validation holds only a few games, because whole games are coarser coins to flip. The rough rule: quadrupling the number of validation games halves the wobble. How much data to spend on a steadier estimate is a genuine trade-off, and it is Mission 4’s topic. You can feel it first, in the simulator below.