A Correct Date Scored Zero

A two-question smoke test on the LoCoMo recall benchmark returned a zero that did not look right. The model's answer contained "May 7, 2023". The gold answer read "7 May 2023". Same date. The normaliser did not match the formats, so the answer scored 0. The scorer was wrong, not the model. ## Where this came from and how much it weighs This was a smoke run, and I want its size on the record. Two questions. Three sessions ingested as plain messages rather than into archival memory, which is not how the framework is meant to be exercised. The best saved run scored 0.5 on a loose string metric. It exists to prove the evaluation path runs end to end, not to rank anything. A defect found in a smoke test is still a defect in the evaluator. It is not evidence about how often the defect fires on the full benchmark, and I have not measured that. ## Why a formatting bug matters more than it sounds Benchmarks get treated as neutral ground. A leaderboard is read as a statement about the systems on it, but it is really a statement about the systems plus the harness that measured them: the answer model, the judge, the retrieval cutoff, the prompt rules, and the string normaliser at the very bottom. If the normaliser penalises correct answers for date format, every ranking downstream inherits it. A system that happens to emit dates in the gold format gains points it did not earn. Nothing in the published number tells you that happened. This is why the lab's first study audits the harness before it trusts any ranking produced by one. Before asking which memory system is best, ask whether the measurement can tell a right answer from a wrong one. Neutral evaluation comes first, or the ranking is measuring luck.