Stefan WesselsandClaude Opus 5
cd9bc15af6
The gauntlet needed two absolute paths on the command line, which made every recorded invocation specific to one machine. It now looks in tools/, then on PATH, then in the usual package-manager prefixes, and --which reports what it found. Engine and book paths resolve relative to the script rather than to the shell, so the whole thing runs from any directory. tools/ is a gitignored drop zone with a tracked README explaining what belongs in it. The tools are fetched, never vendored: they are large, platform specific and under their own licences. The README records the one build gotcha worth knowing, that c-chess-cli's make.py passes an x86-only flag and fails on Apple Silicon. Versions of both the runner and the opponent are now printed with the results. A figure is only reproducible against the opponent that produced it, and a different major Stockfish will move every number in doc/strength.md. Also: the runner is started in tests/. fastchess drops a resume file full of absolute paths into its working directory, and it was landing in the repo root. Disabling autosave does not stop it, and -config is a *load* directive that clobbers the engine setup if you try to use it to redirect the write - so the working directory is the lever, which is free to use now that every path handed to the runner is absolute. Verified against both runners: same games, same result, 8-14-10 either way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tools/
A drop zone for the external programs the strength measurements need. Everything in here except this file is gitignored — the tools are fetched, never vendored. They are large, platform-specific, and under their own licences.
tests/gauntlet.py looks here first, then falls back to whatever is on your PATH, so
putting a tool in this directory is a convenience rather than a requirement. Run
tests/gauntlet.py --which to see what it found.
What goes here
| Directory | What it is |
|---|---|
fastchess/ |
Match runner. The one Stockfish's own testing framework uses. |
c-chess-cli/ |
Match runner. Smaller, plain C, no dependencies. |
stockfish/ |
The reference opponent. |
Either match runner is enough; having both is how a result gets cross-checked against a second implementation, which is worth doing at least once.
Getting them
Stockfish is packaged nearly everywhere — brew install stockfish,
apt install stockfish, or a download from stockfishchess.org on Windows. Record the
version: it is printed in the gauntlet output, and a different major version will move
every number in doc/strength.md.
fastchess — https://github.com/Disservin/fastchess, make in the clone, or grab a
release binary. Windows binaries are published.
c-chess-cli — https://github.com/lucasart/c-chess-cli. Its make.py passes
-mpopcnt, which is x86-only and fails on Apple Silicon; on ARM build it directly instead:
cc -I./src -std=gnu11 -DNDEBUG -O2 -DVERSION=\"local\" src/*.c -o c-chess-cli -lpthread -lm
doc/measuring.md covers what to do with them.