Files
cc65-Chess/tests/main.c
T
Stefan WesselsandClaude Opus 5 e5b640eb4f Price every Sargon level, and close the queen table lead
Everything published against Sargon II was level 1, with one six-game look at
level 6.  Levels 2-5 had never been played, so the competitive band for cc65's
upper levels was unmeasured and no match could be sized.

sargon/levelcost.py times a few opening replies per level: six minutes for all
seven.  Level 3 is 1.5 seconds a Sargon move and level 6 is 91, so a 64-game
screen is 1.6 hours at level 3, 5 to 11 at level 4 and 93 at level 6.  The wall
between an afternoon and a fortnight is between levels 4 and 6.

Two numbers here were wrong.  "45 seconds per Sargon move" at level 6 was
seconds per *ply* - half those plies are cc65's, which answers in 8 ms - so it
is 91 a move, out by two in the direction that matters when sizing a match.
And "an early move is the expensive case" holds at the shallow levels and
inverts at level 6, where a whole game costs 1.9x its opening per move.

A one-factor model fitted the two known anchors within 4% across a range of 150
and then over-predicted level 3 by 87%.  Three measured levels are in the table
and levels 4 and 5 are ranges, because two anchors and a line through them is
not a model.

The first screen off it - cc65 Harder vs level 3, 64 games, 1.6 hours - scores
73.4%, above the band, so level 3 is not the pairing.  The numbers worth
reading are 27 distinct games in 64 against 16-17 before, worst repeat down
from fourteen to six, and one fifty-move draw where the baseline had fifteen.
The levels also refuse to sit on one scale: interpolation predicted 58%, and
reading back down from 73.4% puts level 6 somewhere cc65 Very Hard should have
scored 90% rather than the 41.7% it did.

And the queen table, recorded as the most promising untested lead in the
project, is closed.  EVAL_QUEENHOME sets sc_pstQueen's d1 from -5 to +5, which
zeroes the ten centipawns the evaluation pays the queen to leave home; the
shipping binaries hash identically with it present.  Over 6144 games a side it
measures -1.09 sigma, the wrong way.  Tripling the incentive instead, which had
to be worse if the square mattered, measures +0.40 sigma.  Both directions run
backwards: the square is inert and the mechanism is not where the queen's
opening cost comes from.

Replicating the measurement that motivated it on 64 fresh games then found the
statistic itself was fragile.  Mate scores clamp at +-10000, and a plain mean
over evaluation deltas is not robust to that: a middlegame king move reads as
-289.3 against a true -23.5, and a pawn move reads as a *gain*.  With the
clamps dropped the queen ties with the bishop at about twice a pawn, not four
times - and a developing bishop move is correct chess, which says this
statistic was never measuring what it was read as.

sargon/piececost.py is that computation with the clamps dropped and a median
printed beside every mean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 14:33:49 -07:00

152 lines
5.0 KiB
C

/*
* main.c
* cc65 Chess - test support
*
* One binary, a few subcommands. "all" is the gate: it runs everything that
* has a pass/fail answer and exits non-zero if anything failed.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "types.h"
#include "globals.h"
#include "testutil.h"
/*-----------------------------------------------------------------------*/
static void usage(const char *argv0)
{
printf("usage: %s <command> [options]\n\n", argv0);
printf(" all run the pass/fail suite (exit != 0 on failure)\n");
printf(" eperft [depth] perft on the new 0x88 core (default 4)\n");
printf(" budget nodes needed to complete each depth, through a game\n");
printf(" qgen capture generator against the filtered full one\n");
printf(" divide <fen> <depth> per-move node counts for the new core\n");
printf(" tactics search finds the obvious moves\n");
printf(" convert won endings finished before the fifty-move rule\n");
printf(" bench search speed on the host\n");
printf(" match [sanity|terms|depth|repeat|drive|endgame|queen] configuration A vs B\n");
printf(" fuzz [seed] [games] random games through the game path, undo/redo checked\n");
printf(" castle castling and en passant rules\n");
printf(" repeat repetition detection and its history\n");
printf(" opening opening randomisation, and that it stops\n");
printf(" selfplay [games] [plies] AI against itself, with timings\n");
printf("\noptions: -v for more detail\n");
}
/*-----------------------------------------------------------------------*/
int main(int argc, char **argv)
{
int failures = 0, verbose = 0, i;
const char *command;
for(i = 1; i < argc; ++i)
if(!strcmp(argv[i], "-v"))
verbose = 1;
if(argc < 2)
{
usage(argv[0]);
return 2;
}
command = argv[1];
if(!strcmp(command, "all"))
{
printf("== cc65 Chess test suite ==\n\n");
failures += test_RunCastle(verbose);
printf("\n");
failures += test_RunRepetition(verbose);
printf("\n");
failures += test_RunOpening(verbose);
printf("\n");
failures += test_RunGameFuzz(1, 150, verbose);
failures += test_RunGameFuzz(5000, 150, verbose);
printf("\n");
failures += test_RunEnginePerft(5, verbose);
printf("\n");
failures += test_RunQuiescenceGen(verbose);
printf("\n");
failures += test_RunSearchTactics(verbose);
printf("\n");
failures += test_RunSearchMateInOne(verbose);
printf("\n");
failures += test_RunSearchConversion(verbose);
printf("\n");
failures += test_RunSearchAlwaysMoves(verbose);
printf("\n");
failures += test_RunMatchSanity(0);
printf("\n");
failures += test_RunSelfPlay(1, 120, 0);
printf("\n== %s ==\n", failures ? "FAILED" : "all green");
return failures ? 1 : 0;
}
if(!strcmp(command, "tactics"))
return test_RunSearchTactics(1) ? 1 : 0;
if(!strcmp(command, "matein1"))
return test_RunSearchMateInOne(1) ? 1 : 0;
if(!strcmp(command, "convert"))
return test_RunSearchConversion(1) ? 1 : 0;
if(!strcmp(command, "alwaysmoves"))
return test_RunSearchAlwaysMoves(1) ? 1 : 0;
if(!strcmp(command, "match"))
{
const char *what = argc > 2 ? argv[2] : "sanity";
if(!strcmp(what, "terms")) return test_RunMatchTerms(verbose);
if(!strcmp(what, "depth")) return test_RunMatchDepth(verbose);
if(!strcmp(what, "time")) return test_RunMatchEqualTime(verbose);
if(!strcmp(what, "endgame")) return test_RunMatchEndgame(verbose);
if(!strcmp(what, "ladder")) return test_RunMatchLadder(verbose);
if(!strcmp(what, "repeat")) return test_RunMatchRepetition(verbose);
if(!strcmp(what, "drive")) return test_RunMatchMateDrive(verbose);
if(!strcmp(what, "queen")) return test_RunMatchQueen(verbose);
return test_RunMatchSanity(verbose);
}
if(!strcmp(command, "bench"))
return test_RunSearchBench(verbose) ? 1 : 0;
if(!strcmp(command, "eperft"))
return test_RunEnginePerft(argc > 2 && argv[2][0] != '-' ? atoi(argv[2]) : 4, verbose) ? 1 : 0;
if(!strcmp(command, "budget"))
return test_RunBudgetSurvey(1);
if(!strcmp(command, "qgen"))
return test_RunQuiescenceGen(1) ? 1 : 0;
if(!strcmp(command, "divide"))
{
if(argc < 4) { usage(argv[0]); return 2; }
test_EnginePerftDivide(argv[2], atoi(argv[3]));
return 0;
}
if(!strcmp(command, "fuzz"))
return test_RunGameFuzz(argc > 2 && argv[2][0] != '-' ? atoi(argv[2]) : 1,
argc > 3 && argv[3][0] != '-' ? atoi(argv[3]) : 200,
verbose) ? 1 : 0;
if(!strcmp(command, "castle"))
return test_RunCastle(verbose) ? 1 : 0;
if(!strcmp(command, "repeat"))
return test_RunRepetition(verbose) ? 1 : 0;
if(!strcmp(command, "opening"))
return test_RunOpening(verbose) ? 1 : 0;
if(!strcmp(command, "selfplay"))
return test_RunSelfPlay(argc > 2 && argv[2][0] != '-' ? atoi(argv[2]) : 1,
argc > 3 && argv[3][0] != '-' ? atoi(argv[3]) : 200,
verbose) ? 1 : 0;
usage(argv[0]);
return 2;
}