GamesBench is a framework-agnostic, reproducible benchmark stack for evaluating long-horizon planning and spatial reasoning in tool-calling LLM and RL agents. Instead of scoring single answers, it measures agent behavior over extended trajectories in interactive game environments.
Why external simulation
When a language model plays a game by imagining the board, two abilities get conflated: planning the next move and simulating the environment's dynamics in its head. GamesBench separates them. The environment is simulated externally and exposed through a tool-calling interface, so the agent always sees ground-truth state and the benchmark isolates planning quality from internal world-simulation errors.
The games
Two canonical suites ship first. Tower of Hanoi stresses sequential planning, deep search trees, and long-horizon execution, with difficulty scaled by disk and peg counts. Sokoban stresses spatial reasoning, geometry, and deadlock avoidance, using both human-authored levels and procedural generation. Both are deterministic, so every trajectory is exactly reproducible and scoreable without human judgment.
Architecture
The stack is three strictly decoupled layers: benchmark orchestration (CLI, batch runners, checkpointing, offline scoring), a game-agnostic LLM harness that runs the episode loop across providers, and the game engines themselves, each implementing a small GameAdapter protocol. The decoupling means environments are reusable outside the benchmark, and generation is separated from scoring, so traces are produced once and can be rescored offline as metrics evolve.
Measurement discipline
Long agent runs fail in boring ways, so the infrastructure treats reproducibility as a feature: native checkpoint and resume for long runs, run manifests that pin git hashes and prompt hashes so any result can be traced to the exact code and prompts that produced it, and regression detection with configurable thresholds for catching quiet capability drift between runs.
Status
Open source under MIT, built in Python with a zero-dependency core. The current release focuses on the benchmark stack itself; a published cross-model comparison is the natural next step and the harness is designed so those numbers are cheap to produce and re-score.