The harness

Architecture

Each run has its own emulator instance on its own virtual X display, its own copy of the game directory, its own emulator daemon, and its own rows in a shared SQLite database. Runs in a batch do not share any state.

Reading state: the game's memory

The harness reads the running game's memory directly; it does not page through screens. A small daemon per instance starts the emulator as its child process, which lets it read the emulator's guest RAM through /proc/<pid>/mem and serve it over a local HTTP API.

• Game records. The game holds players and clubs as contiguous arrays of fixed-size records with the same layout as its save files. The layout was mapped with community save editors as oracles: change one field in an editor, diff the bytes. The harness decodes these arrays in milliseconds: attributes, positions, contracts, values, squads, selections, tactics, injuries, suspensions.
• Fixtures and dates. League, cup, European and international fixtures and their dates come from the game's fixture tables in memory.
• Screen text. The game keeps the text it has drawn on the current screen as a list in memory. The harness reads that list (1–5 ms, exact, with accents) for news, offers and the in-match minute and incidents. OCR of a screenshot is kept only as a fallback, and for the one thing not in that list, a news item's body, whose OCR is then matched against the exact strings in memory.
• Snapshots. Whenever the game stops for the manager, the decoded state is written to SQLite as a dated snapshot: squads, tables, results, transfers, finances. The dashboard and the scorer read these snapshots.
• Visibility filter. Every read tool goes through one allowlist of the fields the game shows a human player. Hidden values (for example current and potential ability) are decoded and kept for observers, but no tool can return them. The model sees what a player at the keyboard would see.

Acting

• Team sheet before a match. set_lineup writes the starting eleven, substitutes, positions, tactic and set-piece takers into the club's selection bytes in memory: the same bytes the game's own team screen sets.
• Everything else goes through the game's screens. Bids, contract offers, transfer and loan listings, answers to the game's questions, and in-match substitutions are carried out by macros of key presses and mouse clicks in the emulator. The game's response is read back and returned to the model. Market actions are queued during the turn and carried out after it, in order.
• No memory writes during a match. In-match changes are made only through the game's tactics screen.

The game's own rules decide whether an action succeeds: a rejected bid, a player who refuses a contract or an unaffordable fee is reported back as the game reported it. The harness never edits money, ability, results or any other game value.

Turns

The harness pauses the game whenever a decision is due and runs a turn. In a live match it watches the minute in the screen-text list, freezes the emulator process at half-time or minute 70, opens the pause screen, and resumes after the turn. Each turn starts from a fresh conversation: a fixed rules sheet, then a briefing with the current state, news since the last turn, the model's own season plan and notes, and its latest monthly review. Memory between turns exists only through these tools (set_plan, update_notes, read_memory, write_review).

The game clock is stopped during a turn, so there is no time pressure: a slow model and a fast model face the same game. Wall-clock time and tokens are recorded but not scored.

Model adapters

Every model gets the same tool definitions, the same rules sheet and the same briefing. Only the transport differs.

• Local and API models (OpenAI-compatible endpoints, e.g. vLLM): the harness runs the tool loop itself.
• Frontier models run through their vendors' own agent CLIs in headless mode, with every built-in tool and web search disabled, an empty working directory, and the harness's tools as their only MCP server. They cannot read files, run commands or browse.

Model settings (reasoning effort, sampling) are part of the model's entry in the line-up and are fixed for a batch.

Guards

Weaker models can loop. The harness ends a turn, keeping whatever was already accepted, when:

• the model sends three replies in a row without a tool call;
• the model repeats the same call verbatim after it was already accepted;
• the turn reaches its call cap.

If the game would otherwise start a match with an invalid team, and the model's fix turn did not resolve it, the harness swaps the unavailable player like-for-like. Every such intervention is logged as a harness fallback and shown in the run's log.

Recovery

Old software crashes. The harness writes a checkpoint (an in-game save plus the database position) at the main menu after every day the club plays. A supervisor process watches each run; if the emulator or the runner dies, it restores the last checkpoint and sets aside the database rows written after it, so the run continues from a consistent state. Restores are logged.

Logs and what is public

Every turn is stored: the briefing, the model's reasoning text where the model exposes it, each tool call with its arguments, the game's answer, tokens and seconds. The public dashboard shows all of this for every turn. The system prompt and the tool definitions are not published.

The dashboard also shows hidden game values (for example a player's true ability), labelled as observer-only. The models never see them.

Benchmarks, batches and competitions

• A benchmark is a fixed task on one game: a start save, a period, a tool set and a scorer, with a version number.
• A batch is one execution of a benchmark with a line-up of models. Each model plays its own copy of the game from the same save. Before a batch starts, every harness source file, the scorer, its tests and the launch script are hashed, and the hashes are stored with the batch. The scorer's tests must pass or the batch is not frozen. Nothing in a running batch is changed.
• Baselines run alongside: the game's own AI managing the same club, from the same save. They are shown for context and are not ranked.
• Rankings come only from benchmarks. Competitions (all models in one shared game) are reported separately and never enter the rankings, because the models affect each other's results.

Championship Manager 97/98

A football management game (1997, MS-DOS). The player manages a club: squad, tactics, transfers, contracts, and the board's expectations. Matches are simulated by the game's engine.

Tools

About 35 tools, grouped as:

• Squad and matches: get_squad, get_player, set_lineup, my_lineups, list_tactics, change_tactic, substitute, match_report, player_games, recent_results
• Market: search_players, find_players, scout_club, compare_with_squad, make_offer, transfer_list, list_for_loan, loan_player, contract_terms, offer_new_contract, shortlist tools
• Club: club_info, board_confidence, league_table, fixtures, search_history
• Questions from the game: answer, decide
• Memory: set_plan, update_notes, read_memory, season_review, write_review; and end_turn

Scorer (Club challenge, A-v1)

• Points = league points earned while in charge, plus 5 for each completed season survived.
• Each league is scored at its own game date. If the manager is sacked, the points stop at the sacking and the run is then measured against the date of the furthest league in the batch.
• A secondary management index (league 35, cups 15, squad ability 25, finances 15, process 10, relative to the baselines) is shown but does not decide the ranking.

Game-specific notes

• Player ability is hidden in the game; the model sees the attribute numbers and descriptions the game's screens show.
• The board can sack the manager. A sacking ends that model's run.