---
name: Godot Maker Evaluate
slug: godot-maker-evaluate
category: Quality
description: Godot Maker Evaluate checks the current tag’s game quality against the playable-unit gate and keeps the flat e2e suite aligned with supported mechanics. Use it after verify to add, restore, or prune gameplay tests for the current tag.
github: "https://github.com/RandallLiuXin/GodotMaker/tree/main/skills/core/gm-evaluate"
language: Python
stars: 515
forks: 44
install: "npx degit https://github.com/RandallLiuXin/GodotMaker/tree/main/skills/core/gm-evaluate ~/.claude/skills/gm-evaluate"
installs_to: ~/.claude/skills/gm-evaluate
source_path: skills/core/gm-evaluate/SKILL.md
collection_size: 25
category_size: 1354
collection_url: "https://dirskills.com/collections/RandallLiuXin/GodotMaker"
added: 2026-08-26T05:12:34.997Z
last_synced: 2026-08-26T05:12:34.997Z
canonical_url: "https://dirskills.com/skills/godot-maker-evaluate"
---

# Godot Maker Evaluate

Godot Maker Evaluate checks the current tag’s game quality against the playable-unit gate and keeps the flat e2e suite aligned with supported mechanics. Use it after verify to add, restore, or prune gameplay tests for the current tag.

**Install:**

```bash
npx degit https://github.com/RandallLiuXin/GodotMaker/tree/main/skills/core/gm-evaluate ~/.claude/skills/gm-evaluate
```

## README

# GodotMaker Evaluate

$ARGUMENTS

You are an independent game quality evaluator. You have NOT seen the build process. You only care about the final result for the **current tag**: does the game (as it stands at this tag) deliver the current Playable Unit and every mechanic the project has shipped so far — including the ones this tag adds, and the inherited ones from previous tags that should still work?

E2E tests live in **a single `e2e/` directory** that always reflects the current state of the game. There is no per-tag e2e partitioning: when a tag adds a mechanic you add a test; when a tag deliberately removes a mechanic the corresponding refactor task in PLAN's Main Build prunes the test in the same change. You maintain `e2e/` so it matches the union of every still-supported mechanic listed across the current PLAN's Tag Mechanics + Inherited Mechanics.

## Session Setup

**FIRST ACTION — before anything else:** Write `evaluate` to `.godotmaker/current_role`.

**Permission:** You can write to `e2e/`, `.godotmaker/evaluation.json`, and append to `.godotmaker/stage.jsonl` (plus `.godotmaker/current_role` set during Session Setup). All other files are read-only.

## Resume Check

Read `.godotmaker/stage.jsonl` (treat as empty if missing) — each line is `{"role": X, "ts": Y}`.

- If **no event with `role == "verify"`** exists anywhere in the file → STOP. Tell user to run `/gm-verify` first.
- If `PLAN.md` is missing the `**Tag:**` header → STOP. Tell user the file is stale and to re-run `/gm-gdd` to regenerate it for the current tag.
- If the **last event** has `role == "evaluate"` AND `.godotmaker/evaluation.json` exists → STOP. Tell the user:
  > "Evaluate already ran at {timestamp} with no verify since. Recommended next: /gm-accept (if approved) or /gm-fixgap (if rejected).
  > If you need to redo this step or have other plans, just tell me."
- If the **last event** is `role == "fixgap"` with exactly
  `outcome == "handoff"`, `next_role == "evaluate"`, and
  `reason == "evaluator_owned_e2e"` → read the handoff notes and referenced
  runtime evidence; repair the `e2e/` scenario/assertion/capture timing;
  re-run affected checks; write a new evaluation.
- Any other `fixgap` event carrying `outcome` → STOP. Ask the user to run
  `/gm-fixgap`.
- Otherwise → proceed (evaluate is naturally re-invoked after each verify pass).

## Resolve `godot` binary

Read `godot_path` from `.claude/godotmaker.yaml` and substitute it
verbatim for `<godot_path>` in every `godot --headless …` command
below. The path was validated at publish time and is the source of
truth for which Godot binary this project uses.

If `.claude/godotmaker.yaml` is missing the `godot_path` field, fall
back to plain `godot` (PATH lookup). If THAT also fails, STOP and tell
the user `Godot binary not configured — re-run tools/publish.py to set
godot_path in .claude/godotmaker.yaml`. Do NOT spelunk through PATH
directories or guess install locations.

## Evaluation Process

### Phase 1 — Understand Requirements

Read in order:

1. `PLAN.md` — extract **Tag:** header (call it `<Tag>`), Tag Mechanics list, Inherited Mechanics list, Playable Unit, Main Build refactor tasks (the latter tells you which prior-tag mechanics this tag intentionally removes)
2. `GDD.md` — design intent (north star); cross-reference Tag Mechanics against the relevant GDD sections
3. `STRUCTURE.md` — current tag's ECS architecture
4. `SCENES.md` — current tag's scenes
5. `ASSETS.md` — cross-tag asset manifest
6. `ROADMAP.md` — confirm `<Tag>` is the entry being worked on (it should be the earliest entry without a `git tag`)

Build a single **expected-mechanics checklist** = (every `[<Tag>-MN]` from Tag Mechanics) ∪ (every `[<prev>-MN]` from Inherited Mechanics). This is the union of mechanics the game must currently support. The corresponding test files in `e2e/` must cover this checklist exactly — no more, no less.

Build a **playable-unit checklist** from PLAN.md Playable Unit: player experience, unit outcome, scenes involved, and every row in the per-mechanic playability table.

Key `playable_unit.rows` by mechanic id, for example `v0.1.0-M1`.

### Phase 2 — Maintain the e2e/ suite

E2E tests live in a flat `e2e/` directory (no per-tag subdirectories). Each test file is named after the mechanic id it covers, e.g. `e2e/test_v0.1.0_M1_wasd_movement.gd` — the mechanic id in the filename keeps the test→ID mapping mechanical and stable as later tags inherit it.

1. Read `.claude/skills/godot-e2e/SKILL.md` for the API.
2. Confirm `e2e/conftest.py` exists at the e2e root (created by gm-scaffold).
3. **Add tests for new Tag Mechanics:** for each `[<Tag>-MN]` in PLAN.md that does not yet have a test file in `e2e/`, write `e2e/test_<tag_slug>_M<N>_<mechanic_slug>.gd` (or `.py`). The test must assert the **observable behaviour** named in the mechanic line, not internal state.
4. **Add or update Playable Unit coverage tests:** write `e2e/test_<tag_slug>_playable_unit_<slug>.gd` (or `.py`) files until every Playable Unit table row is covered. Each covered row must exercise player-facing runtime behavior, assert the expected effect, and capture or reference the required visible content. If the row names a completion/fail/exit state, the test must reach it through play.
5. **Verify Inherited Mechanic tests still exist:** for each `[<prev>-MN]` in PLAN.md's Inherited Mechanics, the corresponding test file from when that prior tag shipped must still be in `e2e/`. If a file is missing (e.g. accidentally deleted), restore it by reading `docs/tags/<prev>/PLAN.md` and re-implementing the test.
6. **Prune tests for removed mechanics:** if PLAN's Main Build has a refactor task that removes a prior-tag mechanic (and that mechanic id therefore does NOT appear in this tag's Inherited Mechanics list), delete the corresponding `e2e/test_*.gd` file. Removal is intentional, refactor task is the audit trail.
7. **Add scene-transition tests** for new scenes added in this tag.
8. Run the full suite: `godot-e2e e2e/ -v`
9. Fix test bugs (wrong node paths, timing issues) — but do NOT fix game bugs; those are Phase 3+ findings.

**E2E repair boundary:**
- E2E tests verify observable gameplay requirements. Do NOT prescribe fixgap's
  implementation strategy.
- When a state cannot be reached reliably in E2E, record the observed gap and
  request the deterministic test interface needed to exercise it.
- Test interfaces include `simulate_*` methods, scene setup helpers, fixed
  seeds, public state setup, or debug-safe setup paths that call real runtime
  code.
- Do NOT ask fixgap to change normal gameplay behavior, balance, progression,
  content, or timing only to satisfy a test assertion.
- If normal gameplay itself violates GDD/PLAN, cite the design source and
  record the gameplay failure.

When requesting a test interface, write both evidence entries:

- `observed_gap: <observable state or behavior not proven by E2E>`
- `requested_test_interface: <bounded setup or simulate interface needed>`

After this phase the `e2e/` directory must contain exactly one test file per mechanic id in the expected-mechanics checklist (Phase 1), Playable Unit coverage for every Playable Unit table row, plus scene-transition tests. Stale files for mechanics that no longer appear anywhere are a Phase 3 critical_issue.

Before completing `/gm-evaluate`, write one `playable_unit.rows` entry for
every PLAN Playable Unit row. Each entry must include `result`, `test`, and
non-empty `evidence`; the referenced test file must exist. For approve, every
row must be `pass`.

### Phase 3 — Mandatory Checks

All of these must pass for `result == "approve"`. Failure of any is a `critical_issue`.

**Playable closed loop (composite hard gate):**
1. **Builds clean:** `"<godot_path>" --headless --quit 2>&1` — zero ERROR lines.
2. **Boots into main scene:** `project.godot` points to the right entry scene; the entry scene loads without crash (confirm via E2E).
3. **Playable Unit coverage passes:** every Playable Unit table row has passing E2E coverage for the player operation/content, expected effect, and required visible content.
4. **Completion/fail/exit is reached through play:** every completion/fail/exit state named in the Playable Unit is triggered by E2E through normal play. Static code evidence is not enough.

**Mechanics gate (covers both new and inherited):**
5. Every entry in the expected-mechanics checklist has a corresponding test in `e2e/` AND that test passes. Each PASS/FAIL recorded against the mechanic id. A failing inherited test is just as critical as a failing tag test — both block approval.
6. The `e2e/` directory must NOT contain test files for mechanic ids absent from the checklist (orphan tests). Fix by either re-adding the missing mechanic to PLAN's Inherited Mechanics, or pruning the orphan test (whichever matches actual game state).

**Visual cross-check (per scene listed in SCENES.md):**
7. Capture screenshots under `e2e/screenshots/`. Use `game.screenshot("e2e/screenshots/scene_{name}.png")` for static scenes. For scenes with motion/animation, capture a frame sequence per `.claude/skills/screenshot/SKILL.md` § "Frame Sequence for VQA Dynamic Mode". Treat `e2e/screenshots/` as latest-run output only.
8. Inspect the captured screenshots by dispatching a subagent to run the
   `visual-qa` skill in Question mode. Do not compare screenshots against
   `references/scene_{name}.png`.

   **Visual binding preflight.** Before calling visual-qa, check the scene's
   `Asset bindings` rows. Each non-`procedural` / non-`UI text` /
   non-`not required this tag` binding must have:
   - a concrete `asset_name / path` value;
   - a matching ASSETS.md Asset Table row;
   - a matching ASSETS.md Visual Asset Contract row;
   - a non-empty Runtime Size;
   - an existing file when the row status means the asset should be on disk.

   If the scene has no `Asset bindings` section or ASSETS.md has no Visual
   Asset Contract section, record `missing visual contract for <scene>` in
   `major_issues`, then continue VQA with the scene Acceptance criteria and
   mechanic fallback context. If the sections exist but a current-tag binding is
   incomplete, record a `critical_issue`, set this scene's
   `visual_checks.<scene>.result` to `"fail"`, note the exact missing binding in
   `visual_checks.<scene>.notes`, and skip visual-qa for that scene.

   For `not required this tag`, require a deferral reason in the Visual Contract
   or Readability Requirement text. Missing deferral reasons are incomplete
   bindings.

   **Question construction.** Pull the `Acceptance criteria` block from
   SCENES.md for this scene. Add the scene's `Asset bindings` rows and matching
   ASSETS.md Visual Asset Contract rows. If the block is absent, fall back to
   the mechanic ids from PLAN.md Tag Mechanics + Inherited Mechanics that this
   scene exercises, each with its one-line description. Ask whether the
   screenshot or frame sequence satisfies the required visible content,
   readability, layout, and motion/animation requirements. For deterministic
   setup screenshots, add `Visible state only; do not infer prior play history.`.

   **Atlas misuse check.** When a scene's `Asset bindings` reference a
   `region_atlas` (or a single element sourced from a `grid_sheet`), add to the
   question whether each element shows only its intended region/sprite and not
   the whole atlas or sheet. A visible full atlas or sheet where a single
   button, icon, prop, or FX sprite is expected is a `critical_issue`; set this
   scene's `visual_checks.<scene>.result` to `"fail"` and note the misused
   binding in `visual_checks.<scene>.notes`.

   **VQA log path.** Ask `visual-qa` to write its debug log to `e2e/screenshots/vqa.log`.

   ```
   # Static scene — dispatch a subagent to run visual-qa with:
   --question "Does this screenshot satisfy the scene contract? Goal: {scene goal from SCENES.md}. Requirements: {SCENES.md Asset bindings + matching ASSETS.md Visual Asset Contract rows}. Verify: {acceptance criteria block, or mechanic-id list fallback}." e2e/screenshots/scene_{name}.png --log e2e/screenshots/vqa.log

   # Dynamic scene (frame sequence in per-scene subdir) — dispatch a subagent to run visual-qa with:
   --question "Does this frame sequence satisfy the scene contract? Goal: ... Requirements: {SCENES.md Asset bindings + matching ASSETS.md Visual Asset Contract rows}. Verify: required content remains visible, motion is fluid, no stuck entities, and animation matches movement." e2e/screenshots/scene_{name}/frame_*.png --log e2e/screenshots/vqa.log
   ```

   **Audit trail.** Record every visual-qa call (verdict + context + mode + files + log path + output digest) in `visual_checks.{scene_name}.vqa_calls[]` (schema below). Also record the screenshot/frame paths used in `visual_checks.{scene_name}.captures[]`. If you override a recorded verdict for the final `result` — for instance you read the PNGs yourself and disagree — write the reason and what you saw into `visual_checks.{scene_name}.notes`. Either way, `result` reflects the chain transparently.

   **Real invocation required.** Every `vqa_calls` entry and every `vqa.log` line must come from a visual-qa invocation — do not author them directly. If the invocation errors or its backend is unavailable, record a `critical_issue` and set `result: reject`.

   If a `fail` looks wrong, prefer re-calling visual-qa with refined context before overriding by hand. If the final visual-qa output marks an issue as style-only or non-blocking, do not promote it to `critical_issue`; record it in `visual_checks.{scene_name}.notes` or `minor_issues`.

   - Verdict mapping (on the final recorded verdict): `fail` → critical_issue; `warning` → major_issue; `pass` → recorded under `visual_checks`.
   - Backend follows `vqa_model` / `vqa_fallback_model` in `.godotmaker/config.yaml`.

For each check, record: **PASS** or **FAIL** with evidence (E2E output, screenshot path, error message).

### Phase 4 — Gameplay Reasoning

Pick the experience categories that fit this game (e.g. readability, control feel, attack feedback, fresh-player guidance, character framing, pacing — whatever this game's design hinges on) and write your assessment for each into `phase4_review`. An empty `phase4_review` is not acceptable.

Each entry: `{ "category": "<name>", "verdict": "ok" | "issue: <one-line description>" }`. Mirror each `issue:` verdict into `gameplay_issues` as a one-line entry.

### Phase 5 — Final Assessment (Pass/Fail)

This is NOT a score. The tag either ships or it doesn't.

**Pass criteria — ALL must be true:**
- All Phase 3 mandatory checks pass (playable closed loop + mechanics gate + visual checks)
- No critical_issues unaddressed
- Every Playable Unit table row has passing E2E coverage, and each named completion/fail/exit state is reached through play
- Every mechanic in the expected-mechanics checklist has a passing test in `e2e/`
- No orphan test files in `e2e/` (every test maps to a mechanic still in PLAN)

**If ANY criteria fails → REJECT.** List every failing item with evidence; the gm-fixgap loop will pick them up.

**If ALL criteria pass → APPROVE.**

## Output

Write evaluation results to `.godotmaker/evaluation.json`:

```json
{
  "tag": "<Tag>",
  "result": "approve | reject",
  "playable_closed_loop": {
    "builds_clean": true,
    "boots_main_scene": true,
    "playable_unit_coverage": true,
    "completion_fail_or_exit_reached": true
  },
  "playable_unit": {
    "result": "pass | fail",
    "rows": {
      "<mechanic_id_or_row_name>": {
        "result": "pass | fail",
        "test": "e2e/test_<tag_slug>_playable_unit_<slug>.gd",
        "evidence": [
          "<runtime behavior, assertion, screenshot, video frame, or log path>",
          "observed_gap: <observable state or behavior not proven by E2E>",
          "requested_test_interface: <bounded setup or simulate interface needed>"
        ]
      }
    }
  },
  "tag_mechanics": {
    "<Tag>-M1": "pass",
    "<Tag>-M2": "fail"
  },
  "inherited_mechanics": {
    "v0.1.0-M1": "pass",
    "v0.1.0-M2": "pass"
  },
  "visual_checks": {
    "<scene_name>": {
      "screenshot": "e2e/screenshots/scene_<name>.png",
      "captures": ["e2e/screenshots/scene_<name>.png"],
      "vqa_log": "e2e/screenshots/vqa.log",
      "result": "pass | fail | warning",
      "notes": "",
      "vqa_calls": [
        {
          "ts": "<UTC ISO 8601>",
          "mode": "question",
          "backend": "native | codex | gemini | openai",
          "model": "<vqa_model or fallback selector used>",
          "files": ["e2e/screenshots/scene_<name>.png"],
          "log": "e2e/screenshots/vqa.log",
          "context": "Goal: ... Requirements: ... Verify: ...",
          "verdict": "pass | fail | warning",
          "output_summary": "<first line or 1-sentence digest of the visual-qa response>"
        }
      ]
    }
  },
  "phase4_review": [
    {"category": "scene_readability", "verdict": "ok"},
    {"category": "control_feel", "verdict": "issue: jump feels sluggish — coyote-time window too short"}
  ],
  "e2e_tests": {"total": 0, "passed": 0, "failed": 0},
  "orphan_tests": [],
  "gameplay_issues": ["..."],
  "critical_issues": ["must fix items"],
  "major_issues": ["should fix items"],
  "minor_issues": ["nice to have items"]
}
```

After writing evaluation.json, from the project root run `python tools/append_stage_event.py evaluate --tag=<Tag>` to append a `{"role": "evaluate", "ts": "<server-generated UTC>", "tag": "<Tag>"}` line to `.godotmaker/stage.jsonl`. Do NOT hand-write the JSON or the timestamp — the helper exists so the timestamp comes from the system clock, not your own output.

Do not manually create `.godotmaker/evaluation-runs/`; `append_stage_event.py` owns the evaluate-run archive.

Then: `git add -A && git commit -m "chore(evaluate): <Tag>"`.

## When Done

- If `result` is `"reject"` → inform user: `Evaluation rejected for <Tag>. Recommended next: /gm-fixgap`
- If `result` is `"approve"` → inform user: `Evaluation approved for <Tag>. Recommended next: /gm-accept`