--- name: Power Analysis Loop slug: power-analysis-loop category: AI Engineering description: Power Analysis Loop sizes a single two-arm comparison by simulation, checks the design for validity flaws, and writes a preregistration before data collection. Use it for A/B tests, simple RCTs, or two-model evaluations with one primary outcome. github: "https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/power-analysis" language: Python stars: 166 forks: 19 install: "npx degit https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/power-analysis ~/.claude/skills/power-analysis" installs_to: ~/.claude/skills/power-analysis source_path: loops/power-analysis/SKILL.md collection_size: 25 category_size: 3670 collection_url: "https://dirskills.com/collections/gaasher/Agent-Loop-Skills" added: 2026-09-08T05:34:44.209Z last_synced: 2026-09-08T05:34:44.209Z canonical_url: "https://dirskills.com/skills/power-analysis-loop" --- # Power Analysis Loop Power Analysis Loop sizes a single two-arm comparison by simulation, checks the design for validity flaws, and writes a preregistration before data collection. Use it for A/B tests, simple RCTs, or two-model evaluations with one primary outcome. **Install:** ```bash npx degit https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/power-analysis ~/.claude/skills/power-analysis ``` ## README # Power Analysis Loop A **power-analysis-and-preregister** loop for a **two-arm comparison**. The artifact is the study's statistical plan; the feedback signal is two parts — **statistical power** (estimated by Monte-Carlo simulation of the planned test) and a **count of validity flaws**. Each iteration simulates power, solves for the sample size that reaches the target, audits the design for flaws, and revises — until power clears the target **and** the flaw list is empty. The deliverable is a sample-size justification plus a preregistration that pins the hypothesis, primary outcome, analysis, sample size, and stopping rule before any data is seen. ## Scope & limitations This loop does exactly three things, in a loop: **(1)** computes power and required sample size for a **two-group comparison** by simulation, **(2)** runs a fixed **validity checklist** over the design, and **(3)** writes a **preregistration**. The vendored power model (`tools/power_sim.py`) covers **two-sample mean** (continuous outcome) and **two-proportion** (binary outcome) tests only. It is **not** a general experiment designer. It does **not** handle factorial, repeated-measures, clustered/multilevel, time-series, adaptive, or survival designs; it does **not** pick your outcome measure or manipulation from domain knowledge; and it does **not** analyze data you have already collected. For those, the power numbers here do not apply — use a design-appropriate power method. If the study is not a simple two-arm comparison, say so and stop rather than reporting a power that does not match the planned analysis. ## When to use Use this to size and preregister one two-arm comparison whose primary outcome is a continuous mean or a binary rate. Default to powering for the **minimal effect of interest** the user states; if they are unsure of that effect, help them set it from a baseline and a smallest-meaningful difference rather than an optimistic guess — a design "powered" for an effect bigger than reality is a fiction. If the study is not a two-arm comparison, stop and point to a design-appropriate method. ## Setup Resolve bindings interactively. If `loop.run.yaml` exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the `AskUserQuestion` tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write `loop.run.yaml` (format: `examples/run.example.yaml`) and confirm the values before creating any other files. | binding | meaning | default | how to infer | |---|---|---|---| | `` | the claim the experiment tests | — | ask the user | | `` | primary outcome type + minimal effect of interest: continuous (`baseline_mean`, `sd`, `min_effect`) **or** binary (`baseline_rate`, `min_lift`) | — | ask; this fixes the effect size power is computed at | | `` | power the design must clear | `0.80` | — | | `` | significance level | `0.05` | — | | `` | invocation of the vendored simulator | `python3 /tools/power_sim.py --design --effect [--sd \| --baseline ] --alpha --n ` | — | | `` | output design + preregistration file | `/design.md` | — | | `` | where design + ledger live | `./sandbox` | — | | `` | max iterations | 8 | — | `` prints one JSON object, `{"power", "n_per_group", ...}`. **Run** it to get the power; never estimate power by hand. ## The loop Copy this checklist and tick items off: - [ ] Iteration 0 — draft the design to ``; record nothing as final. - [ ] **Simulate** power: run `` at the current `n` and the assumed effect. - [ ] **Solve N**: if power `< `, re-run at larger `n` (step up, then bisect) until it clears. - [ ] **Audit validity**: list every flaw from the checklist below. - [ ] **Revise**: fix the highest-priority flaw, set `n` to the power-adequate value, update `` (+ Preregistration section). - [ ] Append a ledger row; stop when power clears the target **and** no flaws remain, or at ``. **Iteration 0 — draft.** Write a first design to ``: the arms/conditions, the unit of analysis and how units are assigned, the primary outcome and the exact planned test, the assumed effect size (from ``), and a first sample-size guess. Record nothing as final yet. **Then, until stop (power met + no flaws, or budget):** 1. **Simulate power.** Run `` at the current per-group `n` and the assumed effect, with the `--design` matching the planned test. Record the achieved power. 2. **Solve N.** If `power < `, re-run the simulation at larger `n` — step up (e.g. double), then bisect — until power clears the target, and adopt that `n`. 3. **Audit validity.** Check the design against the checklist and list every flaw found: - **Confounding / no control** — is there a concurrent control group, or is the comparison against a historical/other-source baseline that differs in other ways? - **Randomization** — are units randomly assigned? If not, selection bias threatens any effect. - **Selection / sampling** — is the sample representative of the population the claim is about? - **Multiple comparisons** — more than one outcome/subgroup tested without correction? - **Optional stopping / peeking** — is there a pre-specified stopping rule, or will analysis run repeatedly until significant? - **Outcome & analysis pre-specification** — are the primary outcome and its single planned test fixed in advance (not chosen after seeing data)? - **Measurement** — is the outcome measured reliably and blind to condition where possible? 4. **Revise.** Fix the highest-priority flaw (or a tightly-coupled pair that cannot be fixed independently, such as adding a concurrent control and randomizing assignment to it) and set `n` to the power-adequate value. Update ``, including a **Preregistration** section: hypothesis, primary outcome, the one planned analysis, sample size + how it was derived, randomization scheme, and the stopping rule. 5. **Log** one ledger row and continue. **Stop** when `power ≥ ` **and** the flaw list is empty, or at ``. Report the final design + preregistration, the achieved power and required `n`, and — if stopping on budget — the flaws still outstanding. ## Ledger `/ledger.tsv`, tab-separated, never commas in the text. Header: ``` iter n_per_group power open_flaws change ``` Example: ``` iter n_per_group power open_flaws change 0 50 0.50 2 draft: volunteers vs last-year cohort, n=50 1 100 0.80 1 solved n for 80% power at d=0.4 2 100 0.80 0 randomized concurrent control; pre-specified single primary outcome + stopping rule ``` Report the **best** iteration: the final design, the achieved power and required `n`, and any flaws still open if stopping on budget. ## Constraints - **Power is computed at the minimal effect of interest**, not an optimistic one, because a design powered for an effect bigger than reality detects nothing real — and the `--design` in the simulation must match the test named in the design. Do not edit `tools/power_sim.py`. - **A design does not pass on power alone** — an adequately powered but confounded or non-randomized design still fails; both gates (power and flaws) must clear. - **Preregister before data**, so the eventual test is confirmatory rather than chosen after seeing results: the analysis, outcome, sample size, and stopping rule are fixed in advance. - **One primary outcome and one planned test** drive the power and the verdict; secondary analyses are labeled exploratory. - The sandbox is self-contained — no `../` escapes. Do not pause the loop to ask whether to continue.