--- name: Traffic Capture slug: traffic-capture category: Automation description: "Traffic Capture assesses a web app, fingerprints its framework and protections, and records full HTTP traffic into raw-traffic.json. Use it as Phase 1 when you need a target site's API surface discovered or captured." github: "https://github.com/ItamarZand88/CLI-Anything-WEB/tree/main/cli-anything-web-plugin/skills/capture" language: Python stars: 215 forks: 45 install: "npx degit https://github.com/ItamarZand88/CLI-Anything-WEB/tree/main/cli-anything-web-plugin/skills/capture ~/.claude/skills/capture" installs_to: ~/.claude/skills/capture source_path: cli-anything-web-plugin/skills/capture/SKILL.md collection_size: 25 category_size: 1860 collection_url: "https://dirskills.com/collections/ItamarZand88/CLI-Anything-WEB" added: 2026-09-04T05:25:51.406Z last_synced: 2026-09-04T05:25:51.406Z canonical_url: "https://dirskills.com/skills/traffic-capture" --- # Traffic Capture Traffic Capture assesses a web app, fingerprints its framework and protections, and records full HTTP traffic into raw-traffic.json. Use it as Phase 1 when you need a target site's API surface discovered or captured. **Install:** ```bash npx degit https://github.com/ItamarZand88/CLI-Anything-WEB/tree/main/cli-anything-web-plugin/skills/capture ~/.claude/skills/capture ``` ## README # Traffic Capture (Phase 1) Assess the site, then capture comprehensive HTTP traffic. This skill combines site assessment with full traffic recording in a single browser session. Copy this checklist and check off items as you complete them: ``` Phase 1 Progress: - [ ] Prerequisites: playwright-cli available, checkpoint restored if one exists - [ ] Step 1: Browser open on target URL - [ ] Step 2: Fingerprint run + auth handled + profile classified + assessment.md written - [ ] Step 3: Full capture — exploration targets met for the profile - [ ] Step 4: Trace parsed, validate-capture.py PASSED (the gate) - [ ] Step 5: Browser closed, checkpoint + phase-state marked complete ``` --- ## CRITICAL EXECUTION RULES > **NEVER use `run_in_background: true` for ANY playwright-cli command.** > All playwright-cli commands must run in the foreground with appropriate timeouts. > Background execution causes task ID tracking failures — the command completes > before you can read the output. See `references/playwright-cli-commands.md` > for the timeout table. > **NEVER use `eval` for complex expressions.** `eval` fails silently on ternaries, > comma operators, and multi-branch logic with "not well-serializable" errors. > Use `run-code` instead. See `references/framework-detection.md` for details. > **ESM context — no `require()`.** `run-code` uses ESM. Use `await import('fs')` > instead of `require('fs')`. See `references/playwright-cli-commands.md`. --- ## Prerequisites (Hard Gate) Do NOT start unless: - [ ] playwright-cli is available (`npx @playwright/cli@latest --version`) - [ ] Target URL is known **Default capture method:** playwright-cli tracing (standard workflow below). ### Optional `--mitmproxy` mode When the `--mitmproxy` flag was passed to `/cli-anything-web` (or the capture needs untruncated bodies / enhanced metadata), follow this workflow with the step substitutions in `references/mitmproxy-mode.md` — Steps 1, 3, and 4 differ; everything else is identical. When playwright-cli itself is unavailable (e.g., no Node), fall back to chrome-devtools-mcp via `launch-chrome-debug.sh` — see HARNESS.md Tool Hierarchy. ### Public API Shortcut If the target site has a **documented public REST/JSON API** (e.g., Hacker News Firebase API, Dev.to API, Reddit API, Wikipedia API), browser capture is optional: 1. Probe the API endpoints directly with `httpx` or `curl` 2. Save responses as `/traffic-capture/raw-traffic.json` 3. Skip to Phase 2 (methodology) This applies when: - API docs exist (OpenAPI/Swagger, developer docs page, `/api/` prefix) - The API is publicly accessible without browser-specific auth - Endpoints return JSON (not HTML) If unsure whether a public API exists, proceed with browser capture as normal. ### Resume from Checkpoint Before starting, check if a previous capture session exists: ```bash python ${CLAUDE_PLUGIN_ROOT}/scripts/capture-checkpoint.py restore ``` If a checkpoint exists, read the `guidance` field and **resume from the last completed step** instead of starting over. This prevents duplicate work when sessions are interrupted. --- ## Step 1: Setup ```bash # Create output directory mkdir -p /traffic-capture # Clear any stale sessions npx @playwright/cli@latest kill-all 2>/dev/null || true npx @playwright/cli@latest -s= open --headed --persistent # Note: heavy SPAs (Next.js, React) may show "TimeoutError: page._snapshotForAI" on open. # This is non-fatal — verify with: npx @playwright/cli@latest list # # IMPORTANT — "Browser opened with pid..." in command output means the daemon # RE-ATTACHED to the existing browser, NOT that a new session was created. # Do NOT re-navigate or restart when you see this. The session is still open. ``` > **`--mitmproxy` mode:** use the Step 1 substitution in > `references/mitmproxy-mode.md` (start the proxy, then open through it). **Do NOT ask the user to log in yet** — Step 2 will determine if auth is needed. --- ## Step 2: Site Fingerprint (Single Command) Run the all-in-one site fingerprint command instead of individual eval calls. This is faster, more reliable, and detects framework + protection + iframes + auth requirements in one shot. **Use the script file** — multi-line JS with arrow functions and optional chaining fails in playwright-cli's single-line command parser. The script file approach has been tested and works reliably: ```bash npx @playwright/cli@latest -s= run-code "$(grep -v '^\s*//' ${CLAUDE_PLUGIN_ROOT}/scripts/site-fingerprint.js | tr '\n' ' ')" ``` > **IMPORTANT:** The `site-fingerprint.js` script must be loaded via the command > above. Do NOT copy-paste the JS inline — it will fail with SyntaxError. > The `grep -v` strips comments and `tr` joins lines for single-line execution. ### Interpret fingerprint results The fingerprint returns four groups: `framework`, `protection`, `auth`, `iframes`. Map each `true` flag to the next action: | Group | Action | |---|---| | **framework** | See `references/framework-detection.md` for the full protocol table (`googleBatch` / `nextPages` / `nextApp` / `nuxt` / `spaRoot`). | | **protection** | See `references/protection-detection.md` — **always start at the escalation ladder at the top** (plain httpx → curl_cffi → curl_cffi + cookies → camoufox → hybrid). | | **auth** | Table below (Auth detection section). | | **iframes** | If `iframeCount > 0`, see `references/playwright-cli-advanced.md` for the in-iframe re-run snippet. | Claude-facing shortcuts: - `googleBatch: true` → generate `rpc/` subpackage (batchexecute protocol). - `cloudflareManagedChallenge: true` → tier 4 (camoufox) is required; `curl_cffi` alone will fail. - `awsWaf: true` → capture `aws-waf-token` cookie; use `curl_cffi` for GraphQL, cookie-only for SSR. - `akamai: true` or `datadome: true` → 1–2 s delays between requests are mandatory. - `serviceWorker: true` → note in assessment.md; generated CLI uses `service_workers="block"`. - `iframeCount > 0` → re-run the fingerprint inside the iframe. Google Labs apps (Stitch / MusicFX / ImageFX) follow this pattern — parent has `WIZ_global_data`, iframe has the real app. **Note:** `snapshot` and `click ` auto-resolve iframes. Only drop down to `run-code` for iframe interaction when built-in commands fail. ### Auth detection (BEFORE exploration) Check the fingerprint auth fields: | Condition | Meaning | Action | |-----------|---------|--------| | `hasLoginButton && !hasUserMenu` | Login required, not logged in | Ask user to log in NOW | | `hasUserMenu` | Already logged in | Proceed to capture | | `!hasLoginButton && !hasUserMenu` | No auth needed (public site) | Skip auth, proceed | **If auth is needed:** 1. Tell the user: "This site requires login. Please log in in the browser window." 2. Wait for user confirmation 3. Save auth state and tighten permissions (CLAUDE.md mandates `chmod 600`): ```bash npx @playwright/cli@latest -s= state-save /traffic-capture/-auth.json chmod 600 /traffic-capture/-auth.json ``` **If NO auth is needed:** Skip directly to Step 2b. ### 2b. Classify Site Profile Based on fingerprint results AND what you see in the UI, classify the site: | Profile | Auth? | Operations | Exploration Focus | |---------|-------|-----------|-------------------| | **Auth + CRUD** | Yes | Create, Read, Update, Delete | Full CRUD per resource | | **Auth + Generation** | Yes | Generate, Poll, Download | Generation lifecycle + projects | | **Auth + Read-only** | Yes | Read, Search, Export | Read operations + auth flow | | **No-auth + CRUD** | No/Optional | Full CRUD | Skip auth, full CRUD | | **No-auth + Read-only** | No | Read, Search | Minimal capture | ### 2c. Quick API Probe (Force SPA Navigation Trick) Start a SHORT trace, click 3-4 internal links, stop. This reveals hidden API endpoints that SSR hides on initial page load. ```bash npx @playwright/cli@latest -s= tracing-start npx @playwright/cli@latest -s= click npx @playwright/cli@latest -s= click npx @playwright/cli@latest -s= click npx @playwright/cli@latest -s= tracing-stop # Quick parse to see what endpoints appeared (saved alongside the full capture # so it survives the session — don't output to /tmp). python ${CLAUDE_PLUGIN_ROOT}/scripts/parse-trace.py .playwright-cli/traces/ --latest \ --output /traffic-capture/probe-traffic.json ``` This probe trace is separate from the full capture in Step 3 — Step 3 will start a fresh trace that overwrites the `.network` file in `.playwright-cli/traces/`. The parsed `probe-traffic.json` is kept in `traffic-capture/` so it stays available for cross-referencing during Step 4. Check the probe results — what API patterns did you find? See `references/api-discovery.md` for the priority chain and decision tree. ### 2d. Write Assessment Summary Create `/traffic-capture/assessment.md` to consolidate all findings: ```markdown # Site Assessment: - **URL**: - **Framework**: - **Protocol**: - **Protection**: - **Auth required**: - **Iframes**: ) / no> - **Site profile**: - **Capture strategy**: - **Key observations**: ``` --- ## Step 3: Full Traffic Capture Now do the comprehensive capture based on what Step 2 revealed. ```bash # Optional: Start HAR recording alongside trace for standard-format capture # HAR files enable mitmproxy2swagger (auto OpenAPI spec) and third-party analysis tools npx @playwright/cli@latest -s= run-code "async page => { await page.context().routeFromHAR('/traffic-capture/capture.har', { update: true, updateContent: 'embed', updateMode: 'full' }); return 'HAR recording started'; }" # Start fresh trace for full capture (note the trace ID from output!) npx @playwright/cli@latest -s= tracing-start # Output: "trace-" — record this ID ``` > **`--mitmproxy` mode:** skip `tracing-start` and HAR recording — the proxy > is already capturing (see `references/mitmproxy-mode.md`). > **HAR recording is optional but recommended.** It produces a standard HAR file > alongside the trace. This enables `mitmproxy2swagger` to auto-generate an > OpenAPI spec: `pip install mitmproxy2swagger && mitmproxy2swagger -i capture.har -o api-spec.yaml -p ` > The HAR file is saved when the browser context is closed (Step 5). ### Exploration by site profile Use the **concrete targets** in `references/exploration-checklists.md` for the profile identified in Step 2b. Each profile has an explicit entry count, distinct-path count, and WRITE-op target that `validate-capture.py` (Step 4) will enforce. Minimum bar across all profiles: - ≥ 15 entries, ≥ 3 distinct URL paths, protocol ≠ `unknown` - ≥ 1 WRITE op (unless the site is genuinely read-only — pass `--read-only` to the validator) - < 50% error rate (dominant 4xx/5xx means auth or rate-limit failure) ### Pacing for protected sites If any of `cloudflare`, `cloudflareManagedChallenge`, `akamai`, `datadome`, `awsWaf`, or `rateLimit` fired in the fingerprint, **leave 1–2 s between clicks / form submits**. Faster exploration triggers per-IP challenges within ~30 requests and corrupts the trace. ### General interaction rules - **Click by ref (from snapshot) is most reliable:** `snapshot` → note ref → `click ` - **Refs go stale** — always take a fresh snapshot before clicking - **For localized UIs** (Hebrew, Arabic, etc.) — use refs or data-testid, not text - **For iframe-embedded apps** — `snapshot` + `click ` auto-resolves iframes - **Wait after generation** — if the app generates content async, wait for ≥ 15 s before the next action, otherwise the polling loop won't appear in the trace: ```bash npx @playwright/cli@latest -s= run-code "async page => { await page.waitForTimeout(15000); return 'waited'; }" ``` - **Debounced inputs** — after typing a search query, pause 1–2 s before the next action; submitting immediately misses the auto-complete endpoint. --- ## Step 4: Stop, Save, Parse ```bash npx @playwright/cli@latest -s= tracing-stop ``` **If `tracing-stop` fails:** retry once with a 15s timeout; if it fails again the trace is lost — restart the trace at Step 3. Never retry more than twice. Full decision tree: `skills/shared/RECOVERY.md` §tracing-stop Failure (error signatures: `references/playwright-cli-tracing.md`). ```bash python ${CLAUDE_PLUGIN_ROOT}/scripts/parse-trace.py \ .playwright-cli/traces/ --latest \ --output /traffic-capture/raw-traffic.json # parse-trace.py now auto-runs analyze-traffic.py and produces: # - /traffic-capture/raw-traffic.json (raw request/response data) # - /traffic-capture/traffic-analysis.json (auto-detected protocol, auth, endpoints) # Gate — validate the capture before declaring Phase 1 complete. # This check enforces: ≥15 entries, ≥3 distinct paths, protocol ≠ unknown, # ≥1 WRITE op (add --read-only if the site is genuinely read-only), <50% error rate. python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-capture.py # OR for genuinely read-only sites: # python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-capture.py --read-only ``` If `parse-trace.py` fails or produces an empty/static-only raw-traffic.json, follow `skills/shared/RECOVERY.md` §parse-trace Failure. If `validate-capture.py` returns a non-zero exit code, **do not proceed to Step 5**. Map each failed gate to its targeted remediation in `skills/shared/RECOVERY.md` §validate-capture Non-Zero Exit (e.g., <15 entries → capture more pages; <3 distinct paths → exercise more features; no WRITE op → perform a create/update/delete in the UI). Re-open the browser (Step 1), fill the gaps, then re-run Step 4. Only mark the capture complete after the validator passes (or warns, with your explicit sign-off on each warning). For deeper inspection: ```bash python ${CLAUDE_PLUGIN_ROOT}/scripts/analyze-traffic.py \ /traffic-capture/raw-traffic.json --summary ``` > **`--mitmproxy` mode:** replace the parse/analyze block with the Step 4 > substitution in `references/mitmproxy-mode.md` (stop-proxy already includes > analysis; the same validate-capture gate applies). --- ## Step 5: Close ```bash npx @playwright/cli@latest -s= close # Mark the capture checkpoint AND the pipeline phase complete python ${CLAUDE_PLUGIN_ROOT}/scripts/capture-checkpoint.py update --step complete python ${CLAUDE_PLUGIN_ROOT}/scripts/phase-state.py complete --phase capture \ --output traffic-capture/raw-traffic.json ``` --- ## If an endpoint is missing — USE THE FEATURE Don't grep JS bundles. Start a new trace → screenshot → click the button → fill → submit → stop → parse. The browser IS the API documentation. --- ## Fallback **Fallback:** If playwright-cli is not available, see HARNESS.md Tool Hierarchy for chrome-devtools-mcp fallback instructions. --- ## Next Step When capture is complete (raw-traffic.json has WRITE operations, or the site is read-only with only GET requests), invoke `methodology` to analyze the traffic and build the CLI. --- ## References Gate failures (tracing-stop, parse-trace, validate-capture, phase-state): `skills/shared/RECOVERY.md`. Implementation rules: `skills/shared/CONVENTIONS.md`. See `references/` for: - `playwright-cli-commands.md` — command syntax, timeouts, ESM rules - `playwright-cli-tracing.md` — trace file format, recovery protocol - `playwright-cli-sessions.md` — named sessions, auth persistence - `playwright-cli-advanced.md` — waits, iframes, localized UIs, downloads - `framework-detection.md` — framework → protocol table - `protection-detection.md` — anti-bot escalation ladder (curl_cffi → camoufox → hybrid) - `api-discovery.md` — protocol priority chain, decision tree - `exploration-checklists.md` — per-profile capture targets with concrete numbers - `mitmproxy-mode.md` — step substitutions for `--mitmproxy` capture