---
name: Tokf Filter Authoring
slug: tokf-filter-authoring
category: Automation
description: Tokf Filter Authoring helps you create and modify tokf filter TOML files for command-output compression. Use it when defining command matches, step ordering, templates, JSON or section parsing, and placement conventions.
github: "https://github.com/mpecan/tokf/tree/main/crates/tokf-cli/skills/tokf-filter"
language: Rust
stars: 196
forks: 19
install: "npx degit https://github.com/mpecan/tokf/tree/main/crates/tokf-cli/skills/tokf-filter ~/.claude/skills/tokf-filter"
installs_to: ~/.claude/skills/tokf-filter
source_path: crates/tokf-cli/skills/tokf-filter/SKILL.md
collection_size: 4
category_size: 1956
collection_url: "https://dirskills.com/collections/mpecan/tokf"
added: 2026-09-05T05:31:11.382Z
last_synced: 2026-09-05T05:31:11.382Z
canonical_url: "https://dirskills.com/skills/tokf-filter-authoring"
---

# Tokf Filter Authoring

Tokf Filter Authoring helps you create and modify tokf filter TOML files for command-output compression. Use it when defining command matches, step ordering, templates, JSON or section parsing, and placement conventions.

**Install:**

```bash
npx degit https://github.com/mpecan/tokf/tree/main/crates/tokf-cli/skills/tokf-filter ~/.claude/skills/tokf-filter
```

## README

# tokf Filter Authoring

You are an expert at writing tokf filter files. tokf is a config-driven CLI that compresses command output before it reaches an LLM context. Filters are TOML files that define how to process a command's output.

When the user asks you to create or modify a filter, follow this guide exactly. Produce valid, idiomatic TOML that matches the schema described below.

---

## Section 1 — What a Filter File Is

A filter file is a TOML file that describes:
- Which command(s) it applies to (`command`)
- How to transform the raw output (steps, applied in a fixed order)
- What to emit on success vs. failure

Filters live in three places, searched in priority order:

1. `.tokf/filters/` — project-local (repo-level overrides)
2. `~/.config/tokf/filters/` — user-level overrides
3. Built-in library (embedded in the tokf binary)

First match wins. Use `tokf which "cargo test"` to see which filter would activate for a given command.

---

## Section 2 — Processing Order

Steps execute in this fixed order — **do not rearrange them**:

1. **`match_output`** — whole-output substring checks; if matched, short-circuits the entire pipeline and emits immediately
2. **`[[replace]]`** — per-line regex transforms applied to every line, in array order
3. **`strip_ansi` / `trim_lines`** — per-line cleanup (ANSI stripping, whitespace trimming)
4. **`skip` / `keep`** — line-level filtering (drop or retain lines by regex)
5. **`dedup` / `dedup_window`** — collapse duplicate consecutive lines
6. **`lua_script`** — Luau escape hatch; runs after dedup, before JSON/section/parse
7. **`[json]`** — JSON extraction via `JSONPath`; when configured, replaces section/parse/chunk
8. **`[[section]]` OR `[parse]`** — structured extraction (these are mutually exclusive; section is a state machine, parse is a declarative grouper). Skipped when `[json]` is configured.
9. **`[[chunk]]`** — block-based structured extraction with per-block aggregation, grouping, and tree output (runs on raw output, alongside sections). Skipped when `[json]` is configured.
10. **Exit-code branch** — `[on_success]` or `[on_failure]` depending on exit code
11. **`[fallback]`** — if neither `on_success` nor `on_failure` produced output
12. **`strip_empty_lines` / `collapse_empty_lines`** — post-processing cleanup on the final output

Within `[on_success]` and `[on_failure]`, fields are processed as:
- `head` / `tail` → trim lines
- `skip` / `extract` → further filter
- `aggregate` → reduce collected sections
- `output` → final template render

---

## Section 3 — Top-Level Fields Reference

| Field | Type | Default | Description |
|---|---|---|---|
| `command` | string or array of strings | required | Command pattern(s) to match. Supports `*` wildcard. |
| `run` | string | (same as command) | Override the actual command executed. Use `{args}` to forward arguments. |
| `match_output` | array of tables | `[]` | Whole-output checks. Short-circuit on first match. |
| `[[replace]]` | array of tables | `[]` | Per-line regex replacements, in order. |
| `skip` | array of strings (regex) | `[]` | Drop lines matching any regex. |
| `keep` | array of strings (regex) | `[]` | Retain only lines matching any regex. (Inverse of skip.) |
| `dedup` | bool | `false` | Collapse consecutive identical lines. |
| `dedup_window` | integer | `0` (off) | Dedup within a sliding window of N lines. |
| `strip_ansi` | bool | `false` | Strip ANSI escape sequences before skip/keep. |
| `trim_lines` | bool | `false` | Trim leading/trailing whitespace from each line. |
| `lua_script` | table | (absent) | Luau escape hatch. |
| `[json]` | table | (absent) | JSON extraction via `JSONPath`. When configured, replaces `[[section]]`/`[parse]`/`[[chunk]]`. |
| `[[section]]` | array of tables | `[]` | State-machine section collectors. |
| `[[chunk]]` | array of tables | `[]` | Block-based structured extraction with per-block aggregation and grouping. |
| `[parse]` | table | (absent) | Declarative structured parser (branch + group). |
| `[on_success]` | table | (absent) | Output branch for exit code 0. |
| `[on_failure]` | table | (absent) | Output branch for non-zero exit. |
| `[output]` | table | (absent) | Top-level output template (used by `[parse]`). |
| `[fallback]` | table | (absent) | Fallback when no branch matched. |
| `strip_empty_lines` | bool | `false` | Remove all blank lines from the final output. |
| `collapse_empty_lines` | bool | `false` | Collapse consecutive blank lines into one. |
| `show_history_hint` | bool | `false` | Append a hint line after filtered output pointing to the full output in history. |
| `[[variant]]` | array of tables | `[]` | Context-aware delegation to specialized child filters. |

---

## Section 4 — Step Types

### 4.1 `match_output` — Whole-Output Short-Circuit

Check the entire raw output for a substring. If matched, emit a fixed string and stop — no further processing.

```toml
match_output = [
  { contains = "Everything up-to-date", output = "ok (up-to-date)" },
  { contains = "rejected", output = "✗ push rejected (try pulling first)" },
]
```

- `contains`: literal substring to search for (case-sensitive)
- `output`: string to emit if matched
- `{line_containing}` template variable: the first line that contains the substring

```toml
match_output = [
  { contains = "error", output = "Error on: {line_containing}" },
]
```

**When to use**: for well-known one-liner outcomes that make the rest of filtering irrelevant (e.g., "already up to date", "nothing to push", "authentication failed").

---

### 4.2 `[[replace]]` — Per-Line Regex Transforms

Applied to every line, in array order, before skip/keep. Use to reformat noisy lines.

```toml
[[replace]]
pattern = '^(\S+)\s+\S+\s+(\S+)\s+(\S+)'
output = "{1}: {2} → {3}"

[[replace]]
pattern = '^\s+Compiling (\S+) v(\S+)'
output = "compiling {1}@{2}"
```

- `pattern`: Rust regex (RE2 syntax, no lookaheads)
- `output`: template with `{1}`, `{2}`, … for capture groups; `{0}` is the full match
- If the pattern doesn't match a line, that line passes through unchanged
- Invalid patterns are silently skipped at runtime

**When to use**: when a line contains useful information but in a verbose format — reformat it rather than dropping it.

---

### 4.3 `skip` / `keep` — Line Filtering

`skip` drops lines matching any regex. `keep` retains only lines matching any regex. They compose:

```toml
skip = [
  "^\\s*Compiling ",
  "^\\s*Downloading ",
  "^\\s*$",
]

keep = ["^error", "^warning"]
```

- Both are arrays of regex strings
- Applied after `[[replace]]`
- `skip` is checked first, then `keep`
- A line must pass both: not skipped, and (if keep is non-empty) matching keep

**When to use**: `skip` for removing known noise patterns; `keep` for allow-listing (e.g., keep only lines that start with `error` or `warning`).

Also available inside `[on_success]` and `[on_failure]` for branch-level filtering.

---

### 4.4 `dedup` / `dedup_window` — Deduplication

```toml
dedup = true           # collapse consecutive identical lines
dedup_window = 10      # dedup within a 10-line sliding window
```

- `dedup = true`: removes consecutive duplicate lines (like `uniq`)
- `dedup_window = N`: deduplicates within a sliding window of N lines (catches near-consecutive repeats)
- They are independent; you can use both

**When to use**: for commands that emit repetitive progress lines (e.g., `npm install` printing the same package multiple times, spinner frames, repeated warnings).

---

### 4.5 `lua_script` — Luau Escape Hatch

For logic that pure TOML cannot express: numeric math, multi-line lookahead, conditional branching.

```toml
[lua_script]
lang = "luau"
source = '''
if exit_code == 0 then
    return "passed"
else
    local msg = output:match("Error: (.+)") or "unknown error"
    return "FAILED: " .. msg
end
'''
```

Or load the script from an external file:

```toml
[lua_script]
lang = "luau"
file = "scripts/my-filter.luau"
```

The `file` path resolves relative to the current working directory. Exactly one of `source` or `file` must be set.

**Globals available**:
- `output` (string): the full output after skip/keep/dedup
- `exit_code` (integer): the command's exit code
- `args` (table of strings): the arguments passed to the command

**Return semantics**:
- Return a string → replaces output, skips remaining TOML pipeline
- Return `nil` → fall through to `[[section]]` / `[parse]` / `[on_success]` / `[on_failure]`

**Sandbox**: `io`, `os`, and `package` are blocked. No filesystem or network access. Standard math/string/table libraries are available.

**When to use**: only when no TOML step can express the logic. Most filters do not need this. Consider it after exhausting `match_output`, `skip/keep`, `[[replace]]`, `[[section]]`, and `[parse]`.

---

### 4.6 `[json]` — JSON Extraction via `JSONPath`

For commands that produce JSON output (e.g. `kubectl get pods -o json`, `gh api`, `docker inspect`). Extracts values using `JSONPath` (RFC 9535) queries and produces template variables and structured collections.

```toml
[json]

[[json.extract]]
path = "$.items[*]"
as = "pods"

[[json.extract.fields]]
field = "metadata.name"
as = "name"

[[json.extract.fields]]
field = "status.phase"
as = "phase"

[on_success]
output = "Pods ({pods_count}):\n{pods | each: \"  {name}: {phase}\" | join: \"\\n\"}"
```

**`[[json.extract]]` fields**:

| Field | Type | Required | Description |
|---|---|---|---|
| `path` | string | yes | `JSONPath` expression (RFC 9535), e.g. `"$.items[*]"`, `"$.version"` |
| `as` | string | yes | Variable name to bind the result to |
| `fields` | array of tables | no | Sub-field extraction for each matched object |

**`[[json.extract.fields]]` fields**:

| Field | Type | Required | Description |
|---|---|---|---|
| `field` | string | yes | Dot-separated path within each object (e.g. `"metadata.name"`, `"containers.0.name"`). Not JSONPath — uses simple dot-notation. Supports numeric array indices. |
| `as` | string | yes | Variable name for the extracted value |

**Result mapping**:
- **Single scalar** → `vars["as_name"] = string_value` (no count, no chunk)
- **Array** → `ChunkData::Flat` collection + `{as_name_count}` variable
- **Objects without `fields`** → top-level scalars auto-flattened into chunk items
- **Objects with `fields`** → named fields extracted per item

**Pipeline behavior**: when `[json]` is configured, `[[section]]`, `[parse]`, and `[[chunk]]` are skipped. JSON replaces line-based structural processing. Extracted vars and chunks flow into `[on_success]`/`[on_failure]` template rendering.

**Error handling**: invalid JSON input → extraction skipped, pipeline falls back to raw output (templates are not rendered). Invalid JSONPath → rule silently skipped, other rules still run. Empty array with `fields` → emits `{as_name_count} = "0"`.

**When to use**: when the command produces structured JSON output and you need to extract specific fields. Prefer this over `[parse]` + `skip`/`keep` for JSON-native commands.

---

### 4.7 `[[section]]` — State-Machine Section Collector

The most powerful step. Defines a state machine that collects lines into named variables as it scans top-to-bottom.

```toml
[[section]]
name = "failures"
enter = "^failures:$"      # regex: start collecting when this matches
exit = "^failures:$"       # regex: stop collecting when this matches (after start)
split_on = "^\\s*$"        # regex: split collected lines into blocks on blank lines
collect_as = "failure_blocks"

[[section]]
name = "summary"
match = "^test result:"    # regex: collect only lines matching this (no enter/exit)
collect_as = "summary_lines"
```

**Fields**:
| Field | Required | Description |
|---|---|---|
| `name` | yes | Identifier for this section (used in error messages) |
| `enter` | no | Regex to start collecting (state transitions to "inside") |
| `exit` | no | Regex to stop collecting (state transitions to "outside") |
| `match` | no | Collect any line matching this regex, without enter/exit state |
| `split_on` | no | Split collected lines into blocks when this regex matches |
| `collect_as` | yes | Variable name to bind the result to |

**Accessing collected variables in templates**:
| Expression | Type | Description |
|---|---|---|
| `{name}` | string | Full collected text joined with newlines |
| `{name.lines}` | collection | Individual lines as a list |
| `{name.blocks}` | collection | Blocks split by `split_on` |
| `{name.count}` | integer | Number of blocks (or lines if no split_on) |

**When to use**: when the output has distinct sections with clear start/end markers — test failure blocks, error sections, file change groups.

---

### 4.8 `[[chunk]]` — Block-Based Structured Extraction

Chunks split raw output into repeating structural blocks (e.g., per-crate test suites in a Cargo workspace), extract structured data per-block, and produce named collections for template rendering. Like sections, chunks operate on the raw (unfiltered) command output — skip/keep patterns do not affect chunk processing.

```toml
[[chunk]]
split_on = "^\\s*Running "       # regex marking the start of each chunk
include_split_line = true         # include the splitting line in the chunk (default: true)
collect_as = "suites_detail"      # name for the structured collection
group_by = "crate_name"           # merge chunks sharing this field value
children_as = "children"          # preserve original items as nested collection

[chunk.extract]
pattern = 'unittests.+deps/([\w_-]+)-'  # extract a field from the header line
as = "crate_name"
carry_forward = true              # inherit value from previous chunk when pattern doesn't match

[[chunk.body_extract]]
pattern = 'Running\s+(.+?)\s+\('
as = "suite_name"

[[chunk.aggregate]]
pattern = '(\d+) passed'          # aggregates run within each chunk's lines
sum = "passed"

[[chunk.aggregate]]
pattern = '^test result:'
count_as = "suite_count"
```

**Fields**:

| Field | Type | Required | Description |
|---|---|---|---|
| `split_on` | string (regex) | yes | Regex marking the start of each chunk |
| `include_split_line` | bool | no | Whether the splitting line is part of the chunk (default: `true`) |
| `collect_as` | string | yes | Name for the resulting structured collection |
| `extract` | table | no | Extract a named field from the header line (`pattern` + `as`) |
| `body_extract` | array of tables | no | Extract fields from body lines (`pattern` + `as`, first match wins) |
| `aggregate` | array of tables | no | Per-chunk aggregation rules (`pattern` + `sum`/`count_as`) |
| `group_by` | string | no | Merge chunks sharing the same field value, summing numeric fields |
| `children_as` | string | no | When set with `group_by`, preserve original items as a nested collection |

**`carry_forward`** (on `extract` or `body_extract`): when a chunk's pattern doesn't match, inherit the value from the most recent chunk that did. Useful when boundary markers (like `Running unittests`) identify a group, and subsequent chunks should inherit that identity.

**Structured collections in templates**: each item has named fields accessible in `each` pipes:

```toml
[on_success]
output = """\
{suites_detail | each: "  {crate_name}: {passed} passed ({suite_count} suites)" | join: "\\n"}"""
```

**Tree output with `children_as`**: groups preserve their child items for nested template rendering:

```toml
[on_success]
output = """\
{suites_detail | each: "  {crate_name}: {passed} passed\\n{children | each: \"    {suite_name}: {passed} passed\" | join: \"\\n\"}" | join: "\\n"}"""
```

**When to use**: when output contains repeating structural blocks with per-block data you want to aggregate and display. Common for workspace build tools (Cargo, Gradle, Nx) where output is organized by sub-project.

---

### 4.9 `[parse]` — Declarative Structured Parser

Alternative to `[[section]]` for commands with table-like output. Declaratively extracts a header field and groups remaining lines.

```toml
[parse]
branch = { line = 1, pattern = '## (\S+?)(?:\.\.\.(\S+))?(?:\s+\[(.+)\])?$', output = "{1}" }

[parse.group]
key = { pattern = '^(.{2}) ', output = "{1}" }
labels = { "M " = "modified", "??" = "untracked", "D " = "deleted" }

[output]
format = """
{branch}{tracking_info}
{group_counts}"""
group_counts_format = "  {label}: {count}"
empty = "clean — nothing to commit"
```

**`[parse]` fields**:
| Field | Description |
|---|---|
| `branch` | Extract a single value from a specific line (`line`, `pattern`, `output`) |
| `[parse.group]` | Group remaining lines by a key pattern |

**`[parse.group]` fields**:
| Field | Description |
|---|---|
| `key` | `{ pattern, output }` — extract the grouping key from each line |
| `labels` | Map from raw key string to human-readable label |

**`[output]` fields** (used with `[parse]`):
| Field | Description |
|---|---|
| `format` | Template string for the overall output |
| `group_counts_format` | Template for each group entry: `{label}`, `{count}` |
| `empty` | String to emit when no lines were grouped |

**When to use**: for commands like `git status`, `docker ps`, `kubectl get` — table-formatted output where you want to extract a header and count/group rows.

---

### 4.10 `[on_success]` / `[on_failure]` — Exit Code Branches

These branches run after all top-level steps. They have their own sub-fields:

```toml
[on_success]
output = "ok ✓ {2}"          # template; collected variables are available
head = 20                     # keep first N lines
tail = 10                     # keep last N lines
skip = ["^\\s*$"]            # additional line filtering
extract = { pattern = '(\S+)\s*->\s*(\S+)', output = "ok ✓ {2}" }

# Singular form (one rule):
aggregate = { from = "summary_lines", pattern = 'ok\. (\d+) passed', sum = "passed", count_as = "suites" }

# Plural form (multiple rules):
# [[on_success.aggregates]]
# from = "summary_lines"
# pattern = 'ok\. (\d+) passed'
# sum = "passed"
# count_as = "suites"
#
# [[on_success.aggregates]]
# from = "summary_lines"
# pattern = '(\d+) failed'
# sum = "failed"

[on_failure]
tail = 10
output = "FAILED: {summary_lines | join: \"\\n\"}"
```

**Branch sub-fields**:
| Field | Description |
|---|---|
| `output` | Template string for the output. Has access to all `[[section]]` and `[[chunk]]` variables. `{output}` = the filtered output text. |
| `head` | Keep first N lines of filtered output |
| `tail` | Keep last N lines of filtered output |
| `skip` | Array of regexes to filter output lines within this branch |
| `extract` | `{ pattern, output }` — find first match, render template with capture groups |
| `aggregate` | Reduce collected section lines into numeric summaries (singular form) |
| `aggregates` | Array of aggregate rules (plural form — use `[[on_success.aggregates]]`) |

**`aggregate` / `aggregates` fields**:
| Field | Description |
|---|---|
| `from` | Variable name (a `collect_as` result from `[[section]]`) |
| `pattern` | Regex with one capture group to extract a number |
| `sum` | Variable name to bind the sum to |
| `count_as` | Variable name to bind the count (number of lines matched) to |

Both singular `aggregate` and plural `aggregates` can be used together — they are merged at runtime.

**When to use**: Always. Every filter should have at least one of `[on_success]` or `[on_failure]`. Use `[on_success]` to produce a clean summary. Use `[on_failure]` to show enough context to diagnose the issue.

---

### 4.11 `[fallback]` — Last Resort

Emits output when neither `[on_success]` nor `[on_failure]` produced anything.

```toml
[fallback]
tail = 5
```

**When to use**: as a safety net when you have complex branching logic. Ensures tokf never silently swallows output.

---

### 4.12 `[[variant]]` — Context-Aware Filter Delegation

Some commands are wrappers around different underlying tools (e.g. `npm test` may run Jest, Vitest, or Mocha). A parent filter can declare `[[variant]]` entries that delegate to specialized child filter