---
name: Wayback Archiver
slug: wayback-archiver
category: Data
description: Query and explore a self-hosted personal web archiving system. Search archived pages, view snapshots, and access content as Markdown.
github: "https://github.com/icodeface/wayback-archiver/tree/main/skill.md"
language: Go
stars: 32
forks: 5
install: "git clone https://github.com/icodeface/wayback-archiver"
added: 2026-07-16T03:20:40.806Z
last_synced: 2026-07-16T03:20:40.806Z
canonical_url: "https://dirskills.com/skills/wayback-archiver"
---

# Wayback Archiver

Query and explore a self-hosted personal web archiving system. Search archived pages, view snapshots, and access content as Markdown.

**Install:** `git clone https://github.com/icodeface/wayback-archiver`

## README

# Wayback Archiver

Wayback Archiver is a self-hosted personal web archiving system that captures and preserves web pages with full fidelity — HTML, CSS, JavaScript, images, fonts, and all resources.

**Repository**: https://github.com/icodeface/wayback-archiver

## When to use

Use this skill when the user wants to:
- Search archived web pages
- View snapshots of a specific URL
- List recent archives or filter by date range
- Get details about a specific archived page
- Read page content in Markdown format (for AI/LLM consumption)
- Explore the archiving system's data

## Prerequisites

Before using this skill, ensure the Wayback Archiver server is running:

- **Chrome** or **Firefox** with [Tampermonkey](https://www.tampermonkey.net/) extension (v5.3+)
- **Database** (optional): SQLite (default, zero-config) or PostgreSQL 14+ (for remote deployment)

## Quick Start

### 1. Download Pre-built Binaries

Download from [Releases](https://github.com/icodeface/wayback-archiver/releases):

- **Server binary**: `wayback-server-<os>-<arch>.tar.gz` (or `.zip` for Windows)
- **Userscript**: `wayback-userscript.js`

Extract the server binary:

```bash
# macOS/Linux
tar -xzf wayback-server-*.tar.gz

# Windows: extract the .zip file
```

> **Building from source?** See [docs/BUILD.md](https://github.com/icodeface/wayback-archiver/blob/main/docs/BUILD.md).

### 2. Setup Database

By default, Wayback Archiver uses SQLite (zero-config, single-file database). For remote deployment with multiple users, PostgreSQL is recommended.

**Option 1: SQLite (Default)**

No setup required! The database file will be created automatically at `./data/wayback.db` on first run.

**Option 2: PostgreSQL**

```bash
# Configure database connection in .env
echo "DB_TYPE=postgres" >> .env
echo "DB_NAME=wayback" >> .env
```

Server startup automatically creates the configured PostgreSQL database and tables. If the database does not exist yet, the configured user must have permission to create databases.

> **Upgrade note for existing installs:** Any installation upgrading from **`< 1.1.0`** to `1.1.0` or later needs the `pages.snapshot_state` column. The server applies this automatically during startup when the database user has sufficient privileges. Otherwise, run the following idempotent SQL manually first:
>
> ```sql
> ALTER TABLE pages ADD COLUMN IF NOT EXISTS snapshot_state VARCHAR(16);
> UPDATE pages SET snapshot_state = 'ready' WHERE snapshot_state IS NULL OR snapshot_state = '';
> ALTER TABLE pages ALTER COLUMN snapshot_state SET DEFAULT 'pending';
> ALTER TABLE pages ALTER COLUMN snapshot_state SET NOT NULL;
> ```

### 3. Start Server

```bash
./wayback-server
```

Server runs at `http://localhost:8080` by default.

### 4. Install Browser Script

1. Download `wayback-userscript.js` from [Releases](https://github.com/icodeface/wayback-archiver/releases)
2. Open Tampermonkey dashboard
3. Create new script and paste the content
4. Save and enable

> **Chrome users:** Enable "Allow user scripts" in Tampermonkey's extension settings (right-click icon → Manage extension). Firefox does not require this.

## API Usage

Base URL: `http://localhost:8080`

### Authentication (Optional)

When `AUTH_PASSWORD` is set, use HTTP Basic Auth:
- **Username**: `wayback`
- **Password**: `$AUTH_PASSWORD`

### Endpoints

#### List All Pages

```bash
curl "http://localhost:8080/api/pages?limit=100&offset=0"
```

#### Search Pages

```bash
curl "http://localhost:8080/api/search?q=$KEYWORD"
```

#### Filter by Date Range

```bash
curl "http://localhost:8080/api/pages?from=2026-03-01&to=2026-03-12"
```

#### Get Page Details

```bash
curl "http://localhost:8080/api/pages/$PAGE_ID"
```

#### Get Page Content as Markdown

Returns the page body as clean Markdown (strips scripts, nav, footer, etc.). Ideal for AI/LLM consumption.

```bash
curl "http://localhost:8080/view/$PAGE_ID/md"
```

#### Create Public Share Link

Creates an unauthenticated link for one fixed snapshot. The returned token is only shown once.
The `/view/:id` snapshot header also has a Share button that creates and copies the public URL, then exposes Revoke for that share.

```bash
curl -X POST "http://localhost:8080/api/pages/$PAGE_ID/shares" \
  -H "Content-Type: application/json" \
  -d '{}'
```

#### Get Timeline for URL

```bash
curl "http://localhost:8080/api/pages/timeline?url=$ENCODED_URL"
```

#### View Archived Page

Open in browser:
```
http://localhost:8080/view/$PAGE_ID
```

## Data Models

### Page

```typescript
{
  id: number;
  url: string;
  title: string;
  captured_at: string;      // ISO 8601 timestamp
  html_path: string;
  content_hash: string;     // SHA-256
  first_visited: string;    // ISO 8601 timestamp
  last_visited: string;     // ISO 8601 timestamp
  created_at: string;       // ISO 8601 timestamp
}
```

### Resource

```typescript
{
  id: number;
  url: string;
  content_hash: string;     // SHA-256
  resource_type: string;    // "css", "js", "image", "font", etc.
  file_path: string;
  file_size: number;
  first_seen: string;       // ISO 8601 timestamp
  last_seen: string;        // ISO 8601 timestamp
}
```

## Storage Layout

```
data/
├── html/                     # HTML snapshots, organized by date
│   └── 2026/03/12/
│       └── <timestamp>_<hash>.html
└── resources/                # Deduplicated static resources
    └── ab/cd/
        └── <sha256>.css
```

## Features

- **High-fidelity replay**: CSSOM serialization, computed styles inlining
- **Full-page capture**: HTML, CSS, JS, images, fonts
- **Cross-origin resource recovery**: Server-side download of CORS-blocked resources
- **Content-hash deduplication**: SHA-256 based, shared resources stored once
- **Version history**: Multiple snapshots per URL, distinguished by timestamp
- **Smart dedup**: Session-level + server-level prevents redundant captures
- **Dynamic content support**: Captures live DOM state, auto-updates on significant mutations
- **SPA-aware**: Detects SPA navigation, resets capture state per route
- **Anti-refresh protection**: Archived pages frozen (timers, WebSockets, navigation APIs neutralized)
- **Hardened cross-origin iframe capture**: Embedded cross-origin documents are still captured, but bridge requests are signed and frame HTML returns over a private `MessageChannel`
- **Capture body guardrail**: `/api/archive` and `/api/archive/:id` reject decompressed JSON bodies above 32 MiB
- **Private page filter**: Browser side skips loopback, RFC1918, link-local, ULA/loopback IPv6, and `.local` pages

## Testing

See [docs/BUILD.md](https://github.com/icodeface/wayback-archiver/blob/main/docs/BUILD.md) for build and test instructions.

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Server won't start | Check database configuration; for PostgreSQL ensure it's running and database exists |
| Pages not archiving | Verify Tampermonkey script is enabled and server is reachable |
| Missing resources | Check proxy settings if behind corporate firewall |
| Authentication errors | Verify `AUTH_PASSWORD` env var matches your curl credentials |

## Configuration

Create `.env` file in the project root (or set environment variables directly):

The server automatically loads `.env` from the working directory if it exists.

```bash
# Database
DB_TYPE=sqlite  # "sqlite" (default) or "postgres"
DB_PATH=./data/wayback.db  # SQLite database file path (when DB_TYPE=sqlite)

# PostgreSQL configuration (when DB_TYPE=postgres)
DB_HOST=localhost
DB_PORT=5432
# DB_USER defaults to your current system username; falls back to postgres if unavailable
# DB_USER=
DB_PASSWORD=
DB_NAME=wayback
DB_SSLMODE=disable

# Server
SERVER_HOST=0.0.0.0  # 默认 127.0.0.1，设置 0.0.0.0 监听所有网卡
SERVER_PORT=8080

# Storage
DATA_DIR=./data

# Resource metadata cache
RESOURCE_METADATA_CACHE_MB=256  # Optional; RESOURCE_CACHE_MB is a legacy alias

# Authentication (optional)
AUTH_PASSWORD=

# Debug APIs (disabled by default; enable only while troubleshooting)
ENABLE_DEBUG_API=false
```

## License

MIT