---
name: URL
slug: url
category: Automation
description: URL fetches web pages and extracts clean body text from HTML or PDF URLs. Use it when a source is a web address and you need readable text for downstream processing.
github: "https://github.com/axoviq-ai/synthadoc/tree/main/synthadoc/skills/url"
language: Python
stars: 1012
forks: 122
install: "npx degit https://github.com/axoviq-ai/synthadoc/tree/main/synthadoc/skills/url ~/.claude/skills/url"
installs_to: ~/.claude/skills/url
source_path: synthadoc/skills/url/SKILL.md
collection_size: 10
category_size: 1523
collection_url: "https://dirskills.com/collections/axoviq-ai/synthadoc"
added: 2026-08-21T05:14:32.567Z
last_synced: 2026-08-21T05:14:32.567Z
canonical_url: "https://dirskills.com/skills/url"
---

# URL

URL fetches web pages and extracts clean body text from HTML or PDF URLs. Use it when a source is a web address and you need readable text for downstream processing.

**Install:**

```bash
npx degit https://github.com/axoviq-ai/synthadoc/tree/main/synthadoc/skills/url ~/.claude/skills/url
```

## README

# URL Skill

Fetches a web URL using `httpx`, strips navigation/script/style tags with
`BeautifulSoup`, and returns clean body text. PDF URLs are extracted with
`pypdf` (primary) and `pdfminer.six` (fallback).

## Setup

```bash
pip install httpx beautifulsoup4

# Optional — needed only if you ingest PDF URLs:
pip install pypdf pdfminer.six
```

## Standalone usage

```python
import asyncio
from synthadoc.skills.url.scripts.main import UrlSkill

skill = UrlSkill()

async def main():
    result = await skill.extract("https://example.com/article")
    print(result.text)          # clean body text
    print(result.metadata)      # {"url": "https://..."}

asyncio.run(main())
```

`DomainBlockedException` is raised when the site returns HTTP 401, 403, or
429. Catch it to log and skip the domain:

```python
from synthadoc.skills.base import DomainBlockedException

try:
    result = await skill.extract(url)
except DomainBlockedException as e:
    print(f"Blocked: {e.domain} (HTTP {e.status_code})")
```

## When this skill is used

- Source starts with `https://` or `http://`
- User intent contains: `fetch url`, `web page`, `website`