Web Scraper

by shobcoder

Web Scraper is an Automation skill for Claude Code, published by shobcoder in shob.

582 stars0 forkson shobcoder/shobAdded 2026/08/25Repository updated 2026/08/22

View source on GitHub Download repo ZIP

ai-agentsanthropicantigravityantrygravitryclaude-codeclicodexcodex-clicursor-aimcpopencodeshobshob-aiskills

Install in seconds

Install Web Scraper

Copy Web Scraper into your Claude Code skills folder. Run the command in your terminal, or review the source on GitHub before installing.

terminal

npx degit https://github.com/shobcoder/shob/tree/main/skills/web-scraper ~/.claude/skills/web-scraper

Requires Node.js. Downloads this skill only — not the rest of the repository — into your Claude Code skills folder.

Without Node.js

git clone https://github.com/shobcoder/shob.git

Clones the whole repository, then copy the skill’s own directory into your skills folder yourself.

In this catalog

Source file: skills/web-scraper/SKILL.md in shobcoder/shob
Installs to: ~/.claude/skills/web-scraper
Collection: One of 14 skills cataloged from this repository
Category: Automation — 2226 skills

What Web Scraper does

Web Scraper extracts content and structured data from websites. Use it to scrape pages, crawl sites, collect tables or lists, and monitor website changes.

Web Scraper is cataloged under Automation on DirSkills. Web Scraper comes from a repository tagged ai-agents, anthropic, antigravity, antrygravitry and claude-code.

Documentation

README

Excerpt

Extract text content from web pages
Crawl entire websites
Collect structured data
Research and gather information
Monitor website changes
Extract tables and lists

Tools Available

Content Extraction

// Use extract_content_from_websites for structured extraction
// Supports batch processing of multiple URLs
// Returns JSON format with extracted content

Task Format

{
    tasks: [
        {
            url: "https://example.com",
            prompt: "Extract specific information",
            task_name: "optional_name"
        }
    ]
}

Usage Patterns

This is the opening of the README. Read the full README on GitHub.

Frequently asked about Web Scraper

What else does shobcoder publish alongside Web Scraper?
Web Scraper is one of 14 skills that DirSkills catalogs from shobcoder/shob, the repository it ships in. Its siblings there include Brainstorming, Caveman and Cloudflare Agents SDK. Each one is a separate skill with its own page in this directory, installs the same way Web Scraper does, and is maintained by shobcoder in that same repository. The rest of the collection is listed on the shobcoder/shob page.
How does Web Scraper compare to other Automation skills?
Web Scraper ranks #1266 by stars among the 2226 Automation skills in this catalog. The most-starred ones next to it are Autonomous Loops, Autonomous Agent Harness and Automation Audit Ops. DirSkills ranks by the star count of the repository each skill ships in, so that order reflects how popular those repositories are rather than any review of Web Scraper against them. Open each page to compare what they document and how they install.