0xMassi/webclaw

Overview

A Rust-based tool for fast, local-first web content extraction. It provides a CLI, REST API, and MCP server for scraping, crawling, and extracting structured data. Designed to feed content directly into LLM workflows.

Best for

Best for
Developers needing fast, local web content extraction for LLM pipelines

Use cases

• Extracting structured data from web pages for LLM training datasets
• Crawling websites to build custom knowledge bases
• Integrating web content into AI pipelines via REST API or MCP server

How to use

Install

npx create-webclaw

Tools exposed

• WEBCLAW_API_KEY
• OLLAMA_HOST
• OPENAI_API_KEY
• OPENAI_BASE_URL
• ANTHROPIC_API_KEY
• ANTHROPIC_BASE_URL
• WEBCLAW_PROXY
• WEBCLAW_PROXY_FILE

Tested with

Claude Desktop, Claude Code, Cursor, Windsurf, ChatGPT

Notes

A Rust-based tool for fast, local-first web content extraction. It provides a CLI, REST API, and MCP server for scraping, crawling, and extracting structured data. Designed to feed content directly into LLM workflows.

1,269 stars on GitHub. Last updated 2026-05-31. Licensed AGPL-3.0.

Use cases

• Extracting structured data from web pages for LLM training datasets
• Crawling websites to build custom knowledge bases
• Integrating web content into AI pipelines via REST API or MCP server

Pros

• Fast performance due to Rust implementation
• Local-first design keeps data private and offline
• Multiple interfaces (CLI, API, MCP) for flexible integration

Cons

• Requires Rust toolchain to build from source
• Limited community and documentation compared to established scraping tools
• Primarily optimized for LLM use cases, not general-purpose web scraping

Indexed from awesome-mcp-servers-punkpeye and enriched against its public facts.