andyliszewski/webcrawl-mcp

Overview

A Python-based MCP server that crawls websites and returns structured content. It exposes tools for fetching and parsing web pages, making site data accessible to AI agents via the Model Context Protocol.

Best for

Best for
Developers needing a quick MCP-compatible web crawler for prototyping or small-scale data extraction

Use cases

• Extract text and metadata from a given URL for analysis
• Crawl multiple pages of a site to build a content index
• Feed web content into an AI agent for summarization or research

How to use

Install

pip install webcrawl-mcp

Tools exposed

• webcrawl_scrape
• webcrawl_search
• webcrawl_map
• webcrawl_crawl
• USER_AGENT
• REQUEST_TIMEOUT
• FIRECRAWL_API_KEY
• FIRECRAWL_API_URL
• FALLBACK_ON_TRANSPORT_ERROR
• POLITE_MODE

Tested with

Claude Desktop, Claude Code, Cursor, Cline, Continue

Notes

A Python-based MCP server that crawls websites and returns structured content. It exposes tools for fetching and parsing web pages, making site data accessible to AI agents via the Model Context Protocol.

10 stars on GitHub. Last updated 2026-05-15. Licensed MIT.

Use cases

• Extract text and metadata from a given URL for analysis
• Crawl multiple pages of a site to build a content index
• Feed web content into an AI agent for summarization or research

Pros

• Simple Python implementation easy to extend or debug
• Leverages MCP standard for integration with compatible clients
• Lightweight with minimal dependencies

Cons

• Limited to 10 GitHub stars indicates early-stage or niche adoption
• No built-in rate limiting or politeness controls for crawling
• Requires manual setup and configuration for non-trivial sites

Indexed from awesome-mcp-servers-punkpeye and enriched against its public facts.