pragmar/mcp-server-webcrawl

Overview

This is an MCP server written in Python that connects web crawler data and archives to MCP clients. It exposes crawl results and archived web content through the Model Context Protocol for use in AI and developer tools.

Best for

Best for
Developers who need to expose web crawl archives through the Model Context Protocol

Use cases

• Integrate web crawl archives into AI assistants via MCP
• Query historical web content from crawlers in automated workflows
• Build MCP-based tools that access crawled data sets

Notes

This is an MCP server written in Python that connects web crawler data and archives to MCP clients. It exposes crawl results and archived web content through the Model Context Protocol for use in AI and developer tools.

41 stars on GitHub. Last updated 2026-05-31.

Use cases

• Integrate web crawl archives into AI assistants via MCP
• Query historical web content from crawlers in automated workflows
• Build MCP-based tools that access crawled data sets

Pros

• Lightweight Python implementation with focused functionality
• Open source with MIT license for flexible use
• Simple integration for developers already using MCP

Cons

• Small community with only 41 stars
• Limited to MCP protocol, not a general web crawling solution
• Requires self-hosting and management of crawl data sources

Indexed from awesome-mcp-servers-punkpeye and enriched against its public facts.