Grubbomatic/crawl-readiness-mcp

Overview

An MCP server that brings crawl readiness checks to AI assistants. It includes tools for validation and generation of robots.txt, schema, and llms.txt files. Works with Claude Desktop, Cursor, Cine, and other MCP clients.

Best for

Best for
SEO developers who want AI assistants to manage crawl readiness files.

Use cases

• Check website crawl readiness for search engines
• Generate robots.txt and llms.txt files
• Validate schema and robots configurations

How to use

Tools exposed

• CRAWL_READINESS_API_KEY
• CRAWL_READINESS_API_BASE

Tested with

Claude Desktop, Cursor, Windsurf, Cline, Continue, ChatGPT

Example client config

{\n  "mcpServers": {\n    "crawl-readiness": {\n      "command": "npx",\n      "args": ["-y", "crawl-readiness-mcp"]\n    }\n  }\n}

Notes

An MCP server that brings crawl readiness checks to AI assistants. It includes tools for validation and generation of robots.txt, schema, and llms.txt files. Works with Claude Desktop, Cursor, Cine, and other MCP clients.

0 stars on GitHub. Last updated 2026-09-22. Licensed MIT.

Use cases

• Check website crawl readiness for search engines
• Generate robots.txt and llms.txt files
• Validate schema and robots configurations

Pros

• Supports multiple MCP clients
• Combines validators and generators in one package
• Targets a specific SEO infrastructure need

Cons

• Zero stars indicate limited community adoption
• No clear evidence of ongoing maintenance
• Typo in the official description suggests rough edges

Indexed from awesome-mcp-servers-punkpeye and enriched against its public facts.