Grubbomatic/crawl-readiness-mcp

Overview

An MCP server that brings crawl readiness checks to AI assistants. It includes tools for validation and generation of robots.txt, schema, and llms.txt files. Works with Claude Desktop, Cursor, Cine, and other MCP clients.

Best for

Best for
SEO developers who want AI assistants to manage crawl readiness files.

Use cases

• Check website crawl readiness for search engines
• Generate robots.txt and llms.txt files
• Validate schema and robots configurations

How to use

Tools exposed

• CRAWL_READINESS_API_KEY
• CRAWL_READINESS_API_BASE

Clients listed in this record (not verified here)

Claude Desktop, Cursor, Windsurf, Cline, Continue, ChatGPT

Example client config

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ]
    }
  }
}

Notes