Sriram-PR/doc-scraper

Overview

Go web crawler that scrapes documentation sites and converts the content into clean Markdown for LLM ingestion, such as RAG pipelines or training data. It targets documentation pages specifically and outputs structured Markdown files.

Best for

Best for
Developers who need a quick, no-frills way to turn documentation sites into Markdown for LLM-based applications.

Use cases

• Scrape documentation sites into Markdown for RAG knowledge bases
• Convert multiple doc pages into a consistent format for LLM fine-tuning
• Build local corpora from public docs for offline retrieval

How to use

Tools exposed

• default_user_agent
• default_delay_per_host
• num_workers
• num_image_workers
• max_requests
• max_requests_per_host
• output_base_dir
• state_dir
• max_retries
• initial_retry_delay
• max_retry_delay
• global_crawl_timeout
• per_page_timeout
• skip_images
• max_image_size_bytes
• max_page_size_bytes
• enable_jsonl_output
• jsonl_output_filename
• enable_incremental
• crawl_history_retention

Tested with

Claude Desktop, Claude Code, Cursor, Continue

Notes

Go web crawler that scrapes documentation sites and converts the content into clean Markdown for LLM ingestion, such as RAG pipelines or training data. It targets documentation pages specifically and outputs structured Markdown files.

98 stars on GitHub. Last updated 2026-09-06. Licensed Apache-2.0.

Use cases

• Scrape documentation sites into Markdown for RAG knowledge bases
• Convert multiple doc pages into a consistent format for LLM fine-tuning
• Build local corpora from public docs for offline retrieval

Pros

• Simple, focused tool written in Go with fast execution
• Produces clean Markdown suitable for LLM context windows
• Lightweight and easy to integrate into existing pipelines

Cons

• Limited to documentation sites; not a general-purpose scraper
• Small community and minimal documentation beyond the repo
• No built-in scheduling or incremental crawling features

Indexed from awesome-mcp-servers-punkpeye and enriched against its public facts.