Mcp Doc Scraper

7 MIT

FreeCommunity

AI Systems

MCP Server for scraping web documentation and converting it to markdown.

What is Mcp Doc Scraper

mcp-doc-scraper is a Model Context Protocol (MCP) server designed to scrape documentation from web URLs and convert it into markdown format using jina.ai’s conversion service.

Use cases

Use cases include generating markdown files from online API documentation, converting user manuals from HTML to markdown, and creating documentation for software projects from web sources.

How to use

To use mcp-doc-scraper, install it via Smithery or clone the repository. After setting up a virtual environment and installing dependencies, run the server using Python. Use the ‘scrape_docs’ tool to input a URL and specify an output path for the markdown file.

Key features

Key features include scraping documentation from any web URL, converting HTML documentation to markdown format, saving the converted documentation to a specified output path, and integration with the Model Context Protocol (MCP).

Where to use

mcp-doc-scraper can be used in software development, technical writing, and any field that requires converting web-based documentation into a more manageable markdown format.

Clients Supporting MCP

The following are the main client software that supports the Model Context Protocol. Click the link to visit the official website for more information.

Claude Desktop: Official desktop application from Anthropic, natively supports MCP protocol. claude.ai

Cherry Studio: Cross-platform desktop client supporting multiple LLM providers, built-in MCP server support. cherry-ai.com

LobeChat: Modern open-source ChatGPT/LLMs UI, supports MCP protocol integration. lobehub.com

DeepChat: Cross-platform desktop AI assistant, compatible with MCP protocol, focusing on privacy and efficiency. deepchat.thinkinai.xyz

5ire: Cross-platform open-source desktop intelligent assistant MCP client, supports local knowledge base and MCP server. 5ire.app

View More MCP Clients

Overview

What is Mcp Doc Scraper

mcp-doc-scraper is a Model Context Protocol (MCP) server designed to scrape documentation from web URLs and convert it into markdown format using jina.ai’s conversion service.

Use cases

Use cases include generating markdown files from online API documentation, converting user manuals from HTML to markdown, and creating documentation for software projects from web sources.

How to use

Key features

Where to use

mcp-doc-scraper can be used in software development, technical writing, and any field that requires converting web-based documentation into a more manageable markdown format.

Clients Supporting MCP

The following are the main client software that supports the Model Context Protocol. Click the link to visit the official website for more information.

Claude Desktop: Official desktop application from Anthropic, natively supports MCP protocol. claude.ai

Cherry Studio: Cross-platform desktop client supporting multiple LLM providers, built-in MCP server support. cherry-ai.com

LobeChat: Modern open-source ChatGPT/LLMs UI, supports MCP protocol integration. lobehub.com

DeepChat: Cross-platform desktop AI assistant, compatible with MCP protocol, focusing on privacy and efficiency. deepchat.thinkinai.xyz

5ire: Cross-platform open-source desktop intelligent assistant MCP client, supports local knowledge base and MCP server. 5ire.app

View More MCP Clients

Content

Doc Scraper MCP Server

A Model Context Protocol (MCP) server that provides documentation scraping functionality. This server converts web-based documentation into markdown format using jina.ai’s conversion service.

Features

Scrapes documentation from any web URL
Converts HTML documentation to markdown format
Saves the converted documentation to a specified output path
Integrates with the Model Context Protocol (MCP)

Installation

Installing via Smithery

To install Doc Scraper for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @askjohngeorge/mcp-doc-scraper --client claude

Clone the repository:

git clone https://github.com/askjohngeorge/mcp-doc-scraper.git
cd mcp-doc-scraper

Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows, use: venv\Scripts\activate

Install the dependencies:

pip install -e .

Usage

The server can be run using Python:

python -m mcp_doc_scraper

Tool Description

The server provides a single tool:

Name: scrape_docs
Description: Scrape documentation from a URL and save as markdown
Input Parameters:
- url: The URL of the documentation to scrape
- output_path: The path where the markdown file should be saved

Project Structure

doc_scraper/
├── __init__.py
├── __main__.py
└── server.py

Dependencies

aiohttp
mcp
pydantic

Development

To set up the development environment:

Install development dependencies:

pip install -r requirements.txt

The server uses the Model Context Protocol. Make sure to familiarize yourself with MCP documentation.

License

MIT License

Dev Tools Supporting MCP

The following are the main code editors that support the Model Context Protocol. Click the link to visit the official website for more information.

Zed: High-performance collaborative code editor, supports MCP protocol, providing a smooth programming experience. zed.dev

Cursor: AI code editor built on VS Code, supports MCP protocol for context-aware programming. cursor.com

Windsurf: AI code editor from Codeium, integrates MCP protocol to provide intelligent code assistance. windsurf.com

Continue: Open-source AI programming assistant plugin, supports VS Code and JetBrains, compatible with MCP protocol. continue.dev

Trae: AI-driven code editor, supports MCP protocol, focusing on enhancing developer programming experience. trae.ai

View More MCP Dev Tools

Tools

No tools

Comments

Recommend MCP Servers

Tavily MCP Server The Tavily MCP server provides: search, extract, map, crawl tools Real-time web search capabilities through the tavily-search tool Intelligent data extraction from web pages via the tavily-extract tool Powerful web mapping tool that creates a structured map of website Web crawler that systematically explores websites.

MCP Server Chart This is a TypeScript-based MCP server that provides chart generation capabilities. It allows you to create various types of charts through MCP tools. You can also use it in Dify.

GitHub MCP Server MCP Server for the GitHub API, enabling file operations, repository management, search functionality, and more.

Brave Search MCP Server Web and local search using Brave's Search API

Firecrawl MCP Server Advanced web scraping with JavaScript rendering, PDF support, and smart rate limiting

Context7 MCP LLMs rely on outdated or generic information about the libraries you use. You get:

Slack MCP server Channel management and messaging capabilities

Sequential Thinking MCP Server Dynamic and reflective problem-solving through thought sequences

Fetch MCP Server A Model Context Protocol server that provides web content fetching capabilities.

Playwright MCP A Model Context Protocol (MCP) server that provides browser automation capabilities using [Playwright](https://playwright.dev). This server enables LLMs to interact with web pages through structured accessibility snapshots, bypassing the need for screenshots or visually-tuned models.

View All MCP Servers