What is Mcp Scraper

mcp_scraper is a command-line tool and MCP server designed for scraping websites and converting HTML content into Markdown format. It utilizes Mozilla’s Readability library for extracting meaningful content from web pages.

Use cases

Use cases for mcp_scraper include converting web articles into Markdown for documentation, processing local HTML files for content management systems, and integrating web scraping functionalities into applications.

How to use

To use mcp_scraper, you can operate it in CLI mode by running commands such as ‘scrape https://example.com’ to print output to the console, or ‘scrape --html-file input.html’ to convert local HTML files. It can also be run as an MCP server by executing ‘npm start’.

Key features

Key features of mcp_scraper include the ability to extract content using Readability, convert HTML to high-quality Markdown with TurndownService, securely handle HTML by removing harmful scripts, and support both command-line and MCP server modes.

Where to use

mcp_scraper can be used in various fields such as web development, content management, and digital publishing, where there is a need to convert web content into Markdown format for easier editing and integration.

Website Scraper

A command-line tool and MCP server for scraping websites and converting HTML to Markdown.

Features

• Extracts meaningful content from web pages using Mozilla’s Readability library (the same engine used in Firefox’s Reader View)
• Converts clean HTML to high-quality Markdown with TurndownService
• Securely handles HTML by removing potentially harmful script tags
• Works as both a command-line tool and an MCP server
• Supports direct conversion of local HTML files to Markdown

Installation

# Install dependencies
npm install

# Build the project
npm run build

# Optionally, install globally
npm install -g .

Usage

CLI Mode

# Print output to console
scrape https://example.com

# Save output to a file
scrape https://example.com output.md

# Convert a local HTML file to Markdown
scrape --html-file input.html

# Convert a local HTML file and save output to a file
scrape --html-file input.html output.md

# Show help
scrape --help

# Or run via npm script
npm run start:cli -- https://example.com

MCP Server Mode

This tool can be used as a Model Context Protocol (MCP) server:

# Start in MCP server mode
npm start

Code Structure

• src/index.ts - Core functionality and MCP server implementation
• src/cli.ts - Command-line interface implementation
• src/data_processing.ts - HTML to Markdown conversion functionality

API

The tool exports the following functions:

// Scrape a website and convert to Markdown
import { scrapeToMarkdown } from './build/index.js';

// Convert HTML string to Markdown directly
import { htmlToMarkdown } from './build/data_processing.js';

async function example() {
  // Web scraping
  const markdown = await scrapeToMarkdown('https://example.com');
  console.log(markdown);
  
  // Direct HTML conversion
  const html = '<h1>Hello World</h1><p>This is <strong>bold</strong> text.</p>';
  const md = htmlToMarkdown(html);
  console.log(md);
}

License

ISC