What is Mcp Playwright Scraper

mcp-playwright-scraper is an MCP server designed to scrape web content and convert it into Markdown format. It leverages Playwright for headless browser automation, BeautifulSoup for HTML parsing, and Pypandoc for high-quality conversion to Markdown.

Use cases

Use cases include extracting content from blogs, news articles, or any website and converting it into Markdown for easy editing and integration into static site generators or documentation platforms.

How to use

To use mcp-playwright-scraper, you can either run it using uv or install it via pip. After installation, you can scrape content by calling the ‘scrape_to_markdown’ tool with the required URL parameter.

Key features

Key features include headless browser automation for modern web pages, HTML parsing and cleanup, and high-quality conversion from HTML to Markdown. It also supports SSL certificate verification.

Where to use

mcp-playwright-scraper can be used in web scraping applications, content management systems, and any scenario where web content needs to be converted into Markdown format for documentation or publishing.

mcp-playwright-scraper

A Model Context Protocol (MCP) server that scrapes web content and converts it to Markdown.

Overview

This MCP server provides a simple tool for scraping web content and converting it to Markdown format. It uses:

• Playwright: For headless browser automation to handle modern web pages including JavaScript-heavy sites
• BeautifulSoup: For HTML parsing and cleanup
• Pypandoc: For high-quality HTML to Markdown conversion

Tools

The server implements a single tool:

• scrape_to_markdown: Scrapes content from a URL and converts it to Markdown
  • Required parameter: url (string) - The URL to scrape
  • Optional parameter: verify_ssl (boolean) - Whether to verify SSL certificates (default: true)

Installation

Using uv (recommended)

When using uv no specific installation is needed. We will
use uvx to directly run mcp-playwright-scraper.

Using PIP

Alternatively you can install mcp-playwright-scraper via pip:

pip install mcp-playwright-scraper

After installation, you can run it as a script using:

python -m mcp_playwright_scraper

Prerequisites

• Python 3.11 or higher
• Playwright browser dependencies
• Pandoc (optional, will be automatically installed by pypandoc if possible)

After installation, you need to install Playwright browser dependencies:

playwright install --with-deps chromium

Configuration

Usage with Claude Desktop

Add this to your claude_desktop_config.json:

Usage with Claude Code

# Basic syntax
$ claude mcp add mcp-playwright-scraper -- uvx mcp-playwright-scraper

# Alternatively, with pip installation
$ claude mcp add mcp-playwright-scraper -- python -m mcp_playwright_scraper

Usage with Zed

Add to your Zed settings.json:

Usage with Cursor

1. Open Cursor Settings
   • Navigate to Cursor Settings > Features > MCP
   • Click the “+ Add New MCP Server” button
2. Configure the Server
   • Name: mcp-playwright-scraper
   • Type: Select stdio
   • Command: Enter one of the following:

uvx mcp-playwright-scraper

python -m mcp_playwright_scraper

Usage

Once configured in Claude Desktop, you can explicitly use the scraper with a prompt like:

Use the mcp-playwright-scraper to scrape the content from https://example.com and summarize it.

Debugging

You can use the MCP inspector to debug the server:

npx @modelcontextprotocol/inspector uvx mcp-playwright-scraper

Or if you’ve installed the package in a specific directory or are developing on it:

cd path/to/mcp-playwright-scraper
npx @modelcontextprotocol/inspector uv run mcp-playwright-scraper

Upon launching, the Inspector will display a URL that you can access in your browser to begin debugging.

Development

Building and Publishing

To prepare the package for distribution:

1. Sync dependencies and update lockfile:

uv sync

2. Build package distributions:

uv build

This will create source and wheel distributions in the dist/ directory.

3. Publish to PyPI:

uv publish

Note: You’ll need to set PyPI credentials via environment variables or command flags:

• Token: --token or UV_PUBLISH_TOKEN
• Or username/password: --username/UV_PUBLISH_USERNAME and --password/UV_PUBLISH_PASSWORD

License

This MCP server is licensed under the Apache License, Version 2.0. You are free to use, modify, and distribute the software, subject to the terms and conditions of the Apache License 2.0. For more details, please see the LICENSE file in the project repository or visit http://www.apache.org/licenses/LICENSE-2.0.