Webscrapemcpserver
What is Webscrapemcpserver
WebScrapeMCPServer is a web crawling server designed to automate the process of extracting data from websites. It operates as part of the MCP architecture, allowing users to configure and manage web scraping tasks efficiently.
Use cases
Common use cases include scraping product prices from e-commerce sites, gathering news articles from various publishers, and collecting data for academic research. It can also be used to monitor website changes over time.
How to use
To use WebScrapeMCPServer, clone the repository from GitHub, install the necessary dependencies using npm, and configure the server by creating a .env file with the required environment variables. Finally, start the server and use the provided ‘crawl’ tool to initiate web scraping tasks.
Key features
Key features include customizable crawl settings such as maximum depth, request delay, and concurrent requests. It also allows users to specify whether to follow links during the crawl, providing flexibility in data extraction.
Where to use
WebScrapeMCPServer can be used in various fields such as data analysis, market research, competitive analysis, and content aggregation. It is particularly useful for businesses and developers needing to gather information from multiple web sources.
Clients Supporting MCP
The following are the main client software that supports the Model Context Protocol. Click the link to visit the official website for more information.
Overview
What is Webscrapemcpserver
WebScrapeMCPServer is a web crawling server designed to automate the process of extracting data from websites. It operates as part of the MCP architecture, allowing users to configure and manage web scraping tasks efficiently.
Use cases
Common use cases include scraping product prices from e-commerce sites, gathering news articles from various publishers, and collecting data for academic research. It can also be used to monitor website changes over time.
How to use
To use WebScrapeMCPServer, clone the repository from GitHub, install the necessary dependencies using npm, and configure the server by creating a .env file with the required environment variables. Finally, start the server and use the provided ‘crawl’ tool to initiate web scraping tasks.
Key features
Key features include customizable crawl settings such as maximum depth, request delay, and concurrent requests. It also allows users to specify whether to follow links during the crawl, providing flexibility in data extraction.
Where to use
WebScrapeMCPServer can be used in various fields such as data analysis, market research, competitive analysis, and content aggregation. It is particularly useful for businesses and developers needing to gather information from multiple web sources.
Clients Supporting MCP
The following are the main client software that supports the Model Context Protocol. Click the link to visit the official website for more information.
Content
Web Crawler MCP Server Deployment Guide
Prerequisites
- Node.js (v18+)
- npm (v9+)
Installation
-
Clone the repository:
git clone https://github.com/jitsmaster/web-crawler-mcp.git cd web-crawler-mcp -
Install dependencies:
npm install -
Build the project:
npm run build
Configuration
Create a .env file with the following environment variables:
CRAWL_LINKS=false MAX_DEPTH=3 REQUEST_DELAY=1000 TIMEOUT=5000 MAX_CONCURRENT=5
Running the Server
Start the MCP server:
npm start
MCP Configuration
Add the following to your MCP settings file:
{
"mcpServers": {
"web-crawler": {
"command": "node",
"args": [
"/path/to/web-crawler/build/index.js"
],
"env": {
"CRAWL_LINKS": "false",
"MAX_DEPTH": "3",
"REQUEST_DELAY": "1000",
"TIMEOUT": "5000",
"MAX_CONCURRENT": "5"
}
}
}
}Usage
The server provides a crawl tool that can be accessed through MCP. Example usage:
{
"url": "https://example.com",
"depth": 1
}Configuration Options
| Environment Variable | Default | Description |
|---|---|---|
| CRAWL_LINKS | false | Whether to follow links |
| MAX_DEPTH | 3 | Maximum crawl depth |
| REQUEST_DELAY | 1000 | Delay between requests (ms) |
| TIMEOUT | 5000 | Request timeout (ms) |
| MAX_CONCURRENT | 5 | Maximum concurrent requests |
Dev Tools Supporting MCP
The following are the main code editors that support the Model Context Protocol. Click the link to visit the official website for more information.
