What is Mcp Web Scraper

mcp-web-scraper is a general-purpose web scraping tool designed to extract page content from a specified URL.

Use cases

Use cases for mcp-web-scraper include gathering product information from e-commerce sites, extracting news articles for sentiment analysis, and compiling data for research purposes.

How to use

To use mcp-web-scraper, simply provide the target URL you wish to scrape. For Python developers, it is recommended to use the ‘fetch’ tool, while JavaScript developers can utilize ‘puppeteer’.

Key features

Key features of mcp-web-scraper include its ability to scrape various web pages, support for both Python and JavaScript environments, and ease of integration with existing projects.

Where to use

mcp-web-scraper can be used in various fields such as data analysis, market research, content aggregation, and competitive analysis.

mcp-web-scraper

general purpose web scraper that will scrape a given URL for page content

tool suggestion

fetch for Python devs
puppeteer for Javascript devs

VS Code

Steps to Get it Working

1. Install the uv Python package manager:

pip install uv

2. Install the MCP server fetch package:

uv pip install mcp-server-fetch

3. Ensure the uvx command is available: The uvx command should be available after installing uv. If not, you might need to add it to your PATH or install an additional component.

Testing the Installation

You can verify the installation by running:

uvx mcp-server-fetch --help

Running Within VS Code

Once installed, you can:

1. Open the Command Palette (Cmd+Shift+P)
2. Run “Copilot: Configure MCP Servers”
3. Select the “fetch” server to activate it
   Troubleshooting
   If VS Code can’t find the uvx command:

Make sure it’s in your PATH
Try using the full path to the command in your mcp.json file
Check the VS Code Output panel for error messages
If you continue having issues, you might need to check the GitHub repository mentioned in the comment: https://github.com/modelcontextprotocol/servers/tree/main/src/fetch

Developer Setup Guide: MCP Web Tools Integration with VS Code & GitHub Copilot

Prerequisites

• Python 3.8+ installed and available in your PATH
• VS Code (latest recommended)
• GitHub Copilot extension installed in VS Code
• Git (for cloning the repo)
• pip (Python package manager)

1. Clone the Repository

# Replace <your-repo-url> with the actual repository URL
git clone <your-repo-url>
cd mcp-web-scraper

2. Install Python dependencies:

pip install -r requirements.txt

3. Verify uvx Command

Ensure the uvx command is available (installed by the uv package):

which uvx

If not found, try:

pip install uv --force-reinstall

4. VS Code Configuration

a. Open the Project in VS Code

code .

b. Check .vscode/mcp.json

Ensure your .vscode/mcp.json contains the following (adjust path if needed):

{
  "servers": {
    "fetch": {
      "type": "stdio",
      "command": "uvx",
      "args": ["mcp-server-fetch"]
    },
    "web-tools": {
      "type": "stdio",
      "command": "python",
      "args": ["${workspaceFolder}/mcp_server_web_scraper.py"]
    }
  }
}

5. Enable MCP Servers in Copilot

1. Open the Command Palette (Cmd+Shift+P on Mac, Ctrl+Shift+P on Windows/Linux)
2. Type and select: Copilot: Configure MCP Servers
3. Enable both fetch and web-tools servers

6. Using the Tools in Copilot Chat

Open Copilot Chat in VS Code and use:

#web-tools get_page_title url=https://example.com

or

#web-tools search_new_relic_docs query="install java apm"

The search_new_relic_docs tool will return the top documentation results and a content sample from each page.

7. Troubleshooting

• If you see errors about uvx not found, ensure your Python environment’s bin directory is in your PATH.
• Check the VS Code Output panel (View > Output > Copilot) for logs.
• Ensure all dependencies are installed in the same Python environment VS Code is using.

8. Adding More Tools

To add more tools, define new functions in mcp_server_web_scraper.py using the @mcp.tool() decorator.

That’s it!
Your team can now use and extend the MCP web tools server with GitHub Copilot in VS Code. If you need more help, check the project README or ask the project maintainer.