What is Web Scout Mcp

web-scout-mcp is a powerful MCP server extension that provides web search and content extraction capabilities, integrating DuckDuckGo search functionality and URL content extraction into your MCP environment.

Use cases

Use cases include developing AI assistants that can search the web for information, extracting relevant content from multiple web pages for research purposes, and automating data collection from online sources.

How to use

To use web-scout-mcp, install it globally or locally via npm, and then run the command ‘web-scout-mcp’ in the command line. You can also configure it in your MCP client’s config.json file.

Key features

Key features include DuckDuckGo search for fast and privacy-focused web searches, content extraction for clean text from web pages, parallel processing for multiple URL handling, memory optimization, rate limiting to avoid API blocks, and robust error handling.

Where to use

web-scout-mcp can be used in various fields such as AI development, web scraping, data analysis, and any application that requires web search and content extraction functionalities.

Web Scout MCP Server

An MCP server for web search using DuckDuckGo and content extraction, with support for multiple URLs and memory optimizations.

✨ Features

• 🔍 DuckDuckGo Search: Fast and privacy-focused web search capability
• 📄 Content Extraction: Clean, readable text extraction from web pages
• 🚀 Parallel Processing: Support for extracting content from multiple URLs simultaneously
• 💾 Memory Optimization: Smart memory management to prevent application crashes
• ⏱️ Rate Limiting: Intelligent request throttling to avoid API blocks
• 🛡️ Error Handling: Robust error handling for reliable operation

📦 Installation

Installing via Smithery

To install Web Scout for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @pinkpixel-dev/web-scout-mcp --client claude

Global Installation

npm install -g @pinkpixel/web-scout-mcp

Local Installation

npm install @pinkpixel/web-scout-mcp

🚀 Usage

Command Line

After installing globally, run:

web-scout-mcp

With MCP Clients

Add this to your MCP client’s config.json (Claude Desktop, Cursor, etc.):

{
  "mcpServers": {
    "web-scout": {
      "command": "npx",
      "args": [
        "-y",
        "@pinkpixel/web-scout-mcp"
      ]
    }
  }
}

🧰 Tools

The server provides the following MCP tools:

🔍 DuckDuckGoWebSearch

Initiates a web search query using the DuckDuckGo search engine and returns a well-structured list of findings.

Input:

• query (string): The search query string
• maxResults (number, optional): Maximum number of results to return (default: 10)

Example:

{
  "query": "latest advancements in AI",
  "maxResults": 5
}

Output:
A formatted list of search results with titles, URLs, and snippets.

📄 UrlContentExtractor

Fetches and extracts clean, readable content from web pages by removing unnecessary elements like scripts, styles, and navigation.

Input:

• url: Either a single URL string or an array of URL strings

Example (single URL):

{
  "url": "https://example.com/article"
}

Example (multiple URLs):

{
  "url": [
    "https://example.com/article1",
    "https://example.com/article2"
  ]
}

Output:
Extracted text content from the specified URL(s).

🛠️ Development

# Clone the repository
git clone https://github.com/pinkpixel-dev/web-scout-mcp.git
cd web-scout-mcp

# Install dependencies
npm install

# Build
npm run build

# Run
npm start

📚 Documentation

For more detailed information about the project, check out these resources:

• OVERVIEW.md - Technical overview and architecture
• CONTRIBUTING.md - Guidelines for contributors
• CHANGELOG.md - Version history and changes

📋 Requirements

• Node.js >= 18.0.0
• npm or yarn

📄 License

This project is licensed under the MIT License.

_{Made with ❤️ by Pink Pixel
✨ Dream it, Pixel it ✨}