What is Unstructured API MCP Server

The Unstructured MCP Server is a service designed for interacting with the Unstructured API, enabling users to manage and operate various data sources and workflows. It provides a set of tools to list, create, update, and delete source and destination connectors, as well as workflows that facilitate data processing tasks.

Use cases

The server can be utilized in various scenarios, including data ingestion from multiple sources like S3, Azure, and Google Drive, and exporting processed data to destinations like Pinecone and MongoDB. It’s particularly useful for automating workflows that integrate machine learning models with diverse data sources and destinations.

How to use

To set up the Unstructured MCP Server, users need to have Python 3.12+, an environment management tool like uv, and an API key from Unstructured. The server can be run directly using the provided command-line tools or through integration with applications like Claude Desktop. After configuration, users can call specific API endpoints to list sources, create workflows, and manage jobs effectively.

Key features

The Unstructured MCP Server boasts features like listing various sources and destinations, creating and managing workflows, running jobs asynchronously, and integrating with Firecrawl for web crawling and text generation tasks. It supports multiple connectors, ensuring flexibility in data handling.

Where to use

This server can be deployed in environments where automated data processing is required, such as data engineering pipelines, machine learning model training setups, and web data extraction tasks. It is suitable for developers and data scientists aiming to streamline their workflows and enhance data accessibility.

Unstructured API MCP Server

An MCP server implementation for interacting with the Unstructured API. This server provides tools to list sources and workflows.

Available Tools

Below is a list of connectors the UNS-MCP server currently supports, please see the full list of source connectors that Unstructured platform supports here and destination list here. We are planning on adding more!

To use the tool that creates/updates/deletes a connector, the credentials for that specific connector must be defined in your .env file. Below is the list of credentials for the connectors we support:

Firecrawl Source

Firecrawl is a web crawling API that provides two main capabilities in our MCP:

1. HTML Content Retrieval: Using invoke_firecrawl_crawlhtml to start crawl jobs and check_crawlhtml_status to monitor them
2. LLM-Optimized Text Generation: Using invoke_firecrawl_llmtxt to generate text and check_llmtxt_status to retrieve results

How Firecrawl works:

Web Crawling Process:

• Starts with a specified URL and analyzes it to identify links
• Uses the sitemap if available; otherwise follows links found on the website
• Recursively traverses each link to discover all subpages
• Gathers content from every visited page, handling JavaScript rendering and rate limits
• Jobs can be cancelled with cancel_crawlhtml_job if needed
• Use this if you require all the info extracted into raw HTML, Unstructured’s workflow cleans it up really well :smile:

LLM Text Generation:

• After crawling, extracts clean, meaningful text content from the crawled pages
• Generates optimized text formats specifically formatted for large language models
• Results are automatically uploaded to the specified S3 location
• Note: LLM text generation jobs cannot be cancelled once started. The cancel_llmtxt_job function is provided for consistency but is not currently supported by the Firecrawl API.

Note: A FIRECRAWL_API_KEY environment variable must be set to use these functions.

Installation & Configuration

This guide provides step-by-step instructions to set up and configure the UNS_MCP server using Python 3.12 and the uv tool.

Prerequisites

• Python 3.12+
• uv for environment management
• An API key from Unstructured. You can sign up and obtain your API key here.

Using uv (Recommended)

No additional installation is required when using uvx as it handles execution. However, if you prefer to install the package directly:

uv pip install uns_mcp

Configure Claude Desktop

For integration with Claude Desktop, add the following content to your claude_desktop_config.json:

Note: The file is located in the ~/Library/Application Support/Claude/ directory.

Using uvx Command:

{
  "mcpServers": {
    "UNS_MCP": {
      "command": "uvx",
      "args": [
        "uns_mcp"
      ],
      "env": {
        "UNSTRUCTURED_API_KEY": "<your-key>"
      }
    }
  }
}

Alternatively, Using Python Package:

{
  "mcpServers": {
    "UNS_MCP": {
      "command": "python",
      "args": [
        "-m",
        "uns_mcp"
      ],
      "env": {
        "UNSTRUCTURED_API_KEY": "<your-key>"
      }
    }
  }
}

Using Source Code

1. Clone the repository.

2. Install dependencies:

   uv sync
   

3. Set your Unstructured API key as an environment variable. Create a .env file in the root directory with the following content:

   UNSTRUCTURED_API_KEY="YOUR_KEY"
   

   Refer to .env.template for the configurable environment variables.

You can now run the server using one of the following methods:

uvx pip install -e .

{
  "mcpServers": {
    "UNS_MCP": {
      "command": "uvx",
      "args": [
        "uns_mcp"
      ]
    }
  }
}

{
  "mcpServers": {
    "UNS_MCP": {
      "command": "ABSOLUTE/PATH/TO/.local/bin/uv",
      "args": [
        "--directory",
        "ABSOLUTE/PATH/TO/YOUR-UNS-MCP-REPO/uns_mcp",
        "run",
        "server.py"
      ]
    }
  }
}

uv run python minimal_client/client.py uns_mcp/server.py

Additional Local Client Configuration

Configure the minimal client using environmental variables:

• LOG_LEVEL="ERROR": Set to suppress debug outputs from the LLM, displaying clear messages for users.
• CONFIRM_TOOL_USE='false': Disable tool use confirmation before execution. Use with caution, especially during development, as LLM may execute expensive workflows or delete data.

Debugging tools

Anthropic provides MCP Inspector tool to debug/test your MCP server. Run the following command to spin up a debugging UI. From there, you will be able to add environment variables (pointing to your local env) on the left pane. Include your personal API key there as env var. Go to tools, you can test out the capabilities you add to the MCP server.

mcp dev uns_mcp/server.py

If you need to log request call parameters to UnstructuredClient, set the environment variable DEBUG_API_REQUESTS=false.
The logs are stored in a file with the format unstructured-client-{date}.log, which can be examined to debug request call parameters to UnstructuredClient functions.

Add terminal access to minimal client

We are going to use @wonderwhy-er/desktop-commander to add terminal access to the minimal client. It is built on the MCP Filesystem Server. Be careful, as the client (also LLM) now has access to private files.

Execute the following command to install the package:

npx @wonderwhy-er/desktop-commander setup

Then start client with extra parameter:

uv run python minimal_client/client.py "http://127.0.0.1:8080/sse" "@wonderwhy-er/desktop-commander"
# or
make sse-client-terminal

Using subset of tools

If your client supports using only subset of tools here are the list of things you should be aware:

• update_workflow tool has to be loaded in the context together with create_workflow tool, because it contains detailed description on how to create and configure custom node.

Known issues

• update_workflow - needs to have in context the configuration of the workflow it is updating either by providing it by the user or by calling get_workflow_info tool, as this tool doesn’t work as patch applier, it fully replaces the workflow config.

CHANGELOG.md

Any new developed features/fixes/enhancements will be added to CHANGELOG.md. 0.x.x-dev pre-release format is preferred before we bump to a stable version.

Troubleshooting

• If you encounter issues with Error: spawn <command> ENOENT it means <command> is not installed or visible in your PATH:
  • Make sure to install it and add it to your PATH.
  • or provide absolute path to the command in the command field of your config. So for example replace python with /opt/miniconda3/bin/python