What is Automation Mcp

Automation MCP is a Model Context Protocol (MCP) server that enables AI models to perform complete desktop automation on macOS, allowing control over mouse, keyboard, screen, and window management.

Use cases

Use cases include automating repetitive tasks, testing applications by simulating user interactions, creating AI-driven assistants that can navigate and control the desktop, and performing data analysis through screen content capture.

How to use

To use Automation MCP, install the furi tool and run ‘furi add ashwwwin/automation-mcp’ followed by ‘furi start automation-mcp’. Alternatively, clone the repository, install dependencies using Bun, and start the server with ‘bun run index.ts’.

Key features

Key features include mouse control (click, move, scroll, drag), keyboard input (including shortcuts), screenshot capabilities, window management (focus, move, resize, minimize), UI element interaction, screen color analysis, and image detection.

Where to use

Automation MCP can be used in various fields such as software testing, automated user interface interactions, AI assistant development, and any task requiring automated control of macOS environments.

🤖 Automation MCP

Automation MCP is a Model Context Protocol (MCP) server that provides AI models with complete desktop automation capabilities on macOS. It enables AI assistants to:

• 🖱️ Control your mouse (click, move, scroll, drag)
• ⌨️ Type and send keyboard input (including system shortcuts)
• 📸 Take screenshots and analyze screen content
• 🪟 Manage windows (focus, move, resize, minimize)
• 🎯 Interact with UI elements through coordinates
• 🎨 Analyze screen colors and highlight regions
• 🔍 Wait for images to appear on screen

🚀 Quick start

Make sure you have furi installed, and then run the following command:

furi add ashwwwin/automation-mcp

followed by:

furi start ashwwwin/automation-mcp

and you’re done! (or you can just use the furi desktop app for no cli).

🥲 Normal start (without furi)

Prerequisites

• Bun runtime - Install with: curl -fsSL https://bun.sh/install | bash

1. Clone and Install

git clone https://github.com/ashwwwin/automation-mcp.git
cd automation-mcp
bun install

2. Start the Server

# Start with HTTP transport (recommended for web apps)
bun run index.ts

# Or start with stdio transport (for command line tools)
bun run index.ts --stdio

3. Grant Permissions

On first run, macOS will ask for permissions. You must grant these for full functionality:

1. Accessibility - Allows keyboard/mouse control
2. Screen Recording - Enables screenshots and screen analysis

Or manually enable in: System Settings → Privacy & Security → Accessibility/Screen Recording

🛠️ Available Tools

🖱️ Mouse Control

• mouseClick - Click at coordinates with left/right/middle button
• mouseDoubleClick - Double-click at coordinates
• mouseMove - Move cursor to position
• mouseGetPosition - Get current cursor location
• mouseScroll - Scroll in any direction
• mouseDrag - Drag from current position to target
• mouseButtonControl - Press/release mouse buttons
• mouseMovePath - Follow a smooth path with multiple points

⌨️ Keyboard Input

• type - Type text or press key combinations
• keyControl - Advanced key press/release control
• systemCommand - Common shortcuts (copy, paste, undo, save, etc.)

📸 Screen Capture & Analysis

• screenshot - Capture full screen, regions, or specific windows
• screenInfo - Get screen dimensions
• screenHighlight - Highlight screen regions visually
• colorAt - Get color of any pixel
• waitForImage - Wait for images to appear (template matching)

🪟 Window Management

• getWindows - List all open windows
• getActiveWindow - Get current active window
• windowControl - Focus, move, resize, minimize windows

🔒 Security & Permissions

1. Accessibility - Required for:

   • Mouse clicks and movement
   • Keyboard input simulation
   • Window management

2. Screen Recording - Required for:

   • Taking screenshots
   • Screen analysis
   • Color detection

🚀 Integration Examples

With Claude Desktop + furi

If you’ve already configured furi with Claude Desktop, you don’t need to do anything.

Add to your MCP configuration:

{
  "mcpServers": {
    "furi": {
      "command": "furi",
      "args": [
        "connect"
      ]
    }
  }
}

With Claude Desktop

Add to your MCP configuration:

{
  "mcpServers": {
    "automation": {
      "command": "bun",
      "args": [
        "run",
        "/path/to/automation-mcp/index.ts",
        "--stdio"
      ]
    }
  }
}

🐛 Troubleshooting

Common Issues

Permission Denied Errors

• Ensure Accessibility and Screen Recording permissions are granted
• Ensure Xcode Command Line Tools: xcode-select --install

🙋‍♂️ Support

Having issues? Check the troubleshooting section above or open an issue with:

• Your operating system and version
• Error messages
• Steps to reproduce