What is Mcp Pyautogui

mcp-pyautogui is an MCP server designed for PyAutoGUI, enabling control over mouse, keyboard, and screen interactions for automation tasks on a host machine.

Use cases

Use cases for mcp-pyautogui include automating repetitive tasks, testing user interfaces, creating bots for data entry, and enhancing productivity in software development.

How to use

To use mcp-pyautogui, install it via pip with ‘pip install mcp-pyautogui’. After installation, configure it in your ‘claude_desktop_config.json’ file to integrate with your workflow.

Key features

Key features include mouse control (movements, clicks, drags), keyboard input simulation (single keys, text, hotkeys), screenshot capture, mouse position retrieval, and platform-specific logic handling.

Where to use

mcp-pyautogui can be used in fields such as desktop automation, AI development, and any application requiring automated interaction with graphical user interfaces.

MCP Server for PyAutoGUI

mcp-pyautogui implements a Model Context Protocol (MCP) server for PyAutoGUI. It provides control of mouse, keyboard, and screen interactions, allowing AI systems to automate tasks on a host machine.

The server runs over stdio transport and provides tools for clicking, typing, taking screenshots, and more, with a focus on simplicity and reliability. It’s designed for developers and AI enthusiasts looking to integrate desktop automation into their workflows.

Features

• Control mouse movements, clicks, and drags.
• Simulate keyboard input (single keys, text, hotkeys).
• Capture screenshots and retrieve mouse position.
• Get the current operating system for platform-specific logic.
• Consistent error handling with boolean success indicators (where applicable).

Prerequisites

• Python 3.11+

You’ll also have to grant the relevant permissions.

Installation

Use pip to install the server.

pip install mcp-pyautogui

You can then run the following comand to determine the location at which the server was installed:

which mcp-pyautogui

# Sample output:
# /Users/bob/miniconda3/envs/desk/bin/mcp-pyautogui

Use that location when you’re adding mcp-pyautogui to your claude_desktop_config.json file. Here’s a sample configuration:

{
  "mcpServers": {
    "mcp-pyautogui": {
      "command": "/Users/bob/miniconda3/envs/desk/bin/mcp-pyautogui",
      "args": [
        ""
      ]
    }
  }
}

Usage

If you followed the above installation steps, you don’t need to do anything else. Just use Claude Desktop and you should see all the tools this MCP server offers automatically. You could test this server by saying something like “can you click at (200,200) ?”.

The server runs over stdio transport, meaning it communicates through standard input/output. To use it, connect an MCP-compatible client (e.g., an AI assistant or custom script) that can send JSON-RPC commands and receive responses.

Example Commands

• Click at (100, 200): click(100, 200) → Returns true if successful.
• Type “hello”: type_text("hello") → Returns true.
• Take a screenshot: take_screenshot("screenshot.png") → Saves to screenshot.png.
• Get mouse position: get_mouse_position() → Returns (500, 300) (example).
• Copy text: hotkey("ctrl c") → Returns true.
• Get current operating system: get_os() → Returns “Windows” (if on Windows).

Full List of Available Tools

• click(x, y): Left-click at coordinates (x, y).
  • Parameters: x: int, y: int
  • Return Type: bool
• right_click(x, y): Right-click at coordinates (x, y).
  • Parameters: x: int, y: int
  • Return Type: bool
• move_to(x, y): Move mouse to coordinates (x, y) over 3 seconds.
  • Parameters: x: int, y: int
  • Return Type: bool
• drag_to(x, y, duration): Drag mouse to (x, y) over specified duration (default 1.0s).
  • Parameters: x: int, y: int, duration: float
  • Return Type: bool
• type_text(text): Type the given text.
  • Parameters: text: str
  • Return Type: bool
• press_key(key): Press and release a single key (e.g., 'enter', 'space', 'a').
  • Parameters: key: str
  • Return Type: bool
• hotkey(keys): Press multiple keys together (e.g., 'ctrl c'). Space-separated string.
  • Parameters: keys: str
  • Return Type: bool
• scroll(amount): Scroll up (positive) or down (negative) by amount.
  • Parameters: amount: int
  • Return Type: bool
• take_screenshot(filename): Save a screenshot to the specified file (e.g., 'screen.png').
  • Parameters: filename: str
  • Return Type: bool
• get_mouse_position(): Get current mouse coordinates as (x, y).
  • Parameters: None
  • Return Type: str
• get_os(): Get the current OS name (e.g., 'Windows', 'macOS', 'Linux').
  • Parameters: None
  • Return Type: str
• double_click(x, y): Perform a double-click at the given (x, y) coordinates.
  • Parameters: x: int, y: int
  • Return Type: bool
• get_screen_size(): Get the screen resolution as (width, height).
  • Parameters: None
  • Return Type: str
• pixel_color(x, y): Get the RGB color of the pixel at (x, y). Returns (r, g, b).
  • Parameters: x: int, y: int
  • Return Type: str