What is Mymcpautogui

MyMcpAutoGui is a GUI automation tool that empowers AI agents by providing them with the ability to interact with desktop applications through the Model Context Protocol (MCP). It acts as a bridge between MCP-enabled clients and the user’s graphical user interface.

Use cases

Use cases include automating GUI testing processes, enabling AI agents to perform tasks on behalf of users, and creating intelligent assistants that can manage desktop applications effectively.

How to use

To use MyMcpAutoGui, simply install it and run a server command to start the integration with your MCP-compatible client. The setup is straightforward and requires minimal configuration.

Key features

Key features include comprehensive GUI control, screen perception capabilities, window management, user interaction tools, and seamless integration with MCP clients like Cursor.

Where to use

MyMcpAutoGui can be used in various fields such as software testing, automation of repetitive tasks, and enhancing AI assistants for better user interaction.

PyMCPAutoGUI 🖱️⌨️🖼️ - GUI Automation via MCP

Supercharge your AI Agent’s capabilities! ✨ PyMCPAutoGUI provides a bridge between your AI agents (like those in Cursor or other MCP-compatible environments) and your computer’s graphical user interface (GUI). It allows your agent to see the screen 👁️, control the mouse 🖱️ and keyboard ⌨️, and interact with windows 🪟, just like a human user!

Stop tedious manual GUI tasks and let your AI do the heavy lifting 💪. Perfect for automating repetitive actions, testing GUIs, or building powerful AI assistants 🤖.

🤔 Why Choose PyMCPAutoGUI?

• 🤖 Empower Your Agents: Give your AI agents the power to interact directly with desktop applications.
• ✅ Simple Integration: Works seamlessly with MCP-compatible clients like the Cursor editor. It’s plug and play!
• 🚀 Easy to Use: Get started with a simple server command. Seriously, it’s that easy.
• 🖱️⌨️ Comprehensive Control: Offers a wide range of GUI automation functions from the battle-tested PyAutoGUI and PyGetWindow.
• 🖼️ Screen Perception: Includes tools for taking screenshots and locating images on the screen – let your agent see!
• 🪟 Window Management: Control window position, size, state (minimize, maximize), and more. Tidy up that desktop!
• 💬 User Interaction: Display alert, confirmation, and prompt boxes to communicate with the user.

🛠️ Supported Environments

• Operating Systems: Windows, macOS, Linux (Requires appropriate dependencies for pyautogui on each OS)
• Python: 3.11+ 🐍
• MCP Clients: Cursor Editor, any client supporting the Model Context Protocol (MCP)

🚀 Getting Started - It’s Super Easy!

1. Installation (Recommended: Use a Virtual Environment!)

Using a virtual environment keeps your project dependencies tidy.

# Create and activate a virtual environment (example using venv)
python -m venv .venv
# Windows PowerShell
.venv\Scripts\Activate.ps1
# macOS / Linux bash
source .venv/bin/activate

# Install using pip (from PyPI or local source)
# Make sure your virtual environment is active!
pip install pymcpautogui # Or pip install . if installing from local source

(Note: pyautogui might have system dependencies like scrot on Linux for screenshots. Please check the pyautogui documentation for OS-specific installation requirements.)

2. Running the MCP Server

Once installed, simply run the server from your terminal:

# Make sure your virtual environment is activated!
python -m pymcpautogui.server

The server will start and listen for connections (defaulting to port 6789). Look for this output:

INFO:     Started server process [XXXXX]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:6789 (Press CTRL+C to quit)

Keep this terminal running while you need the GUI automation magic! ✨

✨ Seamless Integration with Cursor Editor

Connect PyMCPAutoGUI to Cursor (@ symbol) for GUI automation directly within your coding workflow.

1. Open MCP Configuration: In Cursor, use the Command Palette (Ctrl+Shift+P or Cmd+Shift+P) and find “MCP: Open mcp.json configuration file”.

2. Add PyMCPAutoGUI Config: Add or merge this configuration into your mcp.json. Adjust paths if needed (especially if Cursor isn’t running from the project root).

   (Tip: If mcp.json already exists, just add the "PyMCPAutoGUI": { ... } part inside the mcpServers object.)

3. Save mcp.json. Cursor will detect the server.

4. Automate! Use @PyMCPAutoGUI in Cursor chats:

   Example:
   @PyMCPAutoGUI move_to(x=100, y=200)
@PyMCPAutoGUI write(text='Automating with AI! 🎉', interval=0.1)
@PyMCPAutoGUI screenshot(filename='current_screen.png')
@PyMCPAutoGUI activate_window(title='Notepad')

🧰 Available Tools

PyMCPAutoGUI exposes most functions from pyautogui and pygetwindow. Examples include:

• Mouse 🖱️: move_to, click, move_rel, drag_to, drag_rel, scroll, mouse_down, mouse_up, get_position
• Keyboard ⌨️: write, press, key_down, key_up, hotkey
• Screenshots 🖼️: screenshot, locate_on_screen, locate_center_on_screen
• Windows 🪟: get_all_titles, get_windows_with_title, get_active_window, activate_window, minimize_window, maximize_window, restore_window, move_window, resize_window, close_window
• Dialogs 💬: alert, confirm, prompt, password
• Config ⚙️: set_pause, set_failsafe

For the full list and details, check the pymcpautogui/server.py file or use @PyMCPAutoGUI list_tools in your MCP client.