What is Screenpilot

ScreenPilot is an MCP server designed to give Large Language Models (LLMs) full control over your device by providing a screen automation toolkit for interacting with graphical user interfaces.

Use cases

Use cases for ScreenPilot include automating repetitive tasks, creating educational tools, and developing fun interactive applications that require user interface manipulation.

How to use

To use ScreenPilot, install Python 3.12, clone the repository, create and activate a virtual environment, install the required packages, and configure the MCP server settings in Claude AI desktop.

Key features

Key features include screen capture and analysis, mouse control (clicking and positioning), keyboard input (typing, key presses, and hotkeys), scrolling, element detection, and action sequences.

Where to use

ScreenPilot can be used in various fields such as automation, education, and entertainment, where interaction with graphical user interfaces is required.

ScreenPilot

MCP server to let LLM take full control on your device by providing screen automation toolkit for controlling and interacting with graphical user interfaces. Good for automation, education and having fun.

Main Features

• 📷 Screen capture and analysis
• 🖱️ Mouse control (clicking, positioning)
• ⌨️ Keyboard input (typing, key presses, hotkeys)

watch demo

https://github.com/user-attachments/assets/c18380c0-b3dd-4b7c-925d-28ef205ca11f

Installation

0. Install python 3.12
1. Clone the repository:

   git clone https://github.com/Mtehabsim/ScreenPilot.git
   

2. create virtiual environment

python -m venv venv

3. activate the env

venv\Scripts\activate

4. Install the required packages:

   pip install -r requirements.txt
   

5. Open Claude AI desktop
6. file -> settings -> developer -> edit config
7. open config file and paste this

{
    "mcpServers": {
        "device-controll": {
            "command": "pathToEnv\\venv\\Scripts\\python.exe",
            "args": [
                "pathToProject\\ScreenPilot\\main.py"
            ]
        }
    }
}

8. Replace
       “pathToEnv\venv\Scripts\python.exe” → with the full path to your python.exe
       “pathToProject\ScreenPilot\main.py” → with the full path to your main.py file

9. Save the config file.

10. Open Claude AI Desktop.

11. Go to File → Exit

12. You can now open Claude AI Desktop and enjoy ScreenPilot.

Available Tools

• Screen Capture: Take screenshots and get screen information
• Mouse Control: Move the mouse and perform clicks
• Keyboard Actions: Type text, press keys, and use hotkey combinations
• Scrolling: Scroll in different directions and to specific positions
• Element Detection: Check if elements exist on screen and wait for them to appear
• Action Sequences: Perform multiple actions in sequence

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.