What is Mac Use

mac-use is a project designed to enable native macOS control for AI applications, specifically for managing disparate agents through local computer use.

Use cases

Use cases include automating tasks on macOS, controlling applications through AI, and integrating various LLMs for enhanced productivity and system management.

How to use

To use mac-use, clone the repository, set up a virtual environment, install dependencies, configure your API keys in a .env file, and run the Streamlit app to access the interface.

Key features

Key features include native macOS GUI interaction, screen capture, keyboard and mouse control, support for multiple LLM providers, a Streamlit-based interface, automatic screen resolution scaling, and file system interaction capabilities.

Where to use

mac-use is applicable in fields such as AI development, automation, and any scenario requiring direct interaction with macOS systems for managing AI agents.

overlord

AI Overlord, managing your disparate agents through local computer use. This project allows AI to control macOS natively, providing direct system control through native macOS commands and utilities.

[!CAUTION]
This comes with obvious risks. Overlord can control everything on your Mac. Please be careful.

Features

• Native macOS GUI interaction (no Docker required)
• Screen capture using native macOS commands
• Keyboard and mouse control through cliclick
• Multiple LLM provider support (Anthropic, Bedrock, Vertex)
• Streamlit-based interface
• Automatic screen resolution scaling
• File system interaction and editing capabilities

Prerequisites

• macOS Sonoma 15.7 or later
• Python 3.12+
• Homebrew (for installing additional dependencies)
• cliclick (brew install cliclick) - Required for mouse and keyboard control

Setup Instructions

1. Clone the repository and navigate to it:

git clone https://github.com/hanzoai/overlord.git
cd overlord

2. Create and activate a virtual environment:

python3.12 -m venv venv
source venv/bin/activate

3. Run the setup script:

chmod +x setup.sh
./setup.sh

4. Install Python requirements:

pip install -r requirements.txt

Running the Demo

Set up your environment and Anthropic API key

1. In a .env file add:

API_PROVIDER=anthropic
ANTHROPIC_API_KEY=<key>
WIDTH=800
HEIGHT=600
DISPLAY_NUM=1

Set the screen dimensions (recommended: stay within XGA/WXGA resolution), and put in your key from Anthropic Console.

2. Start the Streamlit app:

streamlit run streamlit.py

The interface will be available at http://localhost:8501

Screen Size Considerations

We recommend using one of these resolutions for optimal performance:

• XGA: 1024x768 (4:3)
• WXGA: 1280x800 (16:10)
• FWXGA: 1366x768 (~16:9)

Higher resolutions will be automatically scaled down to these targets to optimize model performance. You can set the resolution using environment variables:

export WIDTH=1024
export HEIGHT=768
streamlit run streamlit.py

[!IMPORTANT]
The Beta API used in this reference implementation is subject to change. Please refer to the API release notes for the most up-to-date information.