Mtehabsim/ScreenPilot

Overview

A Python tool that enables an AI agent to control a device by interacting with the screen and interface just as a human would. It captures screen content and performs actions such as clicks and keystrokes to automate workflows.

Best for

Best for
Developers who need to automate GUI-heavy workflows using AI

Use cases

• Automate repetitive GUI tasks without manual scripting
• Perform end-to-end testing of desktop applications
• Automate data entry across multiple applications

How to use

Install

pip install -r requirements.txt

Tested with

Claude AI desktop

Example client config

{\n    "mcpServers": {\n        "device-controll": {\n            "command": "pathToEnv\\venv\\Scripts\\python.exe",\n            "args": [\n                "pathToProject\\ScreenPilot\\main.py"\n            ]\n        }\n    }\n}

Notes

A Python tool that enables an AI agent to control a device by interacting with the screen and interface just as a human would. It captures screen content and performs actions such as clicks and keystrokes to automate workflows.

53 stars on GitHub. Last updated 2025-08-15.

Use cases

• Automate repetitive GUI tasks without manual scripting
• Perform end-to-end testing of desktop applications
• Automate data entry across multiple applications

Pros

• Open source and free to use
• Works at the screen level, no API integration needed
• Written in Python, easy to extend or integrate

Cons

• Small community and limited stars, early stage
• May fail with dynamic or non-standard interfaces
• Requires screen access, not suitable for headless environments

Indexed from awesome-mcp-servers-punkpeye and enriched against its public facts.