NON906/omniparser-autogui-mcp

Overview

A Python tool for automating on-screen GUI operations by parsing visual elements. It uses omniparser and MCP to interpret screen content and execute actions. Designed for developers to script GUI workflows without native API support.

Best for

Best for
Developers needing visual-based GUI automation for non-standard or legacy applications

Use cases

• Automate repetitive desktop workflows
• Control legacy or inaccessible GUI applications
• Integrate GUI automation into larger MCP-based systems

How to use

Tools exposed

• git
• uv

Example client config

{\n  "mcpServers": {\n    "omniparser_autogui_mcp": {\n      "command": "uv",\n      "args": [\n        "--directory",\n        "D:\\CLONED_PATH\\omniparser-autogui-mcp",\n        "run",\n        "omniparser-autogui-mcp"\n      ],\n      "env": {\n        "PYTHONIOENCODING": "utf-8",\n        "OCR_LANG": "en"\n      }\n    }\n  }\n}

Notes

A Python tool for automating on-screen GUI operations by parsing visual elements. It uses omniparser and MCP to interpret screen content and execute actions. Designed for developers to script GUI workflows without native API support.

67 stars on GitHub. Last updated 2025-03-02. Licensed MIT.

Use cases

• Automate repetitive desktop workflows
• Control legacy or inaccessible GUI applications
• Integrate GUI automation into larger MCP-based systems

Pros

• Open source and extensible in Python
• Visual parsing approach works across different GUI frameworks
• Integrates with MCP for model-controlled automation

Cons

• Limited community and documentation due to low star count
• May struggle with dynamic or non-standard UI elements
• Requires careful setup for specific screen resolutions and layouts

Indexed from awesome-mcp-servers-punkpeye and enriched against its public facts.