What Claude Computer Use Actually Is

Strip the launch announcement away and Claude Computer Use is one capability bolted onto the standard Messages API. You send Claude a screenshot of a desktop, plus a goal written in natural language, and Claude returns structured tool calls that represent actions like clicking, typing, scrolling, or taking the next screenshot. Your code executes those actions against a real computer, captures the new screen state, and feeds it back to Claude. The model then sees what changed and decides the next step.

Under the hood it is the same chat completion loop you already know. The difference is that one of the available tools is a computer tool with sub-actions like left_click, type, screenshot, key, and scroll. The model emits calls to that tool instead of answering in prose. Your orchestration code does the actual work of moving the mouse, pressing keys, and reading the screen buffer.

A few things follow from this design. First, Claude is not “seeing” your actual computer unless you wire it up to one. It sees whatever image bytes you hand it, and it controls whatever input device you give it access to. Second, latency is real. Each round trip involves a screenshot upload, a model inference call, a tool execution, and another screenshot. Typical loop latency for this kind of agent sits in the 3 to 8 second range depending on screenshot size and model choice. Third, the model is not running a browser. It is pretending to be a user. Anything a human could do with a mouse and keyboard is in scope, which is broader and messier than most people expect.

In practice this means Computer Use is a control loop, not an API for “doing computer things.” Your job is to build the loop, choose the environment, and decide what Claude is and is not allowed to touch.

Setup and Authentication

You will need four things before any of this works. An Anthropic API key with billing enabled, the Anthropic Python SDK installed locally, a Linux or macOS machine the model can drive, and a way to take screenshots and synthesize input events.

Install the SDK with pip.

Then export your key as an environment variable so the SDK picks it up automatically.

For the screen and input layer you have three realistic options. The first is Anthropic’s hosted reference implementation, which spins up a Docker container with a virtual display, a browser, and the tooling already wired up. This is the fastest path to a working demo and lives in the anthropic-cookbook repository under computer-use-demo. The second option is a self-hosted container you build yourself, often using Xvfb for the virtual framebuffer, xdotool for input synthesis, and a screenshot tool like scrot or gnome-screenshot. The third option is a managed sandbox from providers such as E2B or Docker Sandbox, which give you a disposable Linux desktop over an API.

For most readers the cookbook demo is the right starting point. Clone it, set ANTHROPIC_API_KEY, run the provided Dockerfile, and you get a web UI plus a Python loop that drives a virtual machine. From there you can swap in your own environment or your own tools without rewriting the orchestration.

Authentication is the standard Anthropic pattern. No OAuth, no service accounts, no scopes beyond the API key itself. The model in use for this capability is a Claude variant with vision, currently accessed through the same messages endpoint as text models. Pricing per token is in the range you would expect for a vision-capable model, higher than text-only equivalents because every screenshot is processed as image input.

First Working Example

The minimal loop has four pieces. A system prompt that defines the agent’s job, a tool definition block, a call to messages.create, and a loop that handles tool calls and sends the next screenshot.

Below is the shape of the code in plain terms rather than as a runnable block, so you can drop it into your editor in your own style.

You start a conversation with a system prompt such as “You are a helpful assistant that can control a Linux desktop. Complete the user’s task by taking screenshots and issuing tool calls.” Then you add a tools array containing the computer tool definition. Anthropic publishes a reference tool schema in their docs and cookbook. The tool takes parameters like action, coordinate, and text depending on which sub-action the model wants to run.

On the first turn you send a user message that contains the goal, for example “Open Firefox and search for the weather in Berlin.” You also attach the initial screenshot as an image block. The model responds with a tool call, typically screenshot first to confirm orientation, followed by left_click on a coordinate, then type to enter text.

Your code parses the tool call, dispatches it to your local environment, and waits for the action to complete. Then it captures a fresh screenshot, appends it to the conversation as a new image block, and sends the request again. You keep looping until the model emits text content with no tool call, which signals the task is done.

Two implementation details matter. First, you must pass the conversation’s full message history back on every request. The model has no memory across calls, so the loop is its only continuity. Second, screenshots should be reasonably sized. Full 4K captures blow through the image token budget. Resizing to something in the 1280 by 800 to 1920 by 1080 range keeps costs and latency sane without hurting accuracy noticeably.

Key Settings That Matter

The defaults will get you a demo. The settings below are where real production use diverges from toy examples.

System prompt. This is your main steering wheel. The model is sensitive to phrasing. Telling it “always take a screenshot before clicking” or “never close windows you did not open” cuts down on a meaningful number of mistakes. Include any environment specifics too, like “the address bar is at the top of the browser” or “this desktop only has a terminal.”

Max tokens for the response. Computer Use tool calls are not large in themselves, but the model sometimes narrates its plan in prose before acting. Raising max tokens above the default gives it room to think. For complex tasks a value around 4096 to 8192 is typical.

Coordinate system. Screenshots come back in whatever resolution the capture tool produces. Claude returns coordinates in the same pixel space. If you capture at a different size than the model expects, clicks land in the wrong place. Either capture at a known fixed resolution or write a coordinate transformer. The reference demo uses 1280 by 800 with a scaling factor the docs explain.

Loop termination. Without a hard cap on iterations the loop can run forever, racking up cost. Set a max iteration count, a wall-clock timeout, and a token spend cap. The model will sometimes claim a task is done when it is not, so also require a final screenshot or a specific text marker before accepting completion.

Allowed actions. The computer tool exposes more actions than you usually want. You can pass a reduced action list to limit what the model is allowed to invoke. Restricting to screenshot, left_click, type, and key covers most browser-based work and removes the ability to do anything destructive.

Safety prompts. Anthropic’s reference prompts include guardrails that ask the model to refuse certain classes of action. Keep these in place. They are not perfect, but they are the cheapest defense layer you will get.

Model choice. The current version that supports Computer Use is documented in the API reference. Older Claude models will reject the tool schema. If you build the loop against the wrong model name every request will 400.

Where It Shines

Browser-based research is the obvious win. Tasks like “find the top three competitors for X, capture their pricing pages, and write a summary” work because the steps map cleanly onto clicks and scrolls, and the model has been exposed to a wide variety of web layouts during training.

Form filling across legacy web apps is a close second. When an internal tool has no API and no clean export, Computer Use lets you drive it the same way a human would. This is the kind of work that consumes hours of analyst time on systems nobody has time to integrate properly.

Visual QA is a genuine fit. You can have the model navigate to a staging environment, exercise a flow, and report visual problems. This is not a replacement for proper test automation, but for one-off checks it is much faster than writing Selenium.

Data entry from PDFs or images into structured systems is the most common pattern we see working in production. The model reads a document, then drives a CRM or ERP through its UI. The combination of vision and action is what makes this tractable.

Where It Fails

It is slow. Even well-tuned loops run roughly five to ten times slower than a human at the same task. For anything time sensitive, the math rarely works.

It hallucinates UI. The model sometimes clicks on a button that is not there, types into a field that does not exist, or reports success on a task it never completed. You need a verification step, which usually means another model call or a screenshot diff.

It is expensive at scale. Per-task token spend is in the tens of cents to low dollars range depending on iteration count. Fine for ten tasks a day. Painful at ten thousand.

It cannot reliably handle precise spatial tasks. Dragging a slider to an exact position, drawing a shape, or selecting a specific row in a dense table are all weak spots. Coordinate accuracy drifts as UI density increases.

It is bad at anything where one mistake is unrecoverable. If the wrong click publishes a post, deletes a record, or sends an email, you should not let an autonomous loop near it without a human-in-the-middle checkpoint.

It fails open. If the loop breaks or the screenshot is malformed, the model often improvises rather than failing clearly. Build error handling that surfaces these cases instead of swallowing them.

Practical Workflow Pattern

The pattern that actually works in production is staged. First you run Computer Use in a sandbox with no real data, no production credentials, and no ability to affect anything external. You treat the first iteration of any agent as a demo, not a deployment.

Second you add logging. Every screenshot, every tool call, every model output gets written to disk. When something goes wrong, which it will, you need the full transcript to debug it.

Third you wrap the loop with a supervisor. This is usually a smaller, cheaper model that reviews the screenshot after each action and decides whether the agent is on track, off track, or done. Off track means stopping and asking for help. Done means accepting the result.

Fourth you add a human checkpoint before any action that crosses a trust boundary. Saving a file, clicking a “submit” button, sending a message, all of it. The model proposes, the human disposes. This sounds slow until you realize most agent failures happen at exactly these moments.

Fifth you measure outcomes, not outputs. A successful loop ran 200 tool calls. So what. Did it produce the right artifact? Did it touch the right records? Build your metrics around task completion and accuracy, not around the agent itself.

This is the shape of work we keep coming back to with these tools. Pick a narrow job, give it a controlled environment, supervise it tightly, and only widen the scope once the failure modes are understood. Anything else is a demo dressed up as a product.

If you want the playbook other teams are using with Claude and Codex right now, grab the free Working With Claude field guide. Download it here.