UI Automation for AI Agents: Why I Run It on Separate VMs

Why does UI automation for AI agents need its own machine?

When an agent drives a browser or a native app on the computer I am sitting at, it takes over the mouse and the keyboard focus. I cannot type into another window without breaking its click. So I stop working and watch it, which defeats the point of handing the task over.

A virtual machine fixes that at the root. The agent gets its own screen, its own browser and its own logged-in accounts, and my desktop stays mine. My rule: if a task needs a screen, it goes to the pool. My own desktop is for the work I do myself.

The pool runs Codex, OpenAI’s coding agent, and it has nine hosts. The sections below cover what they are, how a job gets onto one, and the places where the arrangement still fails.

What does the pool of nine hosts look like?

Six hosts run Linux and three run macOS. They differ, and the differences decide which job goes where.

Hosts	System	Where it runs	Screen control
3	Linux desktop	Containers on a rented server	No
3	Linux desktop	Virtual machines on my Mac	No
2	macOS	Virtual machines on my Mac	Yes
1	macOS 15 on Intel	A 2018 MacBook Pro in another home	No, command line only

Every host has a browser and a shell. Only the two macOS virtual machines carry the computer-use components that let an agent take screenshots and click in native apps. Apple allows two macOS guests to run at once on one Mac (the limit is in the macOS license and enforced by its Virtualization framework), so those two are the full set.

The job store is a single list on my Mac. One job runs per host at a time.

How do I start a job on the pool?

One command. A small dispatcher script I wrote picks a free host, starts the task there and tracks it until it finishes. I can ask it which hosts are free or busy, send a task and wait for the answer, restrict a task to the two macOS machines that can control a screen, or mark a host as taken while I use it myself. If no host is free, it says so and I wait.

Routing follows a fixed order. A request for screenshots, clicks or a native macOS app implies the computer-use group, and no flag can send it elsewhere. Other work goes to the container desktops with the most free memory first, then the Linux virtual machines, then the macOS ones, and last to the old laptop.

Each host also stays signed in to the accounts the agent needs, so a job does not start at a login screen. A person does that sign-in once per host, by hand.

How do I know a host can finish a job?

A host that is up and free can still fail a job. A health check, run from my Mac, reads each host and reports whether the agent is signed in, the browser is running and the machine will not go to sleep. It changes nothing.

A background job runs it every 30 minutes and sends a macOS notification that names the host and the failed check, but only for failures that are new since the last run. A virtual machine I have deliberately left switched off does not trigger anything. A pass is not a guarantee: a job can still stop at a sign-in prompt the check did not see.

Where does this setup break or cost me?

Jobs end in one of three ways: done, aborted or lost. Aborted means a person interrupted it or it ran past the dispatcher’s 30-minute ceiling. Lost means the run died without reporting an end, which usually means a container desktop ran out of memory. I retry once, and I check what a job changed before retrying one that wrote anything.

A person sitting at a desktop is invisible to the busy count, so I mark a host as taken before I work on it. Captchas on the container desktops are still mine to solve.

Computer use is the most fragile piece. In early October one of the two macOS machines could not use it because of a setting on the vendor’s side, so it took general work only. I leave that setting alone rather than work around it.

Windows is the other gap. Windows 11 machines are planned for the pool, but the dispatcher cannot send work to Windows yet. Anyone who tests Windows software needs that third system as well.

I have not measured how much time this saves. What I can count is hosts: up to nine jobs can run while I do something else.

When should an agent touch a screen at all?

Less often than the pool might suggest. I keep a standing rule that UI automation never creates, updates or deletes data. Changes go through a service’s documented API or command line, and if the API is too narrow I extend it first. A screen is for read-only inspection and testing.

So the pool does the work that truly needs eyes on a rendered page or a native window, such as checking how a page looks or reading an app that has no interface other than its own. Repeatable work belongs on the command line, which is the point of my practice of running work through CLIs and APIs.

Who is this for, and how do I start in 15 minutes?

It fits someone who already hands browser or desktop tasks to an agent and is tired of losing their own screen while it runs. It is overkill if your agents only touch files and APIs.

To try the smallest version, create one virtual machine on a computer you already own, in whatever tool you use for VMs. Install your agent and a browser in it, sign in to the accounts a typical task needs, and turn off sleep. Then send your next screen-driven task there and keep working on your own desktop. Add a second host and a dispatcher only after the first one has proved useful, and add a health check as soon as a job fails on a host you assumed was ready.

This page is part of the Compounding CEO series, and the matching practice card is Run repeatable work through CLIs and APIs. A related page covers which agent actions go out without my review.

Frequently Asked Questions

Why not let an AI agent control my own desktop?

An agent that drives a browser or a native app on your desktop takes over the mouse and the keyboard focus, so you stop working and watch. On a separate VM the agent has its own screen and you keep yours. That is the reason I send every task that needs a screen to a pool of nine hosts.

Which operating systems does the pool cover?

Six hosts run Linux and three run macOS. Windows is the gap. I authorized two Windows 11 instances on October 5, 2026, but both were still pending setup at that point and are not yet available as job targets.

Can every host in the pool control a screen?

No. Only the two macOS virtual machines on my Mac are set up for screen control. The Linux hosts run shell and browser work, and the physical Intel MacBook takes command-line jobs only.

How do I know a host is ready before I send a job?

A health check runs across the pool every 30 minutes and reports, per host, whether the agent is logged in, the app and browser are present and the machine will not go to sleep. It names the host and the failing check, and I get a notification when a failure is new.