A few months ago, I started building Saarthi because I wanted my coding agents to see and interact with my desktop.
At that time, while building projects, I would ask my agent to fix a few issues on a website, and it would change the code and confirm that the task was done. But I still had to open the site myself to check whether the changes had worked. I was really frustrated and started to wonder if my agent could verify a task itself after performing it, without my help.
Around the same time, I saw demos from the OpenAI team showing Codex using a computer and wondered if I could build something similar for my own desktop. So I decided to try building one for myself.
In this post, I’ll introduce computer use and explore its building blocks through Saarthi.
What is Computer Use?
Before we get into how Saarthi works, let’s clarify what we mean by computer use.
Computer use is the ability of an AI agent to observe a computer’s interface and use tools to interact with it, for example by clicking, typing, or switching between applications.
An agent might observe the interface through screenshots, accessibility information, or a combination of both. Its implementation varies across products.
Where Does Browser Use Fit?
Browser use is when an AI agent carries out tasks within a web browser, such as navigating pages, filling forms, or clicking links.
Browser use focuses on the browser. Computer use extends to the desktop and other applications. In short, browser use can be considered a subset of computer use.
How Does Computer Use Work?
Now that we’ve covered what computer use means, let’s look at how an agent actually uses a computer. When you give it a task, it needs a way to see what’s happening, then interact with the computer and check the results. We can think of that process as a loop:
- Observe: The agent gets information about the current state of the computer from screenshots and other input data sources.
- Decide: It uses that information and the user’s request to decide what to do next.
- Act: It calls a tool to perform an action, such as clicking a button, typing text, or pressing a key.
- Check: The agent looks at the result to see whether the action worked. Based on the result, it can then continue, try something else, or just stop.
What Makes Computer Use Tricky?
The steps above make computer use sound straightforward, but making them work reliably takes more than just putting them in a loop. The agent needs a way to notice when a page changes, wait for it to finish loading, and check whether its last action had the intended effect. Even a click in the right place doesn’t guarantee that the task succeeded.
These were some of the problems I had to think about while building Saarthi. Let’s start with the desktop it runs on, then look at how it observes and interacts with it.
The Desktop Saarthi Was Built For
I built Saarthi for my Linux desktop, which runs Hyprland. If you haven’t come across it before, Hyprland is a Wayland compositor that manages things like windows and workspaces.
Hyprland provides a way to query and control parts of the desktop. Saarthi uses some of those commands, along with other local tools, to help an agent observe the desktop and interact with it.
When an agent calls Saarthi, each command runs as a separate process and returns a result. The agent can then use that result to decide what to do next.
Observe: What Can the Agent See?
Before an agent can perform an action, it needs some idea of what’s currently on the screen. Saarthi gives it a few ways to gather that information:
- Screenshots give the agent a view of the screen or a particular window.
- Accessibility information can describe interface elements exposed by an application, including their names, roles, and sometimes their locations.
- OCR can find visible text in a screenshot and help locate it on the screen.
For example, if the agent is looking for a “Save” button, it might find it through the application’s accessibility information. If that information isn’t available, it could look for the word “Save” using OCR or inspect a screenshot.
Each approach gives the agent a different view of the desktop. Once it has that information, it can decide what to do next.
Decide: What Should the Agent Do Next?
Once the agent has information about the desktop, it uses that information and the user’s request to choose its next step.
This is where the agent and Saarthi have different roles. The agent does the reasoning and decides what to do. Saarthi provides the interface it can use to observe and interact with the desktop. A skill helps the agent to understand which commands are available and how to use them.
For example, if the agent found the “Save” button in the previous step, it might decide to click it. It then asks Saarthi to carry out that action, bringing us to the next part of the loop.
Act: From a Decision to an Action
Once the agent has decided what to do, it needs a way to interact with the desktop. That’s where Saarthi comes in.
For window and workspace operations, Saarthi uses selected Hyprland commands. For typing, it uses wtype, and for moving the pointer and clicking, it uses input tools such as ydotool. Saarthi checks the requested operation, carries it out, and then returns a result to our agent.
Finding the Right Place to Click
One problem I ran into was how to make it easier for an agent to choose a location on the screen. The approach I chose was to put a numbered grid over a screenshot.
The number of rows and columns can be adjusted depending on how precise the agent needs to be. The agent can look at the image, pick the cell containing its target, and then ask Saarthi to click that cell. Saarthi then maps the number to the cell’s center and sends the click action.
For example, if the agent sees a power button in cell 25, it can ask Saarthi to click cell 25. The grid helps translate that choice into its screen coordinates.
Once Saarthi has sent an action, there’s still one question left: did it actually work? That brings us to the last step.
Check: Did It Work?
Sending a click is one thing. Knowing whether it did what the agent intended is another.
After an action, the agent needs a fresh observation to check the result. If it clicked “Save,” for example, it might look for a “Saved” message or another visible change in the application.
Saarthi provides a few ways to help with this. It can use OCR to wait for text to appear or disappear, compare screenshots until the screen stops changing, and capture the screen before and after an action.
The agent can use those results to create a timeline of events and decide what comes next. It can either continue with the task, retry the action, or stop, reassess the situation, and inform the user.
How It All Fits Together
The agent decides what to do, and Saarthi gives it ways to observe and interact with my desktop. Together, they can work through a task one step at a time.
That’s what Saarthi looks like. It reached this point gradually as I added tools and changed how the agent could use them, and it has a long way to go.
How the Design Grew
Saarthi didn’t start with all of these pieces in place. The early work focused on the basics, like managing workspaces and checking what happened after an action.
Over the next few days, I added accessibility inspection, ways to wait for text or for the screen to become stable, and mouse controls. Browser and terminal controls followed, giving the agent more ways to work with the applications I used.
The way agents accessed Saarthi changed too. It started with an MCP interface, but I later moved it to a CLI. I’ll get into why I made that change shortly.
Why I Moved from MCP to a CLI
Saarthi originally exposed its tools through MCP. That worked, but as I added more tools, their descriptions and schemas took up more of the agent’s context, even when it only needed a few of them for a task.
I wanted the agent to discover commands as it needed them. So I moved Saarthi to a CLI: the agent could start with a single entry point and ask for help with a particular command when necessary. The CLI still uses the same underlying handlers and validation schemas. This decision was also inspired by seeing many MCP tools change to CLIs and the rise in the use of skills.
The switch came with trade-offs. Screenshots now return as file paths instead of inline images, and some state that used to live in the MCP server had to move into local files. For my use case, the flexibility of loading command details when needed made the change worthwhile.
Closing Thoughts
Saarthi started with a simple question: could I give my coding agents eyes to see and hands to help them check their own work? Building it gave me a better understanding of what it takes to make an agent interact with a desktop.
As of October 2026, when I’m writing this post, computer use has progressed rapidly and has become a common tool integrated with coding agents.
I hope this post gives you a starting point for understanding computer use and experimenting with it on your own desktop.

