I built an AI agent that does real work on your PC: 5 things that broke along the way

Posted on Oct 17 • Originally published at kyboro.com

AI tools tell you what to do. I wanted one that actually does it — opens the browser, edits the files, runs the terminal command — while I watch.

That became Kyboro: you describe a task in plain English (typos and Hinglish welcome), and it runs it on your own machine.

The shell had a timeout, the WebSocket had one, the cloud task had one. When an outer timeout was shorter than an inner one, the cloud gave up while the command was still running — the task just vanished with no error.

Fix: a strict ladder — shell < socket < cloud. Every inner layer fails first and reports why.

To catch a stuck agent I hashed each tool call with all its arguments. But a stuck agent retries with tiny variations — a different timestamp, one changed word — so every attempt looked brand new.

Fix: hash the shape of the action, not its content.

Deleting files, sending messages, paying: the agent must stop and ask. But the decision is made in the cloud and the user is on the desktop. If the confirmation can't make the round trip, the agent either hangs forever or acts without asking. Both are unacceptable.

Fix: a dedicated confirm bridge, with a clear "no answer = no action" rule.

element.click() from injected JS gets ignored by a lot of modern sites, and half the buttons live inside shadow DOM where normal selectors can't reach.

Fix: trusted input through the Chrome DevTools Protocol, plus selectors that pierce shadow roots.

Asking users to reinstall for every UI fix kills retention. So UI files update by content hash — the app compares hashes on launch and pulls only what changed. The catch: anything packaged inside the Electron bundle still needs a full release, so you have to know which side of that line every file sits on.

Windows + Mac, plus a Chrome extension. It's early, and I'd love brutal feedback:

👉 Try one real task you'd normally do by hand, and tell me where it breaks.