I Built Chunks: an On-Device AI That Sees My Mac Screen and Acts Without Stealing My Mouse

This is a submission for the MLH x DEV Writing Challenge

Chunks is a native macOS menu-bar and notch companion. You talk to it, it looks at your screen, explains what's going on, and can act on your apps for you. And it does all of that without grabbing your cursor or sending your screen to a cloud server.

I built it because the "computer use" agents I tried annoyed me. They warp your physical mouse, so you can't touch your laptop while they work. They take 30 to 60 seconds per step. They ship full-screen screenshots off to someone else's backend. Meanwhile, normal AI chat sits in a browser tab and knows nothing about the app you're stuck in.

I wanted something that felt like a friend sitting next to me: it sees what I see, points at the right button, and does the boring clicks only when I say yes.

Stack: Swift 6, AppKit, SwiftUI, ScreenCaptureKit, ApplicationServices (AXUIElement), AVFoundation, Speech.framework, Ollama, Gemini API.

It's backed by a dependency-free suite of 710+ unit tests and live OS probes ( --probe-local, --probe-ax, --probe-semantic ).

60-second demo: https://drive.google.com/file/d/1gIBZRcIgqB2CSNadLQVepUED_W06cSpv/view?usp=sharing

Code: https://github.com/Rishideep-Mandavilli/chunks

I entered the Gemma track, and Gemma is what makes the "private" part real. Running Gemma 4 2B and 4B quantized models through Ollama meant I could do screen understanding fully offline on a MacBook. A few things I learned:

Gemini Flash stays as an optional fast cloud mode, but Chunks works end to end without it.

I built Chunks for Hacktoberfest Hack Day × AI in Vizag on November 1, 2026, in the Gemma use-case track. Honestly, I didn't present it as well as I wanted, and Chunks didn't place. But I did win the special prize for my LinkedIn post about the day, which was a nice surprise.

What I'll remember most is the energy of a room full of people building with open models on a Sunday, and the moment Chunks typed into Notes while my hands were off the trackpad. Next time I'll practise the demo as hard as I practised the code.

What's next: a signed release so anyone can try it, and better small-model accuracy on crowded screens.

Thanks for reading! If you've fought with computer-use agents too, I'd love to hear what broke for you.