LocalLens — The Unseen AI Ghost Automating Your Browser Behind the Scenes

Hacktoberfest Open-Source AI Challenge: Ghost in the Machine Submission 👻🌼

*This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine*

I built LocalLens, a privacy-first, on-device AI browser extension (MV3) that acts like an invisible digital spirit inside your browser—turning natural language commands into autonomous DOM interactions without transmitting a single byte of screen state to remote servers.

The "Ghost in the Machine" theme represents an invisible, autonomous entity operating silently behind the physical interface. Most AI agents act like cloud phantoms—remote entities hovering over third-party servers, demanding full access to your DOM trees, credentials, and raw screenshots.

LocalLens flips this paradigm. It brings the "ghost" entirely into your own machine. Operating local perception via client-side WebGPU and quantized open-weights, LocalLens silently watches the page, redacts sensitive PII on render like an invisible filter, and executes complex clicks, inputs, and navigations behind the scenes.

I built this for my developer friend Ankit, who needed an intelligent agent to automate repetitive administrative portals and documentation forms—without giving third-party cloud APIs visibility into private keys or student records.

LocalLens is a privacy-first, locally-deployable browser automation agent. It bridges a Chrome MV3 extension with a Vision-Language Model (VLM) backend to autonomously execute natural language tasks on any webpage (e.g., "Submit the login form").

The system is split into a robust frontend extension and an intelligent Python backend: