I Built a 3D Office Where Local AI Agents Share One RTX 4090

Signup Write New Story I Built a 3D Office Where Local AI Agents Share One RTX 4090 by Bohdan Bozhenko by Bohdan Bozhenko | @ bogdus1k Software Engineer - backend & AI/ML. First-Class BSc Computing. I write about backend, AI/LLM engineering and what I ship.

Software Engineer - backend & AI/ML. First-Class BSc Computing. I write about backend, AI/LLM engineering and what I ship.

I built a 3D isometric office where each employee is a local AI agent running on a single RTX 4090. I assign a task, the agent plans, researches, drafts and self-checks, and I approve the result or send it back with a comment. The 3D scene was the easy part. The hard part was making a 27B model that shares one GPU with its colleagues stop inventing facts about me and do the work I asked for.

AI Office is a local web app. You open a browser and there is a 3D isometric office. Each figure at a desk is an AI agent. I assign a task, and the agent works through a fixed pipeline: plan, research, draft, self-check. I read the output, approve it, or send it back with a comment. The model runs on my machine; the only traffic that leaves it is web search.

The idea came from WareTrack, a warehouse-management demo that Dilum Sanjaya posted on X in October 2026. WareTrack is a simulation. In AI Office the agents do real work: they call a language model, hit a search engine, and produce a document I can read.

I built it for a simple reason: I delegate better when I can see the work happening. A chat window tells me a task is "in progress"; an office shows me who is working on it, who is waiting, and who has finished.

- Next.js 15, React Three Fiber, Zustand (frontend)

- Ollama running qwen3.8:27b (q4_K_M, 16K context window, set locally) on one RTX 4090, 24 GB

Every service runs on localhost. I built it in about three days across eight stages with Claude Code. At version 1.0 the repo sits at 70 commits, 270 backend tests, and 41 layout checks.

PostgreSQL is the only source of truth. The 3D scene never decides where a figure stands or what it is doing. It renders.

1. PostgreSQL holds the state - which agent is at which desk, what task it is on, its status and when that status began.

2. The FastAPI backend exposes that state through an API.

3. A Zustand store in the browser reads from the API and holds the current snapshot.

4. React Three Fiber reads from the store and draws the scene.

Walking and driving animations are a pure function of state and time: position = f(state, t). Reload the page or open a second tab and every figure is in the same position, within a millimetre. The scene is a view, not a simulation.

One RTX 4090 means one inference request at a time. If two agents need the model, they queue. In practice, two agents alternate every 10 to 20 seconds.

I gave the agents a kitchen. If an agent's wait for the model exceeds 30 seconds, it stands up, walks to the coffee machine, and waits there. The coffee queue is the GPU queue, made visible.

A 30-second wait does not mean something is broken. It means more than one agent is ahead in the queue. The agent is not stuck; it is in line. The kitchen is just the queue rendered as a place you can see.

Before I added instancing, the scene issued 893 draw calls for roughly 13k triangles. After switching to instanced meshes, draw calls dropped to 117 while the triangle count grew to 64k.

Average CPU work per frame with six figures moving at once is about 1 ms at 165 Hz; p99 was 1.7 ms. I did not measure GPU time separately, so I will not pretend I have a full picture of the frame budget.

On the inference side, the 27B model occupies about 17 GB of my 24 GB VRAM and produces roughly 100 tokens per second. A typical task finishes in about two minutes; with web research in the loop, three to five. Before committing to a model I benchmarked two candidates on the same card: qwen3.8:27b and muse-glimmer:30b. I chose qwen3.8:27b.

One incident is worth describing. Speed fell from about 100 to 25–30 tokens per second with no code change. An orphaned model-server process was holding a second copy of the model in VRAM. It had been left behind when Ollama was force-stopped during a test. Killing it restored the baseline.

The first test runs produced three errors. The model wrote that Dumfries is "roughly 45 minutes from Glasgow", which is false. It mentioned a tool that was not in my profile. It described me as still completing a degree I already hold.

The fixes were small and specific. I put today's date into the system prompt. I added a line-by-line self-check stage that runs before the result reaches me. And I added two house rules that go into the system prompt at every stage:

- Every statement about the director must be backed by a specific line of the profile, the task or the comments. Do not infer or estimate anything else: no distances or travel times, no availability, no tools, roles or responsibilities that are not listed, no skill levels. If a detail would help, mark it [NEEDS INPUT:...].

Each clause in the second rule is there because of a specific mistake.

One invented phrase still got through the self-check. The phrase was "available to start immediately", which I had never said. Final review stays with me.

Before any page is fetched, the target address is checked after DNS resolution. If any resolved address is local or private, the request is refused:

def is_public_ip(address: str) -> bool: ip = ipaddress.ip_address(address.split("%", 1)) if isinstance(ip, ipaddress.IPv6Address): mapped = ip.ipv4_mapped or ip.sixtofour or (ip.teredo if ip.teredo else None) if mapped is not None: ip = mapped return ip.is_global and not ip.is_multicast and not ip.is_reserved async def resolve_public(target: Target, resolver: Resolver = system_resolver) -> str: """Returns the IP to connect to; refuses when any resolved address is local or private.""" try: ipaddress.ip_address(target.host.strip("[]")) addresses = [target.host.strip("[]")] except ValueError: try: addresses = await resolver(target.host, target.port) except OSError as err: raise BlockedUrl(f"cannot resolve '{target.host}'") from err if not addresses: raise BlockedUrl(f"cannot resolve '{target.host}'") for address in addresses: if not is_public_ip(address): raise BlockedUrl(f"'{target.host}' points to a local or private address") return addresses

The fetcher then connects to the checked IP itself and sends the real host name in the Host header and in SNI, so the name cannot resolve to something else between the check and the request. Every redirect goes through the same check.

Page text reaches the model as data wrapped in delimiters, not as instructions. That reduces the chance that a hostile page talks the model into doing something; it does not remove it.

Job boards answered 403 to page reads, so job search moved to official job APIs.

Stage state lives in the database. If the backend restarts mid-task, the interrupted stage is marked as interrupted and the same stage starts again. The agent does not re-plan or re-research.

What still needs me is a short list. I approve the final output. I catch what the self-check misses. I decide whether a task is actually done. I assign the next task.

The first real task I gave an agent was a job search: junior software roles in Glasgow and Edinburgh, four searches through two job APIs. It came back with 34 vacancies and recommended none of them.

Most were training courses advertised as jobs. The agent quoted the lines that gave them away, such as "the cost of the training is the responsibility of the learner". The rest were graduate schemes I had excluded, or engineering roles that had nothing to do with software. It also flagged one role from its own notes that it could not find in the job list, and asked me to confirm it before trusting it.

An empty list was the correct answer. My search terms were too narrow, and the agent said so.

The model is fast and sometimes wrong, so the last check is mine. The office makes it obvious where that check is needed.

I Added Every Index, but My Telemetry Pipeline Was Still Quadratic