How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox

Posted on Oct 31 • Originally published at tappers.hashnode.dev

💡 Methodology & Eval Infrastructure Deep Dive

This article is the Methodology & Architecture companion to our AI Labo Agent Harness Engineering Benchmark Series (41 Patterns Tested). Here we explain how we designed an isolated evaluation infrastructure combining Proxmox VE (0-NIC VMs), QEMU Guest Agent ( virtio-serial ), a budget-enforcing Mac relay, and Linux UID / systemd sandboxing to prevent test peeking, reward hacking, and API key leaks—along with a candid look at the current security boundaries and remaining limitations of our setup.

When benchmarking autonomous AI coding agents fairly and safely, engineers inevitably run into a three-way dilemma (trilemma):

To solve these challenges simultaneously, we combined a mini-PC running Proxmox VE with a macOS controller over a 1-to-1 Tailscale link to build the AI Labo Evaluation Sandbox:

Anyone who has evaluated coding agents at scale has seen Reward Hacking firsthand: the agent reports that all tests pass on Turn 1, but git diff reveals that instead of fixing the bug, it modified the test runner or weakened an assertion.

Running the Agent (Generator) and the Grader (Verifier) under the same user and writable filesystem introduces four structural failure modes:

💡 Why Proxmox VMs + systemd Instead of Docker Containers? You can achieve similar logical separation in Docker by splitting agent and grader containers, passing --network none and --read-only, and keeping secrets on a host proxy. We chose Proxmox VMs + systemd for two reasons:

The infrastructure is split across a macOS Controller/Relay and a Proxmox VE Host connected via a 1-to-1 peer-restricted Tailscale VPN:

+-------------------------------------------------------------------------+ | 1. Controller & Relay Host (macOS) | | - OS Keychain / 0400 Secret --> Loaded once into memory at startup | | - Experiment Orchestrator --> Drives Phase 1 (ISO) & Phase 2 (Run) | | - Private API Relay --> Enforces call, token & micro-USD caps | | - Evidence Store --> Verifies & saves 7 artifacts + SHA256 | +------------------------------------+------------------------------------+ | 1-to-1 Peer-Restricted VPN | (SSH Control + Peer-Pinned HTTP) v +-------------------------------------------------------------------------+ | 2. Virtualization Host (Proxmox VE Mini-PC) | | - Host Controller (driver.py) --> Allowlisted qm/pvesh + Watchdogs | | - Local ISO Builder --> Verified.deb cache -> iso9660 CDs | | - Golden Base VM (VM9300) --> Stopped Debian 13 clone source | | | | +-------------------------------------------------------------------+ | | | 3.

Disposable Sandbox VM (Full Clone per Experiment / 0 NICs) | | | | - serial0 (/dev/ttyS0): One-way boot & QGA readiness signal | | | | - virtio-serial (QGA): Bidirectional RPC via `qm guest exec` | | | | - ide2 & ide3 (CD-ROM): Read-only offline packages & payload | | | | | | | | +-----------------------------+ +----------------------------+ | | | | | Agent Sandbox (UID: labexp) | | Grader Sandbox (labjudge) | | | | | | - PrivateNetwork=yes | | - Runs AFTER agent exits | | | | | | - ProtectSystem=strict | | - Evaluates 20 hidden cases| | | | | | - InaccessiblePaths=/opt/.. | | + 6 boundary mutants | | | | | | /evaluation (EACCES) | | - Writes grader-output.json| | | | | +-----------------------------+ +----------------------------+ | | | +-------------------------------------------------------------------+ | +-------------------------------------------------------------------------+ Enter fullscreen mode Exit fullscreen mode Full Component Matrix (15 Components, Privileges, and Security Roles) Environment Component Execution Identity Primary Role Security & Integrity Meaning External Cloud Cloud LLM API Managed Service countTokens & generateContent Accepts HTTPS only from the Mac relay; unreachable from the guest VM. Mac Controller OS Keychain / Secret Security.framework / 0400 Stores LLM API key Never written to.env or VM disks; read once into relay memory at startup. Mac Controller Orchestrator Local user Runs Phase 1 (ISO) & Phase 2 (VM run) Invokes the Proxmox host driver over SSH and verifies final evidence hashes.

Mac Controller Private API Relay Local user ( serve_once.py ) Proxies LLM calls & enforces budgets Enforces max calls, input/output token caps ( maxOutputTokens ), and micro-USD caps. Mac Controller Evidence Storage Local filesystem Stores 7 evidence files + SHA256 manifest Preserves tamper-evident records of prompts, actions, token usage, and test results. Private VPN Tailscale 1-to-1 Link Encrypted Mesh VPN Mac ⇔ Proxmox control & relay traffic Pins source peer IP ( 100.x.x.x ) so LAN or external hosts cannot hit the relay.

Proxmox Host Host Controller root ( driver.py ) Clones VM, drives QGA, runs watchdogs Executes only allowlisted qm / pvesh commands with stage watchdogs. Proxmox Host Local ISO Builder builder ( builder.py ) Packs verified.deb cache & code into ISOs Verifies.deb SHA-256 hashes from signed InRelease ➔ Packages metadata. Proxmox Host Golden Base VM Stopped ( VM9300 ) Clean Debian 13 clone source Kept stopped with pinned config SHA-256 to eliminate cross-run drift.

Proxmox Host Audit Broker VM Isolated VM ( VM100 ) Holds ledger hash pins Verified before and after every experiment to ensure host state integrity. Sandbox VM (0 NIC) serial0 ( /dev/ttyS0 ) Kernel / Bootstrap One-way boot progress notification Emulated 16550 UART used only during initial boot to signal when offline QGA installation finishes. Sandbox VM (0 NIC) virtio-serial (QGA) root ( guest_rpc.py ) Bidirectional host-guest RPC channel Uses org.qemu.guest_agent.0 (.qga socket) via qm guest exec to exchange JSON.

Sandbox VM (0 NIC) Virtual CD-ROMs × 2 Read-only ( iso9660 ) Delivers.deb deps ( ide2 ) & payload ( ide3 ) Immutable media inside the guest; install.py verifies payload SHA-256 before extraction. Sandbox VM (0 NIC) Agent Sandbox Unprivileged labexp Runs the AI agent under test (Mode A / B) ProtectSystem=strict mounts / read-only; InaccessiblePaths blocks access to evaluation/. Sandbox VM (0 NIC) Grader Sandbox Unprivileged labjudge Runs independent grading suite after agent exits Starts only after agent.service stops; evaluates snapshots against 20 hidden test cases.

3. Core Mechanism 1: OS-Level Separation via Linux UIDs and systemd Inside each experiment VM, the Agent ( labexp ) and the Independent Grader ( labjudge ) run under separate Linux UIDs and separate systemd mount namespaces.

Before starting the agent, the root installer ( guest/install.py ) sets strict filesystem ownership and permissions:

Below are the actual systemd unit files from experiment E20 ( VM224 ):

[Unit] Description = AI Lab E20 Episodic Gotchas Memory Isolated Agent Runner (VM224) [Service] Type = oneshot User = labexp20 Group = labexp20 WorkingDirectory = /var/lib/ai-lab-e20-vm224-01 ExecStart = /usr/bin/python3 -I -B /opt/ai-lab-e20-vm224-01/guest/run_guest.py Restart = no KillMode = control-group TimeoutStartSec = 240 NoNewPrivileges = yes PrivateTmp = yes PrivateDevices = yes PrivateNetwork = yes ProtectSystem = strict ProtectHome = yes CapabilityBoundingSet = AmbientCapabilities = # Under ProtectSystem=strict, ONLY outbox and the 6 pre-created target.py files are writable: ReadWritePaths = /var/lib/ai-lab-e20-vm224-01/outbox ReadWritePaths = /var/lib/ai-lab-e20-vm224-01/work_cold_start/validator.py ReadWritePaths = /var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py ReadWritePaths = /var/lib/ai-lab-e20-vm224-01/work_cold_start/statement.py ReadWritePaths = /var/lib/ai-lab-e20-vm224-01/work_episodic_memory/validator.py ReadWritePaths = /var/lib/ai-lab-e20-vm224-01/work_episodic_memory/settlement.py ReadWritePaths = /var/lib/ai-lab-e20-vm224-01/work_episodic_memory/statement.py # Grading code (evaluation/), initial fixtures, and evidence dirs are overlaid with 0000 mounts: InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/root InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/evidence InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/grade InaccessiblePaths = /opt/ai-lab-e20-vm224-01/evaluation InaccessiblePaths = /opt/ai-lab-e20-vm224-01/fixtures Enter fullscreen mode Exit fullscreen mode Independent Grader Unit ( ai-lab-e20-01-grade.service ) Only after agent.service terminates and the root RPC handler ( guest_rpc.py ) copies a snapshot of the edited files into grade/ does the grader service launch as labjudge20:

[Unit] Description = AI Lab E20 Episodic Gotchas Memory Isolated Grader (VM224) [Service] Type = oneshot User = labjudge20 Group = labjudge20 WorkingDirectory = /var/lib/ai-lab-e20-vm224-01/grade ExecStart = /usr/bin/python3 -I -B /opt/ai-lab-e20-vm224-01/evaluation/grade_comparison.py Restart = no KillMode = control-group TimeoutStartSec = 30 NoNewPrivileges = yes PrivateTmp = yes PrivateDevices = yes PrivateNetwork = yes ProtectSystem = strict ProtectHome = yes CapabilityBoundingSet = AmbientCapabilities = # The grader can only write to the single grader-output.json file: ReadWritePaths = /var/lib/ai-lab-e20-vm224-01/grader-output.json InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/root InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/evidence InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/inbox InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/outbox InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/work_cold_start InaccessiblePaths = /var/lib/ai-lab-e20-vm224-01/work_episodic_memory Enter fullscreen mode Exit fullscreen mode This layered setup enforces three protections during agent execution:

💡 How We Use Mutation Testing ( Mutation Kill Rate ) In experiments where the agent submits both production code and a test suite ( test_suite.py in E06 Blind TDD, E07, E08, E10 ), passing functional tests isn't enough—we also need to know if the agent wrote weak, self-serving tests. Our grader ( judges.py ) runs the agent's test_suite.py against 6 deliberate boundary mutants of the reference implementation (e.g., changing >= 10000 to > 10000, removing isinstance(x, bool) checks, omitting -0 formatting) to measure the exact Mutation Kill Rate ( 0/6 to 6/6 ).

Because the experiment VM has zero network interfaces (verified at boot by asserting that /sys/class/net contains only lo ), it is a network-isolated VM whose sole external channel is an inspected host-mediated relay.

We use two distinct serial mechanisms between the Proxmox host and the guest VM for different phases of the lifecycle:

Because the Mac relay ( serve_once.py ) mediates every LLM call outside the VM, the agent cannot tamper with token accounting. Before each generateContent call, the relay calls countTokens to verify that cumulative_inputs + prompt_tokens <= MAX_INPUT_TOKENS and cumulative_outputs + maxOutputTokens <= MAX_OUTPUT_TOKENS.

Upon receiving the response, the relay extracts promptTokenCount, candidatesTokenCount, and thoughtsTokenCount (reasoning tokens) from usageMetadata, computing billed_out = candidatesTokenCount + thoughtsTokenCount and recording exact costs in micro-USD ( 1e-6 USD ) against a hard session cap (e.g., 350,000 micro-USD = $0.35 ).

While this architecture is highly effective for benchmarking structured-action coding agents, evaluating untrusted, adversarial agents with arbitrary code/shell execution requires addressing several remaining security boundaries. We document these limitations openly for anyone building on this design:

While running 40+ consecutive VM benchmarks, we encountered two non-obvious operational traps worth sharing:

When invoking /usr/sbin/qm guest exec <vmid> -- /usr/bin/python3 -c <script_source> from the Proxmox host, early builds of E21 that included Japanese UTF-8 characters inside Python docstrings consistently failed with VM qga command 'guest-exec' failed - got timeout (exit code 255), leaving guest-ping unresponsive afterward.

While virtio-serial itself is a byte stream and Proxmox's QGA client prepends guest-sync-delimited ( \xff delimiter resynchronization), passing large multi-kilobyte scripts containing multi-byte UTF-8 characters directly as command-line arguments to Proxmox's /usr/sbin/qm guest exec CLI (which is implemented in Perl under the hood) triggered encoding/buffer handling stalls.

While using an AI coding assistant on our development machine to debug Proxmox socket states and systemd sandbox rules (not inside the benchmarked VMs themselves), the Gemini API occasionally blocked responses with:

This request was blocked by Gemini's filters. They can occasionally trigger by mistake on safe coding, security, or biology-related queries.

This is a known false positive ( promptFeedback.blockReason: "PROHIBITED_CONTENT" ) documented on the Google AI Developers Forum that cannot be disabled via standard safety threshold sliders. Because the safety classifier evaluates the full conversation history, accumulating multiple inline snippets referencing raw hypervisor sockets ( *.qmp, *.qga ) and namespace boundary checks ( InaccessiblePaths, runuser ) likely resembled VM-escape or privilege-escalation exploit development. Switching our debugging workflow to invoke only standard /usr/sbin/qm guest exec CLI wrappers and pre-committed scripts eliminated these false positives completely (and zero filter blocks ever occurred inside the 41 benchmark runs).

Every metric reported in our benchmark series ( E04 – E44 ) is extracted directly from the 7 JSON files collected from the VM and verified against manifest.json:

Ready to see what we discovered when we ran 41 agent harness patterns through this sandbox? Check out the master benchmark index and our #1 ranked pattern deep dive: