Building a Sub-Millisecond AI Gateway in Go for Autonomous Coding Agents
When building workflows with autonomous coding agents (like Claude Code, Cursor, or OpenCode), agent execution loops issue dozens of rapid LLM calls per minute. Standard Python-based proxies introduce 40ms–100ms of latency per hop, balloon to hundreds of megabytes in memory, and easily choke during
When building workflows with autonomous coding agents (like Claude Code, Cursor, or OpenCode), agent execution loops issue dozens of rapid LLM calls per minute. Standard Python-based proxies introduce 40ms–100ms of latency per hop, balloon to hundreds of megabytes in memory, and easily choke during parallel agent streaming.
To solve this, I built GoRoute ( https://github.com/ziuus/goroute ) — an ultra-lightweight, high-throughput AI gateway and dynamic LLM router written in Go.
go install github.com/ziuus/goroute@latest goroute --port 8080 --config config.yaml Enter fullscreen mode Exit fullscreen mode Or via Docker:
docker run -p 8080:8080 -v $( pwd ) /config.yaml:/app/config.yaml ghcr.io/ziuus/goroute:latest Enter fullscreen mode Exit fullscreen mode Point your agent or SDK to http://localhost:8080/v1 and you are ready.
GoRoute is 100% open source under the MIT License.
Feedback, issue reports, and benchmarks are warmly welcomed! If you are building agentic tools or multi-model pipelines, give it a try.h