ChatGPT serves 800 million users from one Postgres primary. Here's how 🐘

Posted on Oct 27 • Originally published at mahmad.hashnode.dev

800 million people use ChatGPT, and every write still lands on one PostgreSQL primary. No sharding.

When I read OpenAI's post Scaling PostgreSQL to power 800 million ChatGPT users by Bohan Zhang, I couldn't stop thinking about it. So I turned it into an interactive diagram: robots carry queries between the machines, and you can break things to see what happens.

How OpenAI scaled PostgreSQL for 800 million ChatGPT users, animated: the first design and today’s, in normal traffic, a cache-miss spike and a primary failure.

Here's the short version of how it works.

ChatGPT's database load grew more than 10x in a year. One primary handled every write and many reads, and incidents kept following the same pattern: something upstream spiked the load (a cache failure, a burst of expensive joins, a write storm). Queries slowed down, requests timed out, and the retries added even more load.

Reads go to nearly 50 read replicas around the world, and the primary keeps the writes. Requests are also split into high- and low-priority replicas, so a heavy low-priority job can't slow down the important ones.

Write-heavy work that can be sharded moved to Azure Cosmos DB. And new tables aren't allowed on Postgres anymore: new workloads start on the sharded systems.

Azure Postgres allows 5,000 connections, and opening one was slow. PgBouncer keeps connections open and reuses them, so connecting dropped from 50 ms to 5 ms on average.

When a popular cache key expires, thousands of requests miss at once and all hit the database. With cache locking, one request refills the key and the others wait for it.

The lesson I took away: you don't always need to shard. A single primary goes very far if you keep everything you can off it.

👉 mahmad.me/blocks/interactive/chatgpt-postgres

All the facts come from OpenAI's post. The live numbers in the simulation are illustrative.