Run GLM-5.3-Flash (321B) locally with Project Maya and EXL3 on one RTX 5090

GLM-5.3-Flash is a 321B mixture-of-experts model, and its 3.05 bpw EXL3 quant is 125 GB, too big to fit in my workstation's 128 GB of RAM next to the OS. It still runs locally here: on one RTX 5090 with 128 GB of RAM, with a 512K-token context, at about 25 tokens per second. This guide shows how, with Project Maya, and what we measured on the way.

Every number below was measured on our own machine on October 15, 2026. Where a number comes from someone else, I name the source.

This is the second of two articles. The first one covers Qwen3.8-Flash-Next with Strata at 79 tokens per second, and before that I wrote about Qwen3.8-Flash-Next with EXL3 and TabbyAPI on 40+ GB of RAM.

GLM-5.3-Flash is an open-weight model from zai-org under the MIT license. Maya's README describes it as 321B parameters with about 18B active per token and a context of up to 1M tokens. It has a hybrid attention design and an MTP layer for drafting.

18B active parameters is three times the roughly 6B that the Qwen3.8-Flash-Next model card lists. That explains most of the speed difference below: more of the model has to move through the CPU and over PCIe for every token.

I tried TabbyAPI first, with turboderp's GLM-5.3-Flash EXL3 quants. ExLlamaV3 keeps CPU experts in a RAM arena and has no SSD tier for experts. So the whole model has to fit in VRAM plus RAM, next to the OS. On 128 GB, the largest quant that worked was the self-calibrated 2.60 bpw revision:

The 3.05 bpw quant (125 GB) does not fit that way. You need a runtime that can leave part of the experts on the SSD.

Project Maya is a fork of Strata, built for this one model. Its engine keeps the experts your conversation uses most on the GPU, the next ones in RAM and the rest on the NVMe drive, and moves them while you chat. It measures your hardware and tunes itself, exposes OpenAI- and Anthropic-compatible APIs, and has a dashboard with a live monitor.

Maya ships its own GGUF quants (Maya-S 96.5 GB, Maya-M 116 GB, Maya-L 156 GB). It can also read turboderp's EXL3 quants in place from their safetensors, which its README marks as experimental and NVIDIA-only. I chose the EXL3 3.05 bpw route because I already trusted that quant from the TabbyAPI tests, and because Maya then decoded as fast as the smaller sc_2.60 quant did in TabbyAPI.

git clone https://github.com/mw00/project-maya.git cd project-maya./maya.sh --check Enter fullscreen mode Exit fullscreen mode --check prints the GPUs, driver, CUDA toolkit, compiler, RAM and CPU. Then run setup non-interactively:

./maya.sh --setup --plain --yes --gpu 0 \ --context 131072 --model EXL3-3.05bpw \ --models-dir ~/Models/maya --download-model \ --no-vision --draft mtp --no-start Enter fullscreen mode Exit fullscreen mode What happens, according to Maya's setup docs and what we saw:

Two settings go into the args of maya-exl3-3.05bpw.json:

"args": [ "--glm-pack", "/home/you/Models/maya/EXL3-3.05bpw/pack", "--max-context", "524288", "--expert-cache", "auto", "--prefill", "auto", "--kv-streaming", "on", "--vram-reserve-mib", "3072" ] Enter fullscreen mode Exit fullscreen mode --vram-reserve-mib 3072: the VRAM Maya's expert pool leaves free. The default is 700 MiB. Our desktop runs on the same card, and with the VRAM full the screen flickered, so we keep 3 GB free everywhere. --max-context 524288: GLM-5.3-Flash supports up to 1M tokens, so this is not rope scaling, just a bigger limit.

With --kv-streaming on, extra context costs RAM (about 13 KB per token, judging by the size of the sessions Maya saved to disk), not VRAM. A new --setup run writes this file again. Maya's docs say it keeps settings like the reserve, but keep a backup anyway.

Maya pins most of the free RAM for its expert tier. On this machine it leaves about 4-5 GB free. That collides with two Linux defaults:

So we start it in a separate slice, without a memory limit:

systemd-run --user --scope --collect --slice = llm.slice \ --unit = "maya- $( date +%s ) " -p MemoryMax = infinity \./run-maya-exl3-3.05bpw.sh --plain Enter fullscreen mode Exit fullscreen mode --plain keeps the output as plain text instead of Maya's full-screen dashboard, which is easier to log. Stop it with systemctl --user stop 'maya-*.scope'. The start takes a few minutes. Then:

curl -s http://127.0.0.1:8080/v1/models curl -s http://127.0.0.1:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "glm-5.3-flash", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 2000}' Enter fullscreen mode Exit fullscreen mode The dashboard is at http://127.0.0.1:8080/#monitor and shows decode and prefill speed, the VRAM hit rate and where the experts live. Keep the server on 127.0.0.1; if you open it to your network, use --host 0.0.0.0 together with --api-key.

Maya's calibration measures how to split the work between the CPU and the PCIe link, and how many CPU threads to use:

./maya.sh --calibrate --no-start Enter fullscreen mode Exit fullscreen mode It loads the full model, so run it with Maya stopped and in the same way as the server (in llm.slice, without a memory limit). It refuses to run while more than 1 GiB of VRAM is in use on the GPU, to make sure no other engine is loaded. A desktop on the same card already takes about 2 GB, so on our machine it never started. I raised that threshold locally in busy_gpus() in maya.py from 1024 to 3072 MiB. That is a local patch: check it after every update, and only use it if you are sure nothing else heavy is on the card.

The calibration took about 10 minutes (Maya estimates 10-15). Its full output on our machine:

the engine's own split (PCIe share 0.60): 23.7 tok/s PCIe share 0.00: 21.1 tok/s PCIe share 0.10: 24.6 tok/s PCIe share 0.25: 23.9 tok/s PCIe share 0.50: 24.1 tok/s PCIe share 0.75: 23.3 tok/s 16 CPU threads: 24.6 tok/s 12 CPU threads: 21.9 tok/s 11 CPU threads: 20.0 tok/s 8 CPU threads: 18.1 tok/s 4 CPU threads: 13.7 tok/s confirming against the engine's own settings... [ok] tuned for this PC: STRATA_GLM_PCIE_SHARE=0.10 (25.7 tok/s) Enter fullscreen mode Exit fullscreen mode It stores the result as an environment variable in the config. In the sweep, a PCIe share of 0.10 beat the engine's own split by about 4% (24.6 vs 23.7), and the confirmation run measured 25.7. Fewer than 16 CPU threads hurt a lot.

How we measured: a client-side streaming bench against the OpenAI endpoint. Eight runs per setting, alternating a ~400-word Dutch prose prompt and a coding prompt (binary search with tests), greedy decoding, 512 output tokens, median reported. Prefill comes from a needle-in-a-haystack test at the given prompt length. The needle and filler text are fixed per length, and Maya caches prompt prefixes (also on disk, across restarts), so a later run can be flattered by an earlier one: read the prefill numbers as approximate. Nothing else heavy ran on the machine.

The first two rows ran with the 700 MiB default reserve, and you can see the VRAM peak at the edge of the 31.8 GiB card. With the 3 GB reserve, the peak stayed at or below 29.5 GiB. Reading 506K tokens took 262 seconds. Decoding at 512K was as fast as at 262K, because the KV cache streams from RAM.

Compared to TabbyAPI on the same machine: about the same decode speed as the smaller sc_2.60 quant (25.5, from TabbyAPI's server log rather than this client bench), with a bigger quant, about 1.7x the prefill speed, and twice the context.

For reference, Maya's README lists "30+ tokens/s" for an RTX 5090 with a Ryzen 9 7950X and 96 GB of DDR5, with Maya's own Maya-M quant. That is a different quant and a different bench, so do not compare the numbers one to one.

We ran the same five coding tasks with hidden tests that we used for the Qwen setups: three trials each, sandboxed, reasoning effort medium, and each server's default sampling (Maya temperature 1.0 and top_p 0.95, Strata greedy, TabbyAPI temperature 0.7).

The one miss was invalid JSON (a trailing comma) in one trial of an incident-report task; all the code tasks passed. These tasks are too easy to say which model is smarter. What they do show is that a 3-bit GLM on a desktop produces working code, and that it takes about 1.5x the time per task because of the slower decode.

Nothing else fits next to it. Maya pins RAM until about 4-5 GB is free and holds 29-30 GB of VRAM. Do not run a second model runtime, a game or a GPU miner next to it. Switching models means stopping one and starting the other.

Wait for the VRAM after a stop. VRAM takes several seconds to come back after a runtime stops. Start the next one too early and it fails a VRAM check or gets a smaller expert pool.

Unstable RAM corrupts downloads. Our four DIMMs at their 6,000 MT/s EXPO profile were not stable: kernel oopses, crashes, and most of a model download written in that time was corrupt. We run 5,600 MT/s now. Maya checks its own downloads against SHA-256; do the same for anything you fetch yourself.

Measure with fresh prompts. A repeated long prompt hits the prefix cache. One of our runs reported 70,624 tokens per second of "prefill" for exactly that reason.

Reasoning effort and sampling. Setup wrote "reasoning_effort": "high" with temperature 1.0 and top_p 0.95 as server defaults in the config. Long reasoning at 25 tokens per second adds up, so for interactive work consider sending a lower reasoning_effort from the client. On the Qwen setup, medium scored best on our own bench; for GLM we have not measured that yet.

On this one machine we now have three ways to run a large model, and we switch with a stop and a start script:

The Strata setup, with its calibration A/B and our own nine-task bench, is in the companion article: Run Qwen3.8-Flash-Next at 79 tok/s with Strata and a 5-bit quant on 128 GB of RAM.

If you are deciding what hardware to buy for running AI locally, Voltti has a practical guide on the trade-offs between local and cloud AI for business use.