Run your LLM evals every night on a GPU that only exists for a minute

Most eval suites run when someone remembers to run them. That's usually right before a release, which is the worst time to find out a model regressed two weeks ago.

The fix is boring: run the evals every night, on a GPU that exists only while they run, and read the number in the morning. This post sets that up with Runcrate's CLI: one command creates a nightly schedule, each run is a job on a fresh GPU that stops when the eval exits, and a run costs about a cent for the example here. I ran every command below on 2026-10-21 and the outputs are copied from my terminal.

I'm using EleutherAI's lm-evaluation-harness on a small model so the run is quick: Qwen/Qwen2.5-0.5B-Instruct on 200 questions of ARC-Easy. Swap in your own model and tasks; the scheduling part doesn't change.

The command that runs on the GPU machine:

pip install -q uv && uv venv -q /opt/env &&. /opt/env/bin/activate && \ uv pip install -q "lm-eval[hf]" torch --torch-backend = cu128 && \ lm_eval --model hf --model_args pretrained = Qwen/Qwen2.5-0.5B-Instruct \ --tasks arc_easy --limit 200 --batch_size 16 Enter fullscreen mode Exit fullscreen mode The first line makes a clean virtualenv with uv, so the eval doesn't fight with whatever Python packages the machine image ships. --torch-backend=cu128 picks the PyTorch build that matches the machine's CUDA 12.8 driver. I tried a plain pip install first and got a PyTorch wheel that refused the driver, so pin this to your driver.

brew install aeonmindai/tap/runcrate runcrate login Enter fullscreen mode Exit fullscreen mode login opens the browser. You pick a workspace and an environment, and everything the CLI starts runs and bills there.

runcrate schedule create --cron '0 2 * * *' --tz Europe/London --name nightly-eval \ --gpu A6000 --max-runtime 30m -- \ 'pip install -q uv && uv venv -q /opt/env &&. /opt/env/bin/activate && uv pip install -q "lm-eval[hf]" torch --torch-backend=cu128 && lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct --tasks arc_easy --limit 200 --batch_size 16' Enter fullscreen mode Exit fullscreen mode What it printed:

Schedule created: nightly-eval ID: e18aeb6f-6d15-4f51-804b-65b258ce71e1 Kind: job When: 0 2 * * * (Europe/London) Overlap: skip Enabled: yes Next run: 2026-10-22 02:00 BST GPU: 1x A6000 Max runtime: 30m Enter fullscreen mode Exit fullscreen mode A few things worth knowing about those flags:

runcrate schedule run-now e18aeb6f-6d15-4f51-804b-65b258ce71e1 Enter fullscreen mode Exit fullscreen mode Started job nightly-eval (ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33). Follow it with: runcrate jobs logs -f ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 Enter fullscreen mode Exit fullscreen mode Run now starts the same job straight away and doesn't move the next scheduled run. I also let a schedule fire on its own while testing: it was due at 02:00 BST and launched at 02:02, which matches the docs (schedules are checked every 2 minutes).

runcrate jobs logs prints the job's output. The end of it:

hf ({'pretrained': 'Qwen/Qwen2.5-0.5B-Instruct'}), gen_kwargs: ({}), limit: 200.0, num_fewshot: None, batch_size: 16 | Tasks |Version|Filter|n-shot| Metric | |Value| |Stderr| |--------|------:|------|-----:|--------|---|----:|---|-----:| |arc_easy| 1|none | 0|acc |↑ |0.565|± |0.0351| | | |none | 0|acc_norm|↑ |0.560|± |0.0352| Enter fullscreen mode Exit fullscreen mode And runcrate jobs status for what it cost:

Name: nightly-eval Status: ● succeeded Exit code: 0 GPU: 1x A6000 Max runtime: 30m Runtime: 1m3s Cost: $0.01 ($0.65/hr) Created: 2026-10-21T01:08:32.934+00:00 Started: 2026-10-21T01:12:50.771+00:00 Ended: 2026-10-21T01:13:53.945+00:00 Enter fullscreen mode Exit fullscreen mode About four minutes went to getting the machine ready and one minute to the eval itself. The max runtime counts from when the command starts, so setup doesn't eat into your 30 minutes. A6000 was $0.65 an hour when I ran this; check the live price before you pick a GPU.

runcrate jobs list shows every night's run with its status, exit code and cost, so a week of runs is one command:

ID NAME STATUS GPU EXIT RUNTIME AGE COST ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 nightly-eval ● succeeded A6000 0 1m 7m ago $0.01 Enter fullscreen mode Exit fullscreen mode Making it a real regression check The toy version prints a number. To make it catch regressions:

runcrate schedule list runcrate schedule pause e18aeb6f-6d15-4f51-804b-65b258ce71e1 runcrate schedule delete e18aeb6f-6d15-4f51-804b-65b258ce71e1 -y Enter fullscreen mode Exit fullscreen mode Pausing stops future runs and leaves anything already running alone.

If you use Claude Code, Codex or Cursor, the Runcrate MCP server has the same schedule tools, so "every night at 2am London time, run my eval on an A6000 for at most 30 minutes" works as a prompt. Setup for each client is on the MCP setup page.

If you set this up for your own model and something gets in the way, tell me. Reply here or write to hello@runcrate.ai.

Written with AI assistance. Every command and output above was run on 2026-10-21.