Fresh Hugging Face download vs local cache: time to first token for a 7B model on a Runcrate L40S

How long does a GPU machine sit idle before a 7B model answers its first request, and how much of that is the download? I rented one L40S on Runcrate today (2026-10-29), timed every phase of starting Qwen2.5-7B-Instruct from a fresh Hugging Face download and from a copy already on the machine's disk, and deleted the machine when the numbers were saved.

Short version: from a fresh download, the first token came 48 to 55 s after the Python process started. With the weights already on local disk it came after 18 to 19 s. The download was 30 s of that (15.2 GB at about 515 MB/s), and the part a cache cannot remove was the 14 s it took to read 15 GB from disk into GPU memory.

import json, os, sys, time T0 = time. time () import torch from huggingface_hub import snapshot_download from transformers import AutoModelForCausalLM, AutoTokenizer r = { " import_s ": round ( time. time () - T0, 1 )} MID = " Qwen/Qwen2.5-7B-Instruct " t = time. time (); path = snapshot_download ( MID, allow_patterns = [ " *.json ", " *.safetensors ", " *.txt " ]) r [ " download_s " ] = round ( time. time () - t, 1 ) t = time. time (); tok = AutoTokenizer. from_pretrained ( path ) model = AutoModelForCausalLM. from_pretrained ( path, torch_dtype = torch. bfloat16, device_map = " cuda " ) torch. cuda. synchronize (); r [ " load_to_gpu_s " ] = round ( time. time () - t, 1 ) t = time. time () ids = tok. apply_chat_template ([{ " role ": " user ", " content ": " Say hello in five words. " }], add_generation_prompt = True, return_tensors = " pt " ). to ( " cuda " ) model. generate ( ids, max_new_tokens = 1, do_sample = False ) torch. cuda. synchronize (); r [ " first_token_s " ] = round ( time. time () - t, 2 ) r [ " total_s " ] = round ( time. time () - T0, 1 ) print ( json. dumps ( r )) Enter fullscreen mode Exit fullscreen mode A fresh start is rm -rf $HF_HOME/hub/models--Qwen--Qwen2.5-7B-Instruct before the run. To evict a cached copy from RAM without root ( /proc/sys/vm/drop_caches was not writable in this container) I called os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) on every blob file.

GPU cost of the wait is the total time at the L40S's hourly price.

The same machine also ran an embedding throughput test and a torch.compile training test; the whole run, about 12 minutes of billed L40S time, was billed $0.11.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.