Llmspeed Benchmark: Pick the Perfect On‑Device LLM Hardware Fast

If you’re tired of guessing whether a single RTX 4090 can sustain 100 req/s for Llama‑2‑7B—or how much electricity a Jetson‑Orin inference server will burn each year—this guide is for you. Llmspeed is an open‑source CLI that lets you run reproducible, hardware‑agnostic benchmarks in minutes, so you can make data‑driven purchasing decisions before you spend a fortune.

In the next few sections we’ll walk through installing llmspeed, running benchmarks on CPUs, GPUs, and edge devices, and turning the raw numbers into actionable insights with a ready‑to‑use spreadsheet calculator. By the end you’ll be able to answer questions like:

# Install the CLI (Python 3.10+ required) pip install llmspeed # Verify the installation llmspeed --version Enter fullscreen mode Exit fullscreen mode Benchmark a 7B model on the default device # The CLI auto‑detects the best device (CPU, Apple M‑series, NVIDIA/AMD GPU) llmspeed run --model meta-llama/Llama-2-7b-chat-hf \ --prompt "Explain quantum tunneling in two sentences." \ --output results.json Enter fullscreen mode Exit fullscreen mode Force a specific backend # Run on the NVIDIA RTX 4090 explicitly llmspeed run --model meta-llama/Llama-2-7b-chat-hf \ --device cuda \ --prompt "Summarize the plot of Inception." \ --output results.json Enter fullscreen mode Exit fullscreen mode Run a batch of tests across multiple devices # Create a YAML config with a list of devices and batch sizes cat > benchmark.yaml << ' EOF ' models: - meta-llama/Llama-2-7b-chat-hf devices: - cpu - cuda - rocm - metal # Apple Silicon batch_sizes: [1, 8, 32] prompts: - "Translate to French: The quick brown fox jumps over the lazy dog." EOF # Execute the whole matrix llmspeed batch --config benchmark.yaml --output matrix_results.json Enter fullscreen mode Exit fullscreen mode All measurements are performed locally; no data ever leaves your machine unless you enable the optional telemetry flag.

Step‑by‑step calculation (using the spreadsheet):