Verteiltes GPU-Cluster aus ungenutzten Apple-Silicon-Macs fuer LLM-Inferenz

A single M2 Max Mac Studio can serve 8‑token‑per‑second Llama‑2‑7B inference, yet most developers leave that compute idle overnight. ## Inventory and Network Topology Start by enumerating every Apple Silicon machine on the LAN. arp-scan --localnet | grep -i apple returns MAC addresses and hostnames. In a typical lab you might see two M2 Max units (38‑core GPU, 400 GB/s bandwidth), one M3 Ultra (80‑core GPU, 800 GB/s), and one M1 Pro (16‑core GPU). Record each host, its GPU core count, and unified memory size in a YAML inventory.

Then run exo cluster init --config inventory.yaml on every node so Exo can build a peer‑to‑peer mesh over mDNS. Action: Create an inventory YAML and run exo cluster init --config inventory.yaml on each Mac to join the cluster. ## Unified Memory as a Shared Tensor Buffer Apple’s unified memory lets the CPU and GPU address the same physical pages, eliminating PCIe copies. llama.cpp 0.3.0 exploits this with --n-gpu-layers. On an M2 Max you can offload 35 of 32 layers for a 7 B model:./llama-server -m model.gguf -c 4096 --n-gpu-layers 35 --port 8080.

The same binary on an M3 Ultra can push all 32 layers ( --n-gpu-layers 32 ) because the 80‑core GPU holds the full weight set. Verify layer residency with llama-server -m model.gguf --verbose 2>&1 | grep "gpu_layers". Action: Set --n-gpu-layers to the maximum layers that fit in GPU memory on each model. ## Orchestration with Ray and Exo Ray handles request routing while Exo shards the model across the mesh.

Launch the head on the most powerful node: ray start --head --port=6379 --dashboard-host=0.0.0.0. On every worker run ray start --address=<head-ip>:6379 --resources='{"GPU": 1}'. Deploy a Ray Serve entry point that forwards each prompt to the actor with the lowest queue length:

@serve.deployment ( ray_actor_options = { " num_gpus ": 0.25 }) class Llama: def __init__ ( self ): self. llm = LlamaCPP ( model_path = " model.gguf ", n_gpu_layers = 35 ) async def __call__ ( self, request ): return self. llm ( request. json ()[ " prompt " ]) Enter fullscreen mode Exit fullscreen mode Action: Deploy a Ray Serve deployment that routes requests to the least‑loaded GPU using ray_actor_options={'num_gpus': 0.25}. ## Monitoring, Autoscaling, and Cost Accounting Collect GPU utilization with Apple’s powermetrics: sudo powermetrics --samplers gpu_power -i 1000 -n 1. Feed the output to a custom Prometheus exporter ( mac_gpu_exporter ) that scrapes every 10 seconds. Four Macs drawing ~120 W total cost roughly $0.015 /kWh at US average rates, so a 24‑hour run costs under $0.04.

Add an alert rule: avg_over_time(mac_gpu_utilization[5m]) > 0.8 to trigger autoscaling via a simple launchd job that wakes sleeping Macs. Action: Add a Prometheus scrape config for mac_gpu_exporter and alert when average utilization exceeds 80% for 5 min. Start tonight by running exo cluster init on every idle Mac and point a single Ray Serve endpoint at the cluster; you will have a working distributed inference pool before the next morning.

If this was useful: I packaged a deeper version as Prompt Engineering for AI Roles: Hiring Manager & Engineer Guide.

This article was drafted with AI assistance.