Production Deployment of DeepSeek-V4 and Kimi K3: 32-GPU Topologies, Memory Math, and Cluster Engineering

Posted on Oct 21 • Originally published at blog.llmgo.top

By 2026, state-of-the-art open-weights foundation models have universally converged on ultra-sparse Mixture-of-Experts (MoE) and hybrid recurrent/compressed attention mechanisms. Flagship architectures such as DeepSeek-V4-Pro and Moonshot's Kimi K3 achieve reasoning parity with proprietary frontier models in mathematical synthesis, extended code generation, and autonomous agent loops.

However, operating these models under stringent enterprise SLAs exposes three fundamental infrastructure challenges:

This guide provides an end-to-end engineering blueprint covering hardware topologies, memory calculations, distributed Ray setups, production startup scripts, Kubernetes manifests, and cost models.

Before provisioning hardware, infrastructure architects must inspect the mathematical mechanics of both target models.

graph TD subgraph DeepSeek_V4["DeepSeek-V4-Pro (61 Layers)"] D_In["Input Tokens"] --> D_CSA["CSA: Compressed Sparse Attention<br/>(Local dense + sparse global tokens)"] D_CSA --> D_HCA["HCA: Heavily Compressed Attention<br/>(Ultra-low-rank latent KV projections)"] D_HCA --> D_MoE["MoE Routing Layer<br/>(Fine-grained Top-K, FP4/FP8 experts)"] end subgraph Kimi_K3["Kimi K3 (93 Layers)"] K_In["Input Tokens"] --> K_KDA["69 Layers of KDA<br/>(Recurrent state updates, sublinear KV growth)"] K_KDA --> K_GMLA["24 Layers of Gated MLA<br/>(Precision long-range multi-head associative retrieval)"] K_GMLA --> K_MoE["MoE Routing Layer<br/>(MXFP4 weights, MXFP8 activations)"] end DeepSeek-V4-Pro (61 Layers): Employs a layered combination of Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA maintains local sliding-window attention alongside dynamically sampled global anchors, while HCA applies aggressive low-rank latent projections to background context. The MoE layer routes to fine-grained experts, activating ~49B parameters per step while keeping the full 1.6T parameter footprint resident across cluster VRAM.

Kimi K3 (93 Layers): Features a hybrid stack of 69 KDA layers + 24 Gated MLA layers. KDA (Kimi Dynamic Attention) maintains recurrent linear states whose memory footprints remain bounded regardless of sequence depth. The subsequent 24 Gated MLA layers handle precise global associative recall.

Built on quantization-aware training (QAT) with native MXFP4 weights and MXFP8 activations, dramatically reducing raw model byte volume. Official Specifications Comparison Specification DeepSeek-V4-Pro Kimi K3 Official Repository deepseek-ai/DeepSeek-V4-Pro moonshotai/Kimi-K3 Total / Active Parameters ~1.6T / 49B ~2.8T / 104B Layer Stack 61 Layers (CSA + HCA) 93 Layers (69 KDA + 24 Gated MLA) Native Checkpoint Format FP4 expert weights, FP8 attention/dense Native MXFP4 weights, MXFP8 activations Native Context Length 128k tokens 200k tokens Recommended Serving Backend vLLM v1 / SGLang FlashInfer vLLM (FlashKDA runtime plugin) / Custom C++ runtime [!CAUTION] Crucial Capacity Realities

Trillion-parameter distributed serving exposes an order-of-magnitude bandwidth disparity: intra-node NVLink delivers 900 GB/s bidirectional bandwidth per GPU, whereas inter-node NDR InfiniBand delivers ~50 GB/s (400 Gbps).

Production deployments typically rely on 4 HGX H100/H800 (8x SXM5 80GB) nodes wired in a rail-optimized non-blocking fabric:

flowchart TB subgraph Fabric["Non-Blocking Datacenter InfiniBand Fabric (Leaf-Spine)"] SW1["Spine Switch 1"] SW2["Spine Switch 2"] Leaf1["Leaf Switch 1"] Leaf2["Leaf Switch 2"] SW1 --- Leaf1 & Leaf2 SW2 --- Leaf1 & Leaf2 end subgraph Node0["Node 0 (Head Node - 8x H100 SXM5)"] GPU0["GPU 0..7 (TP=8)"] NVS0["NVSwitch (900 GB/s)"] NIC0["8x ConnectX-7 400G NICs"] GPU0 <--> NVS0 GPU0 <--> NIC0 end subgraph Node1["Node 1 (Worker 1 - 8x H100 SXM5)"] GPU1["GPU 8..15 (TP=8)"] NVS1["NVSwitch (900 GB/s)"] NIC1["8x ConnectX-7 400G NICs"] GPU1 <--> NVS1 GPU1 <--> NIC1 end subgraph Node2["Node 2 (Worker 2 - 8x H100 SXM5)"] GPU2["GPU 16..23 (TP=8)"] NVS2["NVSwitch (900 GB/s)"] NIC2["8x ConnectX-7 400G NICs"] GPU2 <--> NVS2 GPU2 <--> NIC2 end subgraph Node3["Node 3 (Worker 3 - 8x H100 SXM5)"] GPU3["GPU 24..31 (TP=8)"] NVS3["NVSwitch (900 GB/s)"] NIC3["8x ConnectX-7 400G NICs"] GPU3 <--> NVS3 GPU3 <--> NIC3 end NIC0 <==> Leaf1 NIC1 <==> Leaf1 NIC2 <==> Leaf2 NIC3 <==> Leaf2 Parallel Partitioning Strategy Intra-Node Tensor Parallelism (TP = 8): Matrix multiplications across self-attention projections and MLP dense layers are sharded across 8 intra-node GPUs. Because every token step executes multiple All-Reduce collectives, these must remain on NVLink (3.2 TB/s aggregate per node). Setting TP > 8 across physical nodes is strictly prohibited; doing so pushes All-Reduce over network NICs, increasing per-token latency from milliseconds to seconds.

Inter-Node Pipeline Parallelism (PP = 2 or 4): The transformer layer stack is partitioned across physical nodes. Inter-node transfers are restricted to boundary activation tensors, reducing network volume to megabytes per step, which is easily handled by 400Gbps fabrics. Expert Parallelism (EP): MoE experts are distributed across nodes.

Inter-node token dispatching occurs via All-to-All collectives, which modern engines overlap with dense attention computation. 3. Per-Rank GPU Memory Math and Budget Modeling Capacity planning must evaluate the most constrained single GPU rank rather than aggregate cluster numbers.

The memory consumption on any GPU rank $i$ follows:

$$M_{\text{rank}} = M_{\text{weight}} + M_{\text{activation}} + M_{\text{kv_cache}} + M_{\text{comm_buffer}} + M_{\text{cuda_overhead}}$$

Budget allocation for DeepSeek-V4-Pro under a 32-GPU hybrid topology (TP8 x PP4):

[!NOTE] Capacity Assessment With ~26.0 GB of dedicated KV Cache memory per GPU, and benefiting from DeepSeek-V4's CSA+HCA low-rank latent compression (reducing KV memory by ~80% relative to standard MHA), a 32-GPU cluster comfortably supports over 160,000 active concurrent tokens.

Production deployment across 4 physical nodes (Node 0: 192.168.10.101, Node 1: 192.168.10.102, Node 2: 192.168.10.103, Node 3: 192.168.10.104 ).

# 1. Verify 8 GPUs and NVLink topology nvidia-smi nvidia-smi topo -m # 2. Verify all 8 ConnectX-7 NICs are Active at 400 Gbps (NDR) ibstat | grep -E "(CA '|State:|Physical state:|Rate:)" # 3. Verify shared memory allocation (requires >= 64GB) df -h /dev/shm Enter fullscreen mode Exit fullscreen mode Step 2: Initialize Ray Multi-Node Cluster Execute on Node 0 (Head Node):

#!/usr/bin/env bash ray stop --force ray start --head \ --port = 6379 \ --node-ip-address = 192.168.10.101 \ --num-gpus = 8 \ --object-store-memory = 68719476736 \ --block Enter fullscreen mode Exit fullscreen mode Execute on Worker Nodes (Nodes 1, 2, and 3):

#!/usr/bin/env bash ray stop --force ray start \ --address = '192.168.10.101:6379' \ --node-ip-address = $( hostname -I | awk '{print $1}' ) \ --num-gpus = 8 \ --object-store-memory = 68719476736 \ --block Enter fullscreen mode Exit fullscreen mode Validate cluster topology on the Head Node with ray status (must report 4 Nodes, 32 GPUs ).

#!/usr/bin/env bash set -eo pipefail export CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 export NCCL_DEBUG = INFO export NCCL_IB_DISABLE = 0 export NCCL_IB_HCA = mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 export NCCL_IB_GID_INDEX = 3 export NCCL_CROSS_NIC = 1 export NCCL_SOCKET_IFNAME = eth0 export VLLM_HOST_IP = 192.168.10.101 # Launch DeepSeek-V4-Pro Production Service vllm serve deepseek-ai/DeepSeek-V4-Pro \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 8 \ --pipeline-parallel-size 4 \ --distributed-executor-backend ray \ --dtype auto \ --quantization fp8 \ --gpu-memory-utilization 0.90 \ --max-model-len 32768 \ --max-num-seqs 256 \ --enable-chunked-prefill \ --max-num-batched-tokens 4096 \ --trust-remote-code \ --disable-log-requests Enter fullscreen mode Exit fullscreen mode [!TIP] Parameter Best Practices

For Kimi K3, specify the dedicated recurrent kernel backend:

#!/usr/bin/env bash vllm serve moonshotai/Kimi-K3 \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 8 \ --pipeline-parallel-size 4 \ --distributed-executor-backend ray \ --dtype auto \ --kda-prefill-backend triton \ --gpu-memory-utilization 0.92 \ --max-model-len 65536 \ --trust-remote-code Enter fullscreen mode Exit fullscreen mode 5. Enterprise Kubernetes Deployment with KubeRay In containerized Kubernetes environments, clusters are managed declaratively using the KubeRay Operator:

apiVersion: ray.io/v1 kind: RayCluster metadata: name: deepseek-v4-cluster namespace: llm-serving spec: rayVersion: ' 2.38.0' headGroupSpec: rayStartParams: dashboard-host: ' 0.0.0.0' num-gpus: ' 8' template: spec: nodeSelector: node.kubernetes.io/instance-type: hgx-h100-8gpu containers: - name: ray-head image: vllm/vllm-openai:v0.6.3 resources: limits: nvidia.com/gpu: 8 memory: 512Gi cpu: " 64" requests: nvidia.com/gpu: 8 memory: 256Gi cpu: " 32" volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /root/.cache/huggingface name: model-cache volumes: - name: dshm emptyDir: medium: Memory sizeLimit: 128Gi - name: model-cache persistentVolumeClaim: claimName: nfs-model-weights-pvc workerGroupSpecs: - groupName: gpu-workers replicas: 3 minReplicas: 3 maxReplicas: 3 rayStartParams: num-gpus: ' 8' template: spec: nodeSelector: node.kubernetes.io/instance-type: hgx-h100-8gpu containers: - name: ray-worker image: vllm/vllm-openai:v0.6.3 resources: limits: nvidia.com/gpu: 8 memory: 512Gi cpu: " 64" volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /root/.cache/huggingface name: model-cache volumes: - name: dshm emptyDir: medium: Memory sizeLimit: 128Gi - name: model-cache persistentVolumeClaim: claimName: nfs-model-weights-pvc Enter fullscreen mode Exit fullscreen mode [!IMPORTANT] Shared Memory Configuration Mount /dev/shm backed by medium: Memory (>= 128Gi). NCCL intra-node collectives rely on shared memory for GPU-to-GPU transfers. Default container limits (64MB) trigger immediate Bus error (core dumped) crashes.

Verify latency and throughput under production traffic using vLLM's benchmark_serving.py:

python3 -m vllm.benchmarks.benchmark_serving \ --backend vllm \ --model deepseek-ai/DeepSeek-V4-Pro \ --dataset-name sharegpt \ --dataset-path./ShareGPT_V3_unfiltered_cleaned_split.json \ --num-prompts 1000 \ --request-rate 16 \ --max-concurrency 64 Enter fullscreen mode Exit fullscreen mode Production SLA Baseline Thresholds A healthy 32-GPU H100 cluster must achieve the following metrics:

==================== vLLM Serving Benchmark Summary ==================== Successful requests: 1000 / 1000 (100%) Benchmark duration: 62.5 s Total input tokens: 238,400 Total generated tokens: 196,800 Request throughput: 16.0 req/s Token throughput (total): 6,963.2 tokens/s Token throughput (generation): 3,148.8 tokens/s [Latency Metrics] Time to First Token (TTFT): Mean: 48.2 ms P50: 36.1 ms P95: 112.5 ms P99: 168.0 ms <-- Target: < 250 ms Time per Output Token (TPOT): Mean: 18.4 ms P50: 16.2 ms P95: 24.8 ms P99: 31.5 ms <-- Target: < 40 ms (>= 25 tokens/s) ======================================================================== Enter fullscreen mode Exit fullscreen mode 7. Enterprise TCO Economics: Self-Hosting vs. Commercial APIs flowchart LR A["Daily Enterprise Token Volume"] --> B{"Is daily volume sustained<br/>above 1 Billion tokens?"} B -- "No (Early Stage / Bursty)" --> C["Commercial API Strategy<br/>(Zero CapEx, pay-as-you-go)"] B -- "Yes (Core High-Scale Services)" --> D{"Strict Data Sovereignty<br/>or Custom Weight Needs?"} D -- "Yes" --> E["Self-Hosted 32-GPU Cluster<br/>(On-prem isolation, dedicated compute)"] D -- "No" --> F{"Evaluate Duty Cycle<br/>(Peak-to-Trough Variance)"} F -- "High idle nights (>70%)" --> C F -- "Sustained high load" --> E Total Cost of Ownership (32x H100 80GB) Expense Category Monthly Estimate (USD) Scope and Amortization Model Server Hardware Amortization (CapEx) $38,000 / month 4x 8-GPU nodes on a 3-year linear depreciation with 24/7 OEM support Datacenter Power & Colocation (OpEx) $8,500 / month 10–12 kW per rack, high-tier colocation, PUE 1.25 InfiniBand Network Amortization $4,000 / month Spine/Leaf switches, NDR transceivers, and optical cabling AI Infrastructure Engineering $6,500 / month Allocation of site reliability and infrastructure engineering time Total Monthly TCO ~$57,000 / month Annualized: ~$684,000 USD Break-Even Volume Analysis Assuming commercial APIs average $2.50 per million blended tokens (incorporating weighted prompt caching and completion pricing):

$$\text{Monthly Break-Even Volume} = \frac{\$57,000}{\$2.50 / 10^6} = 22,800,000,000 \text{ tokens (22.8 Billion/month)}$$

$$\text{Daily Break-Even Volume} \approx 760,000,000 \text{ tokens/day}$$

No. Kimi K3 contains 2.8T parameters. In FP8 precision (1 byte per parameter), raw uncompressed weights alone consume 2.8TB, exceeding the nominal physical memory of 32x 80GB GPUs ($32 \times 80 = 2560\text{GB} = 2.56\text{TB}$). This leaves zero memory for CUDA contexts, collective buffers, activations, and KV Cache pools. A 32-GPU deployment is made feasible strictly through Kimi K3's native MXFP4 checkpoint format (~0.5 bytes per parameter, ~1.4TB total weight footprint), preserving ~1TB of cluster memory for KV Cache pools.

Tensor Parallelism requires every GPU in the TP group to perform two high-frequency All-Reduce collectives per transformer layer. Intra-node, this runs across NVLink at 900 GB/s with microsecond latency. Pushing TP across network nodes limits bandwidth to 400Gbps (~50 GB/s)—18 times slower than NVLink—introducing substantial packet processing overhead. This starves GPU Tensor Cores, collapsing generation speeds. Multi-node setups must isolate TP to individual nodes (TP=8) and scale out via Pipeline Parallelism (PP) or Expert Parallelism (EP).

DeepSeek-V3's Multi-Head Latent Attention (MLA) compresses keys and values into a single low-rank latent vector $c_t^{KV}$, mathematically absorbing projection matrices during decoding. DeepSeek-V4-Pro's Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) form a heterogeneous tiered pipeline: CSA preserves localized sliding-window context and sparse global anchors, while HCA applies ultra-low-rank compression to long-range background tokens. This hybrid approach slashes KV memory requirements further while boosting prefill processing throughput across 128k contexts.

When tokens concentrate disproportionately on specific experts, the GPU ranks hosting those experts experience high compute latency, stalling the global All-to-All collective. Infrastructure teams should monitor per-expert token dispatch counters via Prometheus. If significant routing skew emerges, verify that system prompts do not contain bias-inducing repetitive keywords, configure dynamic auxiliary loss coefficients in the engine scheduler, or deploy duplicated redundant instances for heavily loaded experts.

Collective operation timeouts indicate network degradation or a stalled GPU rather than internal NCCL engine bugs. Troubleshoot via:

Originally published at LLM Systems Architecture on blog.llmgo.top.