DeepInfer: DeepSeek Open-Sources Its Production Blackwell Inference Stack with FlashMLA, DeepEP, and DSpark Confidence Scheduling

Technical breakdown

Written by Abrar Akhunji

Serving massive Mixture-of-Experts architectures like DeepSeek-V3, DeepSeek-R1, and DeepSeek-V4.1-Flash at scale pushes hardware architectures to their absolute theoretical limits. Standard open-source inference stacks like vLLM and SGLang are exceptional general-purpose engines, but when running multi-hundred-billion parameter MoE models with Multi-Head Latent Attention (MLA), they hit three distinct architectural bottlenecks:

1. KV Cache Memory Bandwidth Stalls: In standard Multi-Head Attention (MHA) or Grouped-Query Attention (GQA), key and value tensors scale linearly with sequence length and batch size, saturating High Bandwidth Memory (HBM) bandwidth during the autoregressive decode phase.
2. MoE All-to-All Communication Latency: Routing tokens to top-$K$ experts distributed across separate physical GPUs requires continuous all-to-all communication primitives, resulting in severe GPU idle time if communication is not perfectly overlapped with compute.
3. Speculative Decoding Compute Bubbles: Fixed-length speculative decoding (e.g., verifying a static tree of 5 draft tokens) wastes substantial verification FLOPs when early draft tokens are rejected, creating pipeline stalls on busy inference clusters.

DeepInfer solves these problems by bundling an integrated suite of low-level kernels and communication libraries into a unified runtime targeting NVIDIA SM100 (Blackwell B200 / GB200) GPUs and Huawei Ascend 950 NPUs.

+---------------------------------------------------------------------------------------------------------------+
| DEEPINFER ARCHITECTURAL SPECIFICATION                                                                         |
+-----------------------------------+---------------------------------------------------------------------------+
| Architectural Subsystem           | Systems Implementation / Kernel Architecture                              |
+-----------------------------------+---------------------------------------------------------------------------+
| Latent Attention Kernel           | FlashMLA: Computes attention directly on compressed key-value latent c_t   |
| High-Speed Matrix Engine          | DeepGEMM: Low-overhead FP8/FP4 GEMM optimized for Blackwell TMA hardware   |
| MoE Communication Subsystem       | DeepEP: Overlapped All-to-All token dispatch with compute over NVLink/IB  |
| Expert Routing Arbiter            | DeepSelect: Vectorized top-K expert selection kernel                      |
| Dynamic Load Balancer             | EPLB (Expert-Parallel Load Balancer): Dynamic replication of hot experts  |
| Speculative Decoding Scheduler    | DSpark: Confidence-head guided adaptive draft tree expansion              |
| Tokenizer & Stream Parser         | deepseek-recipe: High-speed Rust library with Python bindings             |
| Hardware Precision Targets        | FP8 (E4M3 / E5M2), FP4 (NVFP4), BF16 Fallback                             |
| Target Accelerators               | NVIDIA Blackwell SM100, Huawei Ascend 950 NPU                             |
+-----------------------------------+---------------------------------------------------------------------------+

1. FlashMLA: Computing Attention on Compressed Latent Projections

In standard Transformer attention architectures, the Key and Value states are materialized as distinct matrices of shape $(B, L, h_{kv}, d_k)$. For long contexts, storing these matrices consumes the vast majority of GPU memory.

DeepSeek's Multi-Head Latent Attention (MLA) compresses the Key and Value representations into a single low-dimensional latent vector $c_t^{KV} \in \mathbb{R}^{d_c}$ (where $d_c \ll h_{kv} \times d_k$):

$$c_t^{KV} = W^{DKV} h_t$$

During standard inference in legacy engines, this compressed vector is decompressed back into full Keys and Values in memory prior to the attention calculation:

$$k_t^C = W^{UK} c_t^{KV}, \quad v_t^C = W^{UV} c_t^{KV}$$

This decompression destroys the memory bandwidth benefits of MLA. FlashMLA fundamentally avoids this decompression by mathematically folding the projection matrix $W^{UK}$ directly into the Query projection:

$$q_t^C = W^{UQ} c_t^Q$$

$$S_{t, i} = \frac{1}{\sqrt{d}} \left( q_t^C \left( W^{UK} c_i^{KV} \right)^T + q_t^R \left( k_i^R \right)^T \right)$$

$$S_{t, i} = \frac{1}{\sqrt{d}} \left( \left( q_t^C W^{UK} \right) \left( c_i^{KV} \right)^T + q_t^R \left( k_i^R \right)^T \right)$$

By defining an absorbed query representation:

$$\tilde{q}_t^C = q_t^C W^{UK}$$

FlashMLA computes attention dot products directly between $\tilde{q}_t^C$ and the cached latent vector $c_i^{KV}$. The full keys and values are never materialized in High Bandwidth Memory. This reduces the KV cache footprint by 85.3%, allowing an 8x B200 node to support sequences exceeding 256,000 tokens at high batch sizes without memory exhaustion.

2. DeepEP: Overlapping MoE All-to-All Communication with Tensor Core Compute

In distributed Mixture-of-Experts inference, Expert Parallelism (EP) assigns different experts to different GPUs. If a model routes $K = 8$ experts per token across 8 GPUs, every token must be dispatched to its destination GPU, processed by the local expert FFN, and gathered back:

                            DEEPEP OVERLAPPED EXECUTION FLOW
                            
    Rank i Tensor Cores                     Rank i Network Engine (NVLink / RoCE)
    ───────────────────                     ─────────────────────────────────────
    Compute Local Dense Projections (Q, K, V) ──► Start Asynchronous All-to-All Dispatch
    │                                             │ (Tokens dispatched to remote experts)
    ▼                                             ▼
    Execute FlashMLA Attention Kernel        ◄─── Background Transfer In-Flight
    │                                             │
    ▼                                             ▼
    Local Attention Complete                 ◄─── Tokens Arrive at Remote Buffers
    │                                             │
    ▼                                             ▼
    Compute Local Expert FFNs                ──► Start Asynchronous All-to-All Combine

Low-Latency Communication Primitives

Standard MPI or NCCL all_to_all primitives introduce significant software overhead because they synchronize all ranks at barrier points. DeepEP implements custom peer-to-peer CUDA kernels operating directly over NVLink memory mappings:

• Small token batches bypass NCCL ring buffers, writing directly into pre-allocated circular buffers on destination GPUs using NVLink direct memory access (DMA).
• Kernel execution is divided into two distinct phases: dispatch and combine. Both phases are dispatched on separate CUDA streams, allowing tensor core matrix multiplications (DeepGEMM) to execute concurrently with data transmission.

3. EPLB: The Expert-Parallel Load Balancer

A critical vulnerability in MoE inference is expert load imbalance. In natural language generation, certain experts (such as syntax formatting or common semantic tokens) are invoked 10 times more frequently than niche mathematical or domain-specific experts.

When expert parallelism is statically mapped:

$$\text{Latency}_{\text{step}} = \max_{j \in [1, E]} \left( \text{ExecutionTime}(\text{Expert}_j) \right)$$

The slowest, most congested expert dictates the latency for the entire cluster.

DeepInfer includes the Expert-Parallel Load Balancer (EPLB). EPLB continuously records expert routing histograms over sliding time windows. When an expert exceeds a designated congestion threshold $\theta_{\text{hot}}$, EPLB dynamically clones the expert weights across underutilized GPUs:

Dynamic Expert Replication:
Rank 0: [Expert 1 (Hot), Expert 2 (Cold)]  ──► Rank 0: [Expert 1 (Replica A)]
Rank 1: [Expert 3 (Hot), Expert 4 (Cold)]  ──► Rank 1: [Expert 1 (Replica B)]
Rank 2: [Expert 5 (Idle), Expert 6 (Idle)] ──► Rank 2: [Expert 3 (Replica A)]
Rank 3: [Expert 7 (Idle), Expert 8 (Idle)] ──► Rank 3: [Expert 3 (Replica B)]

Tokens routed to Expert 1 are split evenly across Rank 0 and Rank 1, cutting tail latency by up to 64% during burst traffic.

4. DSpark: Confidence-Scheduled Speculative Decoding

Speculative decoding pairs a fast draft model with a large target model. The draft model predicts a sequence of candidate tokens $(\hat{x}_{t+1}, \dots, \hat{x}_{t+\gamma})$, which the target model verifies in a single forward pass.

In fixed-window speculative decoding, the speculation depth $\gamma$ is a constant (e.g., $\gamma = 5$). However, when generating complex reasoning or code syntax, draft model accuracy drops sharply. If token $\hat{x}_{t+2}$ is rejected, tokens $\hat{x}_{t+3}$ through $\hat{x}_{t+5}$ are discarded, wasting verification compute and saturating tensor cores.

                         DSPARK CONFIDENCE PRUNING LOGIC
                         
    Draft Model Proposes Candidates:
    
    Token 1: "function"     ──► Confidence: 0.98  ──► KEEP (>= tau)
    Token 2: " calculate"   ──► Confidence: 0.94  ──► KEEP (>= tau)
    Token 3: "Tax"          ──► Confidence: 0.89  ──► KEEP (>= tau)
    Token 4: "("            ──► Confidence: 0.52  ──► PRUNE (< tau) - HALT EXPANSION!
    Token 5: [Not Computed]
    
    Result: Target model verifies 3 high-confidence tokens. Zero wasted verification FLOPs.

Mathematical Formulation

DSpark introduces a lightweight Confidence Head trained on intermediate transformer features of the draft model. The head outputs an estimated acceptance probability:

$$\hat{P}_{\text{accept}}(\hat{x}_{t+k}) = \sigma \left( w_c^T h_{t+k}^{\text{draft}} + b_c \right)$$

The DSpark scheduler modulates the speculation depth dynamically:

$$\gamma^* = \min \left\{ k \in [1, \gamma_{\max}] \;\middle|\; \hat{P}_{\text{accept}}(\hat{x}_{t+k}) < \tau_{\text{threshold}} \right\}$$

The confidence threshold $\tau_{\text{threshold}}$ is dynamically adjusted based on server load:

• Under low server load, $\tau$ decreases to encourage aggressive speculation.
• Under high cluster load, $\tau$ increases to prioritize deterministic verification and eliminate GPU compute waste.

This adaptive pruning increases effective decoding throughput by 2.4x compared to non-speculative baselines while reducing speculative verification FLOP waste by 71%.

5. Blackwell SM100 Exploits: TMA and Asynchronous Pipelines

DeepInfer is explicitly designed to capitalize on hardware primitives introduced in the NVIDIA Blackwell SM100 architecture:

Tensor Memory Accelerator (TMA)

In Hopper (H100), copying multi-dimensional tensors from global High Bandwidth Memory to Shared Memory (SRAM) required explicit register-assisted address calculations. Blackwell's TMA hardware unit performs multi-dimensional tensor copies asynchronously directly between HBM and Shared Memory with zero register overhead.

DeepInfer's DeepGEMM kernel utilizes TMA to issue asynchronous tile transfers, prefetching matrix tiles for step $k+2$ while Tensor Cores execute matrix multiplications for step $k$.

2-Bit and 4-Bit NVFP Quantization

DeepInfer supports native hardware NVFP4 precision on Blackwell Tensor Cores. By quantizing non-attention feed-forward weights to 4-bit floating point with block-level scaling factors, memory bandwidth requirements are halved compared to FP8, allowing massive models to operate entirely within on-chip cache hierarchies.

6. Production Benchmark Scorecards

DeepInfer was evaluated against vLLM v0.6.4 and SGLang v0.4.1 on an 8-way NVIDIA B200 (192GB HBM3e per GPU) cluster serving a 560-billion-parameter Mixture-of-Experts model:

+---------------------------------------------------------------------------------------------------------------+
| BENCHMARK EVALUATION: 8x NVIDIA B200 SERVING 560B MoE MODEL                                                   |
+-----------------------------------+--------------------+--------------------+---------------------------------+
| Systems Metric                    | vLLM v0.6.4        | SGLang v0.4.1      | DeepInfer (DeepSeek)            |
+-----------------------------------+--------------------+--------------------+---------------------------------+
| Decode Throughput @ 4k Context    | 62 tok/s/GPU       | 78 tok/s/GPU       | 142 tok/s/GPU                   |
| Decode Throughput @ 64k Context   | 41 tok/s/GPU       | 59 tok/s/GPU       | 128 tok/s/GPU                   |
| Decode Throughput @ 256k Context  | 16 tok/s/GPU       | 32 tok/s/GPU       | 108 tok/s/GPU                   |
| Time-To-First-Token (TTFT, 64k)   | 1,240 ms           | 890 ms             | 340 ms                          |
| Peak KV Cache Memory per Context  | 28.4 GB            | 26.2 GB            | 4.1 GB (FlashMLA)               |
| Speculative Acceptance Rate       | 54.2% (Fixed 5)    | 58.1% (EAGLE-2)    | 81.6% (DSpark Adaptive)         |
| Inter-GPU NVLink Idle Bubble Time | 38.4%              | 24.1%              | 4.2% (DeepEP Overlap)           |
+-----------------------------------+--------------------+--------------------+---------------------------------+

Key Engineering Takeaways:

1. 6.7x Throughput Advantage at 256k Context: At extreme sequence lengths, legacy engines stall due to KV cache memory bandwidth starvation. FlashMLA's compressed latent computation allows DeepInfer to maintain 108 tokens/sec/GPU when competitors drop below 35 tokens/sec.
2. Sub-350ms TTFT at 64k Context: By combining DeepGEMM asynchronous TMA prefetching with FlashMLA prefill kernels, Time-To-First-Token drops to 340ms, a 72% reduction compared to vLLM.
3. NVLink Idle Bubbles Reduced to 4.2%: DeepEP's concurrent execution stream design eliminates almost all network wait time during MoE expert dispatch.

7. Production Deployment: Running DeepInfer

DeepInfer provides a high-throughput runtime daemon that integrates directly with existing inference clusters.

Launching the DeepInfer Distributed Daemon

Run the cluster server across 8 GPUs using the official container image:

docker run --gpus all --ipc=host --net=host \
  -v /models:/models \
  deepseek/deepinfer:latest \
  deepinfer-serve \
  --model-path /models/deepseek-v4.1-flash \
  --tensor-parallel-size 1 \
  --expert-parallel-size 8 \
  --enable-flashmla \
  --enable-deepep \
  --speculative-engine dspark \
  --dspark-confidence-threshold 0.75 \
  --precision fp8 \
  --port 8000

Python Client Integration

DeepInfer exposes an OpenAI-compatible endpoint with streaming capabilities supported by deepseek-recipe:

# -*- coding: utf-8 -*-
"""
DeepInfer High-Throughput Inference Client
"""

import httpx

API_URL = "http://localhost:8000/v1/chat/completions"

def stream_deepinfer_response(prompt: str):
    payload = {
        "model": "deepseek-v4.1-flash",
        "messages": [
            {"role": "system", "content": "You are a senior systems engineer."},
            {"role": "user", "content": prompt}
        ],
        "temperature": 0.2,
        "stream": True,
        "extra_body": {
            "dspark_speculation": True,
            "max_draft_tokens": 5
        }
    }
    
    with httpx.stream("POST", API_URL, json=payload, timeout=60.0) as response:
        for chunk in response.iter_lines():
            if chunk.startswith("data: "):
                data = chunk[6:]
                if data == "[DONE]":
                    break
                print(data, end="", flush=True)

if __name__ == "__main__":
    stream_deepinfer_response("Explain how DeepEP overlaps all-to-all communication with tensor compute.")

8. Architectural Takeaways for Senior Systems Engineers

The release of DeepInfer signals three critical paradigms in modern AI infrastructure:

1. Kernel Co-Design Beats General-Purpose Runtimes: Generic inference runtimes that treat attention, feed-forward layers, and network calls as distinct sequential steps cannot extract peak hardware efficiency. DeepInfer proves that co-designing the model architecture (MLA + MoE) with custom CUDA/TMA kernels (FlashMLA + DeepEP) is required to unlock hardware potential.
2. Fixed Speculation is Dead: Treating speculative decoding as a static window is inefficient for real-world workloads. Confidence scheduling via lightweight prediction heads is the future of test-time acceleration on high-throughput clusters.
3. Latency is a Topology Problem: In multi-GPU MoE systems, bandwidth matters less than synchronization overhead. Overlapping all-to-all token dispatch directly inside the execution streams of local attention blocks transforms network communication from a blocker into a background operation.