vLLM push-based KV transfer improves inference speed
vLLM's new push-based KV transfer reduces latency for disaggregated LLM inference, offering up to 3x better TTFT on AWS EC2.
The introduction of vLLM push-based KV transfer aims to significantly reduce latency for disaggregated large language model (LLM) inference. This update optimizes how data moves between computing instances, helping to speed up both the initial response time and the subsequent generation of text.
What's new
- A new push-mode for the NVIDIA Inference Xfer Library (NIXL) connector.
- The ability to move transfer coordination and setup tasks off the critical path of token generation.
- An architecture that allows the decode instance to register target blocks in advance while the prefill instance is still computing.
- Improved support for RDMA (Remote Direct Memory Access) via AWS Elastic Fabric Adapter (EFA).
Why it matters
When running large language models at scale, developers often use "disaggregated inference." This means splitting the workload into two parts: a "prefill" phase that processes the user's prompt and a "decode" phase that generates the answer. These phases are often run on separate server instances to optimize resources.
The main challenge in this setup is the time it takes to move the "KV cache"—the data the model uses to remember the context of the conversation—from the prefill instance to the decode instance. By using a push-based method, vLLM reduces the time users wait for the first token to appear, making AI applications feel much more responsive, especially under heavy workloads.
Deeper dive
To understand the improvement, it is helpful to understand the KV cache. When an LLM processes a prompt, it creates a key-value (KV) cache so it does not have to re-calculate everything for every new word it generates. In disaggregated inference, this cache must be transferred across a network from the prefill server to the decode server.
In the traditional "pull-mode," the process is sequential and creates a bottleneck. The decode instance must wait for the router to relay metadata, then it must allocate its own destination blocks, and finally, it must initiate a NIXL READ to pull the data. This entire sequence happens on the critical path before the first token can be generated.
The new push-mode removes these steps from the critical path by overlapping the work of the two instances. The decode instance registers its target blocks ahead of time. Meanwhile, the prefill instance uses a background thread to prepare the NIXL descriptors and submit the transfer as soon as the blocks are ready. This removes the "round trip" and the setup time from the decode instance's latency-sensitive phase.
This transfer utilizes the NVIDIA Inference Xfer Library (NIXL) over interconnects that support RDMA, such as AWS EFA. This allows for GPUDirect RDMA and Neuron Direct RDMA, enabling the transfer of KV blocks directly between GPU or AWS Trainium device memory without the need to stage data through the host memory.
Availability
The push-based KV transfer feature is accelerator-agnostic and has been validated on AWS Trainium clusters and Amazon EC2 P5en instances using AWS EFA. In benchmarks using the Qwen3-32B model on Amazon EC2 P5en instances, the push mode delivered up to 3x better time-to-first-token (TTFT) and up to 30 percent better time-per-output-token (TPOT) compared to pull mode.
Source: aws.amazon.com