vLLM bi-directional KV transfer improves agentic workloads
New bi-directional KV cache transfer in vLLM improves TTFT for multi-turn agentic workloads on Amazon EC2.
vLLM has introduced bi-directional KV transfer to optimize multi-turn AI workloads. This improvement helps reduce the time it takes for an AI to respond as conversations grow longer and more complex.
What's new
- Bi-directional transfer: Unlike standard disaggregated serving which only moves data from prefill to decode nodes, this method allows decode nodes to send data back to prefill nodes.
- Reduced recomputation: Prefill nodes can now reuse previously computed context from the decode node instead of re-processing the entire conversation history.
- NIXL Enhancements: The feature uses an extended NVIDIA Inference Xfer Library (NIXL) KV connector to enable prefill nodes to consume remote KV blocks.
- Performance gains: Benchmarks on Amazon EC2 P5en instances showed TTFT reductions of up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8.
Why it matters
For developers building AI agents and conversational assistants, long interaction histories typically lead to higher latency and increased compute costs. As the conversation grows, the model must re-process all previous turns, which consumes more accelerator capacity. Bi-directional KV transfer helps maintain a responsive experience by minimizing this redundant work.
Deeper dive
To understand this, it is helpful to first understand the KV cache. When an AI processes text, it creates a "key-value" cache that acts as a short-term memory of the conversation. In Prefill-decode (P-D) disaggregation, one node (the prefill node) processes the prompt to build this cache, and another node (the decode node) retrieves those blocks to generate the response.
In traditional one-directional setups, the flow only goes from prefill to decode. This creates two inefficiencies: the prefill node cannot retrieve the KV cache entries held by the decode node for the next turn, and the prefill nodes are more likely to experience cache evictions for active sessions. This forces the prefill node to recompute the entire history, including system instructions and tool results, increasing the Time to First Token (TTFT).
The bi-directional approach allows the decode node to return previously computed KV blocks to the prefill node, coordinated by a KV-cache-aware router. This process utilizes AWS Elastic Fabric Adapter (EFA), a low-latency network. On supported instance types, EFA supports GPUDirect RDMA or Neuron Direct RDMA to move KV blocks directly between GPU or AWS Trainium device memory on the prefill and decode nodes.
Availability
This feature is an accelerator-agnostic capability contributed to vLLM. Performance results were validated using Amazon EC2 P5en GPU clusters connected with AWS EFA.
Source: aws.amazon.com