SGLang shares GPU memory across KV cache and recurrent states

SGLang has published a detailed account of Unified Memory, a serving feature for hybrid models that combine full-attention KV cache with recurrent or sliding-window state. Historically, reserving separate GPU-memory pools can strand unused capacity: short requests need more recurrent-state slots, while long prompts create heavy KV demand. Sharing the same memory budget lets the two kinds of state grow toward each other as workload composition changes, potentially fitting more useful sessions without increasing physical GPU memory.

The engineering design involves more than a flexible quota. SGLang places KV pages into contiguous page-major envelopes rather than scattering each page across layer-specific arrays. An allocator can reclaim those blocks and compact fragmented space, while stable virtual IDs point to their updated physical locations. This makes dynamic sharing feasible but also introduces address translation, layout requirements and occasional memory-copy costs. The article notes current attention-backend restrictions, including equal sizes for K and V heads in the supported path.

One publisher experiment demonstrates why eviction policy matters. After caching 28 prefixes and applying a long-request pressure workload on an RTX 5090, the improved shared-allocation stopping rule retained 19 prefixes versus 10 with the previous rule. The 28 returning requests finished 37.9% sooner in that setup. Both arms already used Unified Memory; the comparison specifically measures the eviction decision, not a transition from traditional fixed pools to unified storage. It would be incorrect to describe the result as a universal runtime throughput gain.

The SGLang team also reports that splitting the pressure request into smaller chunks removed that retention advantage, and that median output-token timing across five models changed by roughly one percent in either direction. For operators of long-running agents, the practical benefit is potentially fewer repeated prefills under suitable memory pressure, not a free speedup on every prompt. Teams should benchmark their own prefix-reuse distribution, backend compatibility, memory fragmentation and request mix before treating these figures as a capacity guarantee.