The token is the product, and recompute is waste. When an AI model's KV cache gets evicted during long conversations, the GPU has to rebuild context it already computed, and in a week of real agentic coding traffic we measured, 98% of tokens were cache reads. We tested offloading that cache to flash on a Dell PowerEdge XE7740 running four NVIDIA RTX PRO 6000 Blackwell GPUs and eight Solidigm D7-PS1030 SSDs. The payoff: up to 2.9x the serving throughput of a VRAM-only setup, and once the DRAM tier filled up and started dropping context, flash held 94% of its peak while DRAM fell to 42%. It also cut worst-case first-token wait on a resumed session from 13.9 seconds to 3.2. Full deep-dive at the link in bio.
#AI #Inference #Storage solidigmtechnology