NVIDIA's ReplaySSM cuts state memory traffic in SSM decoding for hybrid language models

NVIDIA

NVIDIA describes ReplaySSM, a way to store the state of state-space model layers as a periodic checkpoint plus a buffer of recent inputs, instead of rewriting the full state at every decoding step. This roughly halves the memory traffic that dominates decoding in these hybrid models, and the method is integrated into TensorRT-LLM, vLLM, SGLang and FlashInfer.

• At batch size 256 the SSM kernel runs 1.46x to 1.80x faster than tuned vLLM kernels, and the full decode step 1.20x to 1.47x faster
• With four draft tokens on two 120B-class models, throughput at batch size 512 is 1.85x and 2.10x vLLM's speculative decoding
• On GPQA-Diamond a shared NVFP4 checkpoint loses 3 to 31 points versus FP32; a split-precision setup brings three of four models within about 2 points
• No speedup at batch size 1, and end-to-end gains are smaller than kernel-level gains; the blog says it runs in production in GLM-5.3-Flash and Kimi-K3