TurboQuant Reality Check: Google's KV Cache Compression Claims vs Actual Improvement

Google announced TurboQuant, a technique for compressing the KV cache in transformer attention layers from 16 bits down to 2.5-3.5 bits per value. The claimed results — 6x memory reduction and 8x speedup — require significant context to evaluate. ## What It Actually Does Two-step compression of the KV cache (the memory that stores previous tokens' key-value pairs during inference): **Step 1: Random Rotation** (from PolarQuant). KV cache vectors have uneven information distribution across dimensions — some carry signal, some are noise, many are correlated. A random orthogonal rotation redistributes information evenly across all dimensions, making the vector more amenable to uniform quantization without information loss. **Step 2: Non-Uniform Quantization.** Instead of evenly spaced quantization levels, TurboQuant uses levels optimized for the actual value distribution (which follows a roughly Gaussian shape). After rotation, the distribution is predictable enough that a pre-computed lookup table works — no per-tensor calibration needed. ## The Marketing vs Reality **"8x speedup"** — Compared against FP32 (32-bit) baselines. Production LLM inference already uses FP16 or BF16 (16-bit). The actual speedup over current practice is roughly 2x, not 8x. **"6x memory savings"** — Applies only to the KV cache, which is a fraction of total inference memory. Model weights (the majority of memory usage) are unaffected. Total memory savings depends heavily on sequence length and batch size. **Unfair benchmarking** — In one comparison, they ran a competitor's technique (KVQuant) on CPU while running TurboQuant on an H100 GPU, then compared speeds. This is not a valid comparison. **No code released** — The technique was announced without open-sourcing any implementation, making independent verification impossible. ## What's Genuinely Useful The random rotation preprocessing step (from PolarQuant) is a real and valuable insight — it makes KV cache quantization more robust. The non-uniform quantization with pre-computed lookup tables eliminates per-tensor calibration overhead. These are genuine engineering contributions, just smaller than the headlines suggest.