HyQuant cuts LLM attention compute cost in half with under 1% accuracy loss

Researchers have developed HyQuant, a hybrid-precision quantization method that reduces the computational cost of large language model attention while keeping accuracy degradation below 1%. The approach quantizes most query, key, and value tensors to low-bit formats while retaining a small subset of critical tokens and a local sliding window in full precision, identified by a lightweight pattern detector. In benchmarks, HyQuant achieved between 1.32x and 3.58x decode-kernel speedup and up to 1.17x end-to-end decode speedup across long-context and reasoning tasks. The overhead introduced by the precision detector accounts for only 3–5% of total runtime, making it practical even on dense GPU workloads. The method also extends to vision-language models and ships as a Triton kernel patch that integrates into existing pipelines with a single import.