OptiQ fixes three bugs to make multi-token prediction speculative decoding work on Apple Silicon
OptiQ set out to add Qwen3.5 and Qwen3.6 multi-token prediction (MTP) support, using the bundled prediction head as a draft model for speculative decoding. Wiring the feature through its serving stack worked at first, but the surrounding pieces needed three separate fixes. The initial design loaded a second copy of the base model weights in unified memory, which roughly doubled peak memory and prevented the 27B model from running on a 24 GB Mac. Sampled decoding also compared two independent samples instead of applying rejection sampling, which sharply cut acceptance rates. Even after switching to rejection sampling, throughput at temperature 1.0 trailed greedy decoding, and the team traced the remaining gap to how sampling truncation was applied in verification.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in