Running a 65B dense coding model on a single GPU: memory bandwidth and quantization trade-offs

A developer tested Ripple-65B, a dense 65-billion-parameter open coding model, on single consumer and workstation GPUs using llama.cpp. Because every parameter is active for each token, generation speed is limited mainly by how quickly the card can read memory, reaching about 14 to 16 tokens per second on a 48 GB RTX A6000. The model's mostly windowed attention keeps the key-value cache small enough that a 256K-token prompt fit on the same card once the cache was quantized. The support landed in llama.cpp in November, and the author recommends pinning a specific release that includes prebuilt CUDA binaries to speed up automated deployments. On a 24 GB RTX 3090, a 2-bit file ran faster at 23 to 31 tokens per second, though passing a simple agent test did not prove reliability on fresh coding tasks.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in