Groot-19B-A3B Runs 256K Context on One RTX 5090, With a Thin Memory Margin

GrootTeam claims its Groot-19B-A3B model can hold a 256K-token context on a single RTX 5090 GPU, an estimate based on memory sizes. A test using an FP8 build served through vLLM 0.32.0 confirmed the full window started and handled a 205,525-token prompt correctly. Memory headroom proved slim, with peak usage near the card's limit and no room to run other workloads on the GPU. The model also reasons before answering, so its thinking tokens count against output limits and can truncate replies if max_tokens is set too low. The tester noted that the thinking-disable option was not yet supported by their API, so thinking remained on during testing.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in