mtop 1.6 shows in-flight requests and explains slow local model performance

mtop is a terminal tool that monitors local model servers such as Ollama, llama.cpp, LM Studio, vLLM, Lemonade and SGLang, displaying loaded models, VRAM use, GPU status and request throughput. Version 1.6 now displays requests as soon as they begin, along with elapsed time, tokens so far and live tokens-per-second, plus time to first token and decode time on wider terminals. When it detects a cause of slowness, such as a model partly running from system RAM, a full context window, or queued requests, it reports this beneath the models table, and it also flags NVIDIA GPU throttling for heat or power. Per-request metrics are gathered through a local pass-through proxy on 127.0.0.1:4321, while the models and GPU panes work without it. The tool is available through Homebrew, Scoop and WinGet, or as a single MIT-licensed Go binary from GitHub.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in