A silent memory spike with zero log entries revealed a 2.4M-restart rsyslog loop
A network engineer running a containerized lab noticed a brief 130 MB memory spike across four Arista cEOS routers on a Grafana dashboard, with no corresponding log entries to explain it. Initial investigation pointed to a burst of short-lived systemctl processes, but that turned out to be a red herring caused by stale metric series. Deeper digging uncovered that rsyslog had been crash-looping every 100 milliseconds — accumulating over 2.4 million restarts — because a socket conflict with systemd prevented it from binding properly and restart rate-limiting had been disabled. The rsyslog loop was ultimately unrelated to the spike, which was instead traced to ConfigAgent and Sysdb process memory rising in tandem. The investigation highlighted that an absence of log entries is itself a diagnostic signal, and that process-level telemetry can reveal issues that traditional log monitoring misses entirely.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in