How Three Silent Gaps in Loki-Grafana Pipelines Let SSH Attacks Go Undetected

A team running SSH brute-force detection across bastion hosts discovered their Grafana-Loki alerting pipeline was silently failing despite hundreds of failed login attempts per minute being logged. A routine audit revealed the attack had gone unnoticed for days, with no alerts firing and the on-call team completely unaware. Investigation identified three compounding root causes: inconsistent sshd log format coverage in regex rules, missing source IP field extraction in Promtail that caused costly query timeouts, and an alerting rule whose routing label did not match any configured Alertmanager receiver. Alertmanager silently dropped unmatched alerts rather than raising an error, making the failure invisible. The incident highlights how individually minor misconfigurations in a logging pipeline can combine to create complete detection blind spots.