#alerting

Aegis: Silence Is the Product

I run an alerting system on the home server called Aegis, and it watches the containers, the disks, the backups, the mounts, all the boring things that keep a small stack alive. The first version of it worked, in the sense that it never missed anything, but it also sent me way more messages than I actually needed to see.

My rule for it, the actual rule, is this: "Aegis is meant to tell me only when Jon has to do something. Thats it. My expectation is Jon has to action less than once a week, if ever." I said that after living with the first build for a while and noticing I was reading alert messages the way you read spam, skimming past most of them to find the one that mattered. I also asked the obvious question out loud: "Am I expected to read something every day 7 days a week forever? Thats useless." It is a hobby server, and nobody's job depends on it staying up, so a daily digest is a professional-ops habit and I do not run a professional ops team.

Read the full article →