I run an alerting system on the home server called Aegis, and it watches the containers, the disks, the backups, the mounts, all the boring things that keep a small stack alive. The first version of it worked, in the sense that it never missed anything, but it also sent me way more messages than I actually needed to see.
My rule for it, the actual rule, is this: "Aegis is meant to tell me only when Jon has to do something. Thats it. My expectation is Jon has to action less than once a week, if ever." I said that after living with the first build for a while and noticing I was reading alert messages the way you read spam, skimming past most of them to find the one that mattered. I also asked the obvious question out loud: "Am I expected to read something every day 7 days a week forever? Thats useless." It is a hobby server, and nobody's job depends on it staying up, so a daily digest is a professional-ops habit and I do not run a professional ops team.
How it got noisy in the first place
The first version was built detector out, where every check we wrote got its own path to a channel, because that is how you build alerting when you are thinking about coverage first. More detectors meant more messages, and the reviews we did along the way made it worse, because when a message looked cluttered we cleaned up the wording instead of asking why it existed in the first place. We picked better severity words and tightened the formatting and made the digest read better, but what we never actually asked was whether the message needed to exist at all. We had quietly imported daily digests, severity ladders, executive briefs, the whole toolkit from professional operations centers, into a system where the entire point was that a human almost never has to look at it. We were optimizing for readability when the number of messages was the problem.
The rebuild: three gates before anything reaches me
The fix was fewer messages, and every candidate alert now has to clear three gates before it is allowed to become something I see.
The first is hold-down, where a problem has to persist across a run of the detector that owns it, not just show up once and vanish, and that persistence is counted in the detector's own run count rather than wall-clock minutes, so a slow-running check and a fast one both get a fair read before either pages me.
The second is machine-first, and if there is something the system can try on its own it tries that first, inside an escalation window sized to how urgent the problem actually is. A real emergency gets a short window before it escalates, a lower-severity issue gets a longer one, and the lowest tier does not escalate to me automatically at all, I can go look at it whenever I want but it will not come looking for me.
The third gate is an allow-list of things only a human can actually do something about, like a laptop being physically off, or a network mount being down in a way nothing local can remount, or money, when something is about to cost more than it should. If a problem does not land on that list, it is not mine to act on, so it does not get to page me even if it is technically unresolved.
What actually shows up
When something does clear all three gates, the card is written in plain language, what happened, what the machine already tried, and what I specifically need to do about it, with no severity codes and no jargon, and if the same problem is still open later the existing card gets edited in place instead of a new message going out. There is one card per issue, and it gets updated as the issue moves, so there is never a growing pile of duplicates for something that only needed one mention.
Overnight is fully quiet, every severity, no exceptions, because nothing on this box is worth waking up for, and the whole point only holds if it is measured and not assumed, so mentions per week is a number we actually track.
We found a bug in it during a drill. A long-running process was holding onto a full snapshot of the alert state in memory and saving the whole thing back to disk on every button click, and that overwrote changes another part of the system had written in between. It produced three duplicate cards for the same drill event before we caught it. Well, three duplicate cards from a project whose entire point is not sending duplicate cards. The fix was to stop caching a whole-file snapshot across events in a process something else is also writing to, and just read fresh each time. It seems obvious now that it is written down.