Self-Healing
Automation Monitoring
A twice-daily audit that catches a scheduled automation gone silent, a missing schedule, a dead process, a run that produced nothing, fixes what's mechanical on its own, and escalates what actually needs a human. Built after three automations in this same business went silently, invisibly dark.
THE PROBLEM
An automation can fail with zero symptoms.
A daily automation was fully built, tested, and working, except the final "actually install the schedule" step got skipped. Nothing crashed. Nothing errored. It just never ran, and nobody knew until a human noticed a message that should have arrived hadn't. That's the worst kind of failure: silent, and only discovered by the person it was supposed to save time for.
WHAT CHANGED
From "someone notices it's missing" to "it fixes itself."
| No way to know a job silently never started | A registry of every scheduled automation, checked twice a day against what's actually installed. |
| A crashed always-on process looks fine from the outside | An explicit liveness check, not just "is it loaded", a loaded-but-dead job gets caught. |
| A job that ran but produced nothing looked identical to success | Every scheduled job's actual output is checked against its expected fire time. |
| Fixing it meant a human noticing something was missing first | Structural problems, not installed, dead process, stale output, get fixed automatically, before anyone has to notice. |
| Risk of "fixing" a real problem the wrong way | Only mechanical fixes are automatic. Anything needing judgment or a credential only the owner controls gets escalated, never guessed at. |
THE STRUGGLE
Building the fix found two more of the exact same bug.
The same day this was built, running its first real audit against the whole system immediately surfaced two more automations sitting in the identical broken state, fully built, install script sitting right there, never actually executed. Not a one-off mistake by one careless step; a structural gap that was quietly waiting to happen again, every time a new automation got built.
Three separate automations, built independently over different sessions, all failed the exact same way: working code, a working install script, and a final manual step that was easy to skip once and never revisit. None of the three threw an error. None of the three were "broken" in any way a test suite would catch, they were simply never turned on.
HOW IT'S BUILT
A registry, a boundary, and a rule against noise.
Every launchd-scheduled job in the system is registered once, its label, its install script, its trigger script, where its log lands. Twice a day (once before the day's jobs need to fire, once after, to catch anything that silently didn't run), the check walks that registry and asks three questions: is it installed, is it actually alive if it's supposed to be always-on, and did it produce fresh output by its expected time. Anything mechanical, not installed, a dead process, a missed run, gets fixed on the spot. Anything that needs judgment, an expired credential, a script that runs but throws, an out-of-credit API key, gets reported, never silently patched over or retried into a worse state.
THE FOLLOW-THROUGH
Who watches the watcher?
Any monitoring system has one obvious blind spot: itself. This one ran on its own schedule with nothing checking that the schedule was still firing, if its own entry ever got silently removed, or it crashed before sending its report, that would fail exactly as silently as the original bug it exists to catch.
Want this watching your own automations?
Happy to walk through exactly what it would take to wire this into your stack.