Human-in-the-Loop
Email Automation
An AI drafts every reply. A person approves, edits, or skips it over a chat message. And the decision of whether anything actually sends never hinges on a model's judgment — it's decided by plain code reading an explicit command. Two real bugs shipped and were caught live; both are documented below, not hidden.
THE PROBLEM
Drafting doesn't scale. Auto-sending isn't safe.
Every application-related email needed a reply, but writing each one by hand didn't scale, and letting an AI send anything directly to a real person was too much risk to accept outright. The fix: a pipeline where the model only ever drafts, a person approves or edits over Telegram, and whether anything actually sends is decided by plain code reading an explicit command — never by a model's interpretation of what a message meant.
WHAT CHANGED
From every reply by hand to draft-approve-send.
| Every application email needed a manual reply | An AI drafts a reply the moment a labeled email comes in. |
| Risk of the AI inventing qualifications or facts | Drafting is grounded only in a real profile document and refuses to invent — anything the profile doesn't cover becomes an explicit placeholder, not a guess. |
| Risk of an AI accidentally sending something | Approval is parsed by plain regex code — approve, skip, or edit — not an LLM interpreting intent. A real send never hinges on a model's judgment. |
| Replies landing as disconnected new emails | Approved replies thread correctly into the original conversation, not as a new top-level email. |
THE STRUGGLE
Two real bugs, caught in live testing — both in ordinary code, not the AI.
The riskiest part of a human-in-the-loop system usually isn't the model — it's everything wrapped around it. Both bugs below were caught during real testing, before either could cause a real problem, but both are worth knowing about if the same pattern shows up elsewhere.
The system checked a stored Google credential for a missing permission scope by comparing it against itself: the library used to load the token sets its in-memory .scopes field to whatever scopes you ask for, not what's actually saved in the token file. So the staleness check was always comparing the desired scopes against the desired scopes — it could never actually detect a token that was missing a newly added permission.
Fix: read the raw stored token file's scopes directly, instead of trusting the loaded credential object's version of the truth.
A tracker spreadsheet is meant to be created once and reused. Environment variables are loaded into memory a single time when a process starts — so when the system wrote a newly created spreadsheet's ID back to its config file mid-run, the rest of that same run never saw the update. Worse, the step that persists the ID checked for an existing placeholder value and skipped the write when it found one, instead of replacing it. The result: six separate spreadsheets got created, one per application, before anyone caught it.
Fix: cache the ID in-process the moment it's created, and make the persist step actually overwrite the placeholder line rather than skip past it.
HOW IT'S BUILT
Draft with a model. Decide with code.
A labeled inbox is the only intake — no full-inbox scanning, which would be a noisier, more false-positive-prone signal. Each labeled email goes through a single structured drafting call grounded in a real profile document, returning a summary and a draft with explicit placeholders for anything it can't answer. That draft queues into a chat approval flow: approve, skip, or edit, parsed with plain pattern matching, not a model. Only an explicit approve triggers a real send, correctly threaded into the original conversation.
Need this for your own inbox?
The same draft-approve-send pattern applies to support replies, sales follow-up, or any inbox where the reply matters but the send shouldn't be automatic.