Engineering notes · Agent Runtime

Loom: building a persistent AI agent runtime

What it took to keep one AI agent alive, reachable, and honest for months on a single Mac — and what broke along the way.

By Matthew R. Smith · July 2026 · ~2,400 words

The worst outage my agent ever had produced no error. The process was up. The Telegram poller was polling. The terminal pane kept redrawing. And for roughly two days, every message I sent it went nowhere — ingested into an input box, never dispatched, never answered. Every passive health check I had read green the entire time.

That incident is the thesis of this whole project: long-running agents don’t usually fail by crashing. They fail by going silently deaf. A crash is the easy case — a supervisor sees a dead process and restarts it. Deafness is the hard case, because from the outside a deaf agent is nearly indistinguishable from an idle, healthy one. Local CPU is about zero in both states (generation happens server-side), the process tree looks normal, and the screen looks like a screen.

Loom is what I built to survive that. It’s an always-on Claude Code agent that lives in a tmux session on a single Mac, supervised by a stack of small launchd jobs that keep it running, verify it can actually hear me, restart it non-destructively when it can’t, and route failures from a fleet of about 35 scheduled automations — newsletters, site monitors, a small live trading experiment — back to the agent itself to investigate and fix. Its long-term memory is a directory of markdown files. Its channel to me is a Telegram bot. Its ground-truth liveness signal is a probe message it has to act on every twenty minutes.

Nothing about the design is elegant. Almost every mechanism exists because a specific failure forced it, and I think the honest way to present the system is as a residue of incidents rather than an architecture I drew up front.

By the numbers

Why persistence is hard

Running an agent for months instead of minutes surfaces problems that never show up in demos:

Sessions die and take working state with them. A CLI process gets killed, the machine reboots, a quota modal blocks input. If a restart means a blank context, every restart is a partial lobotomy — in-flight conversations, queued tasks, and half-finished commitments vanish. The lived experience of this, from my side of the Telegram chat, was “why do you keep resetting?” That complaint drove a redesign.

Context windows are finite. A session that runs for days accumulates transcript and backlog. The harness compacts old context, which preserves the thread but loses detail; a fresh session loses everything not written to disk. So long-term knowledge cannot live in the context window at all. It has to live somewhere durable.

Silent failure is the default failure. That outage was a job of my own that hung — but the same deaf-agent symptom also shows up for reasons upstream of my code, in the closed-source channel plumbing, where a CLI upgrade didn’t fix it. Either way it’s invisible from the outside, so everything I built around liveness is recovery machinery for problems I often can’t patch, only detect.

Liveness is nearly unobservable from the outside. Screen-scraping the TUI for strings like the input prompt is version-fragile, and CPU tells you nothing. The hardest-won lesson in the project is that passive observation cannot distinguish “idle and healthy” from “deaf.” More on that below, because I learned it the expensive way.

Exactly one process may hold the bot. A Telegram bot token supports one long-poller. Any other Claude invocation on the machine — a cron job, me opening an interactive session in a terminal — can steal the poller and silently deafen the agent. This single constraint caused more incidents than anything else on this list.

Why wrap a terminal at all? Nearly every failure in this post — the pane-scraping, the send-keys races, the modal dismissal, the wedge I can't patch — traces to one decision: I supervise an interactive CLI instead of owning the message pump myself. There's an Agent SDK that would let me do the latter: call the model directly, control dispatch, never scrape a screen. I chose the CLI on purpose. The Claude Code harness hands me tools, a permission model, context compaction, durable sessions, and a Telegram plugin that turns my phone into the agent's console — for free. Rebuilding that on the SDK means re-implementing most of a coding agent before I write a line of my own logic, and the wedge is rare enough that a thin supervision layer is cheaper than that rewrite. It's a deliberate bet, and I know which way it breaks: if the deafness ever got frequent, I'd move message dispatch onto the SDK and keep the harness for everything else. The scaffolding here is the price of not owning the pump — paid knowingly, not by accident.

The architecture

Five pieces, all on one machine: the agent itself, a watchdog (every 30 seconds), a liveness canary (every 20 minutes), a cron sentinel (every 10 minutes), and a serialized injection path that lets the supervisory jobs type into the agent’s session without corrupting it.

                  operator (Telegram, phone)
                      ▲               │
          escalations │               │ messages
          (backstop)  │               ▼
                 Telegram Bot API — exactly ONE long-poller
                      ▲               │
                      │ replies       ▼
┌─────────────┐   ┌───────────────────────────────────┐
│ sentinel    │   │ AGENT: claude + telegram plugin   │
│ scans ~35   │──▶│ in a tmux session                 │
│ jobs' logs, │   │  ├─ poller subtree                │
│ nudges the  │   │  ├─ file-based memory store       │
│ agent to    │   │  └─ tools: runs & fixes the fleet │
│ investigate │   └───────────────┬───────────────────┘
└─────────────┘                   │ pane output → log
                                  ▼
┌───────────────────────────────────────────────────────┐
│ WATCHDOG — every 30 s (passive, process-level):       │
│   reap orphan/foreign pollers · dismiss blocking      │
│   modals · poller gone >3 min, or idle + CPU flat     │
│   ──▶ kill tree, relaunch: RESUME pinned session      │
│       or FRESH, chosen by recorded failure cause      │
│       (thrash fuse: ≥3 relaunches/15 min → FRESH)     │
└───────────────────────────────────────────────────────┘
┌───────────────────────────────────────────────────────┐
│ CANARY — every 20 min (active, end-to-end):           │
│   inject a real message: "run this one command        │
│   to touch the ack file" · no ack in 3 min,           │
│   twice in a row ──▶ pump is dead, recycle            │
└───────────────────────────────────────────────────────┘

Memory is a directory of markdown files. One fact or topic per file, with a one-line index file that gets loaded into the agent’s context at every session start; the individual files are read on demand when an index line looks relevant. No database, no embeddings, no vector search — files are greppable, diffable, editable by the agent with the same tools it uses for everything else, and they survive any process death by construction. In practice the store behaves less like a knowledge base than an incident log with enforcement teeth: the watchdog’s memory file is a numbered list of failure modes, each ending in explicit invariants (“do not re-add this gate,” “never put this call in the hot path”). Because the index is re-read every session, a freshly restarted agent with zero conversational context still knows not to repeat a mistake made weeks earlier by a different session. The cost is that retrieval quality is exactly as good as the librarianship of past sessions. There’s no ranking and no pruning.

Continuity is session-ID pinning. The watchdog persists the agent’s session ID in a durable state directory. On relaunch it either resumes exactly that transcript or mints a fresh session — and the choice is cause-aware. Every kill path records a one-word cause, and a small pure function maps cause to policy: a crashed process or a missing poller resumes (the context is intact; preserve the thread), while a preventive recycle or a repeatedly-wedged session goes fresh (the context is the thing being shed, or is suspect). A thrash fuse forces fresh if there have been three or more relaunches in fifteen minutes, so a bad resume can never drive a restart loop. An earlier version used “resume most recent session,” which could grab a session belonging to some other Claude process on the machine and resume a stranger’s conversation into my Telegram pane. Pinning the exact ID closed that.

The watchdog checks process health. Every 30 seconds: is the tmux session there, is the agent process there (detected via a ps ancestry walk from the pane, because the obvious pgrep approach silently doesn’t work on macOS and left an earlier version of the watchdog broken for weeks), is a blocking modal on screen (dismiss it), is the poller subtree alive, has anything foreign stolen the bot (reap it). Every destructive action requires a definitive positive signal; ambiguity means do nothing and wait for the next tick.

The canary checks function — and this is the load-bearing idea. All of the watchdog’s checks are passive, and the dominant remaining failure is invisible to every one of them: the message pump wedges while the process, the poller, and the pane all look healthy. The only trustworthy liveness signal I found is to make the agent do something. Every twenty minutes, a script injects a message through the same input path real traffic uses: “liveness check — run this one command to touch the ack file, don’t reply on Telegram.” Then it watches for the file’s timestamp to change. The instruction travels with the message, so a brand-new session that has never seen the contract still obeys it. If the ack doesn’t land within three minutes, twice in a row, the pump is dead — regardless of how healthy everything looks — and the agent gets recycled. A passing canary certifies something no passive check can: a message can travel the entire path and cause an action in the world.

The sentinel closes the loop on the fleet. Every ten minutes it scans all ~35 scheduled jobs for nonzero exits and new error output (reading only new bytes past a stored offset, with a hand-tuned pattern-and-anti-pattern list to keep the signal clean). A detected failure doesn’t page me. It becomes a durable work item in a queue, and the sentinel injects a short nudge into the agent’s session: go read the log and the script, decide whether this is current or stale, fix it if safe, and tell me the resolution — not the raw error. It re-nudges every twenty minutes while the item stays open, and only after thirty minutes does it send me one concise backstop message. That inversion is the point of the whole system: I stopped being the message bus between my own cron jobs and my own agent. Failures route to the agent first, the human last.

What broke, and what I learned

This is the section I actually care about, because the system’s credibility lives here.

I shipped a confident false detector and had to amputate it. The nastiest wedge leaves a message sitting in the input box, so I added a passive gate: non-empty input buffer plus flat CPU for long enough equals deaf agent, recycle it. It seemed principled. In production it false-fired on real replies, on the resume boot menu, and on young sessions; every attempt to tighten it made it worse — 5, then 8, then 18, then 32 recycles a day, each one wiping the conversation it existed to protect. A “non-lossy replay” band-aid built on top of it re-injected garbled scrapings of the screen as fabricated user messages, which the agent nearly acted on. The postmortem, verified by cross-referencing the canary log (the gate once killed an agent 87 seconds after the canary had certified it alive), was that the discriminating signal does not exist in pane text plus local CPU. The gate and the replay were both removed, and the memory store now carries a do-not-rebuild invariant. Cost of the detour: several days of my conversations being destroyed every few minutes by the mechanism meant to protect them. Lesson: passive inference about liveness produces detectors that are confident and wrong, and a wrong recovery mechanism is worse than the failure it guards.

Recovery caused an outage as bad as anything it prevented. A per-tick lsof check degraded under load, falsely reported the poller missing, killed the agent, orphaned a poller, which raised load, which made lsof flakier — a positive feedback loop that produced 119 orphaned poller processes, a load average of 211, and a recycle every four and a half minutes for over eight hours. Separately, the canary once detected a wedge correctly but was wired to a resume policy whose fuse window (15 minutes) was shorter than the canary’s own period (about 23 minutes), so it faithfully revived the same wedged session four times in a row — two hours deaf, with every individual mechanism working as designed. The structural lesson: this system breaks in the interactions between individually-correct mechanisms, and that class of bug will recur every time I add one.

Self-oversight has a bootstrapping hole. The sentinel routes failures to the agent — but when the agent itself is the failed component, that routes work into a black hole. The two-day outage that opens this piece was exactly that failure: a backgrounded research workflow hung on its first phase and froze the main session, so the messages I sent queued unprocessed. I was travelling with no access to the machine that runs it, so it stayed deaf until I got home — which is exactly when silent failure is most dangerous, because the operator I was relying on to notice was me, and I wasn’t there. The canary now catches full deafness, but a session that is alive, acking canaries, and simply never getting to its queue is caught by nothing except me noticing.

Everything reads the screen, and that’s permanent fragility. Liveness classification, modal dismissal, idle detection — all of it greps terminal output for strings the CLI happens to print today. It breaks silently whenever the TUI is restyled, which happened once mid-crisis. There is no stable programmatic health API to build on; scraping is the only interface available.

One machine, one of everything. One Mac, one tmux pane, one bot token, one home directory, and no offsite replica of the memory store. A disk failure takes the agent, its memory, its supervisor, and its fleet down together. At personal scale I’ve accepted that single point of failure rather than mitigated it, and I’d rather say so than gesture at redundancy that doesn’t exist.

Some loss is accepted on purpose. A message that reaches the agent but never becomes a transcript turn is unrecoverable by design now — the replay alternative fabricated input, which is worse. Occasionally I re-send a message after a recycle. Dropping beats fabricating.

What this demonstrates

I don’t think Loom shows that persistent agents are easy. The incident log argues the opposite. What it shows is that one is operable, with a specific discipline: active end-to-end probes as the only trusted liveness signal; cause-aware, fuse-bounded recovery instead of blind restarts; durable file state and idempotent supervision ticks, so any component can die at any moment and the next tick reconstructs; failure routing that makes the agent the first responder for its own fleet; and institutional memory encoded exactly where the next session will re-read it, because the operator context resets constantly and the lessons must not.

It also demonstrates the anti-lessons, which I’d argue matter more: passive health inference produces confident false detectors. Recovery mechanisms compose into new failure modes. Every heuristic that reads a UI is a liability with a version number attached.

The honest open questions: the upstream wedge is survived, not fixed; liveness still depends on screen-scraping; memory recall is only as good as a hand-curated index; and there is no second machine. Porting Loom off this one box would force every piece that quietly leans on “it’s all local” — the tmux socket, the process-ancestry walks, the file locks — to grow real interfaces. That would be the honest test of whether this is an architecture or a very well-tended installation. I suspect the answer is “both, in that order,” but the only way to find out is the same way everything else here was found out: run it until it breaks, and write down what broke.