Skip to content
Personal Digital Estate

2026-09-03 PM — Deleted WAL/SHM Inodes (state.db write block)

On September 3, 2026, Slack and Telegram replies started failing with “Sorry, I encountered an unexpected error”. The root cause was not a new state.db corruption: the gateway process born from the September 2 recovery restart (PR16) held deleted WAL/SHM inodes from its first second of life, and 18 hours later every session write began failing with file is not a database. state.db itself stayed healthy throughout; a plain gateway restart resolved it. Data loss: two in-flight turns only.

This is the same failure class as the September 1-2 corruption incident — a WAL/sidecar problem on the state.db write path — but with a different manifestation and a direct lineage: the conditions were created by the previous incident’s own recovery finalization. The incident record for the family stays the September 2 postmortem; this page documents the recurrence and the gaps it exposed.

Incident summary

Section titled “Incident summary”
Aspect Detail
Incident ID INC-2026-09-state.db-recurrence (same family as INC-2026-09-state.db)
Date September 3, 2026 (UTC)
Conditions created 2026-09-02 17:51:52 (recovery-finalization restart)
First write failures 2026-09-03 06:06:40 (Slack), 06:10:38 (Telegram)
First detection 2026-09-03 11:53:38 (user message)
Resolution 2026-09-03 12:36:21 (gateway restart, approved)
Severity Medium — messaging degraded/blocked ~5.5 h; no data corruption
Impact Slack + Telegram turns failed intermittently (06:06-10:55), fully blocked by 11:53; 2 in-flight turns lost their writes
Root cause Proven: gateway process (PID 125784) born with deleted state.db-wal/state.db-shm inode fds at the PR16 recovery restart; unlinked sidecars caused file is not a database on every DB write after 18 h of silent degradation
Data loss None (state.db verified clean; +3 sessions / +53 messages vs the 02:00 snapshot)

Timeline (evidence)

Section titled “Timeline (evidence)”
When (UTC) Event Evidence
Sep 2 17:51:52 Gateway restarts as the final step of the state.db recovery procedure (PR16 finalize) systemd journal: Started hermes-gateway.service
Sep 2 17:51:56 New gateway (PID 125784) logs Rebuilt stale state.db FTS indexes from canonical messages and restored sync triggers gateway journal
Sep 2 17:51 The new process’s state.db-wal / state.db-shm fds are already marked (deleted) — the unlink happened at (or within seconds of) first open, either from the recovery runbook’s Step 6 (rm -f state.db-shm state.db-wal) racing the first open, or from the FTS rebuild’s own sidecar handling /proc/125784/fd symlink targets, fd mtimes Sep 2 17:51
Sep 2 17:51 → Sep 3 06:06 SQLite writes keep working through the unlinked WAL inode — silent degradation, nothing in any log no journal lines
Sep 3 06:06:40 First failures: Session DB append_message failed: file is not a database, async token accounting: apply failed … file is not a database gateway journal
Sep 3 06:06:57-06:07:02 single-entry routing save failed … file is not a database, Failed to create session row, Agent error in session → Slack user sees “Sorry, I encountered an unexpected error” (François’s “Ready?” exchange) gateway journal
Sep 3 06:10:38 Same failure on Telegram (Agent error in session agent:main:telegram:dm:8660959290) gateway journal
Sep 3 08:55 / 09:55 / 10:55:20 Hourly :55 routing-save failures — the 3x canary that preceded user-facing breakage; none alerted gateway journal
Sep 3 11:53:38 User opens TUI session (“I receive this message when asking anything on slack or telegram”) — first detection session state.db
Sep 3 11:52-12:14 Diagnosis: live state.db reads clean (quick_check: ok, 309 sessions / 17,370 messages, integrity_check: ok), gateway PID 125784 holds deleted WAL/SHM inodes — no restore needed agent TUI session
Sep 3 12:36:21 Gateway restarted (user-approved, option 2 from TUI) systemd journal
Sep 3 12:36:24 New PID 144522; 0 deleted fds; Telegram per-profile TLS reconnecting /proc/144522/fd, journal
Sep 3 12:36:30 → Zero file is not a database / routing-save failures since restart; Slack replies healthy journal (verified 17:15)

Root cause analysis

Section titled “Root cause analysis”

Proven — where the deleted inodes came from

Section titled “Proven — where the deleted inodes came from”

The failure class is identical to the September 1-2 incident: WAL/SHM sidecars unlinked while a process holds them open, making the holder’s SQLite writes invisible to readers/checkpoints and eventually failing with file is not a database.

What is new and proven this time is the origin: the deleted-inode state was born with the process itself, at the finalization of the previous recovery, not created by any later operator action. The fd timestamps on PID 125784 show state.db-wal (deleted) and state.db-shm (deleted) dated Sep 2 17:51 — the same minute the process started. The recovery runbook’s Step 6 explicitly removes the sidecars (rm -f state.db-shm state.db-wal), and the first open then ran an FTS rebuild 4 s after start. One of those two actions unlinked the sidecars the fresh process had just opened.

The September 2 postmortem’s “strongest candidate trigger” (the Sep 1 06:52 restart window) shows no FTS-rebuild or sidecar line in the journal — so the original trigger remains unproven — but the mechanism is now confirmed: a service restart that touches state.db sidecars can leave the new process on unlinked inodes, silently, for hours. The runbook’s “Verify and restart” step never checked the process’s file-descriptor state, so the recovery was declared complete while the write path was already poisoned.

Why the monitoring did not fire

Section titled “Why the monitoring did not fire”

The FTS Repair Watchdog (deployed Sep 2 as postmortem action #8) greps for FTS repair remains blocked | FTS indexes remain corrupt | database disk image is malformed. Today’s signature — file is not a database / routing save failed / Failed to create session row — matches none of those patterns. The watchdog stayed silent for 18 h of degradation and 3 hourly canary failures. The write-failure signature must be part of the alert set.

What was NOT happening

Section titled “What was NOT happening”
  • state.db was never corrupted — PRAGMA quick_check and integrity_check both ok; the live DB held more sessions/messages than the 02:00 snapshot (309 vs 306 / 17,370 vs 17,317). No repair, no recovery, no restore was needed or performed.
  • No FTS/trigram damage — the FTS indexes were the healthy ones rebuilt at 17:51:56.

Detection and response

Section titled “Detection and response”
Phase Assessment
Detection Poor. 18 h silent degradation + 3 hourly canary failures before a user noticed. The watchdog watched the wrong signature.
Response Excellent. From first user contact (11:53) to resolution (12:36) in 43 min; correctly diagnosed process state vs database health (no restore attempted); restart resolved it.
Tooling The journal + /proc/<pid>/fd inspection was decisive. A one-line fd check in the runbook would have caught this at finalize time.

What went right / what went wrong

Section titled “What went right / what went wrong”

Went right

Section titled “Went right”
  • No data loss and no risky surgery: the diagnosis distinguished “process holds deleted sidecar inodes” from “database corrupted” within minutes — no restore, no FTS rebuild, no sidecar deletion.
  • Clean restart fixed it: 0 deleted fds, platforms reconnected, zero failures since — restart was the entire remediation.
  • The recovery artifacts were still in place (.malformed-backup-*, .corrupt-*, daily snapshots) — peace of mind, unused.

Went wrong

Section titled “Went wrong”
  • Recovery finalize left the write path poisoned and unverified — the runbook’s Step 7 checked hermes doctor and the rebuild log, but not the gateway’s fd state nor a real session write.
  • Watchdog blind spot — alert patterns covered FTS corruption only; the actual failure signature (file is not a database) was absent.
  • 18 h of silent degradation — a freshly-restarted gateway can write through an unlinked WAL inode for a long time before failing loudly.

Lessons learned

Section titled “Lessons learned”
  1. After ANY state.db recovery or restart, verify the process’s fd state: /proc/<pid>/fd must show zero (deleted) entries for state.db-wal / state.db-shm, and a real session write must succeed. A “clean” hermes doctor does not prove a healthy write path.
  2. file is not a database + healthy quick_check = stale connection / deleted inode → restart the gateway. Do NOT repair, restore, or touch the DB. Touching the database when the file itself is fine is how a trivial incident becomes a real one.
  3. The hourly :55 routing-save failure is a canary — it fired 3x before user-facing breakage. Any single occurrence of routing save failed is alert-worthy.
  4. Watchdogs must match the operation’s failure signature, not the previous incident’s — the FTS patterns were correct for Sep 2; the write-failure signature is the recurrence indicator for this class.

Action items

Section titled “Action items”
# Action Status
1 Add file is not a database / Failed to create session row / Session DB append_message failed to the FTS Repair Watchdog patterns (alert set covers the write-failure signature, not just FTS corruption) Done — cron “state.db Corruption & Write-Failure Watchdog” (every 5 min, no_agent), script ~/.hermes/scripts/fts_repair_watchdog.sh
2 Recovery runbook Step 7: add deleted-inode fd check + real session-write test before declaring recovery complete Done — see state.db Recovery — Procedure, “Deleted WAL/SHM inodes” quick fix
3 Add the “deleted WAL/SHM inode → restart the gateway” case to the runbook’s When-to-use table (symptom: file is not a database + healthy quick_check) Done — see state.db Recovery — Procedure
4 Upstream (NousResearch/hermes-agent): investigate whether the restart-time FTS rebuild (or recovery sidecar handling) unlinks WAL/SHM sidecars a fresh connection holds; the Rebuilt stale state.db FTS indexes path is the prime suspect Open — recommended; documented field-report scope in the Proposed Upstream state.db Report

The upstream report/comment was not posted as part of this documentation PR. The candidate issue is NousResearch/hermes-agent#78182, a closed report with adjacent but different scope. Before posting, follow the official CONTRIBUTING.md: search open and closed issues/PRs, verify the current source, avoid duplicates, and provide a sanitized reproducible environment and evidence chain. The complete proposed-message record is in the Proposed Upstream state.db Report. Keep this page as the incident-specific summary rather than duplicating the full message.

Section titled “Related”