2026-09-02 PM — state.db Corruption Postmortem

On September 1-2, 2026, state.db — the Hermes session store — went through a corruption chain that ended with Hermes refusing every message: Sending a message failed with session storage could not be written. Search had been silently degraded for over 32 hours before that. Recovery was completed with hermes sessions recover from a pre-surgery backup; the outage ended and the only lasting loss was the write window after that snapshot.

This page is the formal incident record. The step-by-step procedure to resolve a similar event is on the state.db Recovery — Procedure page; the verbatim operator log is the Recovery Session Log.

Recurrence — 2026-09-03: the same failure class returned the next day. A WAL/SHM sidecar-inode problem again blocked every session write (file is not a database), but this time the conditions were created by this incident’s own recovery finalization and the database was never corrupted — a gateway restart fixed it. Recorded separately: 2026-09-03 PM — Deleted WAL/SHM Inodes. It exposed two gaps closed since: the runbook now verifies post-restart fd state, and the watchdog now alerts on the write-failure signature.

Incident summary

Aspect	Detail
Incident ID	INC-2026-09-state.db (estate-internal)
Date	September 1-2, 2026 (UTC)
First detection	2026-09-01 07:49:54 (log), full block 2026-09-02 ~16:10
Resolution	2026-09-02 17:51 (gateway restart, doctor clean)
Severity	High — total write block, ephemeral loss of in-flight turns
Impact	Search degraded ~32 h; messaging fully blocked ~1 h 40 min; 244 messages / 4 sessions lost
Root cause	Trigger unproven (strongest candidate: 06:52 restart window). Aggravators proven: live FTS rebuild + WAL/SHM sidecar deletion while processes held the DB open
Data loss	Everything written after the Sep 1 07:58 snapshot (244 messages, 4 sessions)

Impact

Vector	Impact
session_search	Silent breakage from Sep 1 07:49 — looked up nothing / errored. Estate fell back to LIKE search, which went unnoticed
Scheduled cron jobs	cronjob action='run' failed with disk I/O error (only surfaced because a daily report was missing)
Messaging (Slack/Telegram/CLI)	Worked normally while only search was affected; fully blocked from ~16:10 Sep 2 (turn-stopped message)
Data	244 messages / 4 sessions permanently lost (post-07:58 window)
Active sessions at the time	A few in-flight turns were lost when the write path failed (estimated — not counted at the time); transcripts before the snapshot fully preserved

Timeline (evidence)

When (UTC)	Event	Evidence
Sep 1 06:52:00	gateway exits status=75/TEMPFAIL (systemd restart-on-failure) and the dashboard restarts the same second — abrupt service cycling 57 min before the first detected error; strongest candidate trigger for the FTS corruption	systemd journal
Before Sep 1 07:49	trigram FTS index corrupted. No error precedes the first read, so the exact write is not provable from logs — candidate causes are ranked in the root-cause section below	first error at 07:49:54
Sep 1 07:49:34	state.db.bak-20260901 appears — snapshot before the first repair attempt (agent session investigating the broken search). First appearance of a .bak-* is itself a signal that a problem was already underway	file mtime
Sep 1 07:49:54	hermes_state: FTS indexes remain corrupt (fts5: corruption found reading blob ... from table "messages_fts_trigram"); disabled FTS sync and retrying the canonical write — search degraded to LIKE, message writes continue	gateway journal
Sep 1 07:50-08:06	agent-session repairs (manual): probes, raw FTS commands, writable_schema surgery + VACUUM (08:06) — all fail with database disk image is malformed; the live rebuild converts the logical index defect into structural b-tree damage (invalid page numbers 70xxx, Child page depth differs). The 07:58 pre-surgery backup state.db.malformed-backup-20260901_075905 (still clean for canonical tables) is taken by the agent session before its surgery, not by hermes sessions repair. During this window the WAL/SHM sidecars were deleted while processes held them	journal tool-executor lines; file mtimes
Sep 1 08:52 → Sep 2 16:25	385+ deferred FTS rebuilds, one every ~5 min: every attempt finds foreign processes holding state.db (PIDs 42883 = gateway, 123862 = dashboard session API) and defers. Canonical writes keep working, search stays broken, nobody is alerted	Deferred stale state.db FTS rebuild ... (deferral N)
Sep 1 07:58	the 07:58 pre-surgery backup state.db.malformed-backup-20260901_075905 (taken by the agent session, not by hermes sessions repair) — the last clean copy of the canonical tables (303 sessions, 16,812 messages, 50 routing, 190 prompts; FTS b-trees already damaged, irrelevant — recover rebuilds them). The file the recovery ultimately used	recovery JSON report + read-only probe
Sep 2 ~16:10	a canonical message write finally fails — the page damage has reached the sessions / routing / system_prompts tables	the turn-stopped message
Sep 2 16:1x	hermes doctor, then hermes doctor --fix (as documented in the session log) — auto-repair fails, guidance says restore from backup	session log
Sep 2 16:12	hermes sessions repair makes state.db.malformed-backup-20260902_161230; recovery attempt from it: messages 17,056/17,056 OK, but sessions 0/307, gateway_routing 0/52, system_prompts 0/192 (database disk image is malformed)	recovery JSON report
Sep 2 16:2x	recovery from ...malformed-backup-20260901_075905: all tables verified, "verified": true, "loss_detected": false. Corrupt file preserved (state.db.corrupt-20260902), WAL/SHM sidecars removed, recovered DB installed	recovery JSON report
Sep 2 17:51	gateway restart: Rebuilt stale state.db FTS indexes from canonical messages and restored sync triggers; hermes doctor clean (306 sessions, FTS tables present)	gateway journal

Root cause analysis

What FTS5 trigram corruption actually is

state.db stores every message in a plain messages table; searching that table linearly for text fragments would be too slow on 17 000+ rows. SQLite’s FTS5 extension keeps a separate search index (shadow tables) next to it, and the trigram tokenizer cuts every message into all its 3-character subsequences (hermes → her, erm, rme, mes…). Searching “what did we decide about X” then looks up the index instead of scanning messages — this is what powers session_search.

The trigram index and the message table must stay consistent. SQLite detected the opposite at 07:49: corruption found reading blob 2473901162497 from table "messages_fts_trigram" — the b-tree page pointed to an index blob whose content did not match what the tree structure promised. That is an index-internal inconsistency: on-disk pages were overwritten, truncated, or reordered outside SQLite’s own write protocol. It does not touch the messages content itself (that all survived), it breaks search — which is why the fallback to LIKE search kept the estate functional that day.

Why it happened — proven vs unproven (“did we cause it?”)

SQLite protects FTS tables like any other: WAL journal, atomic commits, single-writer locking. An index does not corrupt “spontaneously” — it corrupts when something breaks that file protocol. Split what is proven from what is not:

Unproven — the trigger (before 07:49). Strongest candidate: the 06:52:00 service restart window — the gateway exited status=75/TEMPFAIL (systemd restart-on-failure) and the dashboard restarted the same second, 57 minutes before the first detected error. A sudden stop plus stale-sidecar handling around the restart is the textbook FTS-blob corruption trigger. There is no evidence of a wrong command before 07:49: no OOM, no unclean kill in the logs; the restart was TEMPFAIL-coded (the process asked to restart), not a kill.

Proven — the aggravators (07:49-08:52, operator/agent actions visible in the journal):

1. FTS rebuild against the live DB — INSERT INTO messages_fts(messages_fts) VALUES('rebuild') at 07:50, then writable_schema surgery, while the gateway was still writing. This converted the logical index defect into structural b-tree damage (invalid page numbers 70xxx). It was run as terminal commands through a hermes session, to fix the very error the tools were prompting about — with the “stop the gateway first” warning easy to miss.
2. Sidecar deletion while processes held them — from the first deferral log (08:52), both long-lived processes (gateway 42883, dashboard 123862) show state.db-wal (deleted) / state.db-shm (deleted): the WAL/SHM files were removed while writers kept them open. The holder keeps writing into the now-unlinked inode; checkpoints and new readers can no longer see those pages — the classic recipe for index blobs referencing missing or stale content.

Ruled out: storage-level defect (disk healthy, no I/O errors).

Honest bottom line: the exact write that broke the trigram index is not provable. The strongest admissible chain is: 06:52 restart window (trigger, unproven) → 07:49-08:52 live repair activity (aggravators, proven). The damage that turned it into a full outage was human-caused — but as a reasonable response to a trap the environment sets: the auto-heal can never fire on a multi-process host, and the tool warnings to stop services first are easy to miss without a runbook. Lesson: the sidecars are part of the database; treat them like it, and never work on it live.

Why it escalated into a full outage

Link	What happened	Why
1. Trigger	trigram index corruption, confined to search. Base tables readable; message writes still work	mechanism in “proven vs unproven” above
2. Aggravation	FTS rebuild against the live DB converted the logical defect into structural b-tree damage (invalid page numbers 70xxx)	vtable rebuild walks corrupt shadow b-trees and writes new index pages while the gateway keeps writing; hermes sessions repair could not restore the damaged canonical pages, and VACUUM cannot repair invalid page pointers; the recovered store was later verified as compact v23
3. Stall	auto-heal never fires on this host — 385+ deferred rebuilds over 32 h	“a later SessionDB open rebuilds the indexes” implies the next open closes; gateway + dashboard hold state.db open forever
4. Block	page damage eventually covered sessions / routing / system_prompts, so message writes failed and every turn stopped	damage spread beyond FTS shadow tables into base-table pages

Detection and response

Phase	Assessment
Detection	Poor. Search degradation went unnoticed for 32 h; surfaced only because a scheduled daily report was silently missing, then by a full messaging block. No monitoring on the FTS repair remains blocked log spam
Response	Good once engaged. The escalation path (doctor → repair → recover) with read-only probing of backup candidates (--inspect-only), JSON-report verification before install, and artifact retention was correct and loss-minimizing
Tooling	hermes sessions recover worked as designed: source never modified, separate clean output, verified report. Its docs are the weak link (flags and strategy undocumented upstream)

What went right / what went wrong

Went right

• Recovery was non-destructive end to end: recover copies the source before opening it, so both backup candidates could be probed and compared without burning them.
• A clean canonical snapshot existed: hermes sessions repair made state.db.malformed-backup-20260901_075905 (07:58) before its own surgery — the artifact that made near-lossless recovery possible.
• Base messages content survived: 16,812 messages fully preserved; the loss is only the post-snapshot write window.
• Corrupt files were retained, not deleted: enabled forensic analysis and a fallback if the installed DB proved incomplete.

Went wrong

• Live FTS rebuild converted a logical index defect into structural b-tree damage — the single action that most reduced recovery options and pushed the damage into base tables.
• Sidecar deletion (or at minimum a sidecar mismatch) during the repair window — the strongest admissible trigger for the original corruption.
• Auto-heal deferred 385+ times for 32 h with no alert — the gateway + dashboard hold the DB open forever, so “a later SessionDB open rebuilds” never happens on this host.
• No monitoring on the degradation signals — a watchdog would have reduced the 32 h to minutes.
• Backup coverage gap: the scheduled backup.sh excludes state.db (by design, GitHub repo size); pre-update snapshots and manual copies were the only sources — and the only automatic snapshot at the time was 33 days old. Recovery worked, but the margin was thinner than it should be. Closed since: daily state-db-snapshots/ (02:00 UTC, local, 7-day rotation) provides an at-worst-yesterday fallback.

Lessons learned

1. Never run raw SQL or vtable surgery against state.db while any Hermes process is running — FTS rebuilds, DROP/CREATE, VACUUM included. Use the sanctioned tools, with services stopped.
2. The -wal/-shm sidecars are part of the database. Never delete or replace them while a process has them open; when snapshotting a dirty DB, copy all three files together.
3. On a multi-process host the FTS auto-heal never fires. Treat “FTS repair remains blocked” as an alert condition, not background noise.
4. The newest backup is not automatically the best candidate — a verified recovery from an orderly snapshot beats a fragile one from a damaged file (the attempt-1/attempt-2 lesson, detailed in the procedure).
5. hermes sessions recover --inspect-only is cheap, non-destructive, and should be run against every candidate before choosing.
6. A malformed-backup filename is a provenance label (“snapshot taken from a database flagged as malformed”), not a verdict on the file’s contents — both candidates in this incident carried the name, only one was usable. Probe the tables, never trust the filename.
7. A state.db.bak-* file never appears on its own — it is created only when someone (human or agent session) starts repairing state.db. Seeing one is a signal that a problem has begun: check the store’s health (hermes doctor) rather than ignoring it. It is a repair artifact, not a routine backup.

Backup complementarity

backup.sh (daily, git repo) and the state.db daily snapshot are complementary, not competing:

System	Protects	Excludes
backup.sh daily (hermes-backup)	the irreplaceable: SOUL/CONTEXT/memory/skills/cron/config/scripts/profiles	state.db (too large for GitHub) and secrets (.env, auth.json — Bitwarden)
state.db daily snapshot (state-db-snapshots/)	full session history (yesterday at worst)	everything else

A full restore needs both + manual secret recovery from Bitwarden. Residual gap: the daily state-db snapshots are same-disk (protect against corruption, not disk loss); off-host snapshot shipping (~56 MB/day gz) is not sized for the GitHub repo and remains a documented future option.

Action items

#	Action	Status
1	Add a no_agent watchdog cron grepping logs for FTS repair remains blocked|FTS indexes remain corrupt → Slack alert (dashboard-security-watchdog pattern)	Done — cron “state.db Corruption & Write-Failure Watchdog” (every 5 min, no_agent), script ~/.hermes/scripts/fts_repair_watchdog.sh; it also covers file is not a database, Failed to create session row, and Session DB append_message failed
2	Add the runbook as the standard recovery procedure (this record’s sibling page)	Done — see state.db Recovery — Procedure
3	Quarterly offline maintenance: stop services → hermes sessions optimize-storage → start services → hermes doctor (only if the live layout is not already compact; the Sep 4 check found fts_storage_version = 1, so the command was a no-op)	Recommended
4	Upstream: document hermes sessions recover flags + a recovery walkthrough in the official Hermes docs	Open — recommended (owner: NousResearch/hermes-agent)
5	Keep state.db.bak-* / state.db.malformed-backup-* artifacts at least one week after an incident	Done (retained on the Agent Host)
6	Add a daily lightweight state.db snapshot (WAL-safe sqlite backup, local-only, 7-day rotation) so a future incident falls back to at worst a yesterday state — the 2026 incident’s only automatic fallback was a 33-day-old pre-update snapshot	Done — cron “Hermes state.db Daily Snapshot” (02:00 UTC), script ~/.hermes/scripts/state_db_backup.sh → ~/.hermes/state-db-snapshots/state.db-YYYYMMDD.gz; full health checks run before atomic publication
7	Add a freshness watchdog on the daily snapshot, so a silently-dead backup cron becomes a loud alert (the no_agent daily cron is silent on success AND when dead — indistinguishable without a watchdog)	Done — cron “state.db Snapshot Freshness Watchdog” (every 6h, no_agent), script ~/.hermes/scripts/state_db_snapshot_freshness.sh, alerts if newest snapshot missing or > 48h old
8	Keep a full read-only health watchdog for silent integrity and handle failures	Done — cron “state.db Health Watchdog” (hourly, no_agent, local-only), script ~/.hermes/scripts/state_db_health_watchdog.py; checks full integrity/FK health on all databases and deleted state.db/WAL/SHM descriptors
9	Verify post-restart fd state in the recovery runbook (the Sep 3 recurrence was born at this incident’s own 17:51 finalize restart)	Done — state.db Recovery — Procedure Step 7 now requires a deleted-inode fd check + real session-write test; new “Deleted WAL/SHM inodes — quick fix” section documents the restart-only fix for file is not a database on a healthy DB
10	Document the 2026-09-04 canonical-table recurrence, the rejected unverified recovery output, and the reference-check rule before row deletion	Done — 2026-09-04 recurrence PM
11	Record the upstream field report and documentation follow-up without posting it automatically; respect the official duplicate-search and contribution workflow	Done as documentation — Proposed Upstream state.db Report

Related

• Proposed Upstream state.db Report — the upstream field report and documentation follow-up are recorded there; nothing was posted
• Recovery Session Log — verbatim operator log: the exact commands that recovered the database
• state.db Recovery — Procedure — the step-by-step runbook (which backup to use, verification rules)
• state-db — schema and backup coverage of the session store
• backup — daily backup system (state.db intentionally excluded)
• The agent-host-ops skill keeps an internal repair-ladder reference (references/state-db-fts-corruption-repair.md) with the exact FTS demote/recreate SQL used for forensic copies