state.db Recovery — Procedure

This page is the operational procedure for repairing a damaged state.db (the Hermes session store, schema and backup coverage: state-db). It is the sibling of the Incident Postmortem (2026-09-02), which explains why this can happen. Use this procedure whenever Hermes reports storage problems, hermes doctor flags state.db, or session_search stops working.

When to use this procedure

Symptom	Action
User-facing `No reply: the turn was stopped because session storage could not be written`	Full block — run this procedure now
`hermes doctor`: `state.db FTS write corruption` / `FTS repair is blocked after N deferral(s) by PID(s) [...]`	Degraded — run this procedure soon (with services stopped)
`session_search` returns nothing / errors; dashboard session API 500; cron fires fail with `disk I/O error`	Search-index only — run this procedure at the next maintenance window, then investigate
Log spam every ~5 min: `Deferred stale state.db FTS rebuild while foreign processes hold the database or WAL sidecars (...)` then `state.db FTS repair remains blocked after N deferrals by holder(s) (...)`	Same as degraded
Gateway replies “Sorry, I encountered an unexpected error”; journal shows `file is not a database` / `Failed to create session row` / `routing save failed` — but the DB itself reads clean (`integrity_check: ok`)	Quick fix, not this procedure — stale-connection symptom, see Deleted WAL/SHM inodes — quick fix. Do NOT repair/restore a healthy file
Full `PRAGMA integrity_check` reports `invalid page number`, `never used`, or another structural error while `quick_check` passes	Full procedure — the database can be silently damaged even when shallow probes pass

Anything in this table is also an alert condition — the state.db Corruption & Write-Failure Watchdog (see the postmortem action items) covers both the FTS-corruption and the file is not a database write-failure signatures.

Deleted WAL/SHM inodes — quick fix

A gateway process that was restarted into a state.db whose sidecars were replaced/unlinked can run for hours writing through deleted WAL/SHM inodes (state.db-wal (deleted) in /proc/<pid>/fd), then start failing every write with file is not a database — while state.db itself is perfectly healthy. This is what happened on 2026-09-03 (18 h after the recovery-finalization restart of the Sep 2 incident; details: 2026-09-03 PM).

Detect — the DB is fine, the process is not:

# 1. Is the database itself healthy?
hermes doctor                 # state.db section clean → the file is NOT the problem
python3 -c "import sqlite3; print(sqlite3.connect('/home/hermes/.hermes/state.db').execute('PRAGMA integrity_check').fetchone()[0])"

# 2. Does the gateway hold deleted sidecar inodes?
GW_PID=$(systemctl --user show -p MainPID --value hermes-gateway)
ls -l /proc/$GW_PID/fd | grep -E 'state\.db-(wal|shm)'
#   → "state.db-wal (deleted)" / "state.db-shm (deleted)" = stale-connection corruption

Fix — restart the gateway. No snapshot, no recovery, no restore, no sidecar deletion:

systemctl --user restart hermes-gateway

Verify — new PID holds fresh (non-deleted) fds and a real turn writes:

GW_PID=$(systemctl --user show -p MainPID --value hermes-gateway)
ls -l /proc/$GW_PID/fd | grep -c deleted        # must be 0
# then send a real Slack/Telegram message and confirm a reply

If the full integrity_check also fails, or hermes doctor flags FTS/canonical corruption, continue to the full procedure below instead.

Which backup for which use case

Different Hermes backup mechanisms answer different questions. Choose by situation:

Situation	What to use	Services stopped?	Notes
`state.db` corrupted/malformed (search broken, `database disk image is malformed`, turns won’t save)	`hermes sessions recover --source <backup> --inspect-only` → `--output` (sources: `state-db-snapshots/` daily, `.malformed-backup-`, `.bak-`, pre-update snapshots)	Yes	Rebuilds a clean `state.db` into a separate file; verify `recovery.json` before installing
Migrate to a new host (change provider — the full estate: config, skills, sessions, memory, `.env` secrets, `auth.json`)	`hermes backup` → zip, transfer → `hermes backup import <zip>` on the new host	Yes (stop everything on the old host before zipping; the zip is a consistent snapshot)	The only mechanism that carries `.env` + `auth.json` (restored with 0600 perms). `state-snapshots/` and `backups/` are excluded from the zip
Rebuild a whole Hermes from scratch / disaster recovery (same host, everything lost except local snapshots)	`hermes backup import <zip>` (CLI)	Yes	Overwrites `~/.hermes`; not undoable. Do not use the dashboard restore UI while it is running
Daily safety net for `state.db` (corruption-only, so a future incident falls back to yesterday at worst)	cron `state_db_backup.sh` → `state-db-snapshots/state.db-YYYYMMDD.gz`	No (read-only, WAL-safe)	Automatic; alerts via the freshness watchdog if it goes stale
Daily off-host backup of the irreplaceable (SOUL, memory, skills, cron, config, profiles — NOT `state.db`, NOT secrets)	`backup.sh` → private GitHub repo	No	Survives VPS disk loss for the config and memory state; full session history is NOT in it
Pin a state before an upgrade / experiment	`hermes backup --quick` (or `/snapshot`) → `state-snapshots/`	Recommended	Cheap config+`state.db`+secrets+cron snapshot; also taken automatically before `hermes update`
Secrets alone (`.env`, `auth.json`)	Bitwarden	—	The zip carries them, but a full restore needs the Bitwarden copy as the source of truth

Rule of thumb: restore into a broken-but-present install → hermes sessions recover (state.db only) or backup import (everything); move to a new machine → hermes backup + backup import; ongoing safety net → daily snapshot + backup.sh.

Diagnosis first

hermes doctor — read the state.db section: corruption type (FTS write corruption vs. schema issue vs. full file damage) and which PIDs hold the DB.
hermes sessions repair --check-only — reports whether the database opens cleanly, without modifying anything.

If the damage looks FTS-only, confirm scope with a real scan (shallow probes lie: SELECT * FROM messages_fts LIMIT 0 can pass while a real scan or MATCH raises database disk image is malformed (11)):

python3 -c "
import sqlite3
con = sqlite3.connect('file:/home/hermes/.hermes/state.db?mode=ro', uri=True)
print(con.execute('SELECT count(*) FROM messages_fts_idx').fetchall())
print(con.execute('PRAGMA integrity_check').fetchall()[:5])
"

A successful COUNT(*), quick_check, or recover --inspect-only result is not an installation approval. Keep the original source immutable while probing it. For an unreadable canonical row, enumerate all references (for example, sessions.system_prompt_hash) before any deletion, determine whether the value can be regenerated, and record the data-loss decision.

Recovery procedure

Step 1 — Stop all writers

state.db is a single-writer SQLite database: gateway, dashboard, and any CLI session must be down before any repair or rebuild, or the repair itself can corrupt the file further.

systemctl --user stop hermes-gateway hermes-dashboard
ps aux | grep -i hermes        # inspect every remaining process
lsof /home/hermes/.hermes/state.db 2>/dev/null

Important: stop the two systemd services and then inspect all Hermes PIDs. A foreground CLI, desktop backend, or detached worker can retain a descriptor after the services stop. No Hermes process may hold state.db, state.db-wal, or state.db-shm before a write-side operation.

Step 2 — Snapshot everything first

Copy the main file and the WAL/SHM sidecars when present; they are one SQLite database bundle:

cd ~/.hermes
STAMP=$(date +%Y%m%d_%H%M%S)
cp -a state.db "state.db.bak-$STAMP"
cp -a state.db-wal "state.db.bak-$STAMP-wal" 2>/dev/null || true
cp -a state.db-shm "state.db.bak-$STAMP-shm" 2>/dev/null || true

Also retain the latest daily snapshot and any pre-update or repair-created artifacts. Do not delete the original or sidecars before the recovery decision is complete.

Step 3 — Try the non-destructive repair first

hermes sessions repair --check-only    # report only
hermes sessions repair                 # makes state.db.malformed-backup-<ts> first

On failure it prints the recover guidance; keep going. If the repair succeeds, continue to Step 7 and verify the actual write path. The live estate already uses the compact v23 FTS layout; optimize-storage reported “already on the compact layout” during the September 4 maintenance check.

Step 4 — `hermes sessions recover`: choose the right backup

This is the heart of the procedure. recover is non-destructive: it copies the source (plus its WAL/SHM sidecars) before SQLite opens anything and writes a separate output database — the source is never modified. That means you can probe every candidate cheaply before committing.

Inventory the candidates (newest → oldest):

ls -lt ~/.hermes/state.db* | grep -vE '(-shm|-wal)$|\.lock$|repair-attempts'
ls -lh ~/.hermes/state-db-snapshots/

Candidates are state.db.malformed-backup-* (pre-surgery backups — taken by hermes sessions repair, or by an agent session before manual surgery, as in this incident), state.db.bak-* (snapshot before a repair attempt — taken by an agent session or manually), the daily snapshots state-db-snapshots/state.db-YYYYMMDD.gz (newest is typically the best candidate — decompress to a temp file first), or the corrupt state.db itself — the worked example below shows how they rank.

Name caveat: a malformed-backup filename is a provenance stamp, not a usability verdict. hermes sessions repair labels every backup it takes before surgery from a database it flagged as malformed. Both candidates in the 2026 incident were named malformed-backup — the 07:58 one had perfectly readable canonical tables (tool verdict: all tables available, zero warnings), the 16:12 one did not. Always probe the tables; never trust the filename.

Which file to use first — decision rule

The FTS index state of a candidate is irrelevant — recover rebuilds it from messages. The only thing that matters is: are the canonical tables intact, and how recent is the data? Work through the candidates newest → oldest, probing each:

hermes sessions recover --source ~/.hermes/<candidate> --inspect-only

The report lists which canonical tables are readable — look at sessions, gateway_routing, system_prompts, messages (and read the full JSON: <output>.recovery.json).

Worked example — the three real candidates of the 2026 incident:

How to tell which is older: the filename embeds the snapshot time — malformed-backup-YYYYMMDD_HHMMSS (exact to the minute) or bak-YYYYMMDD (date only, created by manual cp). Sorting the name timestamps descending works within a format, but the two formats DON’T compare directly. When only the date matches (e.g. bak-20260901 vs malformed-backup-20260901_075905 on the same day):

The bak-* has no hour in its name — its mtime is the creation time, because a plain cp stamps the new file with the copy moment (cp -p / rsync -a would preserve the source’s old mtime instead — check with stat -c '%y' <file> before trusting it).

ls -lt ~/.hermes/state.db* sorts by mtime and separates them: here bak-20260901 mtime 07:49 < malformed-backup-20260901_075905 07:58 → the malformed one is newer.

File	Created	sessions	messages	gateway_routing	system_prompts	Verdict
`state.db.malformed-backup-20260902_161230`	Sep 2 16:12	`false` (0/307)	`true` (17,056)	`false` (0/52)	`false` (0/192)	rejected (attempt 1)
`state.db.malformed-backup-20260901_075905`	Sep 1 07:58	`true` (303)	`true` (16,812)	`true` (50)	`true` (190)	used (attempt 2)
`state.db.bak-20260901`	Sep 1 07:49	`true` (303)	`true` (16,774)	`true` (50)	`true` (190)	would have worked — older, slightly less data, never needed

All three were “corrupt” in the FTS sense; only the middle one maximized both intactness and freshness. That is the target: newest candidate with fully intact canonical tables — true on all four.

Situation	Choose
Newest candidate: all canonical tables readable	Recover from it — smallest data gap
Newest candidate: `messages` readable but `sessions` / `gateway_routing` / `system_prompts` fail (`0/N`)	Rewind to the next older candidate. A recovery without session metadata is not a recovery: sessions vanish, routing breaks, messages become orphans. Accept the data gap instead
Multiple candidates are clean	Newest clean one wins
Only the corrupt file left	`--allow-partial` salvage; keep looking for older backups first

Trade-off when no candidate is fully clean:

Source	Result	Cost
Latest (post-damage)	`messages` usually survives; metadata tables likely lost	no date gap, but broken session metadata / routing — not usable as-is
Older clean snapshot	full metadata + all messages up to the snapshot	data gap = everything written after the snapshot

New since 2026-09-02: state-db-snapshots/ (daily 02:00, WAL-safe sqlite backup, gzip, 7-day rotation) is the first candidate family to check — the newest entry is a clean, verified, at-worst-yesterday state. Manual .bak-* / .malformed-backup-* remain fallbacks, and the pre-update snapshot (~/.hermes/state-snapshots/) is the oldest-resort automatic source.

Step 5 — Recover into a separate file and verify the report

hermes sessions recover --source <chosen candidate> --output ~/recovered-state.db

Then open <output>.recovery.json (default <output>.recovery.json) and require all of:

Check	Required value
`"verified"`	`true`
`"integrity_check"`	`["ok"]`
`"foreign_key_check"`	`[]` (empty)
`"loss_detected"`	`false`
`"opens_cleanly"`	`true`

"complete": false or any failed canonical table = do not install. Either rewind to an older candidate or, if nothing better exists, run --allow-partial (every skipped range is recorded in the report) and label the outcome.

Step 6 — Install only a verified output

cd ~/.hermes
mv state.db "state.db.corrupt-$(date +%Y%m%d_%H%M%S)"   # preserve, never delete
rm -f state.db-shm state.db-wal
cp ~/recovered-state.db state.db

This is a write-side operation: all writers must remain stopped. Keep the renamed corrupt file and the recovery report for at least one week.

Step 7 — Restart and verify the actual write path

hermes doctor        # state.db section clean; FTS tables listed; no corruption/staleness flags
systemctl --user start hermes-gateway hermes-dashboard

The first open may rebuild FTS from canonical messages. Verify search works (session_search or a dashboard session query), then manually fire any cron jobs that failed during the outage.

Post-restart verification is mandatory: the gateway can hold deleted WAL/SHM inodes even though the DB and hermes doctor look clean. Verify every Hermes process, not only the systemd MainPID values:

for pid in $(pgrep -f hermes || true); do
    printf 'PID %s: ' "$pid"
    find "/proc/$pid/fd" -lname '* (deleted)' -printf '%f -> %l\n' 2>/dev/null | grep -E 'state\.db|hermes' || true
done

# The expected result is no deleted state.db, state.db-wal, or state.db-shm descriptor.
# Then send a real Slack/Telegram message and confirm the session write succeeds.

If any state.db-wal (deleted) / state.db-shm (deleted) appears, restart the affected process once more and repeat the descriptor and real-write checks. Do not restore the database you just verified as healthy.

Step 8 — Post-incident

Keep all artifacts (state.db.corrupt-*, state.db.malformed-backup-*, state.db.bak-*) for at least one week.
Verify the post-restart descriptor state; the 2026-09-03 recurrence proved that a clean-looking restart can carry deleted WAL/SHM inodes silently.
If the accepted data gap later matters, the retained corrupt file can still be mined with --allow-partial.
Write or update the incident record (postmortems/YYYYMMDD_PM_<topic>.md) and link the operator log when one exists.
After the retention window, clean up recovery outputs (~/recovered-state*.db, ~/recovered-state*.recovery.json) because they duplicate the installed state.db and can reach hundreds of MB.

Command reference

`hermes sessions recover`

Flag	Purpose
`--source PATH`	Source `state.db` or preserved backup to inspect/recover (required). Source + sidecars are copied before SQLite opens anything — the source is never modified
`--output PATH`	New recovery database path (required unless `--inspect-only`)
`--inspect-only`	Only report canonical-table readability; no output database
`--work-dir PATH`	Directory for the disposable source copy (default: beside the output)
`--chunk-size N`	Rows committed per batch (default 1000)
`--allow-partial`	Best-effort salvage across damaged row ranges; every skipped range is recorded
`--report PATH`	JSON report path (default `<output>.recovery.json`)

Sibling maintenance commands

Command	Purpose
`hermes sessions repair` (`--check-only`, `--no-backup`)	Repair a malformed schema so hidden sessions reappear; backup first
`hermes sessions optimize`	Merge FTS5 segments + `VACUUM` — no data change, but run with services stopped (VACUUM on a live DB violates the cardinal rule)
`hermes sessions optimize-storage` (`--no-vacuum`, `--yes`)	Migrate FTS to the compact v23 layout; on an already-compact store, reports that there is nothing to do. Run with services stopped
`hermes sessions repair-routing` (`--apply`, `--max-gap-seconds N`)	Re-attach gateway conversations that lost their routing identity after a corrupt write path
`hermes sessions stats`	Session store statistics

Documentation gap (upstream): the official Hermes docs list recover, repair, optimize, optimize-storage, and repair-routing as one-line entries in the CLI reference but do not document the flags, and the sessions user guide has no recovery walkthrough (it covers only repair-routing). Everything above was reconstructed from hermes sessions recover --help and hermes doctor output during the incident. An upstream PR adding the flag tables and a “Recovering a damaged state.db” walkthrough would close the gap.

This estate has documented the upstream follow-up but has not posted it or opened an upstream issue from this task. The proposed field report and contribution checklist are in the Proposed Upstream state.db Report. The closest existing report is NousResearch/hermes-agent#78182, which is closed and covers a related but different runtime failure path; it must not be presented as a fix for the canonical-table recurrence described here.

Prevention (recap)

Never rebuild FTS / drop a vtable / VACUUM a live database — use the sanctioned tools with services stopped.
Snapshot before any repair, copying sidecars with the main file; never delete live sidecars.
Use the four read-only cron controls: the 5-minute FTS/write-failure watchdog (Slack on new evidence), the hourly full health/handle watchdog (local-only), the 6-hour snapshot-freshness watchdog (Slack on anomaly), and the daily validated snapshot (Slack only on failure).
Quarterly offline maintenance: stop services → inspect current layout → hermes sessions optimize-storage if needed → hermes doctor → restart → verify descriptors and a real write. On this host the compact v23 migration is already complete; the maintenance remains a health checkpoint.
Retention: keep state.db.bak-* and state.db.malformed-backup-* at least one week after an incident. The scheduled backup.sh deliberately excludes state.db; hermes backup (manual zip) and pre-update snapshots include it.

2026-09-02 PM — state.db Corruption Postmortem — why this happens, evidence timeline, lessons
2026-09-03 PM — Deleted WAL/SHM Inodes — process-handle recurrence
2026-09-04 PM — Structural Corruption Recurrence — full-check vs quick-check recurrence and action plan
Recovery Session Log — verbatim operator log
state-db — schema and backup coverage of the session store
backup — daily backup system (state.db intentionally excluded)
Proposing a PR to the Official Hermes Agent Repo — upstream contribution and issue-comment workflow
Proposed Upstream state.db Report — unpublished upstream follow-up
The agent-host-ops skill keeps an internal repair-ladder reference with the exact FTS demote/recreate SQL used for forensic copies