Hermes state.db Corruption & Recovery — Session Log

Date: September 2, 2026 System: VPS running hermes agent (~/.hermes) Outcome: ✅ Resolved — database recovered, verified, and reinstalled

Verbatim operator log — kept as written at the time. One correction for accuracy: the “~5 hours” in section 5 is a timezone/date slip; the actual gap was Sep 1 07:59 → Sep 2 16:12 (~32 h, 244 messages / 4 sessions), as recorded in the postmortem.

---

1. Initial Symptom

Sending a message to the hermes agent failed with:

⚠ No reply: the turn was stopped because session storage could not be written
  (the transcript would have been lost on restart).
  Check the state database health (hermes doctor), then send your message again.

2. Diagnosis

hermes doctor reported:

⚠ state.db FTS repair is blocked after 381 deferral(s) by PID(s) [42883, 123862]
  (stop the listed processes, then run 'hermes sessions optimize-storage' with the gateway stopped)

1. state.db FTS write corruption — run 'hermes doctor --fix' (or 'hermes sessions repair')
   to rebuild the FTS index

3. Repair Attempts (in order, each failing forward to the next)

Step	Command	Result
1	hermes doctor --fix	✗ FTS write corruption, auto-repair failed — restore from backup
2	hermes sessions repair	✗ database disk image is malformed — backup preserved at state.db.malformed-backup-20260902_161230, told to keep state.db and the backup

4. Recovery Attempt #1 — state.db.malformed-backup-20260902_161230

Inspect:

hermes sessions recover --source /home/hermes/.hermes/state.db.malformed-backup-20260902_161230 \
    --inspect-only

Reported "recoverable": true, all tables listed as available (307 sessions, 17,056 messages, 52 gateway_routing, 192 system_prompts).

Full rebuild:

hermes sessions recover --source /home/hermes/.hermes/state.db.malformed-backup-20260902_161230 \
    --output recovered-state.db

Partial failure.

Table	Result
messages	✅ 17,056/17,056
session_model_usage	✅ 394/394
async_delegations	✅ 5/5
state_meta	✅ 6/7 (1 excluded FTS-bookkeeping key, expected)
sessions	❌ 0/307 — database disk image is malformed
gateway_routing	❌ 0/52 — database disk image is malformed
system_prompts	❌ 0/192 — database disk image is malformed

Result: "verified": false, "complete": false — not installed. Corruption in this snapshot sat specifically on the pages backing sessions, gateway_routing, and system_prompts. The large foreign_key_check list of orphaned messages was a downstream symptom of sessions being empty, not new data loss.

5. Recovery Attempt #2 — state.db.malformed-backup-20260901_075905 (older snapshot)

Inspect:

hermes sessions recover --source /home/hermes/.hermes/state.db.malformed-backup-20260901_075905 \
    --inspect-only

Clean — no errors on any table (303 sessions, 16,812 messages, 50 gateway_routing, 190 system_prompts, 383 session_model_usage).

Full rebuild:

hermes sessions recover --source /home/hermes/.hermes/state.db.malformed-backup-20260901_075905 \
    --output recovered-state-075905.db

✅ Complete success.

Table	Copied	Source	Status
sessions	303	303	✅
messages	16,812	16,812	✅
gateway_routing	50	50	✅
system_prompts	190	190	✅
session_model_usage	383	383	✅
async_delegations	5	5	✅
state_meta	5	6	✅ (1 FTS-bookkeeping key excluded, expected)

Verification block: "healthy": true, "integrity_check": ["ok"], "opens_cleanly": true, "foreign_key_check": [], "loss_detected": false, "verified": true.

Trade-off accepted: using the older snapshot cost roughly the last ~5 hours of activity (Sep 1 ~07:59 → Sep 2 ~16:12) — about 244 messages and 4 sessions — in exchange for fully intact session metadata, routing, and system prompts.

Retrieving the report locally (pulled from the VPS to inspect the full JSON rather than reading it over an SSH terminal):

scp hostinger-vps:/home/hermes/recovered-state-075905.db.recovery.json ~/Downloads/

(Used an existing ~/.ssh/config Host alias — hostinger-vps — so port and identity file didn’t need to be specified inline.)

6. Installation

# Confirm blocking processes were stopped
ps -p 42883,123862

# Preserve the corrupt file, don't delete
mv /home/hermes/.hermes/state.db /home/hermes/.hermes/state.db.corrupt-20260902
rm -f /home/hermes/.hermes/state.db-shm /home/hermes/.hermes/state.db-wal

# Install the recovered database
cp /home/hermes/recovered-state-075905.db /home/hermes/.hermes/state.db

# Re-verify
hermes doctor

7. Final Verification (hermes doctor)

✓ ~/.hermes/state.db exists (303 sessions)
  → state.db logical size 113.2 MB, 28,987 pages, 71 free, WAL 0 B
  → 16,812 messages, 303 sessions, journal_mode=wal, 0 process(es) holding the DB open
  → FTS tables: messages_fts, messages_fts_trigram

No corruption or staleness flags on the database section. System restored to a healthy state.

Remaining flagged issues were unrelated to the incident — optional npm vulnerabilities in agent-browser and the web workspace tooling:

cd /home/hermes/.hermes/hermes-agent && npm audit fix --workspaces=false

8. Files to Retain (insurance, not currently needed)

• state.db.corrupt-20260902 — the malformed file that was replaced
• state.db.malformed-backup-20260901_075905 — source of the successful recovery
• state.db.malformed-backup-20260902_161230 (+ -shm/-wal) — source of the failed recovery attempt; still contains full message history if ever needed for forensic cross-reference
• state.db.bak-20260901 — untouched, never used, still a valid fallback if anything looks off later

Known data gap: activity between roughly 2026-09-01 07:59 and 2026-09-02 16:12 (the corruption window) was not recovered.

---

Recommendation: keep the retained files for at least a week before deleting, in case anything in the recovered database turns out to be incomplete or off.

Related

• 2026-09-02 PM — state.db Corruption Postmortem — the formal incident record
• state.db Recovery — Procedure — the runbook derived from this log