Proposed Upstream state.db Report

This page records the upstream follow-up for the September 2026 state.db incidents. It is a documentation artifact only: no issue, comment, or pull request was created or modified in NousResearch/hermes-agent by this work.

Current status

Item	Decision
Candidate target	NousResearch/hermes-agent#78182 — adjacent, but closed
Publication status	Not posted
Last upstream check	September 5, 2026
Local watchdogs	Stay in the estate repository; do not copy personal paths, cron IDs, or credentials upstream
Next action	Re-run the duplicate search and inspect current `main` immediately before any publication

Why the target was considered

Issue #78182 covers a runtime FTS rebuild that reports success without proving that the retried write succeeded, followed by silent rotation of a bounded pending queue.[1] The September 1-3 estate incident produced a related but distinct observation: the deferred FTS rebuild gate listed the gateway’s own long-lived database holder and therefore deferred repeatedly until all services were stopped.[2]

The upstream tracker also contains adjacent reports concerning structural-corruption classification in doctor,[3] WAL sidecar replacement,[4] and unsafe automatic repair.[5] These references make a new duplicate issue inappropriate without a fresh source review and a new duplicate search.

Two findings that must remain separate

A. September 1-3: deferred FTS rebuild blocked by the always-on lifecycle

More than 385 deferral lines accumulated over roughly 32 hours.
The deferral evidence listed the gateway’s own PID among the holders.
The rebuild fired only after a coordinated service stop and subsequent boot window.
This is a lifecycle/design observation about when the deferred rebuild can run.

The original proposed message below was drafted for this finding.

B. September 4: canonical-table corruption hidden from shallow probes

The later recurrence was different. A full PRAGMA integrity_check found damage in the canonical system_prompts table while a count and quick_check could pass. hermes sessions recover --inspect-only returned recoverable: true, but the generated output was rejected because it was not verified or complete and still had a failed canonical-table copy and foreign-key violations.

This finding is documented in the 2026-09-04 state.db recurrence postmortem. It should not be silently merged into the older FTS-lifecycle report: the two observations have different evidence and potentially different causes.

Publication recommendation

The older draft should not be posted unchanged. Issue #78182 is closed, and its latest maintainer comment states that the two runtime/pending-queue halves described there were fixed on main.[1] Before any future communication:

Search open and closed issues and PRs for both mechanisms and their exact error signatures.
Inspect the current upstream source to determine which behaviors are already fixed.
Decide whether the useful output is a comment, a new bug report, or a documentation PR.
Keep the two findings separate unless a current source review proves they share a mechanism.
Sanitize hostnames, filesystem paths, process IDs, session IDs, prompts, credentials, and private logs.
Include the environment, exact symptom, reproduction or observation, expected behavior, actual behavior, and verification evidence.
Ask maintainers whether a dedicated issue is preferable before prescribing a fix from one deployment.

The official contribution guide explicitly requires searching existing issues and PRs, checking the source, including reproduction details, and avoiding duplicates.[6]

Historical proposed comment — not posted

The following is the proposed communication for finding A. It is retained to preserve the reasoning and evidence chain, not as an instruction to publish it. Re-run the duplicate search and refresh the version/status details before using any part of it.

Related but distinct mechanism, surfaced by a production incident on a headless
always-on gateway deployment (September 1-3, 2026) — sharing per the guidance in
CONTRIBUTING.md's "Reporting Issues" section.

**Duplicate check (must be rerun immediately before posting):** searched open/closed
issues and PRs for `FTS deferral`, `state.db malformed`, `optimize-storage`, and
`SessionDB open`. The closest matches included #82616 (different mechanism) and
#78182 (closest, but focused on unverified rebuild success rather than the
long-lived-holder deferral gate). The issue state and upstream code have changed
since this draft was written, so this text is not publication-ready.

**Environment:**
- OS: Linux (VPS, systemd-managed always-on gateway + dashboard)
- Python: 3.11.15
- Hermes version: 0.21.0
- Journal mode: WAL

**What #78182 covers:** the in-place runtime rebuild claiming success without
verifying the retried write, and the one-shot flag burning recovery attempts
for the process lifetime.

**What we additionally observed:** the *deferred* FTS rebuild path (the one
gated on "next SessionDB open with no other holders") is structurally unable
to complete on an always-on gateway deployment. The gateway's own long-lived
connection is counted as a foreign holder blocking the rebuild on every attempt.

Evidence from our incident:
- 385 deferral log lines accumulated silently over roughly 32 hours before the
  write path corrupted (`database disk image is malformed`).
- Deferral entries listed the gateway's own PID among the "foreign processes
  holding the database". The rebuild opens a fresh connection, requires
  exclusive access, sees the daemon's steady-state connection, and defers.
- The rebuild fired exactly once when the gateway restarted after both the
  gateway and dashboard had been stopped for disaster recovery. That boot was
  the first open with zero other holders.
- Because gateway and dashboard hold the file indefinitely in steady state, the
  required window does not open without a coordinated service stop.

**Generalized reproduction:**
1. Run `hermes gateway` and `hermes dashboard` as long-lived systemd services
   sharing one `state.db`.
2. Let both hold their connections open indefinitely.
3. Cause an FTS write-path issue in a disposable test environment.
4. Watch the journal for repeated deferral lines and inspect the listed holders.
5. Confirm whether the deferred rebuild runs while the services stay up and
   whether it runs after a coordinated stop and boot.

This is adjacent to, but not identical with, the one-shot verification issue:
even a correctly verified rebuild cannot run if the deferral gate is unreachable
while the daemon's own persistent connection is open.

A possible upstream design direction is to run the deferred rebuild in a
controlled boot window, before the steady-state connection is established,
rather than waiting for a later open with zero holders. This is a question for
maintainers, not a patch claim from this single deployment.

Upstream documentation proposal

The runtime report and the documentation proposal are separate. The official documentation currently exposes the session maintenance commands but does not provide the complete recovery walkthrough used by this estate. A possible upstream documentation contribution would add:

the flags for hermes sessions recover, repair, optimize, optimize-storage, and repair-routing;
the distinction between --inspect-only and an installable verified output;
a recovery walkthrough that preserves the source and validates canonical tables;
a warning that recoverable: true is not an installation approval;
a warning to stop every writer before repair, FTS rebuild, VACUUM, restore, or sidecar cleanup.

The estate-specific scripts (state_db_backup.sh, fts_repair_watchdog.sh, state_db_snapshot_freshness.sh, and state_db_health_watchdog.py) and cron schedules remain local operational controls. They are not proposed as upstream files.

Sources

[1] NousResearch/hermes-agent#78182 — runtime writes through corrupt state.db [2] Estate postmortem — September 1-2 state.db corruption [3] NousResearch/hermes-agent#88587 — structural-corruption classification in doctor [4] NousResearch/hermes-agent#90806 — WAL sidecars replaced under live holders [5] NousResearch/hermes-agent#93064 — unsafe automatic repair [6] Hermes Agent CONTRIBUTING.md

The upstream references were checked on September 5, 2026. Issue status and scope can change; repeat the search immediately before any future post or upstream PR.