mpackdb

7.0 KB

1. # STATUS
3. ## Operating / Testing
5. - Run the full suite: `npm test` (runs every `tests/*.test.js` with plain node; a test FAILS by throwing / non-zero exit, look for `✗`).
6. - Key concurrency tests (all must stay green):
7. - `tests/concurrent-delete-during-scan.test.js` — the BUG_REPORT race (lane reads during writes)
8. - `tests/causal-lock.test.js` — in-process lock causality + re-entrancy
9. - `tests/multi-process-write.test.js` — 3 spawned processes writing the same db (uses `tests/helpers/multi-process-worker.js`)
10. - `tests/stale-lock.test.js` — crashed-holder lock takeover
11. - `tests/index-path-guard.test.js` — missing/unknown index paths plus external replacement
12. - Races are timing-dependent: when touching locking/refresh, loop the first two 5x.
13. - Consumers: `adryllium-2026-3` and `mpackdb-admin` both symlink here via `file:../mpackdb` — changes are live for them on their next process restart. mpackdb-admin end-to-end: `cd ../mpackdb-admin && node test/write-api.test.js`.
15. ## 2026-07-02 — Concurrency overhaul (multi-lane + multi-process writes)
17. Fixed the BUG_REPORT.md corruption (deleted UTXOs resurrecting under concurrent lane reads/writes) plus everything needed so a second process (mpackdb-admin) can write safely. All in `src/MPackDB.js` / `src/IndexManager.js`:
19. - meta.json: monotonic `version` (refresh only adopts NEWER disk meta; in-memory is authoritative), atomic tmp+rename writes, persisted `schema` (primaryKey+indexes) for external tools.
20. - Locking: causal re-entrancy via AsyncLocalStorage + in-process FIFO queue. Old depth-counter lock let concurrently-started ops into the critical section. Writers refresh under the lock (fresh cross-process nextId/tombstones). 25ms lock poll, stale-lock takeover after `staleLockTimeout` (default 30s, option).
21. - Index coherence across processes: `<name>.idxstate.json` tracks coverage; refresh tail-scans externally appended records into deltas; `persist()` runs under the db lock and dedupes by location; external compaction (inode change) triggers reopen/rebase.
22. - Latent bugs fixed: runtime `compact()` left the write stream on the orphaned inode (subsequent inserts lost); scans waiting on `'drain'` could hang (now await the last write's flush).
23. - NOT changed: version still 1.0.6, nothing committed/published — user decides release. Suite green (115 checks), admin e2e green.
25. Upgrade note: first open of a pre-existing db does a one-time full index rebuild (missing idxstate.json) — large collections (adryllium utxos) will have a slower first start.
27. Open / next:
28. - mpackdb-admin UI (public/index.html) is still read-only; the write API exists (see ../mpackdb-admin/README.md).
29. - External compaction while another process is attached is self-healing but not transactional — keep `compact: false` on shared files (adryllium already does).
30. - `IndexManager.findLastEntry()` / `getAllLocations()` are dead code (unused).
32. ## 2026-07-18 — 1.0.7 source-first release preparation
34. - Bumped package/lockfile to `1.0.7`; source remains the package entrypoint (`index.js` + `src/*`). No `dist`, esbuild, source map or build hook is shipped. Restored the required direct runtime dependency `msgpackr ^1.11.5`.
35. - Corrected README imports (`mpackdb`, not nonexistent `@worldapi.org/mpackdb`), source-first wording and the actual `insert()` return contract (primary-key value when configured, otherwise the record). Source JSDoc now matches.
36. - Verified on Node `v24.16.0` / npm `11.13.0`: `npm test` ran all 19 test files with exit 0, including 10k-record compaction and multi-process writes. Earlier in this session both timing-sensitive concurrency regressions also passed 5/5 repeated runs. An isolated `npm ci` using only the manifest and lockfile passed and resolved `msgpackr 1.11.5`.
37. - Verified the actual packed artifact, not checkout resolution: `npm pack` produced 8 files, 21,971 bytes compressed / 79,552 unpacked; a clean empty consumer installed it and passed import/export, CRUD+restart, causal-lock ordering, and 40 concurrent delete/insert/scan rounds (0 resurrected deletes, 0 lost inserts). `npm audit --omit=dev` reported 0 vulnerabilities.
38. - npm still serves old `1.0.6` (2026-03-21), whose packed artifact was independently reproduced failing causal locking (duplicate records) and hanging the concurrent delete/insert/scan regression. `1.0.7` is prepared but NOT published; after publishing, reinstall `[email protected]` from npm into an empty directory and repeat the artifact checks before considering the release confirmed. Consider deprecating `1.0.6` with a concurrency warning.
40. ## 2026-07-20 — Index-path crash guard
42. - Fixed `readFile(undefined)` from a missing `_indexPaths[field]`: declared paths are initialized synchronously and repaired deterministically on access; undeclared exact/range hints now throw `INDEX_NOT_FOUND` with the requested and configured fields. All direct production path access is centralized in `IndexManager._getIndexPath()`.
43. - Added `tests/index-path-guard.test.js`: forces a declared slot missing for exact and range lookup, checks unknown-field errors, and verifies paths/lookups after another attached instance compacts and replaces the data file.
44. - Actual verification: targeted test exit 0; `npm test` ran all 20 test files with exit 0; clean `mpackdb-1.0.7.tgz` install passed the same public-entrypoint checks. Final `npm pack --dry-run --json`: 8 files, 22,325 bytes compressed / 80,487 unpacked, SHA-1 `4bf1cf5a9fec8f75b8de32b382408735d398abea`.
45. - `BUG_REPORTS.md` records why the original range-immunity and reset-race hypotheses were disproven. Source fix is tested but remains unconfirmed in `miserit-anthar.net`; keep the consumer workaround until that application verifies the packaged fix under its real workload.
47. ## 2026-09-25 — Why 1.0.7 was only half committed; release readiness
49. - Cause: commit `8d9fd51` (2026-08-14, made from an adryllium session for the Deno `node:` import prefixes) swept in the uncommitted `src/` WIP of the July sessions but **only `src/`**. Everything else from 2026-07-02/18/20 stayed uncommitted: package.json 1.0.7 + source-first entry, `dist/` removal, README/ARCHITECTURE, `tests/crud.test.js`, 8 new tests + `tests/helpers/`, STATUS.md, BUG_REPORTS.md. So HEAD had new code but still `version 1.0.6` / `main: dist/mpackdb.min.js` (stale bundle). Working-tree `src/` == HEAD (no src diff).
50. - `db/` in repo root = output of tests run from repo root (cwd-relative `db/...` paths); now ignored via `/db` in `.gitignore`.
51. - Re-verified against current source (Node v24.16.0): `npm test` 20 files, exit 0, 0 `✗`; `concurrent-delete-during-scan`, `causal-lock`, `multi-process-write` each 5/5 green. `npm pack` → 8 files, 22.3 kB / 80.5 kB, shasum `298cd0faef41c4b159cf3f33b7a9c0854a65e5dc`; clean consumer install of the tgz: all exports, insert/delete/update-callback, unique `DUPLICATE_KEY`, reopen, range hint, `INDEX_NOT_FOUND` — OK.
52. - Release steps (user decides): commit all pending files (`git add -A` — `dist/` deletion is intended), `git tag v1.0.7`, `npm publish`, then `npm i [email protected]` into an empty dir and repeat the consumer smoke test. Optionally `npm deprecate [email protected] "concurrency bugs, use >=1.0.7"`.