mpackdb

15.2 KB

1. # MPackDB Architecture
3. ## Overview
5. MPackDB is a fast, local, append-only JSON database that uses MessagePack serialization. It's designed for Node.js/Bun applications that need a simple, file-based database with optional indexing capabilities and smaller file sizes compared to BSON.
7. ## Core Components
9. ### 1. MPackDB Class (`src/MPackDB.js`)
11. The main database class that handles all CRUD operations and coordinates between components.
13. **Key Responsibilities:**
14. - Database initialization and lifecycle management
15. - CRUD operations (insert, update, delete, find)
16. - File locking for concurrent access
17. - Metadata persistence
18. - Compaction of deleted records
20. **Key Properties:**
21. - `_dataPath`: Path to the `.mpack` data file
22. - `_dataStream`: Write stream for append-only operations
23. - `_meta`: Metadata object containing `nextId` and `deleted` offsets
24. - `_indexManager`: Optional IndexManager instance for indexed queries
25. - `_primaryKey`: Name of the primary key field
26. - `_primaryKeyType`: Type of primary key (NUMBER, UUID, STRING)
28. ### 2. IndexManager Class (`src/IndexManager.js`)
30. Manages binary search indexes for fast lookups on indexed fields.
32. **Key Responsibilities:**
33. - Building and maintaining indexes from data files
34. - Binary search on disk-based index files
35. - Delta indexes (in-memory changes not yet persisted)
36. - Tombstone tracking for deleted records
37. - Auto-persistence of indexes
39. **Index File Format:**
40. ```
41. key,offset,length
42. key,offset,length
43. ...
44. ```
46. Each line represents an index entry where:
47. - `key`: The indexed field value
48. - `offset`: Byte offset in the data file
49. - `length`: Length of the record in bytes
51. **Index Types:**
52. - **NUMERIC**: Numeric comparison for sorting/searching
53. - **LEXICAL**: String comparison for sorting/searching
54. - **UNIQUE**: Any index can be marked unique (`!` prefix) to reject duplicates on insert
56. ### 3. Cursor Class (`src/Cursor.js`)
58. Provides an async iterable interface for query results.
60. **Key Responsibilities:**
61. - Lazy evaluation of queries
62. - Support for different iteration modes (record, offset, mixed, raw)
63. - Integration with IndexManager for indexed queries
64. - Filtering with query functions
66. ### 4. MessagePack Utilities (`src/mpack.js`)
68. Wrapper around `msgpackr` with custom utilities.
70. **Key Exports:**
71. - `serialize()`: Encode JavaScript objects to MessagePack binary
72. - `deserialize()`: Decode MessagePack binary to JavaScript objects
73. - `uuid()`: Generate sortable 12-char base36 unique IDs (9 timestamp + 3 random)
74. - `PrimaryKeyType`: Enum for primary key types
75. - `IndexType`: Enum for index types
77. ## Data Flow
79. ### Insert Operation
81. ```
82. 1. User calls db.insert(record)
83. 2. Acquire file lock
84. 3. Auto-generate primary key if needed (numeric/UUID)
85. 4. Check unique index constraints (skip auto-generated PKs)
86. 5. Serialize record to MessagePack
87. 6. Prepend 4-byte size header
88. 7. Append to data file via write stream
89. 8. Add entry to IndexManager (if indexes enabled)
90. 9. Persist metadata (if nextId changed)
91. 10. Release file lock
92. 11. Return primary key or full record
93. ```
95. ### Find Operation (Indexed)
97. ```
98. 1. User calls db.find(primaryKeyValue)
99. 2. Cursor created with query function
100. 3. IndexManager performs binary search on index file
101. 4. Check delta indexes for recent changes
102. 5. Check tombstones for deleted records
103. 6. Read record from data file at found offset
104. 7. Yield record to user
105. ```
107. ### Find Operation (Non-Indexed)
109. ```
110. 1. User calls db.find(queryFn)
111. 2. Cursor created with query function
112. 3. Stream through entire data file
113. 4. Read 4-byte size header
114. 5. Read MessagePack data based on size
115. 6. Deserialize each record
116. 7. Apply query function filter
117. 8. Skip deleted records (check metadata.deleted)
118. 9. Yield matching records to user
119. ```
121. ### Find with Index Hints
123. ```
124. 1. User calls db.find(filterFn, { index: { field, from, to, direction } })
125. or db.find(filterFn, { index: [hint1, hint2, ...] })
126. 2. For each index hint, collect offset sets:
127. - value hint: exact match via get() → offset set
128. - range hint: entries(from, to, direction) → offset set
129. 3. If multiple hints: intersect all offset sets (smallest first)
130. 4. Stream records from the resulting offsets
131. 5. Apply filter function on deserialized records
132. 6. Yield matching records
133. ```
135. ### Bounding Box Query
137. ```
138. 1. User calls db.boundingBox(corners, filter)
139. 2. Extract min/max for x and y from corners (order irrelevant)
140. 3. Delegates to find(filter, { index: [
141. { field: 'x', from: minX, to: maxX },
142. { field: 'y', from: minY, to: maxY }
143. ]})
144. 4. Intersection + streaming as above
145. ```
147. ### Update Operation
149. ```
150. 1. User calls db.update(query, callback, { index })
151. 2. Internally calls delete(query, { index, callback })
152. 3. For each match (using find with optional index hints):
153. a. Mark old record as deleted
154. b. User callback transforms old record → new record
155. c. New record is inserted with same primary key
156. 4. Persist metadata with new deleted offsets
157. ```
159. ### Delete Operation
161. ```
162. 1. User calls db.delete(query, { index })
163. 2. Find matching records (uses index hints if provided, otherwise full scan)
164. 3. Add offsets to metadata.deleted array
165. 4. Remove from indexes (if enabled)
166. 5. Persist metadata
167. ```
169. ### Compact Operation
171. ```
172. 1. Acquire file lock
173. 2. Create temporary data file
174. 3. Stream through all records
175. 4. Write only non-deleted records to temp file
176. 5. Atomically rename temp file to replace original
177. 6. Clear metadata.deleted array
178. 7. Rebuild all indexes from new file
179. 8. Release file lock
180. ```
182. ## File Structure
184. ```
185. data/
186. ├── users.mpack # Main data file (MessagePack records)
187. ├── users.meta.json # Metadata (nextId, deleted offsets)
188. ├── users.lock # Lock file (contains PID)
189. ├── users.id.txt # Index file for 'id' field
190. ├── users.email.txt # Index file for 'email' field
191. └── ...
192. ```
194. ## Serialization Format
196. MPackDB uses MessagePack format from the `msgpackr` npm package with a custom size header:
198. ```
199. [4 bytes: size][MessagePack data]
200. ```
202. Each record is prefixed with a 4-byte little-endian integer indicating the size of the MessagePack data (not including the size prefix itself).
204. **Why the size header?**
205. - MessagePack doesn't include record boundaries in the format
206. - The size header allows streaming reads without parsing the entire file
207. - Enables skipping deleted records efficiently
208. - Matches the pattern used in BSON for consistency
210. ## Locking Mechanism
212. MPackDB uses file-based locking to prevent concurrent writes:
214. 1. Before any write operation, create `{dbPath}.lock` file with `wx` flag (exclusive)
215. 2. Write current process PID to lock file
216. 3. If lock exists, wait 100ms and retry
217. 4. After operation completes, delete lock file
219. The lock is **re-entrant**: if a lock is already held (e.g. inside `withLock()`), nested operations (`insert`, `delete`, etc.) increment a depth counter instead of acquiring a new file lock. The file lock is only released when the outermost holder finishes.
221. ### withLock for Compound Operations
223. `db.withLock(callback)` acquires the lock for the duration of the callback. All DB operations inside the callback reuse the same lock. This makes compound operations like find-then-insert atomic:
225. ```javascript
226. await db.withLock(async () => {
227. const [user] = await db.find(u => u.email === email);
228. if (!user) await db.insert({ email });
229. });
230. ```
232. This ensures only one process can write at a time while allowing multiple readers.
234. ## Index Persistence Strategy
236. Indexes use a two-tier approach:
238. ### Disk Indexes
239. - Sorted index files on disk
240. - Binary searchable for O(log n) lookups
241. - Rebuilt during compaction
243. ### Delta Indexes (In-Memory)
244. - Track changes since last persistence
245. - Checked before disk indexes
246. - Auto-persisted based on:
247. - Time interval (default: 60 seconds)
248. - Change threshold (default: 1000 operations)
250. ### Tombstones
251. - Track deleted records in memory
252. - Prevent returning deleted records from disk indexes
253. - Cleared during compaction
255. ## Performance Characteristics
257. ### Time Complexity
258. - **Insert**: O(1) for append, O(log n) for index update
259. - **Find by primary key (indexed)**: O(log n) binary search
260. - **Find with query function**: O(n) full scan
261. - **Find with single index hint**: O(log n) seek + O(k) scan where k = entries in range
262. - **Find with index intersection**: O(k1 + k2 + ... + min(k)) for collecting + intersecting offset sets, then O(m) disk reads where m = intersection size
263. - **Bounding box**: Same as index intersection with 2 range indexes
264. - **Update**: O(find) + O(1) insert
265. - **Delete**: O(find) + O(1) mark
266. - **Compact**: O(n) full scan + O(n log n) index rebuild
268. ### Space Complexity
269. - Data file grows with inserts (append-only)
270. - Deleted records remain until compaction
271. - Index files: O(n) per indexed field
272. - Delta indexes: O(m) where m = changes since last persist
274. ### File Size Comparison
275. MessagePack typically produces **15-20% smaller files** than BSON for the same data:
276. - More compact integer encoding
277. - Smaller string overhead
278. - Efficient array/map encoding
280. ## Concurrency Model
282. - **Single-writer, multiple-reader** via file locking
283. - Writes are serialized through lock file
284. - Reads can happen concurrently (no locks needed)
285. - Re-entrant locks allow `withLock()` to wrap compound operations atomically
286. - Unique indexes enforce constraints under the write lock (no duplicates even with concurrent inserts)
287. - `refresh()` re-reads `meta.json` before every read so secondary instances see up-to-date state
288. - Index persistence happens asynchronously but safely
290. ## Primary Key Types
292. ### NUMBER (PrimaryKeyType.NUMBER)
293. - Auto-incremented integer
294. - Stored in metadata.nextId
295. - Prefix syntax: `*id`
297. ### UUID (PrimaryKeyType.UUID)
298. - Sortable base36 unique ID (9-char timestamp + 3-char random)
299. - 12-character string format
300. - Prefix syntax: `@id`
302. ### STRING (PrimaryKeyType.STRING)
303. - User-provided string
304. - No auto-generation
305. - Default (no prefix)
307. ## Design Decisions
309. ### Why Append-Only?
310. - **Fast writes**: No seeking, just append
311. - **Crash safety**: Partial writes don't corrupt existing data
312. - **Simple implementation**: No complex update-in-place logic
314. ### Why MessagePack?
315. - **Smaller files**: 15-20% smaller than BSON on average
316. - **Fast serialization**: Comparable or faster than BSON
317. - **Wide language support**: Available in many programming languages
318. - **Simple format**: Easy to implement and debug
319. - **No external binary dependencies**: Pure JavaScript implementation
321. ### Why Custom Size Header?
322. - MessagePack doesn't define record boundaries
323. - Enables efficient streaming without full deserialization
324. - Allows skipping deleted records quickly
325. - Consistent with BSON's approach
327. ### Why File-Based Locking?
328. - **Simple**: No external dependencies
329. - **Cross-process**: Works across multiple Node.js processes
330. - **Portable**: Works on all platforms
332. ### Why Binary Search Indexes?
333. - **Disk-friendly**: Can search large indexes without loading into memory
334. - **Simple format**: Plain text, easy to debug
335. - **Fast lookups**: O(log n) for indexed queries
337. ## MessagePack vs BSON
339. ### Advantages of MessagePack
340. - **Smaller files**: 15-20% size reduction
341. - **Faster reads**: Simpler format, less parsing overhead
342. - **Pure JavaScript**: No native dependencies
343. - **Smaller library**: ~15KB vs ~173KB for BSON
345. ### Advantages of BSON
346. - **ObjectId type**: Built-in unique identifier type
347. - **Date precision**: Millisecond timestamps
348. - **Binary data**: Native binary type
349. - **MongoDB compatibility**: Direct compatibility with MongoDB
351. ### When to Choose MPackDB
352. - File size is a concern
353. - Pure JavaScript dependencies preferred
354. - Don't need MongoDB compatibility
355. - Want faster read performance
357. ### When to Choose BsonDB
358. - Need ObjectId primary keys
359. - MongoDB compatibility desired
360. - Working with binary data
361. - Need precise date/time handling
363. ## Limitations
365. 1. **Single-writer**: Only one write operation at a time
366. 2. **No multi-record transactions**: `withLock` provides atomicity for compound operations on a single DB, but not across multiple databases
367. 3. **No query language**: Must use JavaScript functions for complex queries
368. 4. **Compaction required**: Deleted records consume space until compaction
369. 5. **Index overhead**: Each index doubles storage for that field
370. 6. **No schema validation**: Records can have any structure
372. ## Index Query Design Decisions
374. ### Why `to` but not `limit` or `filter`
376. The `entries()` generator supports `from`, `to`, and `direction` but intentionally has no `limit` or `filter` parameters.
378. **Why `to` is needed:** When collecting offset sets for index intersection, all matching offsets are loaded into a Map. Without `to`, a range scan like `{ field: 'x', from: 0 }` would collect every entry from 0 to the end of the index — potentially millions of offsets when only a small range is needed. `to` stops the scan at the upper bound, keeping the offset set small.
380. **Why `limit` was removed:** `limit` caps the number of results, but the caller doesn't know the right count upfront. Since results are streamed via async generators, the caller can `break` out of the loop at any time, which terminates the generator immediately. This is more flexible than a fixed limit and works naturally with JavaScript's `for await` syntax.
382. **Why `filter` was removed:** With streaming, the caller filters in the loop body after deserialization. A built-in filter would only save one `if` statement per iteration while adding API complexity. The filter function in `find()` handles this at the right layer — after records are fetched from disk.
384. ### Why index intersection uses pre-collected offset sets
386. Index intersection collects full offset sets from each index hint, then intersects them before reading any records from disk. This trades memory (storing offset integers) for disk I/O (avoiding unnecessary record reads).
388. **Trade-off:** For a query like `x: 0..100 AND status: 'active'` on 10 million records, the x-index might yield 500k offsets and the status-index 50k offsets. Intersecting these sets (~4MB of integers in memory) reduces disk reads from 500k to maybe 10k — a massive I/O saving.
390. **Why not stream-and-check:** An alternative would be to stream the primary index and check each candidate against secondary indexes on the fly (O(log n) lookup per candidate). This uses less memory but requires a disk seek per candidate. For large datasets where the secondary index is highly selective, pre-collected intersection wins. For small datasets, the difference is negligible.
392. **Why the caller picks the primary index:** The engine cannot automatically pick the most selective index without scanning all of them first. Instead, the caller specifies which indexes to use via the `index` option. For streaming without intersection (single index), the caller controls the scan with `from`, `to`, `direction`, and `break`.
394. ### Bounding box vs geospatial
396. `boundingBox()` operates on flat 2D coordinates — simple min/max comparisons on x and y axes. This is sufficient for game grids, tile maps, and any coordinate system where geometry is planar. Geospatial indexing (S2 cells, geohash) is only needed for spherical geometry (lat/lng on Earth) where "straight lines" are actually great-circle arcs and distances warp with latitude. Both approaches use the same fundamental pattern internally: coarse index scan → exact geometric filter.
398. ## Future Improvements
400. - Batch insert operations
401. - Async compaction (background process)
402. - Query optimizer for complex filters
403. - Compression support (MessagePack supports extensions)
404. - Replication/backup utilities
405. - Schema validation layer
406. - Custom MessagePack extension types