mpackdb

15.2 KB

1. # MPackDB Architecture
3. ## Overview
5. MPackDB is a fast, local, append-only JSON database that uses MessagePack serialization. It's designed for Node.js/Bun applications that need a simple, file-based database with optional indexing capabilities and smaller file sizes compared to BSON.
7. ## Core Components
9. ### 1. MPackDB Class (`src/MPackDB.js`)
11. The main database class that handles all CRUD operations and coordinates between components.
13. **Key Responsibilities:**
14. - Database initialization and lifecycle management
15. - CRUD operations (insert, update, delete, find)
16. - File locking for concurrent access
17. - Metadata persistence
18. - Compaction of deleted records
20. **Key Properties:**
21. - `_dataPath`: Path to the `.mpack` data file
22. - `_dataStream`: Write stream for append-only operations
23. - `_meta`: Metadata object containing `nextId` and `deleted` offsets
24. - `_indexManager`: Optional IndexManager instance for indexed queries
25. - `_primaryKey`: Name of the primary key field
26. - `_primaryKeyType`: Type of primary key (NUMBER, UUID, STRING)
28. ### 2. IndexManager Class (`src/IndexManager.js`)
30. Manages binary search indexes for fast lookups on indexed fields.
32. **Key Responsibilities:**
33. - Building and maintaining indexes from data files
34. - Binary search on disk-based index files
35. - Delta indexes (in-memory changes not yet persisted)
36. - Tombstone tracking for deleted records
37. - Auto-persistence of indexes
39. **Index File Format:**
40. ```
41. key,offset,length
42. key,offset,length
43. ...
44. ```
46. Each line represents an index entry where:
47. - `key`: The indexed field value
48. - `offset`: Byte offset in the data file
49. - `length`: Length of the record in bytes
51. **Index Types:**
52. - **NUMERIC**: Numeric comparison for sorting/searching
53. - **LEXICAL**: String comparison for sorting/searching
54. - **UNIQUE**: Any index can be marked unique (`!` prefix) to reject duplicates on insert
56. ### 3. Cursor Class (`src/Cursor.js`)
58. Provides an async iterable interface for query results.
60. **Key Responsibilities:**
61. - Lazy evaluation of queries
62. - Support for different iteration modes (record, offset, mixed, raw)
63. - Integration with IndexManager for indexed queries
64. - Filtering with query functions
66. ### 4. MessagePack Utilities (`src/mpack.js`)
68. Wrapper around `msgpackr` with custom utilities.
70. **Key Exports:**
71. - `serialize()`: Encode JavaScript objects to MessagePack binary
72. - `deserialize()`: Decode MessagePack binary to JavaScript objects
73. - `uuid()`: Generate sortable 12-char base36 unique IDs (9 timestamp + 3 random)
74. - `PrimaryKeyType`: Enum for primary key types
75. - `IndexType`: Enum for index types
77. ## Data Flow
79. ### Insert Operation
81. ```
82. 1. User calls db.insert(record)
83. 2. Acquire file lock
84. 3. Auto-generate primary key if needed (numeric/UUID)
85. 4. Check unique index constraints (skip auto-generated PKs)
86. 5. Serialize record to MessagePack
87. 6. Prepend 4-byte size header
88. 7. Append to data file via write stream
89. 8. Add entry to IndexManager (if indexes enabled)
90. 9. Persist metadata (if nextId changed)
91. 10. Release file lock
92. 11. Return primary key or full record
93. ```
95. ### Find Operation (Indexed)
97. ```
98. 1. User calls db.find(primaryKeyValue)
99. 2. Cursor created with query function
100. 3. IndexManager performs binary search on index file
101. 4. Check delta indexes for recent changes
102. 5. Check tombstones for deleted records
103. 6. Read record from data file at found offset
104. 7. Yield record to user
105. ```
107. ### Find Operation (Non-Indexed)
109. ```
110. 1. User calls db.find(queryFn)
111. 2. Cursor created with query function
112. 3. Stream through entire data file
113. 4. Read 4-byte size header
114. 5. Read MessagePack data based on size
115. 6. Deserialize each record
116. 7. Apply query function filter
117. 8. Skip deleted records (check metadata.deleted)
118. 9. Yield matching records to user
119. ```
121. ### Find with Index Hints
123. ```
124. 1. User calls db.find(filterFn, { index: { field, from, to, direction } })
125. or db.find(filterFn, { index: [hint1, hint2, ...] })
126. 2. For each index hint, collect offset sets:
127. - value hint: exact match via get() → offset set
128. - range hint: entries(from, to, direction) → offset set
129. 3. If multiple hints: intersect all offset sets (smallest first)
130. 4. Stream records from the resulting offsets
131. 5. Apply filter function on deserialized records
132. 6. Yield matching records
133. ```
135. ### Bounding Box Query
137. ```
138. 1. User calls db.boundingBox(corners, filter)
139. 2. Extract min/max for x and y from corners (order irrelevant)
140. 3. Delegates to find(filter, { index: [
141. { field: 'x', from: minX, to: maxX },
142. { field: 'y', from: minY, to: maxY }
143. ]})
144. 4. Intersection + streaming as above
145. ```
147. ### Update Operation
149. ```
150. 1. User calls db.update(query, newData, { index })
151. 2. Internally calls delete(query, { index, callback })
152. 3. For each match (using find with optional index hints):
153. a. Mark old record as deleted
154. b. Callback inserts new record with same primary key
155. 4. Persist metadata with new deleted offsets
156. ```
158. ### Delete Operation
160. ```
161. 1. User calls db.delete(query, { index })
162. 2. Find matching records (uses index hints if provided, otherwise full scan)
163. 3. Add offsets to metadata.deleted array
164. 4. Remove from indexes (if enabled)
165. 5. Persist metadata
166. ```
168. ### Compact Operation
170. ```
171. 1. Acquire file lock
172. 2. Create temporary data file
173. 3. Stream through all records
174. 4. Write only non-deleted records to temp file
175. 5. Atomically rename temp file to replace original
176. 6. Clear metadata.deleted array
177. 7. Rebuild all indexes from new file
178. 8. Release file lock
179. ```
181. ## File Structure
183. ```
184. data/
185. ├── users.mpack # Main data file (MessagePack records)
186. ├── users.meta.json # Metadata (nextId, deleted offsets)
187. ├── users.lock # Lock file (contains PID)
188. ├── users.id.txt # Index file for 'id' field
189. ├── users.email.txt # Index file for 'email' field
190. └── ...
191. ```
193. ## Serialization Format
195. MPackDB uses MessagePack format from the `msgpackr` npm package with a custom size header:
197. ```
198. [4 bytes: size][MessagePack data]
199. ```
201. Each record is prefixed with a 4-byte little-endian integer indicating the size of the MessagePack data (not including the size prefix itself).
203. **Why the size header?**
204. - MessagePack doesn't include record boundaries in the format
205. - The size header allows streaming reads without parsing the entire file
206. - Enables skipping deleted records efficiently
207. - Matches the pattern used in BSON for consistency
209. ## Locking Mechanism
211. MPackDB uses file-based locking to prevent concurrent writes:
213. 1. Before any write operation, create `{dbPath}.lock` file with `wx` flag (exclusive)
214. 2. Write current process PID to lock file
215. 3. If lock exists, wait 100ms and retry
216. 4. After operation completes, delete lock file
218. The lock is **re-entrant**: if a lock is already held (e.g. inside `withLock()`), nested operations (`insert`, `delete`, etc.) increment a depth counter instead of acquiring a new file lock. The file lock is only released when the outermost holder finishes.
220. ### withLock for Compound Operations
222. `db.withLock(callback)` acquires the lock for the duration of the callback. All DB operations inside the callback reuse the same lock. This makes compound operations like find-then-insert atomic:
224. ```javascript
225. await db.withLock(async () => {
226. const [user] = await db.find(u => u.email === email);
227. if (!user) await db.insert({ email });
228. });
229. ```
231. This ensures only one process can write at a time while allowing multiple readers.
233. ## Index Persistence Strategy
235. Indexes use a two-tier approach:
237. ### Disk Indexes
238. - Sorted index files on disk
239. - Binary searchable for O(log n) lookups
240. - Rebuilt during compaction
242. ### Delta Indexes (In-Memory)
243. - Track changes since last persistence
244. - Checked before disk indexes
245. - Auto-persisted based on:
246. - Time interval (default: 60 seconds)
247. - Change threshold (default: 1000 operations)
249. ### Tombstones
250. - Track deleted records in memory
251. - Prevent returning deleted records from disk indexes
252. - Cleared during compaction
254. ## Performance Characteristics
256. ### Time Complexity
257. - **Insert**: O(1) for append, O(log n) for index update
258. - **Find by primary key (indexed)**: O(log n) binary search
259. - **Find with query function**: O(n) full scan
260. - **Find with single index hint**: O(log n) seek + O(k) scan where k = entries in range
261. - **Find with index intersection**: O(k1 + k2 + ... + min(k)) for collecting + intersecting offset sets, then O(m) disk reads where m = intersection size
262. - **Bounding box**: Same as index intersection with 2 range indexes
263. - **Update**: O(find) + O(1) insert
264. - **Delete**: O(find) + O(1) mark
265. - **Compact**: O(n) full scan + O(n log n) index rebuild
267. ### Space Complexity
268. - Data file grows with inserts (append-only)
269. - Deleted records remain until compaction
270. - Index files: O(n) per indexed field
271. - Delta indexes: O(m) where m = changes since last persist
273. ### File Size Comparison
274. MessagePack typically produces **15-20% smaller files** than BSON for the same data:
275. - More compact integer encoding
276. - Smaller string overhead
277. - Efficient array/map encoding
279. ## Concurrency Model
281. - **Single-writer, multiple-reader** via file locking
282. - Writes are serialized through lock file
283. - Reads can happen concurrently (no locks needed)
284. - Re-entrant locks allow `withLock()` to wrap compound operations atomically
285. - Unique indexes enforce constraints under the write lock (no duplicates even with concurrent inserts)
286. - `refresh()` re-reads `meta.json` before every read so secondary instances see up-to-date state
287. - Index persistence happens asynchronously but safely
289. ## Primary Key Types
291. ### NUMBER (PrimaryKeyType.NUMBER)
292. - Auto-incremented integer
293. - Stored in metadata.nextId
294. - Prefix syntax: `*id`
296. ### UUID (PrimaryKeyType.UUID)
297. - Sortable base36 unique ID (9-char timestamp + 3-char random)
298. - 12-character string format
299. - Prefix syntax: `@id`
301. ### STRING (PrimaryKeyType.STRING)
302. - User-provided string
303. - No auto-generation
304. - Default (no prefix)
306. ## Design Decisions
308. ### Why Append-Only?
309. - **Fast writes**: No seeking, just append
310. - **Crash safety**: Partial writes don't corrupt existing data
311. - **Simple implementation**: No complex update-in-place logic
313. ### Why MessagePack?
314. - **Smaller files**: 15-20% smaller than BSON on average
315. - **Fast serialization**: Comparable or faster than BSON
316. - **Wide language support**: Available in many programming languages
317. - **Simple format**: Easy to implement and debug
318. - **No external binary dependencies**: Pure JavaScript implementation
320. ### Why Custom Size Header?
321. - MessagePack doesn't define record boundaries
322. - Enables efficient streaming without full deserialization
323. - Allows skipping deleted records quickly
324. - Consistent with BSON's approach
326. ### Why File-Based Locking?
327. - **Simple**: No external dependencies
328. - **Cross-process**: Works across multiple Node.js processes
329. - **Portable**: Works on all platforms
331. ### Why Binary Search Indexes?
332. - **Disk-friendly**: Can search large indexes without loading into memory
333. - **Simple format**: Plain text, easy to debug
334. - **Fast lookups**: O(log n) for indexed queries
336. ## MessagePack vs BSON
338. ### Advantages of MessagePack
339. - **Smaller files**: 15-20% size reduction
340. - **Faster reads**: Simpler format, less parsing overhead
341. - **Pure JavaScript**: No native dependencies
342. - **Smaller library**: ~15KB vs ~173KB for BSON
344. ### Advantages of BSON
345. - **ObjectId type**: Built-in unique identifier type
346. - **Date precision**: Millisecond timestamps
347. - **Binary data**: Native binary type
348. - **MongoDB compatibility**: Direct compatibility with MongoDB
350. ### When to Choose MPackDB
351. - File size is a concern
352. - Pure JavaScript dependencies preferred
353. - Don't need MongoDB compatibility
354. - Want faster read performance
356. ### When to Choose BsonDB
357. - Need ObjectId primary keys
358. - MongoDB compatibility desired
359. - Working with binary data
360. - Need precise date/time handling
362. ## Limitations
364. 1. **Single-writer**: Only one write operation at a time
365. 2. **No multi-record transactions**: `withLock` provides atomicity for compound operations on a single DB, but not across multiple databases
366. 3. **No query language**: Must use JavaScript functions for complex queries
367. 4. **Compaction required**: Deleted records consume space until compaction
368. 5. **Index overhead**: Each index doubles storage for that field
369. 6. **No schema validation**: Records can have any structure
371. ## Index Query Design Decisions
373. ### Why `to` but not `limit` or `filter`
375. The `entries()` generator supports `from`, `to`, and `direction` but intentionally has no `limit` or `filter` parameters.
377. **Why `to` is needed:** When collecting offset sets for index intersection, all matching offsets are loaded into a Map. Without `to`, a range scan like `{ field: 'x', from: 0 }` would collect every entry from 0 to the end of the index — potentially millions of offsets when only a small range is needed. `to` stops the scan at the upper bound, keeping the offset set small.
379. **Why `limit` was removed:** `limit` caps the number of results, but the caller doesn't know the right count upfront. Since results are streamed via async generators, the caller can `break` out of the loop at any time, which terminates the generator immediately. This is more flexible than a fixed limit and works naturally with JavaScript's `for await` syntax.
381. **Why `filter` was removed:** With streaming, the caller filters in the loop body after deserialization. A built-in filter would only save one `if` statement per iteration while adding API complexity. The filter function in `find()` handles this at the right layer — after records are fetched from disk.
383. ### Why index intersection uses pre-collected offset sets
385. Index intersection collects full offset sets from each index hint, then intersects them before reading any records from disk. This trades memory (storing offset integers) for disk I/O (avoiding unnecessary record reads).
387. **Trade-off:** For a query like `x: 0..100 AND status: 'active'` on 10 million records, the x-index might yield 500k offsets and the status-index 50k offsets. Intersecting these sets (~4MB of integers in memory) reduces disk reads from 500k to maybe 10k — a massive I/O saving.
389. **Why not stream-and-check:** An alternative would be to stream the primary index and check each candidate against secondary indexes on the fly (O(log n) lookup per candidate). This uses less memory but requires a disk seek per candidate. For large datasets where the secondary index is highly selective, pre-collected intersection wins. For small datasets, the difference is negligible.
391. **Why the caller picks the primary index:** The engine cannot automatically pick the most selective index without scanning all of them first. Instead, the caller specifies which indexes to use via the `index` option. For streaming without intersection (single index), the caller controls the scan with `from`, `to`, `direction`, and `break`.
393. ### Bounding box vs geospatial
395. `boundingBox()` operates on flat 2D coordinates — simple min/max comparisons on x and y axes. This is sufficient for game grids, tile maps, and any coordinate system where geometry is planar. Geospatial indexing (S2 cells, geohash) is only needed for spherical geometry (lat/lng on Earth) where "straight lines" are actually great-circle arcs and distances warp with latitude. Both approaches use the same fundamental pattern internally: coarse index scan → exact geometric filter.
397. ## Future Improvements
399. - Batch insert operations
400. - Async compaction (background process)
401. - Query optimizer for complex filters
402. - Compression support (MessagePack supports extensions)
403. - Replication/backup utilities
404. - Schema validation layer
405. - Custom MessagePack extension types