mpackdb

9.9 KB

1. # MPackDB Architecture
3. ## Overview
5. MPackDB is a fast, local, append-only JSON database that uses MessagePack serialization. It's designed for Node.js/Bun applications that need a simple, file-based database with optional indexing capabilities and smaller file sizes compared to BSON.
7. ## Core Components
9. ### 1. MPackDB Class (`src/MPackDB.js`)
11. The main database class that handles all CRUD operations and coordinates between components.
13. **Key Responsibilities:**
14. - Database initialization and lifecycle management
15. - CRUD operations (insert, update, delete, find)
16. - File locking for concurrent access
17. - Metadata persistence
18. - Compaction of deleted records
20. **Key Properties:**
21. - `_dataPath`: Path to the `.mpack` data file
22. - `_dataStream`: Write stream for append-only operations
23. - `_meta`: Metadata object containing `nextId` and `deleted` offsets
24. - `_indexManager`: Optional IndexManager instance for indexed queries
25. - `_primaryKey`: Name of the primary key field
26. - `_primaryKeyType`: Type of primary key (NUMBER, UUID, STRING)
28. ### 2. IndexManager Class (`src/IndexManager.js`)
30. Manages binary search indexes for fast lookups on indexed fields.
32. **Key Responsibilities:**
33. - Building and maintaining indexes from data files
34. - Binary search on disk-based index files
35. - Delta indexes (in-memory changes not yet persisted)
36. - Tombstone tracking for deleted records
37. - Auto-persistence of indexes
39. **Index File Format:**
40. ```
41. key,offset,length
42. key,offset,length
43. ...
44. ```
46. Each line represents an index entry where:
47. - `key`: The indexed field value
48. - `offset`: Byte offset in the data file
49. - `length`: Length of the record in bytes
51. **Index Types:**
52. - **NUMERIC**: Numeric comparison for sorting/searching
53. - **LEXICAL**: String comparison for sorting/searching
55. ### 3. Cursor Class (`src/Cursor.js`)
57. Provides an async iterable interface for query results.
59. **Key Responsibilities:**
60. - Lazy evaluation of queries
61. - Support for different iteration modes (record, offset, mixed, raw)
62. - Integration with IndexManager for indexed queries
63. - Filtering with query functions
65. ### 4. MessagePack Utilities (`src/mpack.js`)
67. Wrapper around `msgpackr` with custom utilities.
69. **Key Exports:**
70. - `serialize()`: Encode JavaScript objects to MessagePack binary
71. - `deserialize()`: Decode MessagePack binary to JavaScript objects
72. - `uuid()`: Generate sortable 12-char base36 unique IDs (9 timestamp + 3 random)
73. - `PrimaryKeyType`: Enum for primary key types
74. - `IndexType`: Enum for index types
76. ## Data Flow
78. ### Insert Operation
80. ```
81. 1. User calls db.insert(record)
82. 2. Acquire file lock
83. 3. Auto-generate primary key if needed (numeric/UUID)
84. 4. Serialize record to MessagePack
85. 5. Prepend 4-byte size header
86. 6. Append to data file via write stream
87. 7. Add entry to IndexManager (if indexes enabled)
88. 8. Persist metadata (if nextId changed)
89. 9. Release file lock
90. 10. Return primary key or full record
91. ```
93. ### Find Operation (Indexed)
95. ```
96. 1. User calls db.find(primaryKeyValue)
97. 2. Cursor created with query function
98. 3. IndexManager performs binary search on index file
99. 4. Check delta indexes for recent changes
100. 5. Check tombstones for deleted records
101. 6. Read record from data file at found offset
102. 7. Yield record to user
103. ```
105. ### Find Operation (Non-Indexed)
107. ```
108. 1. User calls db.find(queryFn)
109. 2. Cursor created with query function
110. 3. Stream through entire data file
111. 4. Read 4-byte size header
112. 5. Read MessagePack data based on size
113. 6. Deserialize each record
114. 7. Apply query function filter
115. 8. Skip deleted records (check metadata.deleted)
116. 9. Yield matching records to user
117. ```
119. ### Update Operation
121. ```
122. 1. User calls db.update(query, newData)
123. 2. Find matching records (using find)
124. 3. For each match:
125. a. Mark old record as deleted
126. b. Insert new record with same primary key
127. 4. Persist metadata with new deleted offsets
128. ```
130. ### Delete Operation
132. ```
133. 1. User calls db.delete(query)
134. 2. Find matching records
135. 3. Add offsets to metadata.deleted array
136. 4. Remove from indexes (if enabled)
137. 5. Persist metadata
138. ```
140. ### Compact Operation
142. ```
143. 1. Acquire file lock
144. 2. Create temporary data file
145. 3. Stream through all records
146. 4. Write only non-deleted records to temp file
147. 5. Atomically rename temp file to replace original
148. 6. Clear metadata.deleted array
149. 7. Rebuild all indexes from new file
150. 8. Release file lock
151. ```
153. ## File Structure
155. ```
156. data/
157. ├── users.mpack # Main data file (MessagePack records)
158. ├── users.meta.json # Metadata (nextId, deleted offsets)
159. ├── users.lock # Lock file (contains PID)
160. ├── users.id.txt # Index file for 'id' field
161. ├── users.email.txt # Index file for 'email' field
162. └── ...
163. ```
165. ## Serialization Format
167. MPackDB uses MessagePack format from the `msgpackr` npm package with a custom size header:
169. ```
170. [4 bytes: size][MessagePack data]
171. ```
173. Each record is prefixed with a 4-byte little-endian integer indicating the size of the MessagePack data (not including the size prefix itself).
175. **Why the size header?**
176. - MessagePack doesn't include record boundaries in the format
177. - The size header allows streaming reads without parsing the entire file
178. - Enables skipping deleted records efficiently
179. - Matches the pattern used in BSON for consistency
181. ## Locking Mechanism
183. MPackDB uses file-based locking to prevent concurrent writes:
185. 1. Before any write operation, create `{dbPath}.lock` file with `wx` flag (exclusive)
186. 2. Write current process PID to lock file
187. 3. If lock exists, wait 100ms and retry
188. 4. After operation completes, delete lock file
190. This ensures only one process can write at a time while allowing multiple readers.
192. ## Index Persistence Strategy
194. Indexes use a two-tier approach:
196. ### Disk Indexes
197. - Sorted index files on disk
198. - Binary searchable for O(log n) lookups
199. - Rebuilt during compaction
201. ### Delta Indexes (In-Memory)
202. - Track changes since last persistence
203. - Checked before disk indexes
204. - Auto-persisted based on:
205. - Time interval (default: 60 seconds)
206. - Change threshold (default: 1000 operations)
208. ### Tombstones
209. - Track deleted records in memory
210. - Prevent returning deleted records from disk indexes
211. - Cleared during compaction
213. ## Performance Characteristics
215. ### Time Complexity
216. - **Insert**: O(1) for append, O(log n) for index update
217. - **Find by primary key (indexed)**: O(log n) binary search
218. - **Find with query function**: O(n) full scan
219. - **Update**: O(log n) find + O(1) insert
220. - **Delete**: O(log n) find + O(1) mark
221. - **Compact**: O(n) full scan + O(n log n) index rebuild
223. ### Space Complexity
224. - Data file grows with inserts (append-only)
225. - Deleted records remain until compaction
226. - Index files: O(n) per indexed field
227. - Delta indexes: O(m) where m = changes since last persist
229. ### File Size Comparison
230. MessagePack typically produces **15-20% smaller files** than BSON for the same data:
231. - More compact integer encoding
232. - Smaller string overhead
233. - Efficient array/map encoding
235. ## Concurrency Model
237. - **Single-writer, multiple-reader** via file locking
238. - Writes are serialized through lock file
239. - Reads can happen concurrently (no locks needed)
240. - Index persistence happens asynchronously but safely
242. ## Primary Key Types
244. ### NUMBER (PrimaryKeyType.NUMBER)
245. - Auto-incremented integer
246. - Stored in metadata.nextId
247. - Prefix syntax: `*id`
249. ### UUID (PrimaryKeyType.UUID)
250. - Sortable base36 unique ID (9-char timestamp + 3-char random)
251. - 12-character string format
252. - Prefix syntax: `@id`
254. ### STRING (PrimaryKeyType.STRING)
255. - User-provided string
256. - No auto-generation
257. - Default (no prefix)
259. ## Design Decisions
261. ### Why Append-Only?
262. - **Fast writes**: No seeking, just append
263. - **Crash safety**: Partial writes don't corrupt existing data
264. - **Simple implementation**: No complex update-in-place logic
266. ### Why MessagePack?
267. - **Smaller files**: 15-20% smaller than BSON on average
268. - **Fast serialization**: Comparable or faster than BSON
269. - **Wide language support**: Available in many programming languages
270. - **Simple format**: Easy to implement and debug
271. - **No external binary dependencies**: Pure JavaScript implementation
273. ### Why Custom Size Header?
274. - MessagePack doesn't define record boundaries
275. - Enables efficient streaming without full deserialization
276. - Allows skipping deleted records quickly
277. - Consistent with BSON's approach
279. ### Why File-Based Locking?
280. - **Simple**: No external dependencies
281. - **Cross-process**: Works across multiple Node.js processes
282. - **Portable**: Works on all platforms
284. ### Why Binary Search Indexes?
285. - **Disk-friendly**: Can search large indexes without loading into memory
286. - **Simple format**: Plain text, easy to debug
287. - **Fast lookups**: O(log n) for indexed queries
289. ## MessagePack vs BSON
291. ### Advantages of MessagePack
292. - **Smaller files**: 15-20% size reduction
293. - **Faster reads**: Simpler format, less parsing overhead
294. - **Pure JavaScript**: No native dependencies
295. - **Smaller library**: ~15KB vs ~173KB for BSON
297. ### Advantages of BSON
298. - **ObjectId type**: Built-in unique identifier type
299. - **Date precision**: Millisecond timestamps
300. - **Binary data**: Native binary type
301. - **MongoDB compatibility**: Direct compatibility with MongoDB
303. ### When to Choose MPackDB
304. - File size is a concern
305. - Pure JavaScript dependencies preferred
306. - Don't need MongoDB compatibility
307. - Want faster read performance
309. ### When to Choose BsonDB
310. - Need ObjectId primary keys
311. - MongoDB compatibility desired
312. - Working with binary data
313. - Need precise date/time handling
315. ## Limitations
317. 1. **Single-writer**: Only one write operation at a time
318. 2. **No transactions**: Operations are not atomic across multiple records
319. 3. **No query language**: Must use JavaScript functions for complex queries
320. 4. **Compaction required**: Deleted records consume space until compaction
321. 5. **Index overhead**: Each index doubles storage for that field
322. 6. **No schema validation**: Records can have any structure
323. 7. **No ObjectId type**: Must use UUIDs or numeric IDs
325. ## Future Improvements
327. - Batch insert operations
328. - Async compaction (background process)
329. - Query optimizer for complex filters
330. - Compression support (MessagePack supports extensions)
331. - Replication/backup utilities
332. - Schema validation layer
333. - Custom MessagePack extension types