mpackdb

10.0 KB

1. # MPackDB
3. A fast, local, append-only JSON database with [MessagePack](https://msgpack.org/) serialization for Node.js.
5. ## Features
7. - **🚀 High Performance** - Append-only writes with MessagePack binary serialization
8. - **📇 Flexible Indexing** - Optional numeric and lexical indexes for fast queries
9. - **🔒 Concurrent Access** - File-based locking for safe multi-process access
10. - **💾 Auto-Persistence** - Configurable automatic index persistence
11. - **🗜️ Auto-Compaction** - Removes deleted records on startup
12. - **🎯 Simple API** - Intuitive CRUD operations with async/await
13. - **📦 Zero Dependencies** - Only requires `msgpackr`
15. ## Installation
17. ```bash
18. npm install mpackdb
19. ```
21. ## Quick Start
23. ```javascript
24. import MPackDB from 'mpackdb';
26. // Create a database with auto-increment numeric primary key
27. const db = new MPackDB('data/users', {
28. primaryKey: '*id', // * prefix = numeric auto-increment
29. indexes: ['email', '*age'] // Index email (lexical) and age (numeric)
30. });
32. // Insert records
33. await db.insert({ name: 'Alice', email: '[email protected]', age: 30 });
34. await db.insert({ name: 'Bob', email: '[email protected]', age: 25 });
36. // Find all records
37. for await (const user of db.find()) {
38. console.log(user);
39. }
41. // Find by primary key
42. const users = await db.find(0); // Returns array with Alice
44. // Query with function
45. for await (const user of db.find(r => r.age > 28)) {
46. console.log(user.name); // Alice
47. }
49. // Update records
50. await db.update(0, { age: 31 });
51. await db.update(r => r.age < 26, { status: 'junior' });
53. // Delete records
54. await db.delete(r => r.age < 18);
56. // Close database (persists all changes)
57. await db.close();
58. ```
60. ## API Reference
62. ### Constructor
64. ```javascript
65. new MPackDB(dbFile, options)
66. ```
68. **Parameters:**
69. - `dbFile` (string): Path to database file (without extension)
70. - `options` (object):
71. - `primaryKey` (string): Primary key field name with optional prefix:
72. - `*field` - Numeric auto-increment (e.g., `*id`)
73. - `@field` - UUID (e.g., `@uuid`)
74. - `field` - String (e.g., `username`)
75. - `primaryKeyType` (PrimaryKeyType): Explicit type override
76. - `indexes` (string[]): Fields to index (use prefixes like primary key)
77. - `debug` (boolean): Enable debug logging (default: `false`)
78. - `indexPersistInterval` (number): Auto-persist interval in ms (default: `60000`, `0` to disable)
79. - `indexPersistThreshold` (number): Auto-persist after N changes (default: `1000`)
81. **Examples:**
82. functional style:
83. ```javascript
84. import MPackDB from '@worldapi.org/mpackdb';
86. const db = new MPackDB('data/products', {
87. primaryKey: '*id',
88. indexes: ['category', '*price', '@sku'],
89. indexPersistThreshold: 100
90. });
91. ```
93. OOP style:
94. ```javascript
95. import { MPackDB, Model, PrimaryKeyType } from 'mpackdb';
97. export class Product extends Model {
98. id = 0;
99. name = '';
100. price = 0;
101. category = '';
102. sku = '';
103. }
105. export class Products extends MPackDB {
106. _classToUse = Product;
107. _primaryKey = 'id';
108. _primaryKeyType = PrimaryKeyType.UUID;
109. _indexes = ['category', '*price', '@sku'];
110. }
112. export default products = new Products('data/products');
113. ```
115. ### insert(record, options)
117. Insert a new record into the database.
119. ```javascript
120. await db.insert({ name: 'Alice', age: 30 });
121. // Returns: { id: 0, name: 'Alice', age: 30 }
122. ```
124. **Parameters:**
125. - `record` (object): The record to insert
126. - `options` (object):
127. - `skipPrimaryKey` (boolean): Don't auto-generate primary key
129. **Returns:** Promise<Object> - The inserted record with primary key
131. ### find(query, options)
133. Find records in the database. Returns an async iterable Cursor.
135. ```javascript
136. // Find all
137. for await (const record of db.find()) {
138. console.log(record);
139. }
141. // Find by primary key
142. const users = await db.find(0);
144. // Find with query function
145. for await (const user of db.find(r => r.age > 30)) {
146. console.log(user.name);
147. }
148. ```
150. **Parameters:**
151. - `query` (undefined|string|Function):
152. - `undefined/null` - Returns all records
153. - `string` - Primary key value
154. - `Function` - Query function `(record) => boolean`
155. - `options` (object):
156. - `mode` (string): Return mode - `'record'`, `'raw'`, or `'mixed'`
158. **Returns:** Cursor (async iterable)
160. ### update(query, data, options)
162. Update records matching a query.
164. ```javascript
165. // Update by primary key
166. await db.update(0, { age: 31 });
168. // Update with query function
169. await db.update(r => r.age > 30, { status: 'senior' });
171. // Update with callback
172. await db.update(r => r.age > 30, r => ({ ...r, age: r.age + 1 }));
173. ```
175. **Parameters:**
176. - `query` (string|Function): Primary key or query function
177. - `data` (object|Function): Data to update or callback function
178. - `options` (object):
179. - `upsert` (boolean): Insert if no records match
181. **Returns:** Promise<Object[]> - Array of updated records
183. ### upsert(query, data)
185. Update records or insert if not found.
187. ```javascript
188. await db.upsert(r => r.email === '[email protected]', {
189. name: 'Alice',
190. email: '[email protected]',
191. age: 30
192. });
193. ```
195. ### delete(query, callback)
197. Delete records matching a query.
199. ```javascript
200. // Delete by primary key
201. await db.delete(0);
203. // Delete with query function
204. await db.delete(r => r.age < 18);
206. // Delete with callback
207. await db.delete(r => r.age < 18, async (record) => {
208. console.log('Deleting:', record.name);
209. return record;
210. });
211. ```
213. **Parameters:**
214. - `query` (string|Function): Primary key or query function
215. - `callback` (Function): Optional callback for each deleted record
217. **Returns:** Promise<Object[]> - Array of deleted records
219. ### compact()
221. Compact the database by removing deleted records. This rewrites the data file without tombstones.
223. ```javascript
224. await db.compact();
225. ```
227. **Note:** Compaction happens automatically on database initialization.
229. ### close()
231. Close the database and persist all pending changes. Should be called before process exit.
233. ```javascript
234. await db.close();
235. ```
237. ## Primary Key Types
239. MPackDB supports three primary key types:
241. ### Numeric (Auto-increment)
243. ```javascript
244. const db = new MPackDB('data/users', {
245. primaryKey: '*id' // * prefix
246. });
248. await db.insert({ name: 'Alice' });
249. // { id: 0, name: 'Alice' }
250. ```
252. ### UUID
254. ```javascript
255. const db = new MPackDB('data/sessions', {
256. primaryKey: '@sessionId' // @ prefix
257. });
259. await db.insert({ data: 'session data' });
260. // { sessionId: 'lz7gdcwh9x4r', data: 'session data' }
261. ```
263. ### String
265. ```javascript
266. const db = new MPackDB('data/users', {
267. primaryKey: 'username' // No prefix
268. });
270. await db.insert({ username: 'alice', name: 'Alice' });
271. // { username: 'alice', name: 'Alice' }
272. ```
274. ## Indexes
276. Indexes dramatically improve query performance for large datasets.
278. ### Index Types
280. - **Lexical** (default): String sorting, good for text fields
281. - **Numeric**: Number sorting, good for integers/floats
282. - **UUID**: Treated as lexical (string)
284. ### Creating Indexes
286. ```javascript
287. const db = new MPackDB('data/products', {
288. primaryKey: '*id',
289. indexes: [
290. 'category', // Lexical index
291. '*price', // Numeric index
292. '*stock', // Numeric index
293. '@sku' // UUID index (lexical)
294. ]
295. });
296. ```
298. ### Index Persistence
300. Indexes are automatically persisted based on:
301. - **Threshold**: After N changes (default: 1000)
302. - **Interval**: Every N milliseconds (default: 60000)
303. - **On close**: When `db.close()` is called
305. ```javascript
306. const db = new MPackDB('data/users', {
307. primaryKey: '*id',
308. indexes: ['email'],
309. indexPersistThreshold: 100, // Persist after 100 changes
310. indexPersistInterval: 30000 // Persist every 30 seconds
311. });
312. ```
314. ## File Structure
316. MPackDB creates the following files:
318. ```
319. data/
320. users.mpack # Main data file (MessagePack binary)
321. users.meta.json # Metadata (nextId, deleted offsets)
322. users.id.txt # Primary key index
323. users.email.txt # Email field index
324. users.age.txt # Age field index
325. users.lock # Lock file (temporary)
326. ```
328. ## Concurrency
330. MPackDB uses file-based locking to ensure safe concurrent access:
332. ```javascript
333. // Process 1
334. const db1 = new MPackDB('data/users', { primaryKey: '*id' });
335. await db1.insert({ name: 'Alice' });
337. // Process 2 (waits for lock)
338. const db2 = new MPackDB('data/users', { primaryKey: '*id' });
339. await db2.insert({ name: 'Bob' });
340. ```
342. ## Performance Tips
344. 1. **Use indexes** for frequently queried fields
345. 2. **Adjust persist thresholds** based on your write patterns
346. 3. **Call `compact()`** periodically if you have many deletes
347. 4. **Use numeric indexes** for number fields
348. 5. **Batch operations** when possible
350. ## Examples
352. ### User Management System
354. ```javascript
355. import MPackDB from '@worldapi.org/mpackdb';
357. const users = new MPackDB('data/users', {
358. primaryKey: '*id',
359. indexes: ['email', '*age', 'role']
360. });
362. // Register user
363. await users.insert({
364. email: '[email protected]',
365. name: 'Alice',
366. age: 30,
367. role: 'admin'
368. });
370. // Find by email
371. for await (const user of users.find(u => u.email === '[email protected]')) {
372. console.log('Found user:', user.name);
373. }
375. // Get all admins
376. for await (const admin of users.find(u => u.role === 'admin')) {
377. console.log('Admin:', admin.name);
378. }
380. // Update age
381. await users.update(u => u.email === '[email protected]', { age: 31 });
383. // Delete inactive users
384. await users.delete(u => u.lastLogin < Date.now() - 90 * 24 * 60 * 60 * 1000);
386. await users.close();
387. ```
389. ### Product Catalog
391. ```javascript
392. const products = new MPackDB('data/products', {
393. primaryKey: '*id',
394. indexes: ['category', '*price', '@sku']
395. });
397. // Add products
398. await products.insert({ sku: 'ABC-123', name: 'Laptop', category: 'Electronics', price: 999 });
399. await products.insert({ sku: 'DEF-456', name: 'Mouse', category: 'Electronics', price: 29 });
401. // Find by category
402. for await (const product of products.find(p => p.category === 'Electronics')) {
403. console.log(product.name, product.price);
404. }
406. // Find products under $50
407. for await (const product of products.find(p => p.price < 50)) {
408. console.log('Affordable:', product.name);
409. }
411. // Update price
412. await products.update(p => p.sku === 'ABC-123', { price: 899 });
414. await products.close();
415. ```
417. ## License
419. MIT
421. ## Contributing
423. Contributions are welcome! Please open an issue or submit a pull request.
425. ## Related Projects
427. - [**BsonDB**](https://bsondb.gitoria.worldapi.org) - Similar database using BSON serialization