mpackdb

All repositories: gitoria

Readme Code Pull requests Releases Tickets Settings

Branchmaster

87888725release 1.0.7caramboleyo2026-09-25 23:30

master/README.md

18.1 KB

# MPackDB
A fast, local, append-only JSON database with [MessagePack](https://msgpack.org/) serialization for Node.js.
## Features
- **🚀 High Performance** - Append-only writes with MessagePack binary serialization
- **📇 Flexible Indexing** - Optional numeric and lexical indexes for fast queries
- **🔒 Concurrent Access** - File-based locking for safe multi-process access
- **💾 Auto-Persistence** - Configurable automatic index persistence
- **🗜️ Auto-Compaction** - Removes deleted records on startup
- **🎯 Simple API** - Intuitive CRUD operations with async/await
- **📦 Source-First Package** - Ships readable ESM source with one direct runtime dependency (`msgpackr`); consumers choose how to bundle or minify it
## Installation
```bash
npm install mpackdb
```
## Quick Start
```javascript
import MPackDB from 'mpackdb';
// Create a database with auto-increment numeric primary key
const db = new MPackDB('data/users', {
primaryKey: '*id', // * prefix = numeric auto-increment
indexes: ['email', '*age'] // Index email (lexical) and age (numeric)
});
// Insert records
await db.insert({ name: 'Alice', email: '[email protected]', age: 30 });
await db.insert({ name: 'Bob', email: '[email protected]', age: 25 });
// Find all records
for await (const user of db.find()) {
console.log(user);
}
// Find by primary key
const users = await db.find(0); // Returns array with Alice
// Query with function
for await (const user of db.find(r => r.age > 28)) {
console.log(user.name); // Alice
}
// Update records
await db.update(0, r => { r.age = 31; return r; });
await db.update(r => r.age < 26, r => { r.status = 'junior'; return r; });
// Delete records
await db.delete(r => r.age < 18);
// Close database (persists all changes)
await db.close();
```
## API Reference
### Constructor
```javascript
new MPackDB(dbFile, options)
```
**Parameters:**
- `dbFile` (string): Path to database file (without extension)
- `options` (object):
- `primaryKey` (string): Primary key field name with optional prefix:
- `*field` - Numeric auto-increment (e.g., `*id`)
- `@field` - UUID (e.g., `@uuid`)
- `field` - String (e.g., `username`)
- `primaryKeyType` (PrimaryKeyType): Explicit type override
- `indexes` (string[]): Fields to index. Prefix options:
- `*field` - Numeric index
- `@field` - UUID index (lexical)
- `!field` - Unique index (rejects duplicates)
- `!*field` - Unique numeric index
- `field` - Lexical index (default)
- `debug` (boolean): Enable debug logging (default: `false`)
- `compact` (boolean): Run compaction on init (default: `true`, set `false` whenever another process may have the same files open)
- `indexPersistInterval` (number): Auto-persist interval in ms (default: `60000`, `0` to disable)
- `indexPersistThreshold` (number): Auto-persist after N changes (default: `1000`)
- `staleLockTimeout` (number): Take over lock files older than this many ms — a crashed holder never removes its lock (default: `30000`, `0` to disable)
**Examples:**
functional style:
```javascript
import MPackDB from 'mpackdb';
const db = new MPackDB('data/products', {
primaryKey: '*id',
indexes: ['category', '*price', '@sku'],
indexPersistThreshold: 100
});
```
OOP style:
```javascript
import { MPackDB, Model, PrimaryKeyType } from 'mpackdb';
export class Product extends Model {
id = 0;
name = '';
price = 0;
category = '';
sku = '';
}
export class Products extends MPackDB {
_classToUse = Product;
_primaryKey = 'id';
_primaryKeyType = PrimaryKeyType.UUID;
_indexes = ['category', '*price', '@sku'];
}
export default products = new Products('data/products');
```
### insert(record, options)
Insert a new record into the database.
```javascript
const id = await db.insert({ name: 'Alice', age: 30 });
// Returns: 0 (the generated primary-key value)
```
**Parameters:**
- `record` (object): The record to insert
- `options` (object):
- `skipPrimaryKey` (boolean): Don't auto-generate primary key
**Returns:** Promise<string|number|Object> - The primary-key value when a primary key is configured; otherwise the inserted record
### find(query, options)
Find records in the database. Returns an async iterable Cursor.
The Cursor is both an async iterable (for streaming with `for await`) and a thenable (awaiting it returns an array).
```javascript
// Find all (streaming)
for await (const record of db.find()) {
console.log(record);
}
// Find all (array)
const all = await db.find();
// Find by primary key
const users = await db.find(0);
// Find with filter function (stream)
for await (const user of db.find(r => r.age > 30)) {
console.log(user.name);
}
// Find with single index — only reads records from the index range
for await (const user of db.find(r => r.age <= 50, {
index: { field: 'age', from: 30, to: 50 }
})) {
console.log(user.name);
}
// Find with index (descending)
for await (const user of db.find(null, {
index: { field: 'age', from: 50, direction: 'desc' }
})) {
if (user.age < 18) break;
console.log(user.name);
}
// Find with index intersection — intersects offset sets, then streams only matches
const results = await db.find(null, {
index: [
{ field: 'x', from: 0, to: 100 },
{ field: 'status', value: 'active' }
]
});
```
**Parameters:**
- `query` (undefined|string|number|Function):
- `undefined/null` - Returns all records
- `string/number` - Primary key value
- `Function` - Filter function `(record) => boolean`
- `options` (object):
- `mode` (string): Return mode - `'record'`, `'raw'`, or `'mixed'`
- `index` (object|array): Index hint(s) to narrow disk reads:
- `field` (string): Configured indexed field name. Unknown fields reject with `error.code === 'INDEX_NOT_FOUND'`
- `value` (any): Exact match lookup
- `from` (any): Start key (inclusive) for range scan
- `to` (any): End key (inclusive) for range scan
- `direction` ('asc'|'desc'): Scan direction (default: 'asc')
When `index` is an array, offset sets from each index are intersected before streaming records from disk.
**Returns:** Cursor (async iterable + thenable)
### update(query, callback, options)
Update records matching a query. The callback receives the old record and must return the new record. There is no automatic merging — the callback has full control.
```javascript
// Update by primary key — modify and return
await db.update(0, record => {
record.age = 32;
delete record.address;
return record; // RETURN IS IMPORTANT OR YOUR RECORD WILL BE undefined
});
// Update with query function
await db.update(r => r.age > 30, record => {
record.status = 'senior';
return record;
});
// Spread for partial updates
await db.update(0, record => ({ ...record, age: 32 }));
```
**Parameters:**
- `query` (string|number|Function): Primary key or query function
- `callback` (Function): Receives old record, must return new record
- `options` (object):
- `upsert` (boolean): Insert if no records match
- `index` (object|array): Index hint(s), same as `find()`
**Returns:** Promise<Object[]> - Array of updated records
### upsert(query, callback, options)
Update records or insert if not found. The callback receives `{}` when inserting.
```javascript
await db.upsert(r => r.email === '[email protected]', record => ({
...record,
name: 'Alice',
email: '[email protected]',
age: 30
}));
```
**Parameters:**
- `query` (string|number|Function): Primary key or query function
- `callback` (Function): Receives old record (or `{}` if inserting), must return new record
- `options` (object):
- `index` (object|array): Index hint(s), same as `find()`
### delete(query, options)
Delete records matching a query.
```javascript
// Delete by primary key
await db.delete(0);
// Delete with query function
await db.delete(r => r.age < 18);
// Delete with index hint (avoids full scan)
await db.delete(r => r.status === 'inactive', {
index: { field: 'status', value: 'inactive' }
});
```
**Parameters:**
- `query` (string|Function): Primary key or query function
- `options` (object):
- `index` (object|array): Index hint(s), same as `find()`
**Returns:** Promise<Object[]> - Array of deleted records
### compact()
Compact the database by removing deleted records. This rewrites the data file without tombstones.
```javascript
await db.compact();
```
**Note:** Compaction happens automatically on database initialization.
### boundingBox(corners, filter)
Find records within a bounding box defined by 4 corner coordinates. Requires numeric indexes on `x` and `y` fields. Corners can be in any order — min/max are extracted automatically.
```javascript
const db = new MPackDB('data/sectors', {
primaryKey: '*id',
indexes: ['*x', '*y']
});
// Find all sectors in a bounding box
const sectors = await db.boundingBox([
{ x: -5, y: 5 },
{ x: 5, y: 5 },
{ x: 5, y: -5 },
{ x: -5, y: -5 }
]);
// Streaming
for await (const sector of db.boundingBox([
{ x: 0, y: 0 }, { x: 100, y: 0 },
{ x: 100, y: 100 }, { x: 0, y: 100 }
])) {
console.log(sector);
}
// With additional filter
const active = await db.boundingBox([
{ x: -5, y: 5 }, { x: 5, y: 5 },
{ x: 5, y: -5 }, { x: -5, y: -5 }
], s => s.status === 'active');
```
**Parameters:**
- `corners` (Array<{x: number, y: number}>): 4 corner coordinates
- `filter` (Function): Optional additional filter function
**Returns:** Cursor (async iterable + thenable)
### withLock(callback)
Execute a callback while holding the database lock. The lock is re-entrant: `find`, `insert`, `delete`, `update` called inside the callback reuse the same lock instead of deadlocking.
Use this for compound operations that must be atomic, e.g. find-then-insert.
```javascript
const user = await db.withLock(async () => {
const [existing] = await db.find(u => u.email === email);
if (existing) return existing;
return db.insert({ email, name });
});
```
**Parameters:**
- `callback` (Function): Async function to execute under lock
**Returns:** Promise<any> - The return value of the callback
### close()
Close the database and persist all pending changes. Should be called before process exit.
```javascript
await db.close();
```
## Primary Key Types
MPackDB supports three primary key types:
### Numeric (Auto-increment)
```javascript
const db = new MPackDB('data/users', {
primaryKey: '*id' // * prefix
});
await db.insert({ name: 'Alice' });
// { id: 0, name: 'Alice' }
```
### UUID
```javascript
const db = new MPackDB('data/sessions', {
primaryKey: '@sessionId' // @ prefix
});
await db.insert({ data: 'session data' });
// { sessionId: 'lz7gdcwh9x4r', data: 'session data' }
```
### String
```javascript
const db = new MPackDB('data/users', {
primaryKey: 'username' // No prefix
});
await db.insert({ username: 'alice', name: 'Alice' });
// { username: 'alice', name: 'Alice' }
```
## Indexes
Indexes dramatically improve query performance for large datasets.
### Index Types
- **Lexical** (default): String sorting, good for text fields
- **Numeric**: Number sorting, good for integers/floats
- **UUID**: Treated as lexical (string)
- **Unique**: Rejects duplicate values on insert (any type)
### Creating Indexes
```javascript
const db = new MPackDB('data/products', {
primaryKey: '*id',
indexes: [
'category', // Lexical index
'*price', // Numeric index
'*stock', // Numeric index
'@sku', // UUID index (lexical)
'!email' // Unique lexical index
]
});
```
### Unique Indexes
Unique indexes prevent duplicate values. Primary keys are always unique by default. Use the `!` prefix to make secondary indexes unique:
```javascript
const db = new MPackDB('data/users', {
primaryKey: '*id',
indexes: ['!email', '!username', '*age']
});
await db.insert({ email: '[email protected]', username: 'alice', age: 30 }); // ok
await db.insert({ email: '[email protected]', username: 'bob', age: 25 });
// throws: Error { code: 'DUPLICATE_KEY', field: 'email', value: '[email protected]' }
```
Combine `!` with type prefixes: `!*field` for unique numeric, `!@field` for unique UUID.
Uniqueness is enforced atomically under the write lock, so concurrent inserts cannot create duplicates.
### Index Persistence
Indexes are automatically persisted based on:
- **Threshold**: After N changes (default: 1000)
- **Interval**: Every N milliseconds (default: 60000)
- **On close**: When `db.close()` is called
```javascript
const db = new MPackDB('data/users', {
primaryKey: '*id',
indexes: ['email'],
indexPersistThreshold: 100, // Persist after 100 changes
indexPersistInterval: 30000 // Persist every 30 seconds
});
```
## File Structure
MPackDB creates the following files:
```
data/
users.mpack # Main data file (MessagePack binary)
users.meta.json # Metadata (version, nextId, deleted offsets, schema)
users.id.txt # Primary key index
users.email.txt # Email field index
users.age.txt # Age field index
users.idxstate.json # Index coverage state (how far into the data file the indexes reach)
users.lock # Lock file (temporary)
```
## Concurrency
MPackDB supports concurrent reads and writes **within one process** (multiple async flows on the same instance) and **across processes** (multiple processes opening the same files).
### In-process
Writes on an instance are serialized through an in-process FIFO queue. The lock is *causal*: operations called inside a lock-holding call chain (`find` inside `delete`, `insert` inside `withLock`) re-enter, while unrelated concurrent operations wait their turn. Reads (`find`) are lock-free and always see a consistent snapshot — a concurrent write can never roll back another write's in-flight state.
```javascript
// Safe: concurrent lanes writing to the same store
await Promise.all([
db.delete(spentId),
db.insert(newUtxo),
(async () => { for await (const r of db.find(q)) { ... } })(),
]);
```
For compound read-then-write operations that must be atomic, use `withLock` (see above).
### Multi-process
Cross-process exclusion uses a lock file; visibility uses a version counter in `meta.json` plus incremental index catch-up:
- `meta.json` carries a monotonic `version`. `refresh()` (run before every read and after acquiring the write lock) only reloads when another process persisted a newer version — the local in-memory state stays authoritative otherwise.
- New records appended by other processes are picked up by an incremental tail scan of the data file (`<name>.idxstate.json` tracks index coverage), so **indexed queries see other processes' inserts even before they persist their index files**.
- `meta.json` also stores the collection's **schema** (primary key + indexes), so tools like [mpackdb-admin](https://github.com/caramboleyo/mpackdb-admin) can open and safely write to any collection without knowing its configuration.
- Stale locks from crashed processes are taken over after `staleLockTimeout` (default 30s, `0` disables). Raise it if you run operations holding the lock longer than that (e.g. compacting huge files).
```javascript
// Process 1 — the app
const db1 = new MPackDB('data/users', { primaryKey: '*id', compact: false });
await db1.insert({ name: 'Alice' });
// Process 2 — admin UI, migration script, ... (waits for lock, sees Alice)
const db2 = new MPackDB('data/users', { primaryKey: '*id', compact: false });
await db2.insert({ name: 'Bob' });
```
**Compaction rule:** when multiple processes have the same files open, run with `compact: false` and only compact when you have exclusive access. Compaction replaces the data file via `rename()`; other attached instances detect the inode change on their next `refresh()` and reopen/rebase themselves, but the window is not transactional.
```javascript
// Exclusive owner — may compact on startup (default)
const db = new MPackDB('data/users', { primaryKey: '*id' });
// Anyone sharing the files with another live process — no compaction
const admin = new MPackDB('data/users', { primaryKey: '*id', compact: false });
```
### Upgrading from <= 1.0.6
Old databases lack the meta `version`, the persisted schema and `<name>.idxstate.json`; all three are created automatically on first use. The missing idxstate triggers a **one-time full index rebuild** on the first open — expect a longer first start on large collections.
## Performance Tips
1. **Use indexes** for frequently queried fields
2. **Adjust persist thresholds** based on your write patterns
3. **Call `compact()`** periodically if you have many deletes
4. **Use numeric indexes** for number fields
5. **Batch operations** when possible
## Examples
### User Management System
```javascript
import MPackDB from 'mpackdb';
const users = new MPackDB('data/users', {
primaryKey: '*id',
indexes: ['email', '*age', 'role']
});
// Register user
await users.insert({
email: '[email protected]',
name: 'Alice',
age: 30,
role: 'admin'
});
// Find by email
for await (const user of users.find(u => u.email === '[email protected]')) {
console.log('Found user:', user.name);
}
// Get all admins
for await (const admin of users.find(u => u.role === 'admin')) {
console.log('Admin:', admin.name);
}
// Update age
await users.update(u => u.email === '[email protected]', r => { r.age = 31; return r; });
// Delete inactive users
await users.delete(u => u.lastLogin < Date.now() - 90 * 24 * 60 * 60 * 1000);
await users.close();
```
### Product Catalog
```javascript
const products = new MPackDB('data/products', {
primaryKey: '*id',
indexes: ['category', '*price', '@sku']
});
// Add products
await products.insert({ sku: 'ABC-123', name: 'Laptop', category: 'Electronics', price: 999 });
await products.insert({ sku: 'DEF-456', name: 'Mouse', category: 'Electronics', price: 29 });
// Find by category
for await (const product of products.find(p => p.category === 'Electronics')) {
console.log(product.name, product.price);
}
// Find products under $50
for await (const product of products.find(p => p.price < 50)) {
console.log('Affordable:', product.name);
}
// Update price
await products.update(p => p.sku === 'ABC-123', r => { r.price = 899; return r; });
await products.close();
```
## License
MIT
## Contributing
Contributions are welcome! Please open an issue or submit a pull request.
## Related Projects
- [**BsonDB**](https://bsondb.gitoria.worldapi.org) - Similar database using BSON serialization

Branches

mastermain branch

Latest commits

87888725release 1.0.7caramboleyo2026-09-25 23:30
c4cdb9b6node: import prefixes (Deno compat) + pre-existing index-state WIPcaramboleyo2026-09-25 23:30
0afb8f4bupdate now must be a callbackcaramboleyo2026-03-24 22:19
cde73eb4release 1.0.6caramboleyo2026-03-21 10:41
d01dda02add index hints, intersection, boundingBox; remove findByIndexcaramboleyo2026-03-21 10:39
b8ffc1a0release 1.0.5caramboleyo2026-03-12 19:51
d47876a1reimplemented lost features like indexed find and more testscaramboleyo2026-03-12 19:48
7f08da9afixed insert ignoring model definitioncaramboleyo2026-03-09 00:47
705774a9added flush before findcaramboleyo2026-03-08 22:01
b4db6391initial commitcaramboleyo2026-03-07 13:57