mpackdb

MPackDB

A fast, local, append-only JSON database with MessagePack serialization for Node.js.

Features

• 🚀 High Performance - Append-only writes with MessagePack binary serialization
• 📇 Flexible Indexing - Optional numeric and lexical indexes for fast queries
• 🔒 Concurrent Access - File-based locking for safe multi-process access
• 💾 Auto-Persistence - Configurable automatic index persistence
• 🗜️ Auto-Compaction - Removes deleted records on startup
• 🎯 Simple API - Intuitive CRUD operations with async/await
• 📦 Source-First Package - Ships readable ESM source with one direct runtime dependency (msgpackr); consumers choose how to bundle or minify it

Installation

npm install mpackdb

Quick Start

import MPackDB from 'mpackdb';

// Create a database with auto-increment numeric primary key
const db = new MPackDB('data/users', {
  primaryKey: '*id',  // * prefix = numeric auto-increment
  indexes: ['email', '*age']  // Index email (lexical) and age (numeric)
});

// Insert records
await db.insert({ name: 'Alice', email: '[email protected]', age: 30 });
await db.insert({ name: 'Bob', email: '[email protected]', age: 25 });

// Find all records
for await (const user of db.find()) {
  console.log(user);
}

// Find by primary key
const users = await db.find(0);  // Returns array with Alice

// Query with function
for await (const user of db.find(r => r.age > 28)) {
  console.log(user.name);  // Alice
}

// Update records
await db.update(0, r => { r.age = 31; return r; });
await db.update(r => r.age < 26, r => { r.status = 'junior'; return r; });

// Delete records
await db.delete(r => r.age < 18);

// Close database (persists all changes)
await db.close();

API Reference

Constructor

new MPackDB(dbFile, options)

Parameters:

• dbFile (string): Path to database file (without extension)
• options (object):
• primaryKey (string): Primary key field name with optional prefix:
• *field - Numeric auto-increment (e.g., *id)
• @field - UUID (e.g., @uuid)
• field - String (e.g., username)
• primaryKeyType (PrimaryKeyType): Explicit type override
• indexes (string[]): Fields to index. Prefix options:
• *field - Numeric index
• @field - UUID index (lexical)
• !field - Unique index (rejects duplicates)
• !*field - Unique numeric index
• field - Lexical index (default)
• debug (boolean): Enable debug logging (default: false)
• compact (boolean): Run compaction on init (default: true, set false whenever another process may have the same files open)
• indexPersistInterval (number): Auto-persist interval in ms (default: 60000, 0 to disable)
• indexPersistThreshold (number): Auto-persist after N changes (default: 1000)
• staleLockTimeout (number): Take over lock files older than this many ms — a crashed holder never removes its lock (default: 30000, 0 to disable)

Examples: functional style:

import MPackDB from 'mpackdb';

const db = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: ['category', '*price', '@sku'],
  indexPersistThreshold: 100
});

OOP style:

import { MPackDB, Model, PrimaryKeyType } from 'mpackdb';

export class Product extends Model {
  id = 0;
  name = '';
  price = 0;
  category = '';
  sku = '';
}

export class Products extends MPackDB {
  _classToUse = Product;
  _primaryKey = 'id';
  _primaryKeyType = PrimaryKeyType.UUID;
  _indexes = ['category', '*price', '@sku'];
}

export default products = new Products('data/products');

insert(record, options)

Insert a new record into the database.

const id = await db.insert({ name: 'Alice', age: 30 });
// Returns: 0 (the generated primary-key value)

Parameters:

• record (object): The record to insert
• options (object):
• skipPrimaryKey (boolean): Don't auto-generate primary key

Returns: Promise<string|number|Object> - The primary-key value when a primary key is configured; otherwise the inserted record

find(query, options)

Find records in the database. Returns an async iterable Cursor.

The Cursor is both an async iterable (for streaming with for await) and a thenable (awaiting it returns an array).

// Find all (streaming)
for await (const record of db.find()) {
  console.log(record);
}

// Find all (array)
const all = await db.find();

// Find by primary key
const users = await db.find(0);

// Find with filter function (stream)
for await (const user of db.find(r => r.age > 30)) {
  console.log(user.name);
}

// Find with single index — only reads records from the index range
for await (const user of db.find(r => r.age <= 50, {
  index: { field: 'age', from: 30, to: 50 }
})) {
  console.log(user.name);
}

// Find with index (descending)
for await (const user of db.find(null, {
  index: { field: 'age', from: 50, direction: 'desc' }
})) {
  if (user.age < 18) break;
  console.log(user.name);
}

// Find with index intersection — intersects offset sets, then streams only matches
const results = await db.find(null, {
  index: [
    { field: 'x', from: 0, to: 100 },
    { field: 'status', value: 'active' }
  ]
});

Parameters:

• query (undefined|string|number|Function):
• undefined/null - Returns all records
• string/number - Primary key value
• Function - Filter function (record) => boolean
• options (object):
• mode (string): Return mode - 'record', 'raw', or 'mixed'
• index (object|array): Index hint(s) to narrow disk reads:
• field (string): Configured indexed field name. Unknown fields reject with error.code === 'INDEX_NOT_FOUND'
• value (any): Exact match lookup
• from (any): Start key (inclusive) for range scan
• to (any): End key (inclusive) for range scan
• direction ('asc'|'desc'): Scan direction (default: 'asc')

When index is an array, offset sets from each index are intersected before streaming records from disk.

Returns: Cursor (async iterable + thenable)

update(query, callback, options)

Update records matching a query. The callback receives the old record and must return the new record. There is no automatic merging — the callback has full control.

// Update by primary key — modify and return
await db.update(0, record => {
  record.age = 32;
  delete record.address;
  return record; // RETURN IS IMPORTANT OR YOUR RECORD WILL BE undefined
});

// Update with query function
await db.update(r => r.age > 30, record => {
  record.status = 'senior';
  return record;
});

// Spread for partial updates
await db.update(0, record => ({ ...record, age: 32 }));

Parameters:

• query (string|number|Function): Primary key or query function
• callback (Function): Receives old record, must return new record
• options (object):
• upsert (boolean): Insert if no records match
• index (object|array): Index hint(s), same as find()

Returns: Promise<Object[]> - Array of updated records

upsert(query, callback, options)

Update records or insert if not found. The callback receives {} when inserting.

await db.upsert(r => r.email === '[email protected]', record => ({
  ...record,
  name: 'Alice',
  email: '[email protected]',
  age: 30
}));

Parameters:

• query (string|number|Function): Primary key or query function
• callback (Function): Receives old record (or {} if inserting), must return new record
• options (object):
• index (object|array): Index hint(s), same as find()

delete(query, options)

Delete records matching a query.

// Delete by primary key
await db.delete(0);

// Delete with query function
await db.delete(r => r.age < 18);

// Delete with index hint (avoids full scan)
await db.delete(r => r.status === 'inactive', {
  index: { field: 'status', value: 'inactive' }
});

Parameters:

• query (string|Function): Primary key or query function
• options (object):
• index (object|array): Index hint(s), same as find()

Returns: Promise<Object[]> - Array of deleted records

compact()

Compact the database by removing deleted records. This rewrites the data file without tombstones.

await db.compact();

Note: Compaction happens automatically on database initialization.

boundingBox(corners, filter)

Find records within a bounding box defined by 4 corner coordinates. Requires numeric indexes on x and y fields. Corners can be in any order — min/max are extracted automatically.

const db = new MPackDB('data/sectors', {
  primaryKey: '*id',
  indexes: ['*x', '*y']
});

// Find all sectors in a bounding box
const sectors = await db.boundingBox([
  { x: -5, y: 5 },
  { x: 5, y: 5 },
  { x: 5, y: -5 },
  { x: -5, y: -5 }
]);

// Streaming
for await (const sector of db.boundingBox([
  { x: 0, y: 0 }, { x: 100, y: 0 },
  { x: 100, y: 100 }, { x: 0, y: 100 }
])) {
  console.log(sector);
}

// With additional filter
const active = await db.boundingBox([
  { x: -5, y: 5 }, { x: 5, y: 5 },
  { x: 5, y: -5 }, { x: -5, y: -5 }
], s => s.status === 'active');

Parameters:

• corners (Array<{x: number, y: number}>): 4 corner coordinates
• filter (Function): Optional additional filter function

Returns: Cursor (async iterable + thenable)

withLock(callback)

Execute a callback while holding the database lock. The lock is re-entrant: find, insert, delete, update called inside the callback reuse the same lock instead of deadlocking.

Use this for compound operations that must be atomic, e.g. find-then-insert.

const user = await db.withLock(async () => {
  const [existing] = await db.find(u => u.email === email);
  if (existing) return existing;
  return db.insert({ email, name });
});

Parameters:

• callback (Function): Async function to execute under lock

Returns: Promise<any> - The return value of the callback

close()

Close the database and persist all pending changes. Should be called before process exit.

await db.close();

Primary Key Types

MPackDB supports three primary key types:

Numeric (Auto-increment)

const db = new MPackDB('data/users', {
  primaryKey: '*id'  // * prefix
});

await db.insert({ name: 'Alice' });
// { id: 0, name: 'Alice' }

UUID

const db = new MPackDB('data/sessions', {
  primaryKey: '@sessionId'  // @ prefix
});

await db.insert({ data: 'session data' });
// { sessionId: 'lz7gdcwh9x4r', data: 'session data' }

String

const db = new MPackDB('data/users', {
  primaryKey: 'username'  // No prefix
});

await db.insert({ username: 'alice', name: 'Alice' });
// { username: 'alice', name: 'Alice' }

Indexes

Indexes dramatically improve query performance for large datasets.

Index Types

• Lexical (default): String sorting, good for text fields
• Numeric: Number sorting, good for integers/floats
• UUID: Treated as lexical (string)
• Unique: Rejects duplicate values on insert (any type)

Creating Indexes

const db = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: [
    'category',      // Lexical index
    '*price',        // Numeric index
    '*stock',        // Numeric index
    '@sku',          // UUID index (lexical)
    '!email'         // Unique lexical index
  ]
});

Unique Indexes

Unique indexes prevent duplicate values. Primary keys are always unique by default. Use the ! prefix to make secondary indexes unique:

const db = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['!email', '!username', '*age']
});

await db.insert({ email: '[email protected]', username: 'alice', age: 30 }); // ok
await db.insert({ email: '[email protected]', username: 'bob', age: 25 });
// throws: Error { code: 'DUPLICATE_KEY', field: 'email', value: '[email protected]' }

Combine ! with type prefixes: !*field for unique numeric, !@field for unique UUID.

Uniqueness is enforced atomically under the write lock, so concurrent inserts cannot create duplicates.

Index Persistence

Indexes are automatically persisted based on:

• Threshold: After N changes (default: 1000)
• Interval: Every N milliseconds (default: 60000)
• On close: When db.close() is called

const db = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['email'],
  indexPersistThreshold: 100,  // Persist after 100 changes
  indexPersistInterval: 30000  // Persist every 30 seconds
});

File Structure

MPackDB creates the following files:

data/
  users.mpack          # Main data file (MessagePack binary)
  users.meta.json       # Metadata (version, nextId, deleted offsets, schema)
  users.id.txt          # Primary key index
  users.email.txt       # Email field index
  users.age.txt         # Age field index
  users.idxstate.json   # Index coverage state (how far into the data file the indexes reach)
  users.lock            # Lock file (temporary)

Concurrency

MPackDB supports concurrent reads and writes within one process (multiple async flows on the same instance) and across processes (multiple processes opening the same files).

In-process

Writes on an instance are serialized through an in-process FIFO queue. The lock is causal: operations called inside a lock-holding call chain (find inside delete, insert inside withLock) re-enter, while unrelated concurrent operations wait their turn. Reads (find) are lock-free and always see a consistent snapshot — a concurrent write can never roll back another write's in-flight state.

// Safe: concurrent lanes writing to the same store
await Promise.all([
  db.delete(spentId),
  db.insert(newUtxo),
  (async () => { for await (const r of db.find(q)) { ... } })(),
]);

For compound read-then-write operations that must be atomic, use withLock (see above).

Multi-process

Cross-process exclusion uses a lock file; visibility uses a version counter in meta.json plus incremental index catch-up:

• meta.json carries a monotonic version. refresh() (run before every read and after acquiring the write lock) only reloads when another process persisted a newer version — the local in-memory state stays authoritative otherwise.
• New records appended by other processes are picked up by an incremental tail scan of the data file (<name>.idxstate.json tracks index coverage), so indexed queries see other processes' inserts even before they persist their index files.
• meta.json also stores the collection's schema (primary key + indexes), so tools like mpackdb-admin can open and safely write to any collection without knowing its configuration.
• Stale locks from crashed processes are taken over after staleLockTimeout (default 30s, 0 disables). Raise it if you run operations holding the lock longer than that (e.g. compacting huge files).

// Process 1 — the app
const db1 = new MPackDB('data/users', { primaryKey: '*id', compact: false });
await db1.insert({ name: 'Alice' });

// Process 2 — admin UI, migration script, ... (waits for lock, sees Alice)
const db2 = new MPackDB('data/users', { primaryKey: '*id', compact: false });
await db2.insert({ name: 'Bob' });

Compaction rule: when multiple processes have the same files open, run with compact: false and only compact when you have exclusive access. Compaction replaces the data file via rename(); other attached instances detect the inode change on their next refresh() and reopen/rebase themselves, but the window is not transactional.

// Exclusive owner — may compact on startup (default)
const db = new MPackDB('data/users', { primaryKey: '*id' });

// Anyone sharing the files with another live process — no compaction
const admin = new MPackDB('data/users', { primaryKey: '*id', compact: false });

Upgrading from <= 1.0.6

Old databases lack the meta version, the persisted schema and <name>.idxstate.json; all three are created automatically on first use. The missing idxstate triggers a one-time full index rebuild on the first open — expect a longer first start on large collections.

Performance Tips

1. Use indexes for frequently queried fields
2. Adjust persist thresholds based on your write patterns
3. Call compact() periodically if you have many deletes
4. Use numeric indexes for number fields
5. Batch operations when possible

Examples

User Management System

import MPackDB from 'mpackdb';

const users = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['email', '*age', 'role']
});

// Register user
await users.insert({
  email: '[email protected]',
  name: 'Alice',
  age: 30,
  role: 'admin'
});

// Find by email
for await (const user of users.find(u => u.email === '[email protected]')) {
  console.log('Found user:', user.name);
}

// Get all admins
for await (const admin of users.find(u => u.role === 'admin')) {
  console.log('Admin:', admin.name);
}

// Update age
await users.update(u => u.email === '[email protected]', r => { r.age = 31; return r; });

// Delete inactive users
await users.delete(u => u.lastLogin < Date.now() - 90 * 24 * 60 * 60 * 1000);

await users.close();

Product Catalog

const products = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: ['category', '*price', '@sku']
});

// Add products
await products.insert({ sku: 'ABC-123', name: 'Laptop', category: 'Electronics', price: 999 });
await products.insert({ sku: 'DEF-456', name: 'Mouse', category: 'Electronics', price: 29 });

// Find by category
for await (const product of products.find(p => p.category === 'Electronics')) {
  console.log(product.name, product.price);
}

// Find products under $50
for await (const product of products.find(p => p.price < 50)) {
  console.log('Affordable:', product.name);
}

// Update price
await products.update(p => p.sku === 'ABC-123', r => { r.price = 899; return r; });

await products.close();

License

MIT

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

Related Projects

• **BsonDB** - Similar database using BSON serialization