mpackdb

All repositories: gitoria

Readme Code Pull requests Releases Tickets Settings

Commitd47876a1

d47876a1reimplemented lost features like indexed find and more testscaramboleyo2026-03-12 19:48

d47876a1/README.md

13.5 KB

# MPackDB
A fast, local, append-only JSON database with [MessagePack](https://msgpack.org/) serialization for Node.js.
## Features
- **🚀 High Performance** - Append-only writes with MessagePack binary serialization
- **📇 Flexible Indexing** - Optional numeric and lexical indexes for fast queries
- **🔒 Concurrent Access** - File-based locking for safe multi-process access
- **💾 Auto-Persistence** - Configurable automatic index persistence
- **🗜️ Auto-Compaction** - Removes deleted records on startup
- **🎯 Simple API** - Intuitive CRUD operations with async/await
- **📦 Zero Dependencies** - Only requires `msgpackr`
## Installation
```bash
npm install mpackdb
```
## Quick Start
```javascript
import MPackDB from 'mpackdb';
// Create a database with auto-increment numeric primary key
const db = new MPackDB('data/users', {
primaryKey: '*id', // * prefix = numeric auto-increment
indexes: ['email', '*age'] // Index email (lexical) and age (numeric)
});
// Insert records
await db.insert({ name: 'Alice', email: '[email protected]', age: 30 });
await db.insert({ name: 'Bob', email: '[email protected]', age: 25 });
// Find all records
for await (const user of db.find()) {
console.log(user);
}
// Find by primary key
const users = await db.find(0); // Returns array with Alice
// Query with function
for await (const user of db.find(r => r.age > 28)) {
console.log(user.name); // Alice
}
// Update records
await db.update(0, { age: 31 });
await db.update(r => r.age < 26, { status: 'junior' });
// Delete records
await db.delete(r => r.age < 18);
// Close database (persists all changes)
await db.close();
```
## API Reference
### Constructor
```javascript
new MPackDB(dbFile, options)
```
**Parameters:**
- `dbFile` (string): Path to database file (without extension)
- `options` (object):
- `primaryKey` (string): Primary key field name with optional prefix:
- `*field` - Numeric auto-increment (e.g., `*id`)
- `@field` - UUID (e.g., `@uuid`)
- `field` - String (e.g., `username`)
- `primaryKeyType` (PrimaryKeyType): Explicit type override
- `indexes` (string[]): Fields to index. Prefix options:
- `*field` - Numeric index
- `@field` - UUID index (lexical)
- `!field` - Unique index (rejects duplicates)
- `!*field` - Unique numeric index
- `field` - Lexical index (default)
- `debug` (boolean): Enable debug logging (default: `false`)
- `compact` (boolean): Run compaction on init (default: `true`, set `false` for read-only / secondary instances)
- `indexPersistInterval` (number): Auto-persist interval in ms (default: `60000`, `0` to disable)
- `indexPersistThreshold` (number): Auto-persist after N changes (default: `1000`)
**Examples:**
functional style:
```javascript
import MPackDB from '@worldapi.org/mpackdb';
const db = new MPackDB('data/products', {
primaryKey: '*id',
indexes: ['category', '*price', '@sku'],
indexPersistThreshold: 100
});
```
OOP style:
```javascript
import { MPackDB, Model, PrimaryKeyType } from 'mpackdb';
export class Product extends Model {
id = 0;
name = '';
price = 0;
category = '';
sku = '';
}
export class Products extends MPackDB {
_classToUse = Product;
_primaryKey = 'id';
_primaryKeyType = PrimaryKeyType.UUID;
_indexes = ['category', '*price', '@sku'];
}
export default products = new Products('data/products');
```
### insert(record, options)
Insert a new record into the database.
```javascript
await db.insert({ name: 'Alice', age: 30 });
// Returns: { id: 0, name: 'Alice', age: 30 }
```
**Parameters:**
- `record` (object): The record to insert
- `options` (object):
- `skipPrimaryKey` (boolean): Don't auto-generate primary key
**Returns:** Promise<Object> - The inserted record with primary key
### find(query, options)
Find records in the database. Returns an async iterable Cursor.
```javascript
// Find all
for await (const record of db.find()) {
console.log(record);
}
// Find by primary key
const users = await db.find(0);
// Find with query function
for await (const user of db.find(r => r.age > 30)) {
console.log(user.name);
}
```
**Parameters:**
- `query` (undefined|string|Function):
- `undefined/null` - Returns all records
- `string` - Primary key value
- `Function` - Query function `(record) => boolean`
- `options` (object):
- `mode` (string): Return mode - `'record'`, `'raw'`, or `'mixed'`
**Returns:** Cursor (async iterable)
### update(query, data, options)
Update records matching a query.
```javascript
// Update by primary key
await db.update(0, { age: 31 });
// Update with query function
await db.update(r => r.age > 30, { status: 'senior' });
// Update with callback
await db.update(r => r.age > 30, r => ({ ...r, age: r.age + 1 }));
```
**Parameters:**
- `query` (string|Function): Primary key or query function
- `data` (object|Function): Data to update or callback function
- `options` (object):
- `upsert` (boolean): Insert if no records match
**Returns:** Promise<Object[]> - Array of updated records
### upsert(query, data)
Update records or insert if not found.
```javascript
await db.upsert(r => r.email === '[email protected]', {
name: 'Alice',
email: '[email protected]',
age: 30
});
```
### delete(query, callback)
Delete records matching a query.
```javascript
// Delete by primary key
await db.delete(0);
// Delete with query function
await db.delete(r => r.age < 18);
// Delete with callback
await db.delete(r => r.age < 18, async (record) => {
console.log('Deleting:', record.name);
return record;
});
```
**Parameters:**
- `query` (string|Function): Primary key or query function
- `callback` (Function): Optional callback for each deleted record
**Returns:** Promise<Object[]> - Array of deleted records
### compact()
Compact the database by removing deleted records. This rewrites the data file without tombstones.
```javascript
await db.compact();
```
**Note:** Compaction happens automatically on database initialization.
### findByIndex(field, options)
Walk an index in sorted order. Returns an async iterable Cursor. Streams block by block from disk — does not load the full index into memory.
```javascript
// All users sorted by age
for await (const user of db.findByIndex('age')) {
console.log(user.name, user.age);
}
// Age 25+, first 10
for await (const user of db.findByIndex('age', { from: 25, limit: 10 })) {
console.log(user.name);
}
// Age 18+, only admins, max 5
const admins = await db.findByIndex('age', {
from: 18,
filter: u => u.role === 'admin',
limit: 5
});
```
**Parameters:**
- `field` (string): Indexed field name
- `options` (object):
- `from` (any): Start key (inclusive), uses binary search to skip ahead
- `limit` (number): Max records to return
- `filter` (Function): Filter applied per record during iteration
**Returns:** Cursor (async iterable)
### withLock(callback)
Execute a callback while holding the database lock. The lock is re-entrant: `find`, `insert`, `delete`, `update` called inside the callback reuse the same lock instead of deadlocking.
Use this for compound operations that must be atomic, e.g. find-then-insert.
```javascript
const user = await db.withLock(async () => {
const [existing] = await db.find(u => u.email === email);
if (existing) return existing;
return db.insert({ email, name });
});
```
**Parameters:**
- `callback` (Function): Async function to execute under lock
**Returns:** Promise<any> - The return value of the callback
### close()
Close the database and persist all pending changes. Should be called before process exit.
```javascript
await db.close();
```
## Primary Key Types
MPackDB supports three primary key types:
### Numeric (Auto-increment)
```javascript
const db = new MPackDB('data/users', {
primaryKey: '*id' // * prefix
});
await db.insert({ name: 'Alice' });
// { id: 0, name: 'Alice' }
```
### UUID
```javascript
const db = new MPackDB('data/sessions', {
primaryKey: '@sessionId' // @ prefix
});
await db.insert({ data: 'session data' });
// { sessionId: 'lz7gdcwh9x4r', data: 'session data' }
```
### String
```javascript
const db = new MPackDB('data/users', {
primaryKey: 'username' // No prefix
});
await db.insert({ username: 'alice', name: 'Alice' });
// { username: 'alice', name: 'Alice' }
```
## Indexes
Indexes dramatically improve query performance for large datasets.
### Index Types
- **Lexical** (default): String sorting, good for text fields
- **Numeric**: Number sorting, good for integers/floats
- **UUID**: Treated as lexical (string)
- **Unique**: Rejects duplicate values on insert (any type)
### Creating Indexes
```javascript
const db = new MPackDB('data/products', {
primaryKey: '*id',
indexes: [
'category', // Lexical index
'*price', // Numeric index
'*stock', // Numeric index
'@sku', // UUID index (lexical)
'!email' // Unique lexical index
]
});
```
### Unique Indexes
Unique indexes prevent duplicate values. Primary keys are always unique by default. Use the `!` prefix to make secondary indexes unique:
```javascript
const db = new MPackDB('data/users', {
primaryKey: '*id',
indexes: ['!email', '!username', '*age']
});
await db.insert({ email: '[email protected]', username: 'alice', age: 30 }); // ok
await db.insert({ email: '[email protected]', username: 'bob', age: 25 });
// throws: Error { code: 'DUPLICATE_KEY', field: 'email', value: '[email protected]' }
```
Combine `!` with type prefixes: `!*field` for unique numeric, `!@field` for unique UUID.
Uniqueness is enforced atomically under the write lock, so concurrent inserts cannot create duplicates.
### Index Persistence
Indexes are automatically persisted based on:
- **Threshold**: After N changes (default: 1000)
- **Interval**: Every N milliseconds (default: 60000)
- **On close**: When `db.close()` is called
```javascript
const db = new MPackDB('data/users', {
primaryKey: '*id',
indexes: ['email'],
indexPersistThreshold: 100, // Persist after 100 changes
indexPersistInterval: 30000 // Persist every 30 seconds
});
```
## File Structure
MPackDB creates the following files:
```
data/
users.mpack # Main data file (MessagePack binary)
users.meta.json # Metadata (nextId, deleted offsets)
users.id.txt # Primary key index
users.email.txt # Email field index
users.age.txt # Age field index
users.lock # Lock file (temporary)
```
## Concurrency
MPackDB uses file-based locking to ensure safe concurrent access:
```javascript
// Process 1
const db1 = new MPackDB('data/users', { primaryKey: '*id' });
await db1.insert({ name: 'Alice' });
// Process 2 (waits for lock)
const db2 = new MPackDB('data/users', { primaryKey: '*id' });
await db2.insert({ name: 'Bob' });
```
### Multi-process reads
MPackDB automatically re-reads `meta.json` from disk before every read operation (`refresh()`). This means a second process (e.g. an admin UI) opening the same database files will always see the latest state, including records deleted by the main application. Without this, the in-memory `_meta.deleted` array would go stale and deleted records would reappear in query results.
You can also call `db.refresh()` manually if needed.
**Important:** Secondary/read-only instances should disable compaction with `compact: false`. Compaction replaces the data file via `rename()`, which invalidates the write stream of any other instance pointing at the old file inode.
```javascript
// Main app — owns the data, compacts on startup (default)
const db = new MPackDB('data/users', { primaryKey: '*id' });
// Admin UI or secondary reader — no compaction
const admin = new MPackDB('data/users', { primaryKey: '*id', compact: false });
```
## Performance Tips
1. **Use indexes** for frequently queried fields
2. **Adjust persist thresholds** based on your write patterns
3. **Call `compact()`** periodically if you have many deletes
4. **Use numeric indexes** for number fields
5. **Batch operations** when possible
## Examples
### User Management System
```javascript
import MPackDB from '@worldapi.org/mpackdb';
const users = new MPackDB('data/users', {
primaryKey: '*id',
indexes: ['email', '*age', 'role']
});
// Register user
await users.insert({
email: '[email protected]',
name: 'Alice',
age: 30,
role: 'admin'
});
// Find by email
for await (const user of users.find(u => u.email === '[email protected]')) {
console.log('Found user:', user.name);
}
// Get all admins
for await (const admin of users.find(u => u.role === 'admin')) {
console.log('Admin:', admin.name);
}
// Update age
await users.update(u => u.email === '[email protected]', { age: 31 });
// Delete inactive users
await users.delete(u => u.lastLogin < Date.now() - 90 * 24 * 60 * 60 * 1000);
await users.close();
```
### Product Catalog
```javascript
const products = new MPackDB('data/products', {
primaryKey: '*id',
indexes: ['category', '*price', '@sku']
});
// Add products
await products.insert({ sku: 'ABC-123', name: 'Laptop', category: 'Electronics', price: 999 });
await products.insert({ sku: 'DEF-456', name: 'Mouse', category: 'Electronics', price: 29 });
// Find by category
for await (const product of products.find(p => p.category === 'Electronics')) {
console.log(product.name, product.price);
}
// Find products under $50
for await (const product of products.find(p => p.price < 50)) {
console.log('Affordable:', product.name);
}
// Update price
await products.update(p => p.sku === 'ABC-123', { price: 899 });
await products.close();
```
## License
MIT
## Contributing
Contributions are welcome! Please open an issue or submit a pull request.
## Related Projects
- [**BsonDB**](https://bsondb.gitoria.worldapi.org) - Similar database using BSON serialization

Branches

mastermain branch

Latest commits

d47876a1reimplemented lost features like indexed find and more testscaramboleyo2026-03-12 19:48
7f08da9afixed insert ignoring model definitioncaramboleyo2026-03-09 00:47
705774a9added flush before findcaramboleyo2026-03-08 22:01
b4db6391initial commitcaramboleyo2026-03-07 13:57