mpackdb

All repositories: gitoria

Readme Code Pull requests Releases Tickets Settings

Commitb8ffc1a0

b8ffc1a0release 1.0.5caramboleyo2026-03-12 19:51

b8ffc1a0/ARCHITECTURE.md

11.4 KB

# MPackDB Architecture
## Overview
MPackDB is a fast, local, append-only JSON database that uses MessagePack serialization. It's designed for Node.js/Bun applications that need a simple, file-based database with optional indexing capabilities and smaller file sizes compared to BSON.
## Core Components
### 1. MPackDB Class (`src/MPackDB.js`)
The main database class that handles all CRUD operations and coordinates between components.
**Key Responsibilities:**
- Database initialization and lifecycle management
- CRUD operations (insert, update, delete, find)
- File locking for concurrent access
- Metadata persistence
- Compaction of deleted records
**Key Properties:**
- `_dataPath`: Path to the `.mpack` data file
- `_dataStream`: Write stream for append-only operations
- `_meta`: Metadata object containing `nextId` and `deleted` offsets
- `_indexManager`: Optional IndexManager instance for indexed queries
- `_primaryKey`: Name of the primary key field
- `_primaryKeyType`: Type of primary key (NUMBER, UUID, STRING)
### 2. IndexManager Class (`src/IndexManager.js`)
Manages binary search indexes for fast lookups on indexed fields.
**Key Responsibilities:**
- Building and maintaining indexes from data files
- Binary search on disk-based index files
- Delta indexes (in-memory changes not yet persisted)
- Tombstone tracking for deleted records
- Auto-persistence of indexes
**Index File Format:**
```
key,offset,length
key,offset,length
...
```
Each line represents an index entry where:
- `key`: The indexed field value
- `offset`: Byte offset in the data file
- `length`: Length of the record in bytes
**Index Types:**
- **NUMERIC**: Numeric comparison for sorting/searching
- **LEXICAL**: String comparison for sorting/searching
- **UNIQUE**: Any index can be marked unique (`!` prefix) to reject duplicates on insert
### 3. Cursor Class (`src/Cursor.js`)
Provides an async iterable interface for query results.
**Key Responsibilities:**
- Lazy evaluation of queries
- Support for different iteration modes (record, offset, mixed, raw)
- Integration with IndexManager for indexed queries
- Filtering with query functions
### 4. MessagePack Utilities (`src/mpack.js`)
Wrapper around `msgpackr` with custom utilities.
**Key Exports:**
- `serialize()`: Encode JavaScript objects to MessagePack binary
- `deserialize()`: Decode MessagePack binary to JavaScript objects
- `uuid()`: Generate sortable 12-char base36 unique IDs (9 timestamp + 3 random)
- `PrimaryKeyType`: Enum for primary key types
- `IndexType`: Enum for index types
## Data Flow
### Insert Operation
```
1. User calls db.insert(record)
2. Acquire file lock
3. Auto-generate primary key if needed (numeric/UUID)
4. Check unique index constraints (skip auto-generated PKs)
5. Serialize record to MessagePack
6. Prepend 4-byte size header
7. Append to data file via write stream
8. Add entry to IndexManager (if indexes enabled)
9. Persist metadata (if nextId changed)
10. Release file lock
11. Return primary key or full record
```
### Find Operation (Indexed)
```
1. User calls db.find(primaryKeyValue)
2. Cursor created with query function
3. IndexManager performs binary search on index file
4. Check delta indexes for recent changes
5. Check tombstones for deleted records
6. Read record from data file at found offset
7. Yield record to user
```
### Find Operation (Non-Indexed)
```
1. User calls db.find(queryFn)
2. Cursor created with query function
3. Stream through entire data file
4. Read 4-byte size header
5. Read MessagePack data based on size
6. Deserialize each record
7. Apply query function filter
8. Skip deleted records (check metadata.deleted)
9. Yield matching records to user
```
### findByIndex Operation (Sorted Index Walk)
```
1. User calls db.findByIndex(field, { from, limit, filter })
2. Cursor created with _indexWalk option
3. Binary search on disk to find start position (from key)
4. Stream index entries block by block from disk
5. Merge with in-memory delta entries in sorted order
6. For each entry: read record from data file at offset
7. Skip deleted records, apply filter
8. Yield matching records, stop at limit
```
### Update Operation
```
1. User calls db.update(query, newData)
2. Find matching records (using find)
3. For each match:
a. Mark old record as deleted
b. Insert new record with same primary key
4. Persist metadata with new deleted offsets
```
### Delete Operation
```
1. User calls db.delete(query)
2. Find matching records
3. Add offsets to metadata.deleted array
4. Remove from indexes (if enabled)
5. Persist metadata
```
### Compact Operation
```
1. Acquire file lock
2. Create temporary data file
3. Stream through all records
4. Write only non-deleted records to temp file
5. Atomically rename temp file to replace original
6. Clear metadata.deleted array
7. Rebuild all indexes from new file
8. Release file lock
```
## File Structure
```
data/
├── users.mpack # Main data file (MessagePack records)
├── users.meta.json # Metadata (nextId, deleted offsets)
├── users.lock # Lock file (contains PID)
├── users.id.txt # Index file for 'id' field
├── users.email.txt # Index file for 'email' field
└── ...
```
## Serialization Format
MPackDB uses MessagePack format from the `msgpackr` npm package with a custom size header:
```
[4 bytes: size][MessagePack data]
```
Each record is prefixed with a 4-byte little-endian integer indicating the size of the MessagePack data (not including the size prefix itself).
**Why the size header?**
- MessagePack doesn't include record boundaries in the format
- The size header allows streaming reads without parsing the entire file
- Enables skipping deleted records efficiently
- Matches the pattern used in BSON for consistency
## Locking Mechanism
MPackDB uses file-based locking to prevent concurrent writes:
1. Before any write operation, create `{dbPath}.lock` file with `wx` flag (exclusive)
2. Write current process PID to lock file
3. If lock exists, wait 100ms and retry
4. After operation completes, delete lock file
The lock is **re-entrant**: if a lock is already held (e.g. inside `withLock()`), nested operations (`insert`, `delete`, etc.) increment a depth counter instead of acquiring a new file lock. The file lock is only released when the outermost holder finishes.
### withLock for Compound Operations
`db.withLock(callback)` acquires the lock for the duration of the callback. All DB operations inside the callback reuse the same lock. This makes compound operations like find-then-insert atomic:
```javascript
await db.withLock(async () => {
const [user] = await db.find(u => u.email === email);
if (!user) await db.insert({ email });
});
```
This ensures only one process can write at a time while allowing multiple readers.
## Index Persistence Strategy
Indexes use a two-tier approach:
### Disk Indexes
- Sorted index files on disk
- Binary searchable for O(log n) lookups
- Rebuilt during compaction
### Delta Indexes (In-Memory)
- Track changes since last persistence
- Checked before disk indexes
- Auto-persisted based on:
- Time interval (default: 60 seconds)
- Change threshold (default: 1000 operations)
### Tombstones
- Track deleted records in memory
- Prevent returning deleted records from disk indexes
- Cleared during compaction
## Performance Characteristics
### Time Complexity
- **Insert**: O(1) for append, O(log n) for index update
- **Find by primary key (indexed)**: O(log n) binary search
- **Find with query function**: O(n) full scan
- **Update**: O(log n) find + O(1) insert
- **Delete**: O(log n) find + O(1) mark
- **Compact**: O(n) full scan + O(n log n) index rebuild
### Space Complexity
- Data file grows with inserts (append-only)
- Deleted records remain until compaction
- Index files: O(n) per indexed field
- Delta indexes: O(m) where m = changes since last persist
### File Size Comparison
MessagePack typically produces **15-20% smaller files** than BSON for the same data:
- More compact integer encoding
- Smaller string overhead
- Efficient array/map encoding
## Concurrency Model
- **Single-writer, multiple-reader** via file locking
- Writes are serialized through lock file
- Reads can happen concurrently (no locks needed)
- Re-entrant locks allow `withLock()` to wrap compound operations atomically
- Unique indexes enforce constraints under the write lock (no duplicates even with concurrent inserts)
- `refresh()` re-reads `meta.json` before every read so secondary instances see up-to-date state
- Index persistence happens asynchronously but safely
## Primary Key Types
### NUMBER (PrimaryKeyType.NUMBER)
- Auto-incremented integer
- Stored in metadata.nextId
- Prefix syntax: `*id`
### UUID (PrimaryKeyType.UUID)
- Sortable base36 unique ID (9-char timestamp + 3-char random)
- 12-character string format
- Prefix syntax: `@id`
### STRING (PrimaryKeyType.STRING)
- User-provided string
- No auto-generation
- Default (no prefix)
## Design Decisions
### Why Append-Only?
- **Fast writes**: No seeking, just append
- **Crash safety**: Partial writes don't corrupt existing data
- **Simple implementation**: No complex update-in-place logic
### Why MessagePack?
- **Smaller files**: 15-20% smaller than BSON on average
- **Fast serialization**: Comparable or faster than BSON
- **Wide language support**: Available in many programming languages
- **Simple format**: Easy to implement and debug
- **No external binary dependencies**: Pure JavaScript implementation
### Why Custom Size Header?
- MessagePack doesn't define record boundaries
- Enables efficient streaming without full deserialization
- Allows skipping deleted records quickly
- Consistent with BSON's approach
### Why File-Based Locking?
- **Simple**: No external dependencies
- **Cross-process**: Works across multiple Node.js processes
- **Portable**: Works on all platforms
### Why Binary Search Indexes?
- **Disk-friendly**: Can search large indexes without loading into memory
- **Simple format**: Plain text, easy to debug
- **Fast lookups**: O(log n) for indexed queries
## MessagePack vs BSON
### Advantages of MessagePack
- **Smaller files**: 15-20% size reduction
- **Faster reads**: Simpler format, less parsing overhead
- **Pure JavaScript**: No native dependencies
- **Smaller library**: ~15KB vs ~173KB for BSON
### Advantages of BSON
- **ObjectId type**: Built-in unique identifier type
- **Date precision**: Millisecond timestamps
- **Binary data**: Native binary type
- **MongoDB compatibility**: Direct compatibility with MongoDB
### When to Choose MPackDB
- File size is a concern
- Pure JavaScript dependencies preferred
- Don't need MongoDB compatibility
- Want faster read performance
### When to Choose BsonDB
- Need ObjectId primary keys
- MongoDB compatibility desired
- Working with binary data
- Need precise date/time handling
## Limitations
1. **Single-writer**: Only one write operation at a time
2. **No multi-record transactions**: `withLock` provides atomicity for compound operations on a single DB, but not across multiple databases
3. **No query language**: Must use JavaScript functions for complex queries
4. **Compaction required**: Deleted records consume space until compaction
5. **Index overhead**: Each index doubles storage for that field
6. **No schema validation**: Records can have any structure
## Future Improvements
- Batch insert operations
- Async compaction (background process)
- Query optimizer for complex filters
- Compression support (MessagePack supports extensions)
- Replication/backup utilities
- Schema validation layer
- Custom MessagePack extension types

Branches

mastermain branch

Latest commits

b8ffc1a0release 1.0.5caramboleyo2026-03-12 19:51
d47876a1reimplemented lost features like indexed find and more testscaramboleyo2026-03-12 19:48
7f08da9afixed insert ignoring model definitioncaramboleyo2026-03-09 00:47
705774a9added flush before findcaramboleyo2026-03-08 22:01
b4db6391initial commitcaramboleyo2026-03-07 13:57