SweepStore/README.md

# SweepStore (formerly BinaryTable)

A high-performance binary storage format for Dart applications with efficient memory management and random access capabilities.

## Overview

BinaryTable is a key-value storage system designed for applications requiring fast, compact data persistence. It features a custom binary format with automatic memory management, making it ideal for game save files, caching systems, and other performance-critical data storage needs.

## Features

### Core Storage Types
- **Integers**: 32-bit signed integers
- **Floats**: 32-bit floating-point numbers
- **Strings**: Variable-length UTF-16 strings
- **Uniform Arrays**: Homogeneous arrays of integers or floats with O(1) random access

### Memory Management
- **Free List System**: Automatic memory reclamation and defragmentation
- **Block Merging**: Contiguous free blocks are automatically merged
- **File Truncation**: Unused space at end of file is reclaimed
- **Lift/Drop Mechanism**: Prevents free list fragmentation during operations

### Performance Features
- **Random Access**: Direct addressing for all data types
- **Minimal Seeks**: Optimized file layout reduces disk I/O
- **Compact Format**: No padding or alignment overhead
- **Efficient Arrays**: Uniform arrays use offset calculations for O(1) access

## Usage

### Basic Operations

```dart
// Initialize
File file = File('data.bin');
file.createSync();
BinaryTable table = BinaryTable(file.path);
table.initialise();

// Store values
table["player_level"] = 42;
table["player_name"] = "Hero";
table["score"] = 1337.5;

// Retrieve values
int level = table["player_level"];
String name = table["player_name"];
double score = table["score"];

// Delete entries
table.delete("old_data");
```

### Array Operations

```dart
// Create arrays
table["inventory"] = [10, 20, 30, 40, 50];
table["coordinates"] = [1.5, 2.7, 3.9];

// Random access
table["inventory"][0] = 15;
double y = table["coordinates"][1];

// Dynamic growth
table["inventory"].add(60);
table["inventory"].addAll([70, 80, 90]);

// Pre-sized arrays (recommended for game data)
table["heightmap"] = List.filled(1024 * 1024, 0.0);
```

### Memory Management

```dart
// Reclaim unused space
table.truncate();

// Manual cleanup (usually automatic)
table.delete("unused_key");
table.truncate();
```

## Version History

### Version 3.0 - Current
**Added: Uniform Arrays**
- `BT_UniformArray` class for homogeneous array storage
- O(1) random access for integer and float arrays
- Dynamic array growth with automatic reallocation
- Type safety enforcement for array elements
- Support for empty arrays with type inference on first use

**Improvements:**
- Enhanced memory management for variable-size data
- Optimized file format for array storage
- Better error handling for type mismatches

### Version 2.0
**Added: Persistent Free Lists**
- Free list data persists across program restarts
- Automatic block merging for defragmentation
- File truncation to reclaim unused space
- "Read from end" encoding for free list metadata

**Improvements:**
- Eliminated memory leaks from previous version
- Reduced file size growth over time
- Better space utilization efficiency

### Version 1.0
**Core Features:**
- Basic key-value storage (integers, floats, strings)
- Binary file format with type identification
- Address table for O(1) key lookups
- Volatile garbage collection (in-memory only)

**Initial Implementation:**
- RandomAccessFile-based storage
- Little-endian encoding
- Hash-based key addressing

## File Format

The BinaryTable uses a custom binary format optimized for random access:

### Header Structure
```
[Address Table Pointer: 8 bytes]
[Free List Entry Count: 4 bytes]
```

### Data Layout
- **Address Table**: Hash-to-pointer mappings at variable location
- **Free List**: Memory management metadata at end of file
- **Data Blocks**: Variable-size type-prefixed data scattered throughout

### Type Encoding
- Each value prefixed with type identifier byte
- Variable-length data includes size information
- Arrays store length followed by elements

## Performance Characteristics

### Time Complexity
- **Key Lookup**: O(1) average case
- **Array Access**: O(1) for uniform arrays
- **Insertion**: O(1) amortized
- **Deletion**: O(1) plus defragmentation cost

### Space Efficiency
- **Overhead**: ~13 bytes per key-value pair minimum
- **Arrays**: 5 bytes + (5 bytes × length) for uniform arrays
- **Fragmentation**: Automatically managed and minimized

## Use Cases

### Game Development
- Save file persistence with fast loading
- Level data storage (heightmaps, object placement)
- Player progression and inventory systems
- Settings and configuration data

### General Applications
- High-performance caching systems
- Embedded database applications
- Configuration file storage
- Any scenario requiring fast binary persistence

## Limitations

- **Hash Collisions**: Uses Dart's built-in string hashCode (risk of collisions)
- **File Corruption**: No built-in checksums or validation
- **Concurrent Access**: Single-threaded access only
- **Type System**: Limited compared to JSON (no nested objects, mixed arrays)

## Future Roadmap

- Full JSON parity with nested objects and mixed-type arrays
- Compression support for large data blocks
- Checksums and data integrity validation
- Multi-threaded access with file locking
- Query system for complex data retrieval