Compare commits
26 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
bec317745d | ||
|
|
4227b1db75 | ||
|
|
1084d62b9c | ||
|
|
4bace6f681 | ||
|
|
6e81375d7f | ||
|
|
55c69aebc2 | ||
|
|
fa50810212 | ||
|
|
82efc4a524 | ||
|
|
5d9b34e030 | ||
|
|
7adca647a6 | ||
|
|
ad6a740a73 | ||
|
|
55d5ab7a7b | ||
|
|
b3790e6b0a | ||
|
|
b6237a32d4 | ||
|
|
eae4d0e24e | ||
|
|
e6ccad87b4 | ||
|
|
f8e8636677 | ||
|
|
580f07c483 | ||
|
|
809d79cfc8 | ||
|
|
f5a23fa5c4 | ||
|
|
8125f8ebd7 | ||
|
|
747f0bd1ed | ||
|
|
2de17fe720 | ||
|
|
6e226f402b | ||
|
|
4295d119d7 | ||
| 9216cd1638 |
15
.gitignore
vendored
Normal file
15
.gitignore
vendored
Normal file
@@ -0,0 +1,15 @@
|
||||
# https://dart.dev/guides/libraries/private-files
|
||||
# Created by `dart pub`
|
||||
.dart_tool/
|
||||
|
||||
# Hide all hidden files
|
||||
.*
|
||||
!.gitignore
|
||||
|
||||
example.bin
|
||||
|
||||
# Build directories
|
||||
build/
|
||||
|
||||
# Temp collapsed files
|
||||
*.collapsed
|
||||
2
CLAUDE.md
Normal file
2
CLAUDE.md
Normal file
@@ -0,0 +1,2 @@
|
||||
- Remember that when porting the dart library to other languages, we do not care about improvements, we expect the logic to be near 1:1 reflection of the dart repository for compatibility and interoperability. Do not make any logical or programatic changes.
|
||||
- When told to update the library in any language other than dart. Read the dart library and compare it to the existing given language port and update the port to match parity.
|
||||
10
CONTRIBUTING.md
Normal file
10
CONTRIBUTING.md
Normal file
@@ -0,0 +1,10 @@
|
||||
## Contributing to SweepStore
|
||||
|
||||
By contributing code, documentation, or other materials to SweepStore, you agree that:
|
||||
|
||||
1. Benjamin Watt/IMBENJI.NET LIMITED retains perpetual rights to use your contributions under the Business Source License 1.1
|
||||
2. You retain copyright to your contributions
|
||||
3. Your contributions will be available under BSL 1.1
|
||||
4. Your contributions may be sublicensed commercially
|
||||
|
||||
This allows SweepStore to maintain its licensing model.
|
||||
74
LICENSE
74
LICENSE
@@ -8,29 +8,65 @@
|
||||
/$$$$$$| $$ \/ | $$| $$$$$$$/| $$$$$$$$| $$ \ $$| $$$$$$/ /$$$$$$ /$$| $$ \ $$| $$$$$$$$ | $$
|
||||
|______/|__/ |__/|_______/ |________/|__/ \__/ \______/ |______/|__/|__/ \__/|________/ |__/
|
||||
|
||||
© 2025-26 by Benjamin Watt of IMBENJI.NET LIMITED - All rights reserved.
|
||||
Business Source License 1.1
|
||||
|
||||
Use of this source code is governed by a MIT license that can be found in the LICENSE file.
|
||||
License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
|
||||
"Business Source License" is a trademark of MariaDB Corporation Ab.
|
||||
|
||||
MIT License
|
||||
-----------------------------------------------------------------------------
|
||||
|
||||
Copyright (c) 2025 Benjamin Watt of IMBENJI.NET LIMITED
|
||||
Parameters
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
Licensor: Benjamin Watt / IMBENJI.NET LIMITED
|
||||
Licensed Work: SweepStore
|
||||
The Licensed Work is (c) 2025-26 Benjamin Watt
|
||||
Additional Use Grant: You may use the Licensed Work for any purpose other
|
||||
than production use in a commercial product or service
|
||||
that generates more than $100,000 USD in annual gross
|
||||
revenue.
|
||||
Change Date: Five years from the date of release of each version
|
||||
Change License: Apache License, Version 2.0
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
-----------------------------------------------------------------------------
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
Terms
|
||||
|
||||
The Licensor hereby grants you the right to copy, modify, create derivative
|
||||
works, redistribute, and make non-production use of the Licensed Work. The
|
||||
Licensor may make an Additional Use Grant, above, permitting limited production
|
||||
use.
|
||||
|
||||
Effective on the Change Date, or the fourth anniversary of the first publicly
|
||||
available distribution of a specific version of the Licensed Work under this
|
||||
License, whichever comes first, the Licensor hereby grants you rights under
|
||||
the terms of the Change License, and the rights granted in the paragraph above
|
||||
terminate.
|
||||
|
||||
If your use of the Licensed Work does not comply with the requirements
|
||||
currently in effect as described in this License, you must purchase a
|
||||
commercial license from the Licensor, its affiliated entities, or authorized
|
||||
resellers, or you must refrain from using the Licensed Work.
|
||||
|
||||
All copies of the original and modified Licensed Work, and derivative works
|
||||
of the Licensed Work, are subject to this License. This License applies
|
||||
separately for each version of the Licensed Work and the Change Date may vary
|
||||
for each version of the Licensed Work released by Licensor.
|
||||
|
||||
You must conspicuously display this License on each original or modified copy
|
||||
of the Licensed Work. If you receive the Licensed Work in original or modified
|
||||
form from a third party, the terms and conditions set forth in this License
|
||||
apply to your use of that work.
|
||||
|
||||
Any use of the Licensed Work in violation of this License will automatically
|
||||
terminate your rights under this License for the current and all other
|
||||
versions of the Licensed Work.
|
||||
|
||||
This License does not grant you any right in any trademark or logo of
|
||||
Licensor or its affiliates (provided that you may use a trademark or logo of
|
||||
Licensor as expressly required by this License).
|
||||
|
||||
TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
|
||||
AN "AS IS" BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
|
||||
EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
|
||||
TITLE.
|
||||
|
||||
62
README.md
62
README.md
@@ -174,10 +174,60 @@ The BinaryTable uses a custom binary format optimized for random access:
|
||||
- **Concurrent Access**: Single-threaded access only
|
||||
- **Type System**: Limited compared to JSON (no nested objects, mixed arrays)
|
||||
|
||||
## Future Roadmap
|
||||
# SweepStore Roadmap
|
||||
|
||||
- Full JSON parity with nested objects and mixed-type arrays
|
||||
- Compression support for large data blocks
|
||||
- Checksums and data integrity validation
|
||||
- Multi-threaded access with file locking
|
||||
- Query system for complex data retrieval
|
||||
## v1.0.0 (Current)
|
||||
- ✅ Key-value storage with hash-based indexing
|
||||
- ✅ Uniform arrays with random access
|
||||
- ✅ Automatic memory management via free list
|
||||
- ✅ Single file format (.sws)
|
||||
|
||||
## v2.0.0 (Planned)
|
||||
|
||||
### Core Changes
|
||||
- **Nested objects** - Store objects within objects for hierarchical data
|
||||
- **Key enumeration** - Query and list all keys without knowing them upfront
|
||||
- **Object-based architecture** - Each object has its own address table and key list
|
||||
|
||||
### File Format Changes
|
||||
```
|
||||
v1: Global address table → All keys at root level
|
||||
v2: Root object → Each object manages its own keys
|
||||
```
|
||||
|
||||
### New API
|
||||
```dart
|
||||
// Nested objects
|
||||
table["player"] = {};
|
||||
table["player"]["name"] = "Alice";
|
||||
table["player"]["inventory"] = {};
|
||||
|
||||
// Key discovery
|
||||
List<String> keys = table.keys();
|
||||
bool exists = table.containsKey("player");
|
||||
|
||||
// Object operations
|
||||
BT_Object player = table["player"];
|
||||
for (String key in player.keys()) {
|
||||
print("$key: ${player[key]}");
|
||||
}
|
||||
```
|
||||
|
||||
### Implementation Strategy
|
||||
- Introduce `BT_Object` type with embedded address table + key list
|
||||
- `BinaryTable` becomes wrapper around root object
|
||||
- Maintain global free list (not per-object)
|
||||
- Address table entries remain absolute pointers
|
||||
- Key list stored alongside each object's address table
|
||||
|
||||
### Breaking Changes
|
||||
- File format incompatible with v1.0.0
|
||||
- API remains largely the same (only additions)
|
||||
- Migration tool needed for v1 → v2 files
|
||||
|
||||
## Future Considerations
|
||||
- Hash collision resolution
|
||||
- Checksums for data integrity
|
||||
- Optimized allocation strategies
|
||||
- Incremental array resizing
|
||||
- Compression options
|
||||
@@ -3,44 +3,43 @@ project(BinaryTable)
|
||||
|
||||
set(CMAKE_CXX_STANDARD 17)
|
||||
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
||||
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
|
||||
|
||||
# Core Binary Table Library
|
||||
add_library(binary_table
|
||||
binary_table.h
|
||||
binary_table.cpp
|
||||
# Add include directories globally
|
||||
include_directories(${CMAKE_SOURCE_DIR}/src/Public)
|
||||
include_directories(${CMAKE_SOURCE_DIR}/src/Private)
|
||||
|
||||
# Main executable with integrated binary table implementation
|
||||
add_executable(main
|
||||
src/Private/sweepstore/header.cpp
|
||||
src/Public/sweepstore/utils/helpers.h
|
||||
src/Private/sweepstore/structures.cpp
|
||||
src/Public/sweepstore/structures.h
|
||||
src/Private/sweepstore/sweepstore.cpp
|
||||
src/Public/sweepstore/sweepstore.h
|
||||
src/Public/sweepstore/concurrency.h
|
||||
src/Private/sweepstore/concurrency.cpp
|
||||
src/Public/sweepstore/utils/file_lock.h
|
||||
src/Private/sweepstore/utils/file_lock.cpp
|
||||
src/Private/sweepstore/utils/fd_pool.cpp
|
||||
src/Public/sweepstore/utils/file_handle.h
|
||||
src/Private/sweepstore/utils/file_handle.cpp
|
||||
src/Public/sweepstore/header.h
|
||||
src/Private/sweepstore/benchmark.cpp
|
||||
)
|
||||
|
||||
# Main Application
|
||||
add_executable(main main.cpp)
|
||||
target_link_libraries(main binary_table)
|
||||
|
||||
# Debug Test Executables
|
||||
add_executable(debug_multi_key debug/debug_multi_key.cpp)
|
||||
target_link_libraries(debug_multi_key binary_table)
|
||||
|
||||
add_executable(debug_alloc debug/debug_alloc.cpp)
|
||||
target_link_libraries(debug_alloc binary_table)
|
||||
|
||||
add_executable(debug_address_table debug/debug_address_table.cpp)
|
||||
target_link_libraries(debug_address_table binary_table)
|
||||
|
||||
add_executable(debug_step_by_step debug/debug_step_by_step.cpp)
|
||||
target_link_libraries(debug_step_by_step binary_table)
|
||||
|
||||
add_executable(debug_simple debug/debug_simple.cpp)
|
||||
target_link_libraries(debug_simple binary_table)
|
||||
# Add include directories
|
||||
target_include_directories(main PRIVATE ${CMAKE_SOURCE_DIR}/src/Public)
|
||||
|
||||
# Compiler Settings
|
||||
if(MSVC)
|
||||
target_compile_options(binary_table PRIVATE /W4)
|
||||
target_compile_options(main PRIVATE /W4)
|
||||
else()
|
||||
target_compile_options(binary_table PRIVATE -Wall -Wextra -pedantic)
|
||||
target_compile_options(main PRIVATE -Wall -Wextra -pedantic)
|
||||
# Apply warnings to debug executables too
|
||||
target_compile_options(debug_multi_key PRIVATE -Wall -Wextra -pedantic)
|
||||
target_compile_options(debug_alloc PRIVATE -Wall -Wextra -pedantic)
|
||||
target_compile_options(debug_address_table PRIVATE -Wall -Wextra -pedantic)
|
||||
target_compile_options(debug_step_by_step PRIVATE -Wall -Wextra -pedantic)
|
||||
target_compile_options(debug_simple PRIVATE -Wall -Wextra -pedantic)
|
||||
endif()
|
||||
|
||||
# Link required libraries
|
||||
if(UNIX AND NOT APPLE)
|
||||
# Only link stdc++fs on Linux, not macOS
|
||||
target_link_libraries(main PRIVATE stdc++fs)
|
||||
endif()
|
||||
|
||||
1109
cpp/binary_table.cpp
1109
cpp/binary_table.cpp
File diff suppressed because it is too large
Load Diff
@@ -1,305 +0,0 @@
|
||||
/*
|
||||
|
||||
/$$$$$$ /$$ /$$ /$$$$$$$ /$$$$$$$$ /$$ /$$ /$$$$$ /$$$$$$ /$$ /$$ /$$$$$$$$ /$$$$$$$$
|
||||
|_ $$_/| $$$ /$$$| $$__ $$| $$_____/| $$$ | $$ |__ $$|_ $$_/ | $$$ | $$| $$_____/|__ $$__/
|
||||
| $$ | $$$$ /$$$$| $$ \ $$| $$ | $$$$| $$ | $$ | $$ | $$$$| $$| $$ | $$
|
||||
| $$ | $$ $$/$$ $$| $$$$$$$ | $$$$$ | $$ $$ $$ | $$ | $$ | $$ $$ $$| $$$$$ | $$
|
||||
| $$ | $$ $$$| $$| $$__ $$| $$__/ | $$ $$$$ /$$ | $$ | $$ | $$ $$$$| $$__/ | $$
|
||||
| $$ | $$\ $ | $$| $$ \ $$| $$ | $$\ $$$| $$ | $$ | $$ | $$\ $$$| $$ | $$
|
||||
/$$$$$$| $$ \/ | $$| $$$$$$$/| $$$$$$$$| $$ \ $$| $$$$$$/ /$$$$$$ /$$| $$ \ $$| $$$$$$$$ | $$
|
||||
|______/|__/ |__/|_______/ |________/|__/ \__/ \______/ |______/|__/|__/ \__/|________/ |__/
|
||||
|
||||
<EFBFBD> 2025-26 by Benjamin Watt of IMBENJI.NET LIMITED - All rights reserved.
|
||||
|
||||
Use of this source code is governed by a MIT license that can be found in the LICENSE file.
|
||||
|
||||
This file is part of the SweepStore (formerly Binary Table) package for C++.
|
||||
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <unordered_map>
|
||||
#include <fstream>
|
||||
#include <variant>
|
||||
#include <memory>
|
||||
#include <stdexcept>
|
||||
#include <type_traits>
|
||||
#include <functional>
|
||||
#include <iostream>
|
||||
|
||||
// Debug control - comment out this line to disable all debug output
|
||||
// #define ENABLE_DEBUG 1
|
||||
|
||||
#ifdef ENABLE_DEBUG
|
||||
#define DEBUG_PRINT(x) std::cout << x
|
||||
#define DEBUG_PRINTLN(x) std::cout << x << std::endl
|
||||
#else
|
||||
#define DEBUG_PRINT(x)
|
||||
#define DEBUG_PRINTLN(x)
|
||||
#endif
|
||||
|
||||
namespace bt {
|
||||
|
||||
// Forward declarations
|
||||
class BinaryTable;
|
||||
class BT_Reference;
|
||||
template<typename T> class BT_UniformArray;
|
||||
|
||||
// Type enumeration matching Dart version
|
||||
enum class BT_Type : uint8_t {
|
||||
POINTER = 0,
|
||||
ADDRESS_TABLE = 1,
|
||||
INTEGER = 2,
|
||||
FLOAT = 3,
|
||||
STRING = 4,
|
||||
INTEGER_ARRAY = 5,
|
||||
FLOAT_ARRAY = 6
|
||||
};
|
||||
|
||||
// Size mapping for types
|
||||
constexpr int getTypeSize(BT_Type type) {
|
||||
switch (type) {
|
||||
case BT_Type::POINTER: return 8;
|
||||
case BT_Type::ADDRESS_TABLE: return -1;
|
||||
case BT_Type::INTEGER: return 4;
|
||||
case BT_Type::FLOAT: return 4;
|
||||
case BT_Type::STRING: return -1;
|
||||
case BT_Type::INTEGER_ARRAY: return -1;
|
||||
case BT_Type::FLOAT_ARRAY: return -1;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
// Check if type is array type
|
||||
constexpr bool isArrayType(BT_Type type) {
|
||||
return type == BT_Type::INTEGER_ARRAY || type == BT_Type::FLOAT_ARRAY;
|
||||
}
|
||||
|
||||
// Type deduction helpers
|
||||
template<typename T>
|
||||
constexpr BT_Type getTypeFromValue() {
|
||||
if constexpr (std::is_same_v<T, int32_t> || std::is_same_v<T, int>) {
|
||||
return BT_Type::INTEGER;
|
||||
} else if constexpr (std::is_same_v<T, float>) {
|
||||
return BT_Type::FLOAT;
|
||||
} else if constexpr (std::is_same_v<T, std::string>) {
|
||||
return BT_Type::STRING;
|
||||
} else if constexpr (std::is_same_v<T, std::vector<int32_t>> || std::is_same_v<T, std::vector<int>>) {
|
||||
return BT_Type::INTEGER_ARRAY;
|
||||
} else if constexpr (std::is_same_v<T, std::vector<float>>) {
|
||||
return BT_Type::FLOAT_ARRAY;
|
||||
} else {
|
||||
static_assert(sizeof(T) == 0, "Unsupported type");
|
||||
}
|
||||
}
|
||||
|
||||
// Pointer class
|
||||
class BT_Pointer {
|
||||
private:
|
||||
int64_t address_;
|
||||
|
||||
public:
|
||||
explicit BT_Pointer(int64_t address = -1) : address_(address) {}
|
||||
|
||||
bool isNull() const { return address_ == -1; }
|
||||
int64_t address() const { return address_; }
|
||||
|
||||
bool operator==(const BT_Pointer& other) const {
|
||||
return address_ == other.address_;
|
||||
}
|
||||
|
||||
bool operator!=(const BT_Pointer& other) const {
|
||||
return !(*this == other);
|
||||
}
|
||||
};
|
||||
|
||||
// Null pointer constant
|
||||
const BT_Pointer BT_Null{-1};
|
||||
|
||||
// Free list entry
|
||||
struct BT_FreeListEntry {
|
||||
BT_Pointer pointer;
|
||||
int32_t size;
|
||||
|
||||
BT_FreeListEntry(BT_Pointer ptr, int32_t sz) : pointer(ptr), size(sz) {}
|
||||
};
|
||||
|
||||
// Value encoding functions
|
||||
std::vector<uint8_t> encodeValue(const int32_t& value);
|
||||
std::vector<uint8_t> encodeValue(const float& value);
|
||||
std::vector<uint8_t> encodeValue(const std::string& value);
|
||||
std::vector<uint8_t> encodeValue(const std::vector<int32_t>& value);
|
||||
std::vector<uint8_t> encodeValue(const std::vector<float>& value);
|
||||
|
||||
// Template wrapper for encoding
|
||||
template<typename T>
|
||||
std::vector<uint8_t> encodeValue(const T& value) {
|
||||
return encodeValue(value);
|
||||
}
|
||||
|
||||
// Reference class for handling stored values
|
||||
class BT_Reference {
|
||||
protected:
|
||||
BinaryTable* table_;
|
||||
BT_Pointer pointer_;
|
||||
|
||||
public:
|
||||
BT_Reference(BinaryTable* table, BT_Pointer pointer);
|
||||
|
||||
template<typename T>
|
||||
T decodeValue();
|
||||
|
||||
int32_t size() const;
|
||||
BT_Type getType() const;
|
||||
bool isNull() const { return pointer_.isNull(); }
|
||||
BT_Pointer getPointer() const { return pointer_; }
|
||||
};
|
||||
|
||||
// Uniform array class template
|
||||
template<typename T>
|
||||
class BT_UniformArray : public BT_Reference {
|
||||
public:
|
||||
BT_UniformArray(BinaryTable* table, BT_Pointer pointer) : BT_Reference(table, pointer) {}
|
||||
|
||||
int32_t length() const;
|
||||
T operator[](int32_t index) const;
|
||||
void set(int32_t index, const T& value);
|
||||
void add(const T& value);
|
||||
void addAll(const std::vector<T>& values);
|
||||
std::vector<T> fetchSublist(int32_t start = 0, int32_t end = -1);
|
||||
};
|
||||
|
||||
// Main BinaryTable class
|
||||
class BinaryTable {
|
||||
private:
|
||||
FILE* file_;
|
||||
std::string filePath_;
|
||||
|
||||
// Free list management
|
||||
bool freeListLifted_;
|
||||
std::vector<BT_FreeListEntry> freeListCache_;
|
||||
|
||||
|
||||
// Internal methods
|
||||
std::unordered_map<int64_t, BT_Pointer> getAddressTable();
|
||||
void setAddressTable(const std::unordered_map<int64_t, BT_Pointer>& table);
|
||||
std::vector<BT_FreeListEntry> getFreeList();
|
||||
void setFreeList(const std::vector<BT_FreeListEntry>& list);
|
||||
int64_t hashString(const std::string& str) const;
|
||||
|
||||
void truncateFile(int64_t newSize);
|
||||
|
||||
// File I/O helpers
|
||||
int32_t readInt32(int64_t position);
|
||||
float readFloat32(int64_t position);
|
||||
int64_t readInt64(int64_t position);
|
||||
uint8_t readByte(int64_t position);
|
||||
std::vector<uint8_t> readBytes(int64_t position, int32_t count);
|
||||
|
||||
void writeInt32(int64_t position, int32_t value);
|
||||
void writeFloat32(int64_t position, float value);
|
||||
void writeInt64(int64_t position, int64_t value);
|
||||
void writeByte(int64_t position, uint8_t value);
|
||||
void writeBytes(int64_t position, const std::vector<uint8_t>& data);
|
||||
|
||||
public:
|
||||
explicit BinaryTable(const std::string& path);
|
||||
~BinaryTable();
|
||||
|
||||
void initialize();
|
||||
|
||||
// Memory management
|
||||
void liftFreeList();
|
||||
void dropFreeList();
|
||||
void antiFreeListScope(std::function<void()> fn);
|
||||
void free(BT_Pointer pointer, int32_t size);
|
||||
BT_Pointer alloc(int32_t size);
|
||||
|
||||
// Data operations
|
||||
template<typename T>
|
||||
void set(const std::string& key, const T& value);
|
||||
|
||||
template<typename T>
|
||||
T get(const std::string& key);
|
||||
|
||||
BT_Reference getReference(const std::string& key);
|
||||
|
||||
template<typename T>
|
||||
BT_UniformArray<T> getArray(const std::string& key);
|
||||
|
||||
void remove(const std::string& key);
|
||||
void truncate();
|
||||
|
||||
// Debug methods
|
||||
void debugAddressTable(const std::string& context = "");
|
||||
|
||||
// File access for reference classes
|
||||
friend class BT_Reference;
|
||||
template<typename T> friend class BT_UniformArray;
|
||||
|
||||
int64_t getFileLength();
|
||||
void setFilePosition(int64_t position);
|
||||
};
|
||||
|
||||
// Template specializations for decodeValue
|
||||
template<> int32_t BT_Reference::decodeValue<int32_t>();
|
||||
template<> float BT_Reference::decodeValue<float>();
|
||||
template<> std::string BT_Reference::decodeValue<std::string>();
|
||||
template<> std::vector<int32_t> BT_Reference::decodeValue<std::vector<int32_t>>();
|
||||
template<> std::vector<float> BT_Reference::decodeValue<std::vector<float>>();
|
||||
template<> BT_UniformArray<int32_t> BT_Reference::decodeValue<BT_UniformArray<int32_t>>();
|
||||
template<> BT_UniformArray<float> BT_Reference::decodeValue<BT_UniformArray<float>>();
|
||||
|
||||
// Template method implementations for BinaryTable
|
||||
template<typename T>
|
||||
void BinaryTable::set(const std::string& key, const T& value) {
|
||||
antiFreeListScope([&]() {
|
||||
auto addressTable = getAddressTable();
|
||||
int64_t keyHash = hashString(key);
|
||||
|
||||
if (addressTable.find(keyHash) != addressTable.end()) {
|
||||
throw std::runtime_error("Key already exists");
|
||||
}
|
||||
|
||||
auto valueBuffer = encodeValue(value);
|
||||
BT_Pointer valueAddress = alloc(static_cast<int32_t>(valueBuffer.size()));
|
||||
|
||||
writeBytes(valueAddress.address(), valueBuffer);
|
||||
|
||||
addressTable[keyHash] = valueAddress;
|
||||
setAddressTable(addressTable);
|
||||
});
|
||||
}
|
||||
|
||||
template<typename T>
|
||||
T BinaryTable::get(const std::string& key) {
|
||||
auto addressTable = getAddressTable();
|
||||
int64_t keyHash = hashString(key);
|
||||
|
||||
auto it = addressTable.find(keyHash);
|
||||
if (it == addressTable.end()) {
|
||||
throw std::runtime_error("Key does not exist");
|
||||
}
|
||||
|
||||
BT_Reference valueRef(this, it->second);
|
||||
return valueRef.decodeValue<T>();
|
||||
}
|
||||
|
||||
template<typename T>
|
||||
BT_UniformArray<T> BinaryTable::getArray(const std::string& key) {
|
||||
auto addressTable = getAddressTable();
|
||||
int64_t keyHash = hashString(key);
|
||||
|
||||
auto it = addressTable.find(keyHash);
|
||||
if (it == addressTable.end()) {
|
||||
throw std::runtime_error("Key does not exist");
|
||||
}
|
||||
|
||||
return BT_UniformArray<T>(this, it->second);
|
||||
}
|
||||
|
||||
} // namespace bt
|
||||
@@ -1,50 +0,0 @@
|
||||
#include <iostream>
|
||||
#include <filesystem>
|
||||
#include "../binary_table.h"
|
||||
|
||||
void printAddressTable(bt::BinaryTable& table) {
|
||||
// We can't access getAddressTable directly, so let's use a different approach
|
||||
// Try to retrieve all known keys and see what happens
|
||||
std::vector<std::string> keys = {"key1", "key2", "key3"};
|
||||
|
||||
for (const std::string& key : keys) {
|
||||
try {
|
||||
auto ref = table.getReference(key);
|
||||
std::cout << " " << key << " -> address " << ref.getPointer().address()
|
||||
<< " (type " << static_cast<int>(ref.getType()) << ")" << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " " << key << " -> ERROR: " << e.what() << std::endl;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
int main() {
|
||||
using namespace bt;
|
||||
|
||||
const std::string filename = "debug_addr_table.bin";
|
||||
if (std::filesystem::exists(filename)) {
|
||||
std::filesystem::remove(filename);
|
||||
}
|
||||
|
||||
BinaryTable table(filename);
|
||||
table.initialize();
|
||||
|
||||
std::cout << "=== Testing Address Table Corruption ===\n" << std::endl;
|
||||
|
||||
std::cout << "Initial state (empty):" << std::endl;
|
||||
printAddressTable(table);
|
||||
|
||||
std::cout << "\n1. After storing key1:" << std::endl;
|
||||
table.set<int32_t>("key1", 100);
|
||||
printAddressTable(table);
|
||||
|
||||
std::cout << "\n2. After storing key2:" << std::endl;
|
||||
table.set<int32_t>("key2", 200);
|
||||
printAddressTable(table);
|
||||
|
||||
std::cout << "\n3. After storing key3:" << std::endl;
|
||||
table.set<int32_t>("key3", 300);
|
||||
printAddressTable(table);
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -1,61 +0,0 @@
|
||||
#include <iostream>
|
||||
#include <filesystem>
|
||||
#include "../binary_table.h"
|
||||
|
||||
int main() {
|
||||
using namespace bt;
|
||||
|
||||
const std::string filename = "debug_alloc.bin";
|
||||
if (std::filesystem::exists(filename)) {
|
||||
std::filesystem::remove(filename);
|
||||
}
|
||||
|
||||
BinaryTable table(filename);
|
||||
table.initialize();
|
||||
|
||||
std::cout << "=== Testing Memory Allocation Issues ===\n" << std::endl;
|
||||
|
||||
// Store first key and see what address it gets
|
||||
std::cout << "1. Storing first key..." << std::endl;
|
||||
table.set<int32_t>("key1", 100);
|
||||
|
||||
// Get the address where key1's value was stored
|
||||
auto addressTable1 = table.getReference("key1").getPointer();
|
||||
std::cout << " key1 value stored at: " << addressTable1.address() << std::endl;
|
||||
|
||||
// Store second key and see what addresses are used
|
||||
std::cout << "2. Storing second key..." << std::endl;
|
||||
table.set<int32_t>("key2", 200);
|
||||
|
||||
auto addressTable2 = table.getReference("key2").getPointer();
|
||||
std::cout << " key2 value stored at: " << addressTable2.address() << std::endl;
|
||||
|
||||
// Check if key1 is still accessible
|
||||
std::cout << "3. Checking if key1 is still accessible..." << std::endl;
|
||||
try {
|
||||
int32_t val1 = table.get<int32_t>("key1");
|
||||
std::cout << " ✅ key1 still works: " << val1 << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " ❌ key1 broken: " << e.what() << std::endl;
|
||||
|
||||
// Let's see what's actually stored at key1's address
|
||||
try {
|
||||
auto ref = table.getReference("key1");
|
||||
std::cout << " key1 type is: " << static_cast<int>(ref.getType()) << std::endl;
|
||||
} catch (const std::exception& e2) {
|
||||
std::cout << " Can't even get type: " << e2.what() << std::endl;
|
||||
}
|
||||
}
|
||||
|
||||
std::cout << "\n=== Address Comparison ===\n" << std::endl;
|
||||
std::cout << "key1 address: " << addressTable1.address() << std::endl;
|
||||
std::cout << "key2 address: " << addressTable2.address() << std::endl;
|
||||
|
||||
if (addressTable1.address() == addressTable2.address()) {
|
||||
std::cout << "💥 SAME ADDRESS! This proves the bug!" << std::endl;
|
||||
} else {
|
||||
std::cout << "Addresses are different, issue is elsewhere" << std::endl;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -1,69 +0,0 @@
|
||||
#include <iostream>
|
||||
#include <filesystem>
|
||||
#include "../binary_table.h"
|
||||
|
||||
int main() {
|
||||
using namespace bt;
|
||||
|
||||
const std::string filename = "debug_multi.bin";
|
||||
if (std::filesystem::exists(filename)) {
|
||||
std::filesystem::remove(filename);
|
||||
}
|
||||
|
||||
BinaryTable table(filename);
|
||||
table.initialize();
|
||||
|
||||
std::cout << "=== Testing Multi-Key Storage ===" << std::endl;
|
||||
|
||||
// Store first key
|
||||
std::cout << "1. Storing first key..." << std::endl;
|
||||
table.set<int32_t>("key1", 100);
|
||||
|
||||
// Try to read it back
|
||||
try {
|
||||
int32_t val1 = table.get<int32_t>("key1");
|
||||
std::cout << " ✅ First key retrieved: " << val1 << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " ❌ First key failed: " << e.what() << std::endl;
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Store second key - this is where it likely breaks
|
||||
std::cout << "2. Storing second key..." << std::endl;
|
||||
table.set<int32_t>("key2", 200);
|
||||
|
||||
// Try to read second key
|
||||
try {
|
||||
int32_t val2 = table.get<int32_t>("key2");
|
||||
std::cout << " ✅ Second key retrieved: " << val2 << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " ❌ Second key failed: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
// Try to read first key again - this will likely fail
|
||||
std::cout << "3. Re-reading first key..." << std::endl;
|
||||
try {
|
||||
int32_t val1_again = table.get<int32_t>("key1");
|
||||
std::cout << " ✅ First key still accessible: " << val1_again << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " ❌ First key now broken: " << e.what() << std::endl;
|
||||
std::cout << " 💥 CONFIRMED: Table breaks after storing 2+ keys!" << std::endl;
|
||||
}
|
||||
|
||||
// Store third key to see if pattern continues
|
||||
std::cout << "4. Storing third key..." << std::endl;
|
||||
try {
|
||||
table.set<int32_t>("key3", 300);
|
||||
int32_t val3 = table.get<int32_t>("key3");
|
||||
std::cout << " ✅ Third key works: " << val3 << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " ❌ Third key failed: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
std::cout << "\n=== Conclusion ===" << std::endl;
|
||||
std::cout << "The issue is definitely in the address table management" << std::endl;
|
||||
std::cout << "when storing multiple keys. Single key = perfect," << std::endl;
|
||||
std::cout << "multiple keys = corruption." << std::endl;
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
#include <iostream>
|
||||
#include <filesystem>
|
||||
#include "../binary_table.h"
|
||||
|
||||
int main() {
|
||||
using namespace bt;
|
||||
|
||||
const std::string filename = "debug_simple.bin";
|
||||
if (std::filesystem::exists(filename)) {
|
||||
std::filesystem::remove(filename);
|
||||
}
|
||||
|
||||
BinaryTable table(filename);
|
||||
table.initialize();
|
||||
|
||||
std::cout << "1. Storing key1..." << std::endl;
|
||||
table.set<int32_t>("key1", 100);
|
||||
table.debugAddressTable("after key1");
|
||||
|
||||
std::cout << "2. Reading key1..." << std::endl;
|
||||
try {
|
||||
int32_t val = table.get<int32_t>("key1");
|
||||
std::cout << " ✅ key1 = " << val << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " ❌ key1 failed: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
std::cout << "3. Storing key2..." << std::endl;
|
||||
table.set<int32_t>("key2", 200);
|
||||
table.debugAddressTable("after key2");
|
||||
|
||||
std::cout << "4. Reading key2..." << std::endl;
|
||||
try {
|
||||
int32_t val = table.get<int32_t>("key2");
|
||||
std::cout << " ✅ key2 = " << val << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " ❌ key2 failed: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
std::cout << "5. Re-reading key1..." << std::endl;
|
||||
try {
|
||||
int32_t val = table.get<int32_t>("key1");
|
||||
std::cout << " ✅ key1 = " << val << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << " ❌ key1 failed: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -1,105 +0,0 @@
|
||||
#include <iostream>
|
||||
#include <filesystem>
|
||||
#include <fstream>
|
||||
#include <iomanip>
|
||||
#include "../binary_table.h"
|
||||
|
||||
void dumpFile(const std::string& filename) {
|
||||
std::ifstream file(filename, std::ios::binary);
|
||||
file.seekg(0, std::ios::end);
|
||||
size_t size = file.tellg();
|
||||
file.seekg(0, std::ios::beg);
|
||||
|
||||
std::vector<uint8_t> data(size);
|
||||
file.read(reinterpret_cast<char*>(data.data()), size);
|
||||
|
||||
std::cout << "File size: " << size << " bytes" << std::endl;
|
||||
for (size_t i = 0; i < std::min(size, size_t(80)); i++) {
|
||||
if (i % 16 == 0) std::cout << std::hex << i << ": ";
|
||||
std::cout << std::hex << std::setfill('0') << std::setw(2) << (int)data[i] << " ";
|
||||
if (i % 16 == 15) std::cout << std::endl;
|
||||
}
|
||||
if (size % 16 != 0) std::cout << std::endl;
|
||||
}
|
||||
|
||||
int main() {
|
||||
using namespace bt;
|
||||
|
||||
const std::string filename = "debug_step.bin";
|
||||
if (std::filesystem::exists(filename)) {
|
||||
std::filesystem::remove(filename);
|
||||
}
|
||||
|
||||
BinaryTable table(filename);
|
||||
table.initialize();
|
||||
|
||||
std::cout << "=== Step-by-step Address Table Debug ===\n" << std::endl;
|
||||
|
||||
std::cout << "After initialize():" << std::endl;
|
||||
dumpFile(filename);
|
||||
|
||||
std::cout << "\n1. Before storing key1:" << std::endl;
|
||||
// Try reading the address table header
|
||||
{
|
||||
std::ifstream file(filename, std::ios::binary);
|
||||
int64_t addr;
|
||||
file.read(reinterpret_cast<char*>(&addr), 8);
|
||||
std::cout << "Address table pointer: " << addr << std::endl;
|
||||
}
|
||||
|
||||
std::cout << "\n2. Storing key1..." << std::endl;
|
||||
table.set<int32_t>("key1", 100);
|
||||
|
||||
std::cout << "After storing key1:" << std::endl;
|
||||
dumpFile(filename);
|
||||
|
||||
// Try reading the address table
|
||||
{
|
||||
std::ifstream file(filename, std::ios::binary);
|
||||
int64_t addr;
|
||||
file.read(reinterpret_cast<char*>(&addr), 8);
|
||||
std::cout << "Address table pointer: " << addr << std::endl;
|
||||
|
||||
if (addr != -1) {
|
||||
file.seekg(addr);
|
||||
uint8_t type;
|
||||
int32_t count;
|
||||
file.read(reinterpret_cast<char*>(&type), 1);
|
||||
file.read(reinterpret_cast<char*>(&count), 4);
|
||||
std::cout << "Address table type: " << (int)type << ", count: " << count << std::endl;
|
||||
}
|
||||
}
|
||||
|
||||
std::cout << "\n3. Storing key2..." << std::endl;
|
||||
table.set<int32_t>("key2", 200);
|
||||
|
||||
std::cout << "After storing key2:" << std::endl;
|
||||
dumpFile(filename);
|
||||
|
||||
// Try reading the address table again
|
||||
{
|
||||
std::ifstream file(filename, std::ios::binary);
|
||||
int64_t addr;
|
||||
file.read(reinterpret_cast<char*>(&addr), 8);
|
||||
std::cout << "Address table pointer: " << addr << std::endl;
|
||||
|
||||
if (addr != -1) {
|
||||
file.seekg(addr);
|
||||
uint8_t type;
|
||||
int32_t count;
|
||||
file.read(reinterpret_cast<char*>(&type), 1);
|
||||
file.read(reinterpret_cast<char*>(&count), 4);
|
||||
std::cout << "Address table type: " << (int)type << ", count: " << count << std::endl;
|
||||
|
||||
// Read the entries
|
||||
for (int32_t i = 0; i < count && i < 5; i++) {
|
||||
int64_t keyHash, valueAddr;
|
||||
file.read(reinterpret_cast<char*>(&keyHash), 8);
|
||||
file.read(reinterpret_cast<char*>(&valueAddr), 8);
|
||||
std::cout << "Entry " << i << ": hash=" << keyHash << ", addr=" << valueAddr << std::endl;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
213
cpp/main.cpp
213
cpp/main.cpp
@@ -1,213 +0,0 @@
|
||||
#include <iostream>
|
||||
#include <filesystem>
|
||||
#include "binary_table.h"
|
||||
|
||||
void printBinaryDump(const std::vector<uint8_t>& data) {
|
||||
for (size_t i = 0; i < data.size(); i += 16) {
|
||||
// Address
|
||||
printf("0x%04X (%4zu) | ", static_cast<unsigned int>(i), i);
|
||||
|
||||
// Hex bytes
|
||||
for (int j = 0; j < 16; j++) {
|
||||
if (i + j < data.size()) {
|
||||
printf("%02X ", data[i + j]);
|
||||
} else {
|
||||
printf(" ");
|
||||
}
|
||||
}
|
||||
|
||||
printf(" | ");
|
||||
|
||||
// Integer representation
|
||||
for (int j = 0; j < 16; j++) {
|
||||
if (i + j < data.size()) {
|
||||
printf("%3d ", data[i + j]);
|
||||
} else {
|
||||
printf(" ");
|
||||
}
|
||||
}
|
||||
|
||||
printf(" | ");
|
||||
|
||||
// ASCII representation
|
||||
for (int j = 0; j < 16; j++) {
|
||||
if (i + j < data.size()) {
|
||||
uint8_t byte = data[i + j];
|
||||
if (byte >= 32 && byte <= 126) {
|
||||
printf("%c", static_cast<char>(byte));
|
||||
} else {
|
||||
printf(".");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
printf(" |\n");
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<uint8_t> readFile(const std::string& path) {
|
||||
std::ifstream file(path, std::ios::binary);
|
||||
file.seekg(0, std::ios::end);
|
||||
size_t size = file.tellg();
|
||||
file.seekg(0, std::ios::beg);
|
||||
|
||||
std::vector<uint8_t> data(size);
|
||||
file.read(reinterpret_cast<char*>(data.data()), size);
|
||||
return data;
|
||||
}
|
||||
|
||||
int main() {
|
||||
using namespace bt;
|
||||
|
||||
std::cout << "C++ Binary Table - Reading Dart Reference File" << std::endl;
|
||||
std::cout << "===============================================" << std::endl;
|
||||
|
||||
// Read the file created by Dart
|
||||
const std::string filename = "dart_reference.bin";
|
||||
if (!std::filesystem::exists(filename)) {
|
||||
std::cout << "❌ Reference file not found: " << filename << std::endl;
|
||||
return 1;
|
||||
}
|
||||
|
||||
std::cout << "📁 Reading reference file created by Dart..." << std::endl;
|
||||
auto fileData = readFile(filename);
|
||||
printBinaryDump(fileData);
|
||||
std::cout << "File size: " << fileData.size() << " bytes\n" << std::endl;
|
||||
|
||||
// Try to read the file with C++ implementation
|
||||
try {
|
||||
BinaryTable table(filename);
|
||||
|
||||
std::cout << "🔍 Testing C++ reading of Dart-created file..." << std::endl;
|
||||
|
||||
// Try to read the arrays that Dart created
|
||||
std::cout << "Attempting to read 'int_array'..." << std::endl;
|
||||
try {
|
||||
auto intArray = table.getArray<int32_t>("int_array");
|
||||
std::cout << "✅ int_array found, length: " << intArray.length() << std::endl;
|
||||
|
||||
if (intArray.length() > 0) {
|
||||
std::cout << "First few elements: ";
|
||||
int count = std::min(5, static_cast<int>(intArray.length()));
|
||||
for (int i = 0; i < count; i++) {
|
||||
std::cout << intArray[i] << " ";
|
||||
}
|
||||
std::cout << std::endl;
|
||||
}
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << "❌ Failed to read int_array: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
std::cout << "\nAttempting to read 'float_array'..." << std::endl;
|
||||
try {
|
||||
auto floatArray = table.getArray<float>("float_array");
|
||||
std::cout << "✅ float_array found, length: " << floatArray.length() << std::endl;
|
||||
|
||||
if (floatArray.length() > 0) {
|
||||
std::cout << "First few elements: ";
|
||||
int count = std::min(5, static_cast<int>(floatArray.length()));
|
||||
for (int i = 0; i < count; i++) {
|
||||
std::cout << floatArray[i] << " ";
|
||||
}
|
||||
std::cout << std::endl;
|
||||
}
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << "❌ Failed to read float_array: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
std::cout << "\nAttempting to read 'empty' array..." << std::endl;
|
||||
try {
|
||||
auto emptyArray = table.getArray<int32_t>("empty");
|
||||
std::cout << "✅ empty array found, length: " << emptyArray.length() << std::endl;
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << "❌ Failed to read empty array: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << "❌ Failed to read file: " << e.what() << std::endl;
|
||||
}
|
||||
|
||||
std::cout << "\n" << std::string(50, '=') << std::endl;
|
||||
std::cout << "Testing C++ Writing -> C++ Reading" << std::endl;
|
||||
std::cout << std::string(50, '=') << std::endl;
|
||||
|
||||
// Test C++ writing by creating a simple file
|
||||
const std::string testFilename = "cpp_test.bin";
|
||||
if (std::filesystem::exists(testFilename)) {
|
||||
std::filesystem::remove(testFilename);
|
||||
}
|
||||
|
||||
try {
|
||||
BinaryTable writeTable(testFilename);
|
||||
writeTable.initialize();
|
||||
|
||||
std::cout << "📝 Writing simple data with C++..." << std::endl;
|
||||
|
||||
// Write very simple data first
|
||||
writeTable.set<int32_t>("test_int", 42);
|
||||
std::cout << "✅ Wrote integer" << std::endl;
|
||||
|
||||
// Read it back immediately
|
||||
int32_t readInt = writeTable.get<int32_t>("test_int");
|
||||
std::cout << "✅ Read back integer: " << readInt << std::endl;
|
||||
|
||||
// Write a simple array
|
||||
writeTable.set<std::vector<int32_t>>("simple_array", {1, 2, 3});
|
||||
std::cout << "✅ Wrote simple array" << std::endl;
|
||||
|
||||
auto readArray = writeTable.getArray<int32_t>("simple_array");
|
||||
std::cout << "✅ Read back array, length: " << readArray.length() << std::endl;
|
||||
|
||||
if (readArray.length() > 0) {
|
||||
std::cout << "Array elements: ";
|
||||
for (int i = 0; i < readArray.length(); i++) {
|
||||
std::cout << readArray[i] << " ";
|
||||
}
|
||||
std::cout << std::endl;
|
||||
}
|
||||
|
||||
// Test array operations
|
||||
std::cout << "\n📝 Testing array operations..." << std::endl;
|
||||
readArray.set(0, 99); // Modify first element
|
||||
readArray.add(4); // Add element
|
||||
readArray.addAll({5, 6}); // Add multiple
|
||||
|
||||
std::cout << "After modifications, length: " << readArray.length() << std::endl;
|
||||
std::cout << "Elements: ";
|
||||
for (int i = 0; i < readArray.length(); i++) {
|
||||
std::cout << readArray[i] << " ";
|
||||
}
|
||||
std::cout << std::endl;
|
||||
|
||||
// Test sublist
|
||||
auto sublist = readArray.fetchSublist(0, 3);
|
||||
std::cout << "Sublist (0-3): ";
|
||||
for (auto val : sublist) {
|
||||
std::cout << val << " ";
|
||||
}
|
||||
std::cout << std::endl;
|
||||
|
||||
std::cout << "\n🎉 C++ Implementation Status:" << std::endl;
|
||||
std::cout << "✅ File reading (Dart compatibility)" << std::endl;
|
||||
std::cout << "✅ File writing" << std::endl;
|
||||
std::cout << "✅ Basic data types (int, float, string)" << std::endl;
|
||||
std::cout << "✅ Array storage and retrieval" << std::endl;
|
||||
std::cout << "✅ Array operations (set, add, addAll)" << std::endl;
|
||||
std::cout << "✅ Array sublist fetching" << std::endl;
|
||||
std::cout << "✅ Type-safe template system" << std::endl;
|
||||
std::cout << "✅ Memory-efficient file access" << std::endl;
|
||||
std::cout << "✅ Full interoperability with Dart" << std::endl;
|
||||
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << "❌ C++ write/read test failed: " << e.what() << std::endl;
|
||||
|
||||
// Show the file that was created
|
||||
if (std::filesystem::exists(testFilename)) {
|
||||
std::cout << "\nFile that was created:" << std::endl;
|
||||
auto data = readFile(testFilename);
|
||||
printBinaryDump(data);
|
||||
}
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -1,197 +0,0 @@
|
||||
#include <iostream>
|
||||
#include <cassert>
|
||||
#include <vector>
|
||||
#include <filesystem>
|
||||
#include "binary_table.h"
|
||||
|
||||
void printBinaryDump(const std::string& filename) {
|
||||
std::ifstream file(filename, std::ios::binary);
|
||||
if (!file) {
|
||||
std::cout << "Cannot open file for dump" << std::endl;
|
||||
return;
|
||||
}
|
||||
|
||||
file.seekg(0, std::ios::end);
|
||||
size_t size = file.tellg();
|
||||
file.seekg(0, std::ios::beg);
|
||||
|
||||
std::vector<uint8_t> data(size);
|
||||
file.read(reinterpret_cast<char*>(data.data()), size);
|
||||
file.close();
|
||||
|
||||
std::cout << "\n=== Binary Dump of " << filename << " (" << size << " bytes) ===" << std::endl;
|
||||
|
||||
for (size_t i = 0; i < data.size(); i += 16) {
|
||||
printf("0x%04X | ", static_cast<unsigned int>(i));
|
||||
|
||||
// Hex bytes
|
||||
for (int j = 0; j < 16; j++) {
|
||||
if (i + j < data.size()) {
|
||||
printf("%02X ", data[i + j]);
|
||||
} else {
|
||||
printf(" ");
|
||||
}
|
||||
}
|
||||
|
||||
printf(" | ");
|
||||
|
||||
// ASCII representation
|
||||
for (int j = 0; j < 16; j++) {
|
||||
if (i + j < data.size()) {
|
||||
uint8_t byte = data[i + j];
|
||||
printf("%c", (byte >= 32 && byte <= 126) ? byte : '.');
|
||||
}
|
||||
}
|
||||
|
||||
printf("\n");
|
||||
}
|
||||
std::cout << "=========================" << std::endl;
|
||||
}
|
||||
|
||||
// Test equivalent to Dart's main() function
|
||||
int main() {
|
||||
std::cout << "🧪 C++ Binary Table Parity Test (matching Dart behavior)" << std::endl;
|
||||
std::cout << "=========================================================" << std::endl;
|
||||
|
||||
const std::string filename = "cpp_parity_test.bin";
|
||||
|
||||
// Clean up any existing file
|
||||
std::filesystem::remove(filename);
|
||||
|
||||
try {
|
||||
bt::BinaryTable table(filename);
|
||||
table.initialize();
|
||||
|
||||
std::cout << "\n1. Testing basic data types..." << std::endl;
|
||||
|
||||
// Set basic values
|
||||
table.set<int32_t>("myInt", 42);
|
||||
table.set<float>("myFloat", 3.14f);
|
||||
table.set<std::string>("myString", "Hello, World!");
|
||||
|
||||
// Verify basic values
|
||||
assert(table.get<int32_t>("myInt") == 42);
|
||||
assert(table.get<float>("myFloat") == 3.14f);
|
||||
assert(table.get<std::string>("myString") == "Hello, World!");
|
||||
|
||||
std::cout << "✅ Basic data types work correctly" << std::endl;
|
||||
|
||||
std::cout << "\n2. Testing array operations..." << std::endl;
|
||||
|
||||
// Test array creation and access
|
||||
std::vector<int32_t> testArray = {10, 20, 30, 40, 50};
|
||||
table.set<std::vector<int32_t>>("myArray", testArray);
|
||||
|
||||
auto retrievedArray = table.get<std::vector<int32_t>>("myArray");
|
||||
assert(retrievedArray.size() == 5);
|
||||
for (size_t i = 0; i < retrievedArray.size(); i++) {
|
||||
assert(retrievedArray[i] == testArray[i]);
|
||||
}
|
||||
|
||||
std::cout << "✅ Array storage and retrieval work correctly" << std::endl;
|
||||
|
||||
// Test uniform array operations
|
||||
auto uniformArray = table.getArray<int32_t>("myArray");
|
||||
assert(uniformArray.length() == 5);
|
||||
assert(uniformArray[0] == 10);
|
||||
assert(uniformArray[4] == 50);
|
||||
|
||||
// Test array modification
|
||||
uniformArray.set(2, 999);
|
||||
assert(uniformArray[2] == 999);
|
||||
|
||||
// Test array extension
|
||||
uniformArray.add(60);
|
||||
assert(uniformArray.length() == 6);
|
||||
assert(uniformArray[5] == 60);
|
||||
|
||||
std::cout << "✅ Uniform array operations work correctly" << std::endl;
|
||||
|
||||
std::cout << "\n3. Testing multi-key operations (previously causing corruption)..." << std::endl;
|
||||
|
||||
// Add multiple keys to test address table stability
|
||||
table.set<int32_t>("key1", 100);
|
||||
table.set<int32_t>("key2", 200);
|
||||
table.set<int32_t>("key3", 300);
|
||||
table.set<std::string>("str1", "First");
|
||||
table.set<std::string>("str2", "Second");
|
||||
|
||||
// Verify all keys are accessible
|
||||
assert(table.get<int32_t>("key1") == 100);
|
||||
assert(table.get<int32_t>("key2") == 200);
|
||||
assert(table.get<int32_t>("key3") == 300);
|
||||
assert(table.get<std::string>("str1") == "First");
|
||||
assert(table.get<std::string>("str2") == "Second");
|
||||
|
||||
std::cout << "✅ Multi-key operations work without corruption" << std::endl;
|
||||
|
||||
std::cout << "\n4. Testing remove operations..." << std::endl;
|
||||
|
||||
// Test removal
|
||||
table.remove("key2");
|
||||
|
||||
// Verify removed key is gone
|
||||
try {
|
||||
table.get<int32_t>("key2");
|
||||
assert(false && "Should have thrown exception");
|
||||
} catch (const std::runtime_error&) {
|
||||
// Expected
|
||||
}
|
||||
|
||||
// Verify other keys still work
|
||||
assert(table.get<int32_t>("key1") == 100);
|
||||
assert(table.get<int32_t>("key3") == 300);
|
||||
|
||||
std::cout << "✅ Remove operations work correctly" << std::endl;
|
||||
|
||||
std::cout << "\n5. Testing fetchSublist functionality..." << std::endl;
|
||||
|
||||
auto sublist = uniformArray.fetchSublist(1, 4);
|
||||
assert(sublist.size() == 3);
|
||||
assert(sublist[0] == 20); // myArray[1]
|
||||
assert(sublist[1] == 999); // myArray[2] (modified)
|
||||
assert(sublist[2] == 40); // myArray[3]
|
||||
|
||||
std::cout << "✅ fetchSublist works correctly" << std::endl;
|
||||
|
||||
std::cout << "\n6. Testing free list and truncation operations..." << std::endl;
|
||||
|
||||
// Create some data, then remove it to test free list
|
||||
table.set<int32_t>("temp1", 1000);
|
||||
table.set<int32_t>("temp2", 2000);
|
||||
table.set<int32_t>("temp3", 3000);
|
||||
|
||||
table.remove("temp1");
|
||||
table.remove("temp2");
|
||||
table.remove("temp3");
|
||||
|
||||
// Test truncation
|
||||
table.truncate();
|
||||
|
||||
// Verify original data still accessible
|
||||
assert(table.get<int32_t>("myInt") == 42);
|
||||
assert(table.get<std::string>("myString") == "Hello, World!");
|
||||
assert(table.get<int32_t>("key1") == 100);
|
||||
|
||||
std::cout << "✅ Free list and truncation work correctly" << std::endl;
|
||||
|
||||
std::cout << "\n🎉 ALL TESTS PASSED! C++ implementation has Dart parity!" << std::endl;
|
||||
|
||||
// Print final file dump for verification
|
||||
printBinaryDump(filename);
|
||||
|
||||
// Clean up
|
||||
std::filesystem::remove(filename);
|
||||
|
||||
return 0;
|
||||
|
||||
} catch (const std::exception& e) {
|
||||
std::cout << "❌ Test failed: " << e.what() << std::endl;
|
||||
|
||||
// Print file dump for debugging
|
||||
printBinaryDump(filename);
|
||||
|
||||
std::filesystem::remove(filename);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
121
cpp/src/Private/sweepstore/benchmark.cpp
Normal file
121
cpp/src/Private/sweepstore/benchmark.cpp
Normal file
@@ -0,0 +1,121 @@
|
||||
//
|
||||
// Created by Benjamin Watt on 02/12/2025.
|
||||
//
|
||||
|
||||
#include "sweepstore/sweepstore.h"
|
||||
|
||||
#include <string>
|
||||
#include <filesystem>
|
||||
#include <iostream>
|
||||
#include <thread>
|
||||
#include <queue>
|
||||
#include <mutex>
|
||||
#include <condition_variable>
|
||||
#include <atomic>
|
||||
#include <vector>
|
||||
|
||||
#include "sweepstore/utils/helpers.h"
|
||||
#include "sweepstore/utils/file_handle.h"
|
||||
#include "sweepstore/structures.h"
|
||||
#include "sweepstore/concurrency.h"
|
||||
|
||||
int main() {
|
||||
namespace fs = std::filesystem;
|
||||
|
||||
std::string filePath = "./example.bin";
|
||||
|
||||
Sweepstore sweepstore(filePath);
|
||||
sweepstore.initialise(32);
|
||||
|
||||
preciseSleep(std::chrono::milliseconds(1000));
|
||||
|
||||
std::vector<uint8_t> fileData = loadFile(filePath);
|
||||
std::cout << binaryDump(fileData) << std::endl;
|
||||
|
||||
std::cout << "Concurrent Workers: " << sweepstore.getConcurrencyHeader()->readNumberOfWorkers() << std::endl;
|
||||
std::cout << "Stale Ticket Threshold: " << STALE_HEARTBEAT_THRESHOLD_MS << std::endl;
|
||||
|
||||
SweepstoreConcurrency::initialiseMasterAsync(filePath);
|
||||
|
||||
int iterations = 16;
|
||||
int currentIteration = 0;
|
||||
|
||||
int concurrencyTest = 1;
|
||||
|
||||
// Worker pool infrastructure - created once and reused
|
||||
std::queue<std::function<void()>> taskQueue;
|
||||
std::mutex queueMutex;
|
||||
std::condition_variable queueCV;
|
||||
std::condition_variable completionCV;
|
||||
std::atomic<bool> shutdown{false};
|
||||
std::atomic<int> completedJobs{0};
|
||||
|
||||
// Create 32 persistent worker threads BEFORE timing
|
||||
std::vector<std::thread> workers;
|
||||
for (int i = 0; i < 32; i++) {
|
||||
workers.emplace_back([&]() {
|
||||
while (!shutdown) {
|
||||
std::function<void()> task;
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(queueMutex);
|
||||
queueCV.wait(lock, [&]{ return !taskQueue.empty() || shutdown; });
|
||||
if (shutdown && taskQueue.empty()) return;
|
||||
if (!taskQueue.empty()) {
|
||||
task = std::move(taskQueue.front());
|
||||
taskQueue.pop();
|
||||
}
|
||||
}
|
||||
if (task) {
|
||||
task();
|
||||
completionCV.notify_one();
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
while (true) {
|
||||
|
||||
if (++currentIteration > iterations) {
|
||||
break;
|
||||
}
|
||||
|
||||
completedJobs = 0;
|
||||
|
||||
// Queue tasks
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(queueMutex);
|
||||
for (int i = 0; i < concurrencyTest; i++) {
|
||||
taskQueue.push([i, &sweepstore, &completedJobs]() {
|
||||
sweepstore["key_" + std::to_string(i)] = "value_" + std::to_string(i);
|
||||
++completedJobs;
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Start timing JUST before notifying workers
|
||||
auto start = std::chrono::high_resolution_clock::now();
|
||||
queueCV.notify_all();
|
||||
|
||||
// Wait for completion
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(queueMutex);
|
||||
completionCV.wait(lock, [&]{ return completedJobs >= concurrencyTest; });
|
||||
}
|
||||
|
||||
auto end = std::chrono::high_resolution_clock::now();
|
||||
auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();
|
||||
|
||||
std::cout << "[" << currentIteration << "/" << iterations << "] Completed " << concurrencyTest << " operations in " << duration << " ms." << std::endl;
|
||||
|
||||
concurrencyTest *= 2;
|
||||
}
|
||||
|
||||
// Shutdown workers after all iterations
|
||||
shutdown = true;
|
||||
queueCV.notify_all();
|
||||
for (auto& worker : workers) {
|
||||
worker.join();
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
256
cpp/src/Private/sweepstore/concurrency.cpp
Normal file
256
cpp/src/Private/sweepstore/concurrency.cpp
Normal file
@@ -0,0 +1,256 @@
|
||||
|
||||
|
||||
#include <functional>
|
||||
#include <iosfwd>
|
||||
|
||||
#include "sweepstore/concurrency.h"
|
||||
|
||||
#include <iostream>
|
||||
#include <random>
|
||||
|
||||
#include "sweepstore/header.h"
|
||||
#include "sweepstore/utils/helpers.h"
|
||||
#include "sweepstore/utils/file_handle.h"
|
||||
|
||||
|
||||
uint64_t getRandomOffset(uint64_t maxValue) {
|
||||
static std::random_device rd;
|
||||
static std::mt19937_64 gen(rd());
|
||||
std::uniform_int_distribution<uint64_t> dist(0, maxValue);
|
||||
return dist(gen);
|
||||
}
|
||||
|
||||
int randomId() {
|
||||
// mix timestamp with random for better uniqueness
|
||||
// keep it positive to avoid signed int issues when storing
|
||||
auto now = std::chrono::system_clock::now();
|
||||
auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
|
||||
int32_t time = static_cast<int32_t>(millis & 0xFFFFFFFF); // Get lower 32 bits
|
||||
int32_t random = static_cast<int32_t>(getRandomOffset(0x7FFFFFFF)); // 0 to 0x7FFFFFFF
|
||||
return (time ^ random) & 0x7FFFFFFF;
|
||||
}
|
||||
|
||||
void SweepstoreConcurrency::spawnTicket(SweepstoreFileHandle* _file,
|
||||
const SweepstoreTicketOperation& operation,
|
||||
const uint32_t keyHash,
|
||||
const uint32_t targetSize,
|
||||
const std::function<void()> onApproved,
|
||||
std::string debugLabel
|
||||
) {
|
||||
|
||||
// FileHandle now uses thread-local streams internally - no need to create new handle!
|
||||
// Each thread automatically gets its own fstream from the shared file handle
|
||||
SweepstoreFileHandle* file = new SweepstoreFileHandle(_file->getPath(), std::ios::in | std::ios::out | std::ios::binary);
|
||||
|
||||
/*
|
||||
Useful Functions
|
||||
*/
|
||||
|
||||
/// Logging function
|
||||
auto log = [&](const std::string &message) {
|
||||
std::string prefix = !debugLabel.empty() ? "\033[38;5;208m[Ticket Spawner - " + debugLabel + "]:\033[0m " : "\033[38;5;208m[Ticket Spawner]:\033[0m ";
|
||||
// debugPrint(prefix + message);
|
||||
};
|
||||
|
||||
// Sleep with variance (additive only)
|
||||
auto varySleep = [&](std::chrono::nanoseconds minSleepDuration, std::chrono::nanoseconds variance) {
|
||||
if (variance.count() <= 0) {
|
||||
preciseSleep(minSleepDuration);
|
||||
} else {
|
||||
// Generate random duration within variance
|
||||
uint64_t randomOffset = getRandomOffset(variance.count());
|
||||
preciseSleep(minSleepDuration + std::chrono::nanoseconds(randomOffset));
|
||||
}
|
||||
};
|
||||
|
||||
// Exponential sleep
|
||||
std::unordered_map<std::string, int> expSleepTracker = {};
|
||||
auto expSleep = [&expSleepTracker](const std::string& label) {
|
||||
int count = expSleepTracker[label]; // defaults to 0 if not found
|
||||
int sleepTime = (1 << count); // Exponential backoff
|
||||
sleepTime = std::max(1, std::min(sleepTime, 1000)); // Clamp between 1ms and 1000ms
|
||||
preciseSleep(std::chrono::milliseconds(sleepTime));
|
||||
expSleepTracker[label] = count + 1;
|
||||
};
|
||||
|
||||
// Get the header(s) - using the shared file handle directly
|
||||
SweepstoreHeader header(*file);
|
||||
SweepstoreConcurrencyHeader concurrencyHeader(*file);
|
||||
|
||||
/*
|
||||
Ticket Acquisition
|
||||
*/
|
||||
auto acquireTicket = [&](uint32_t newIdentifier) -> SweepstoreWorkerTicket {
|
||||
|
||||
// Reduce the chance of race condition
|
||||
varySleep(std::chrono::microseconds(500), std::chrono::microseconds(200));
|
||||
|
||||
uint32_t ticketIndex = -1u;
|
||||
|
||||
while (true) {
|
||||
|
||||
uint32_t concurrentWorkers = concurrencyHeader.readNumberOfWorkers();
|
||||
|
||||
for (uint32_t i = 0; i < concurrentWorkers; i++) {
|
||||
|
||||
SweepstoreWorkerTicket ticket = SweepstoreWorkerTicket(i, *file);
|
||||
|
||||
if (!ticket.writable()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
SweepstoreWorkerTicketSnapshot snapshot = ticket.snapshot();
|
||||
|
||||
int identifier = snapshot.identifier;
|
||||
|
||||
bool identifier_unassigned = identifier == 0;
|
||||
bool stale_heartbeat = millisecondsSinceEpoch32() - snapshot.workerHeartbeat > STALE_HEARTBEAT_THRESHOLD_MS;
|
||||
bool is_free = snapshot.state == SweepstoreTicketState::FREE;
|
||||
|
||||
if (identifier_unassigned && stale_heartbeat && is_free) {
|
||||
snapshot.identifier = newIdentifier;
|
||||
snapshot.workerHeartbeat = millisecondsSinceEpoch32();
|
||||
snapshot.state = SweepstoreTicketState::WAITING;
|
||||
ticket.write(snapshot);
|
||||
ticketIndex = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
preciseSleep(std::chrono::milliseconds(2));
|
||||
|
||||
// Ensure we still own the ticket - if not, reset and try again
|
||||
if (ticketIndex != -1u) {
|
||||
SweepstoreWorkerTicketSnapshot verifySnapshot = concurrencyHeader[ticketIndex].snapshot();
|
||||
|
||||
if (verifySnapshot.identifier != newIdentifier) {
|
||||
ticketIndex = -1; // Lost the ticket, try again
|
||||
} else {
|
||||
log("Acquired ticket " + std::to_string(ticketIndex) + " with identifier " + std::to_string(newIdentifier) + ".");
|
||||
return concurrencyHeader[ticketIndex];
|
||||
}
|
||||
}
|
||||
|
||||
expSleep("acquireTicket");
|
||||
}
|
||||
|
||||
};
|
||||
|
||||
|
||||
uint32_t myIdentifier = randomId();
|
||||
|
||||
SweepstoreWorkerTicket myTicket = acquireTicket(myIdentifier);
|
||||
SweepstoreWorkerTicketSnapshot mySnapshot = myTicket.snapshot();
|
||||
mySnapshot.workerHeartbeat = millisecondsSinceEpoch32();
|
||||
mySnapshot.state = SweepstoreTicketState::WAITING;
|
||||
mySnapshot.operation = operation;
|
||||
mySnapshot.keyHash = keyHash;
|
||||
mySnapshot.targetSize = targetSize;
|
||||
myTicket.write(mySnapshot);
|
||||
|
||||
// Wait for approval
|
||||
while (true) {
|
||||
|
||||
SweepstoreWorkerTicketSnapshot snapshot = myTicket.snapshot();
|
||||
|
||||
// Update heartbeat
|
||||
uint32_t currentTime = millisecondsSinceEpoch32();
|
||||
if (currentTime - snapshot.workerHeartbeat > 700) {
|
||||
snapshot.workerHeartbeat = currentTime;
|
||||
myTicket.write(snapshot);
|
||||
}
|
||||
|
||||
// Check if we still own the ticket
|
||||
if (snapshot.identifier != myIdentifier) {
|
||||
|
||||
preciseSleep(std::chrono::milliseconds(10));
|
||||
|
||||
// Re-verify we lost the ticket
|
||||
SweepstoreWorkerTicketSnapshot recheckSnapshot = myTicket.snapshot();
|
||||
if (recheckSnapshot.identifier != myIdentifier) {
|
||||
// log("Lost ownership of ticket " + std::to_string(myTicket.getTicketIndex()) + ", was expecting identifier " + std::to_string(myIdentifier) + " but found " + std::to_string(recheckSnapshot.identifier) + ".");
|
||||
std::cout << "\033[38;5;82m[Ticket Spawner - " << debugLabel << "]:\033[0m Lost ticket " << myTicket.getTicketIndex() << ", respawning..." << std::endl;
|
||||
|
||||
// ReSharper disable once CppDFAInfiniteRecursion
|
||||
spawnTicket(
|
||||
_file,
|
||||
operation,
|
||||
keyHash,
|
||||
targetSize,
|
||||
onApproved,
|
||||
debugLabel
|
||||
);
|
||||
break;
|
||||
}
|
||||
|
||||
// False alarm, continue waiting
|
||||
// log("False alarm, still own ticket " + std::to_string(myTicket.getTicketIndex()) + ".");
|
||||
std::cout << "\033[38;5;82m[Ticket Spawner - " << debugLabel << "]:\033[0m False alarm, still own ticket " << myTicket.getTicketIndex() << "." << std::endl;
|
||||
snapshot = recheckSnapshot;
|
||||
}
|
||||
|
||||
if (snapshot.state == SweepstoreTicketState::APPROVED) {
|
||||
snapshot.state = SweepstoreTicketState::EXECUTING;
|
||||
myTicket.write(snapshot);
|
||||
|
||||
onApproved();
|
||||
|
||||
snapshot.state = SweepstoreTicketState::COMPLETED;
|
||||
myTicket.write(snapshot);
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
varySleep(std::chrono::microseconds(500), std::chrono::microseconds(200));
|
||||
}
|
||||
|
||||
// std::cout << "\033[38;5;82m[Ticket Spawner - " << debugLabel << "]:\033[0m Completed ticket " << myTicket.getTicketIndex() << "." << std::endl;
|
||||
delete file;
|
||||
}
|
||||
|
||||
void SweepstoreConcurrency::initialiseMaster(std::string filePath) {
|
||||
|
||||
auto log = [&](const std::string &message) {
|
||||
debugPrint("\033[38;5;33m[Concurrency Master]:\033[0m " + message);
|
||||
};
|
||||
|
||||
SweepstoreFileHandle file(filePath, std::ios::binary | std::ios::in | std::ios::out);
|
||||
|
||||
SweepstoreHeader header(file);
|
||||
SweepstoreConcurrencyHeader concurrencyHeader(file);
|
||||
|
||||
std::cout << "[Master] Starting master loop" << std::endl;
|
||||
|
||||
while (true) {
|
||||
|
||||
int concurrentWorkers = concurrencyHeader.readNumberOfWorkers();
|
||||
|
||||
for (uint32_t i = 0; i < concurrentWorkers; i++) {
|
||||
|
||||
SweepstoreWorkerTicket ticket(i, file);
|
||||
SweepstoreWorkerTicketSnapshot snapshot = ticket.snapshot();
|
||||
|
||||
if (snapshot.state == WAITING) {
|
||||
log("Found waiting ticket " + std::to_string(i) + "(Key Hash: " + std::to_string(snapshot.keyHash) + ")...");
|
||||
|
||||
// Approve the ticket
|
||||
snapshot.state = APPROVED;
|
||||
ticket.write(snapshot);
|
||||
log("Approved ticket " + std::to_string(i) + ".");
|
||||
} else if (snapshot.state == SweepstoreTicketState::COMPLETED) {
|
||||
log("Ticket " + std::to_string(i) + " has completed. Resetting...");
|
||||
|
||||
// Reset the ticket
|
||||
SweepstoreWorkerTicketSnapshot cleanSnapshot = SweepstoreWorkerTicketSnapshot();
|
||||
ticket.write(cleanSnapshot);
|
||||
log("Reset ticket " + std::to_string(i) + ".");
|
||||
}
|
||||
|
||||
// Handle stale tickets
|
||||
uint32_t currentTime = millisecondsSinceEpoch32();
|
||||
}
|
||||
|
||||
preciseSleep(std::chrono::milliseconds(1));
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
221
cpp/src/Private/sweepstore/header.cpp
Normal file
221
cpp/src/Private/sweepstore/header.cpp
Normal file
@@ -0,0 +1,221 @@
|
||||
|
||||
#include "sweepstore/header.h"
|
||||
|
||||
#include "sweepstore/utils/file_lock.h"
|
||||
#include "sweepstore/utils/helpers.h"
|
||||
|
||||
std::string SweepstoreHeader::readMagicNumber() {
|
||||
file.readSeek(0, std::ios::beg);
|
||||
char buffer[4];
|
||||
file.readBytes(buffer, 4);
|
||||
return std::string(buffer, 4);
|
||||
}
|
||||
|
||||
void SweepstoreHeader::writeMagicNumber(const std::string& magicNumber) {
|
||||
if (magicNumber.size() != 4) {
|
||||
throw std::invalid_argument("Magic number must be exactly 4 characters long.");
|
||||
}
|
||||
file.writeSeek(0, std::ios::beg);
|
||||
file.writeBytes(magicNumber.c_str(), 4);
|
||||
}
|
||||
|
||||
std::string SweepstoreHeader::readVersion() {
|
||||
file.readSeek(4, std::ios::beg);
|
||||
char buffer[12];
|
||||
file.readBytes(buffer, 12);
|
||||
|
||||
// Trim leading and trailing spaces
|
||||
std::string version(buffer, 12);
|
||||
version = trim(version);
|
||||
|
||||
return version;
|
||||
}
|
||||
|
||||
void SweepstoreHeader::writeVersion(const std::string& version) {
|
||||
if (version.size() > 11) {
|
||||
throw std::invalid_argument("Version string must be at most 11 characters long.");
|
||||
}
|
||||
// Pad 1 space to the beginning
|
||||
// And pad to end to make it 12 bytes
|
||||
std::string paddedVersion = " " + version;
|
||||
paddedVersion.resize(12, ' ');
|
||||
|
||||
file.writeSeek(4, std::ios::beg);
|
||||
file.writeBytes(paddedVersion.c_str(), 12);
|
||||
}
|
||||
|
||||
SweepstorePointer SweepstoreHeader::readAddressTablePointer() {
|
||||
file.readSeek(16, std::ios::beg);
|
||||
int64_t address;
|
||||
file.readBytes(reinterpret_cast<char*>(&address), sizeof(address));
|
||||
return address; // Implicit conversion to SweepstorePointer
|
||||
}
|
||||
|
||||
void SweepstoreHeader::writeAddressTablePointer(const SweepstorePointer& ptr) {
|
||||
file.writeSeek(16, std::ios::beg);
|
||||
int64_t address = ptr;
|
||||
file.writeBytes(reinterpret_cast<const char*>(&address), sizeof(address));
|
||||
}
|
||||
|
||||
uint32_t SweepstoreHeader::readFreeListCount() {
|
||||
file.readSeek(24, std::ios::beg);
|
||||
uint32_t count;
|
||||
file.readBytes(reinterpret_cast<char*>(&count), sizeof(count));
|
||||
return count;
|
||||
}
|
||||
|
||||
void SweepstoreHeader::writeFreeListCount(uint32_t count) {
|
||||
file.writeSeek(24, std::ios::beg);
|
||||
file.writeBytes(reinterpret_cast<const char*>(&count), sizeof(count));
|
||||
}
|
||||
|
||||
bool SweepstoreHeader::readIsFreeListLifted() {
|
||||
file.readSeek(28, std::ios::beg);
|
||||
char flag;
|
||||
file.readBytes(&flag, sizeof(flag));
|
||||
return flag != 0;
|
||||
}
|
||||
|
||||
void SweepstoreHeader::writeIsFreeListLifted(bool isLifted) {
|
||||
file.writeSeek(28, std::ios::beg);
|
||||
char flag = isLifted ? 1 : 0;
|
||||
file.writeBytes(&flag, sizeof(flag));
|
||||
}
|
||||
|
||||
void SweepstoreHeader::initialise() {
|
||||
writeMagicNumber("SWPT");
|
||||
writeVersion("undefined");
|
||||
writeAddressTablePointer(SweepstorePointer::NULL_PTR);
|
||||
writeFreeListCount(0);
|
||||
writeIsFreeListLifted(false);
|
||||
file.flush();
|
||||
}
|
||||
|
||||
uint64_t SweepstoreConcurrencyHeader::readMasterIdentifier() {
|
||||
file.readSeek(29, std::ios::beg);
|
||||
uint64_t identifier;
|
||||
file.readBytes(reinterpret_cast<char*>(&identifier), sizeof(identifier));
|
||||
return identifier;
|
||||
}
|
||||
|
||||
void SweepstoreConcurrencyHeader::writeMasterIdentifier(uint64_t identifier) {
|
||||
file.writeSeek(29, std::ios::beg);
|
||||
file.writeBytes(reinterpret_cast<const char*>(&identifier), sizeof(identifier));
|
||||
}
|
||||
|
||||
uint32_t SweepstoreConcurrencyHeader::readMasterHeartbeat() {
|
||||
file.readSeek(37, std::ios::beg);
|
||||
uint32_t heartbeat;
|
||||
file.readBytes(reinterpret_cast<char*>(&heartbeat), sizeof(heartbeat));
|
||||
return heartbeat;
|
||||
}
|
||||
|
||||
void SweepstoreConcurrencyHeader::writeMasterHeartbeat(uint32_t heartbeat) {
|
||||
file.writeSeek(37, std::ios::beg);
|
||||
file.writeBytes(reinterpret_cast<const char*>(&heartbeat), sizeof(heartbeat));
|
||||
}
|
||||
|
||||
uint32_t SweepstoreConcurrencyHeader::readNumberOfWorkers() {
|
||||
file.readSeek(41, std::ios::beg);
|
||||
uint32_t numWorkers;
|
||||
file.readBytes(reinterpret_cast<char*>(&numWorkers), sizeof(numWorkers));
|
||||
return numWorkers;
|
||||
}
|
||||
|
||||
void SweepstoreConcurrencyHeader::writeNumberOfWorkers(uint32_t numWorkers) {
|
||||
file.writeSeek(41, std::ios::beg);
|
||||
file.writeBytes(reinterpret_cast<const char*>(&numWorkers), sizeof(numWorkers));
|
||||
}
|
||||
|
||||
bool SweepstoreConcurrencyHeader::readIsReadAllowed() {
|
||||
file.readSeek(45, std::ios::beg);
|
||||
char flag;
|
||||
file.readBytes(&flag, sizeof(flag));
|
||||
return flag != 0;
|
||||
}
|
||||
|
||||
void SweepstoreConcurrencyHeader::writeIsReadAllowed(bool isAllowed) {
|
||||
file.writeSeek(45, std::ios::beg);
|
||||
char flag = isAllowed ? 1 : 0;
|
||||
file.writeBytes(&flag, sizeof(flag));
|
||||
}
|
||||
|
||||
void SweepstoreConcurrencyHeader::initialise(int concurrentWorkers) {
|
||||
writeMasterIdentifier(0);
|
||||
writeMasterHeartbeat(0);
|
||||
writeNumberOfWorkers(concurrentWorkers);
|
||||
writeIsReadAllowed(true);
|
||||
uint32_t verifyWorkers = readNumberOfWorkers();
|
||||
for (uint32_t i = 0; i < verifyWorkers; i++) {
|
||||
SweepstoreWorkerTicketSnapshot ticket = SweepstoreWorkerTicketSnapshot();
|
||||
ticket.identifier = 0;
|
||||
ticket.workerHeartbeat = 0;
|
||||
ticket.state = SweepstoreTicketState::FREE;
|
||||
ticket.operation = SweepstoreTicketOperation::NONE;
|
||||
ticket.keyHash = 0;
|
||||
ticket.targetAddress = SweepstorePointer::NULL_PTR;
|
||||
ticket.targetSize = 0;
|
||||
|
||||
SweepstoreWorkerTicket ticketWriter = SweepstoreWorkerTicket(i, file);
|
||||
ticketWriter.write(ticket);
|
||||
}
|
||||
file.flush();
|
||||
}
|
||||
|
||||
void SweepstoreWorkerTicket::write(SweepstoreWorkerTicketSnapshot &snapshot) {
|
||||
RandomAccessMemory buffer;
|
||||
|
||||
SweepstoreFileLock lock(file.getPath(), 0, 0, SweepstoreFileLock::Mode::Exclusive);
|
||||
SweepstoreFileLock::Scoped scopedLock(lock);
|
||||
|
||||
buffer.setPositionSync(0);
|
||||
buffer.writeIntSync(snapshot.identifier, 4);
|
||||
buffer.writeIntSync(snapshot.workerHeartbeat, 4);
|
||||
buffer.writeIntSync(static_cast<uint8_t>(snapshot.state), 1);
|
||||
buffer.writeIntSync(static_cast<uint8_t>(snapshot.operation), 1);
|
||||
buffer.writeUIntSync(snapshot.keyHash, 8);
|
||||
buffer.writePointerSync(snapshot.targetAddress, 8);
|
||||
buffer.writeUIntSync(snapshot.targetSize, 4);
|
||||
|
||||
// Pad the rest with zeros if necessary
|
||||
while (buffer.length() < TICKET_SIZE) {
|
||||
buffer.writeIntSync(0, 1);
|
||||
}
|
||||
|
||||
// Prepare data
|
||||
buffer.setPositionSync(0);
|
||||
std::vector<uint8_t> data = buffer.readSync(buffer.length());
|
||||
char* dataPtr = reinterpret_cast<char*>(data.data());
|
||||
|
||||
// Write to file
|
||||
file.writeSeek(getOffset());
|
||||
file.writeBytes(dataPtr, data.size());
|
||||
file.flush();
|
||||
}
|
||||
|
||||
bool SweepstoreWorkerTicket::writable() {
|
||||
SweepstoreFileLock lock(file.getPath(), 0, 0, SweepstoreFileLock::Mode::Exclusive);
|
||||
return lock.isLocked() == false;
|
||||
}
|
||||
|
||||
SweepstoreWorkerTicketSnapshot SweepstoreWorkerTicket::snapshot() {
|
||||
SweepstoreFileLock lock(file.getPath(), 0, 0, SweepstoreFileLock::Mode::Shared);
|
||||
lock.lock();
|
||||
file.readSeek(getOffset());
|
||||
std::unique_ptr<char[]> buffer(new char[TICKET_SIZE]);
|
||||
file.readBytes(buffer.get(), TICKET_SIZE);
|
||||
lock.unlock();
|
||||
RandomAccessMemory ram(reinterpret_cast<uint8_t*>(buffer.get()), TICKET_SIZE);
|
||||
|
||||
SweepstoreWorkerTicketSnapshot snapshot;
|
||||
ram.setPositionSync(0);
|
||||
snapshot.identifier = ram.readUIntSync(4);
|
||||
snapshot.workerHeartbeat = ram.readUIntSync(4);
|
||||
snapshot.state = static_cast<SweepstoreTicketState>(ram.readUIntSync(1));
|
||||
snapshot.operation = static_cast<SweepstoreTicketOperation>(ram.readUIntSync(1));
|
||||
snapshot.keyHash = ram.readUIntSync(8);
|
||||
snapshot.targetAddress = ram.readPointerSync(8);
|
||||
snapshot.targetSize = ram.readUIntSync(4);
|
||||
|
||||
return snapshot;
|
||||
}
|
||||
13
cpp/src/Private/sweepstore/structures.cpp
Normal file
13
cpp/src/Private/sweepstore/structures.cpp
Normal file
@@ -0,0 +1,13 @@
|
||||
|
||||
|
||||
|
||||
#include "sweepstore/structures.h"
|
||||
const SweepstorePointer SweepstorePointer::NULL_PTR = SweepstorePointer(UINT64_MAX);
|
||||
|
||||
bool SweepstorePointer::operator==(const SweepstorePointer &p) {
|
||||
if (this->address == p.address) {
|
||||
return true;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
21
cpp/src/Private/sweepstore/sweepstore.cpp
Normal file
21
cpp/src/Private/sweepstore/sweepstore.cpp
Normal file
@@ -0,0 +1,21 @@
|
||||
//
|
||||
// Created by Benjamin Watt on 24/11/2025.
|
||||
//
|
||||
|
||||
#include "sweepstore/sweepstore.h"
|
||||
|
||||
#include <string>
|
||||
#include <filesystem>
|
||||
#include <iostream>
|
||||
#include <thread>
|
||||
|
||||
#include "sweepstore/utils/helpers.h"
|
||||
#include "sweepstore/utils/file_handle.h"
|
||||
|
||||
void Sweepstore::initialise(int concurrentWorkers) {
|
||||
header->initialise();
|
||||
header->writeVersion("1.1.0.2");
|
||||
concurrencyHeader->initialise(concurrentWorkers);
|
||||
|
||||
debugPrint("Version: " + header->readVersion());
|
||||
}
|
||||
12
cpp/src/Private/sweepstore/utils/fd_pool.cpp
Normal file
12
cpp/src/Private/sweepstore/utils/fd_pool.cpp
Normal file
@@ -0,0 +1,12 @@
|
||||
#include "sweepstore/utils/file_lock.h"
|
||||
#include "sweepstore/utils/file_handle.h"
|
||||
|
||||
// Thread-local FD cache definition for file locking
|
||||
#ifndef _WIN32
|
||||
thread_local std::unordered_map<std::string, int> SweepstoreFileLock::fdCache;
|
||||
#endif
|
||||
|
||||
// Thread-local stream cache definition for file handles
|
||||
#ifndef WITH_UNREAL
|
||||
thread_local std::unordered_map<std::string, std::unique_ptr<std::fstream>> SweepstoreFileHandle::streamCache;
|
||||
#endif
|
||||
172
cpp/src/Private/sweepstore/utils/file_handle.cpp
Normal file
172
cpp/src/Private/sweepstore/utils/file_handle.cpp
Normal file
@@ -0,0 +1,172 @@
|
||||
#include "sweepstore/utils/file_handle.h"
|
||||
|
||||
// Constructor - just stores path and mode, actual stream is created per-thread
|
||||
SweepstoreFileHandle::SweepstoreFileHandle(const std::string& p, std::ios::openmode mode)
|
||||
: path(p)
|
||||
, openMode(mode)
|
||||
#ifdef WITH_UNREAL
|
||||
{
|
||||
IPlatformFile& platformFile = FPlatformFileManager::Get().GetPlatformFile();
|
||||
|
||||
// Map std::ios flags to Unreal flags
|
||||
bool read = (mode & std::ios::in) != 0;
|
||||
bool write = (mode & std::ios::out) != 0;
|
||||
|
||||
if (read && write) {
|
||||
unrealHandle = platformFile.OpenReadWrite(*FString(path.c_str()), true);
|
||||
} else if (write) {
|
||||
unrealHandle = platformFile.OpenWrite(*FString(path.c_str()), false, false);
|
||||
} else {
|
||||
unrealHandle = platformFile.OpenRead(*FString(path.c_str()), false);
|
||||
}
|
||||
|
||||
if (!unrealHandle) {
|
||||
throw std::runtime_error("Failed to open file: " + path);
|
||||
}
|
||||
}
|
||||
#else
|
||||
{
|
||||
// Thread-local streams created on demand in getThreadStream()
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifndef WITH_UNREAL
|
||||
// Get or create the fstream for this thread
|
||||
std::fstream& SweepstoreFileHandle::getThreadStream() {
|
||||
auto it = streamCache.find(path);
|
||||
if (it == streamCache.end() || !it->second || !it->second->is_open()) {
|
||||
// Create new stream for this thread
|
||||
auto stream = std::make_unique<std::fstream>(path, openMode);
|
||||
if (!stream->is_open()) {
|
||||
throw std::runtime_error("Failed to open file: " + path);
|
||||
}
|
||||
streamCache[path] = std::move(stream);
|
||||
return *streamCache[path];
|
||||
}
|
||||
return *it->second;
|
||||
}
|
||||
|
||||
const std::fstream& SweepstoreFileHandle::getThreadStream() const {
|
||||
// Use const_cast to reuse the non-const version
|
||||
return const_cast<SweepstoreFileHandle*>(this)->getThreadStream();
|
||||
}
|
||||
#endif
|
||||
|
||||
// isOpen
|
||||
bool SweepstoreFileHandle::isOpen() const {
|
||||
#ifdef WITH_UNREAL
|
||||
return unrealHandle != nullptr;
|
||||
#else
|
||||
auto it = streamCache.find(path);
|
||||
return it != streamCache.end() && it->second && it->second->is_open();
|
||||
#endif
|
||||
}
|
||||
|
||||
// close
|
||||
void SweepstoreFileHandle::close() {
|
||||
#ifdef WITH_UNREAL
|
||||
if (unrealHandle) {
|
||||
delete unrealHandle;
|
||||
unrealHandle = nullptr;
|
||||
}
|
||||
#else
|
||||
// Close this thread's stream if it exists
|
||||
auto it = streamCache.find(path);
|
||||
if (it != streamCache.end() && it->second && it->second->is_open()) {
|
||||
it->second->close();
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
#if defined(_WIN32) || defined(WITH_UNREAL)
|
||||
// flush
|
||||
void SweepstoreFileHandle::flush() {
|
||||
#ifdef WITH_UNREAL
|
||||
if (unrealHandle) {
|
||||
unrealHandle->Flush();
|
||||
}
|
||||
#else
|
||||
// Windows-specific implementation for guaranteed flush to disk
|
||||
auto& stream = getThreadStream();
|
||||
stream.flush();
|
||||
|
||||
// On Windows, also call sync to push to OS buffers
|
||||
// Then open a Windows HANDLE to the same file and call FlushFileBuffers
|
||||
// This is more reliable than trying to extract the HANDLE from fstream
|
||||
HANDLE h = CreateFileA(
|
||||
path.c_str(),
|
||||
GENERIC_WRITE,
|
||||
FILE_SHARE_READ | FILE_SHARE_WRITE,
|
||||
NULL,
|
||||
OPEN_EXISTING,
|
||||
FILE_ATTRIBUTE_NORMAL,
|
||||
NULL
|
||||
);
|
||||
|
||||
if (h != INVALID_HANDLE_VALUE) {
|
||||
FlushFileBuffers(h);
|
||||
CloseHandle(h);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
// readSeek
|
||||
void SweepstoreFileHandle::readSeek(std::streampos pos, std::ios::seekdir dir) {
|
||||
#ifdef WITH_UNREAL
|
||||
// Unreal doesn't have separate read/write pointers, so just seek
|
||||
int64 unrealPos = static_cast<int64>(pos);
|
||||
if (dir == std::ios::beg) {
|
||||
unrealHandle->Seek(unrealPos);
|
||||
} else if (dir == std::ios::cur) {
|
||||
unrealHandle->Seek(unrealHandle->Tell() + unrealPos);
|
||||
} else if (dir == std::ios::end) {
|
||||
unrealHandle->SeekFromEnd(unrealPos);
|
||||
}
|
||||
#else
|
||||
// Windows - simplified to only seek read pointer
|
||||
auto& stream = getThreadStream();
|
||||
stream.seekg(pos, dir);
|
||||
if (stream.fail()) {
|
||||
stream.clear();
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
// writeSeek
|
||||
void SweepstoreFileHandle::writeSeek(std::streampos pos, std::ios::seekdir dir) {
|
||||
#ifdef WITH_UNREAL
|
||||
// Same as readSeek for Unreal
|
||||
readSeek(pos, dir);
|
||||
#else
|
||||
// Windows - simplified to only seek write pointer
|
||||
auto& stream = getThreadStream();
|
||||
stream.seekp(pos, dir);
|
||||
if (stream.fail()) {
|
||||
stream.clear();
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
// readBytes
|
||||
void SweepstoreFileHandle::readBytes(char* buffer, std::streamsize size) {
|
||||
#ifdef WITH_UNREAL
|
||||
unrealHandle->Read(reinterpret_cast<uint8*>(buffer), size);
|
||||
#else
|
||||
// Windows
|
||||
auto& stream = getThreadStream();
|
||||
stream.read(buffer, size);
|
||||
#endif
|
||||
}
|
||||
|
||||
// writeBytes
|
||||
void SweepstoreFileHandle::writeBytes(const char* buffer, std::streamsize size) {
|
||||
#ifdef WITH_UNREAL
|
||||
unrealHandle->Write(reinterpret_cast<const uint8*>(buffer), size);
|
||||
unrealHandle->Flush(); // Unreal requires explicit flush
|
||||
#else
|
||||
// Windows
|
||||
auto& stream = getThreadStream();
|
||||
stream.write(buffer, size);
|
||||
#endif
|
||||
}
|
||||
#endif // _WIN32 || WITH_UNREAL
|
||||
7
cpp/src/Private/sweepstore/utils/file_lock.cpp
Normal file
7
cpp/src/Private/sweepstore/utils/file_lock.cpp
Normal file
@@ -0,0 +1,7 @@
|
||||
#include "sweepstore/utils/file_lock.h"
|
||||
|
||||
#ifdef _WIN32
|
||||
thread_local std::unordered_map<std::string, HANDLE> SweepstoreFileLock::handleCache;
|
||||
#else
|
||||
thread_local std::unordered_map<std::string, int> SweepstoreFileLock::fdCache;
|
||||
#endif
|
||||
32
cpp/src/Public/sweepstore/concurrency.h
Normal file
32
cpp/src/Public/sweepstore/concurrency.h
Normal file
@@ -0,0 +1,32 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <memory>
|
||||
|
||||
#define STALE_HEARTBEAT_THRESHOLD_MS 5000
|
||||
|
||||
enum SweepstoreTicketOperation : int;
|
||||
class SweepstoreFileHandle;
|
||||
|
||||
namespace SweepstoreConcurrency {
|
||||
|
||||
void spawnTicket(SweepstoreFileHandle* file,
|
||||
const SweepstoreTicketOperation& operation,
|
||||
const uint32_t keyHash,
|
||||
const uint32_t targetSize,
|
||||
const std::function<void()> onApproved,
|
||||
std::string debugLabel = ""
|
||||
);
|
||||
|
||||
void initialiseMaster(std::string filePath);
|
||||
|
||||
inline void initialiseMasterAsync(std::string filePath) {
|
||||
std::thread([filePath]() {
|
||||
initialiseMaster(filePath);
|
||||
}).detach();
|
||||
}
|
||||
|
||||
}
|
||||
143
cpp/src/Public/sweepstore/header.h
Normal file
143
cpp/src/Public/sweepstore/header.h
Normal file
@@ -0,0 +1,143 @@
|
||||
#pragma once
|
||||
|
||||
#include <iosfwd>
|
||||
|
||||
#include "structures.h"
|
||||
#include "utils/file_handle.h"
|
||||
|
||||
constexpr int roundToNearest16(int number) {
|
||||
return (number + 15) & ~15;
|
||||
}
|
||||
|
||||
class SweepstoreHeader {
|
||||
|
||||
private:
|
||||
SweepstoreFileHandle& file;
|
||||
|
||||
public:
|
||||
explicit SweepstoreHeader(SweepstoreFileHandle &fileStream) : file(fileStream) {}
|
||||
|
||||
// Offset 0 - 4 bytes
|
||||
std::string readMagicNumber();
|
||||
void writeMagicNumber(const std::string& magicNumber);
|
||||
|
||||
// Offset 4 - 12 bytes
|
||||
std::string readVersion();
|
||||
void writeVersion(const std::string& version);
|
||||
|
||||
// Offset 16 - 8 bytes
|
||||
SweepstorePointer readAddressTablePointer();
|
||||
void writeAddressTablePointer(const SweepstorePointer& ptr);
|
||||
|
||||
// Offset 24 - 4 bytes
|
||||
uint32_t readFreeListCount();
|
||||
void writeFreeListCount(uint32_t count);
|
||||
|
||||
// Offset 28 - 1 byte
|
||||
bool readIsFreeListLifted();
|
||||
void writeIsFreeListLifted(bool isLifted);
|
||||
|
||||
/**
|
||||
* Initialises the header with default values.
|
||||
*/
|
||||
void initialise();
|
||||
|
||||
};
|
||||
|
||||
constexpr int SWEEPSTORE_COMBINED_STATIC_HEADER_SIZE = roundToNearest16(46);
|
||||
|
||||
struct SweepstoreWorkerTicketSnapshot {
|
||||
|
||||
SweepstoreWorkerTicketSnapshot() :
|
||||
identifier(0),
|
||||
workerHeartbeat(0),
|
||||
state(SweepstoreTicketState::FREE),
|
||||
operation(SweepstoreTicketOperation::NONE),
|
||||
keyHash(0),
|
||||
targetAddress(SweepstorePointer::NULL_PTR),
|
||||
targetSize(0) {}
|
||||
|
||||
// Offset 0 - 4 bytes
|
||||
uint32_t identifier;
|
||||
|
||||
// Offset 4 - 4 bytes
|
||||
uint32_t workerHeartbeat;
|
||||
|
||||
// Offset 8 - 1 byte
|
||||
SweepstoreTicketState state;
|
||||
|
||||
// Offset 9 - 1 byte
|
||||
SweepstoreTicketOperation operation;
|
||||
|
||||
// Offset 10 - 8 bytes
|
||||
uint64_t keyHash;
|
||||
|
||||
// Offset 18 - 8 bytes
|
||||
SweepstorePointer targetAddress;
|
||||
|
||||
// Offset 26 - 4 bytes
|
||||
uint32_t targetSize;
|
||||
};
|
||||
|
||||
class SweepstoreWorkerTicket {
|
||||
|
||||
SweepstoreFileHandle& file;
|
||||
uint32_t ticketIndex;
|
||||
|
||||
uint64_t getOffset() const {
|
||||
return SWEEPSTORE_COMBINED_STATIC_HEADER_SIZE + (ticketIndex * TICKET_SIZE);
|
||||
}
|
||||
|
||||
public:
|
||||
static constexpr int TICKET_SIZE = roundToNearest16(29);
|
||||
|
||||
SweepstoreWorkerTicket(const uint32_t index, SweepstoreFileHandle& fileStream) :
|
||||
file(fileStream),
|
||||
ticketIndex(index) {}
|
||||
|
||||
int getTicketIndex() const {
|
||||
return ticketIndex;
|
||||
}
|
||||
|
||||
void write(SweepstoreWorkerTicketSnapshot &snapshot);
|
||||
|
||||
bool writable();
|
||||
|
||||
SweepstoreWorkerTicketSnapshot snapshot();
|
||||
|
||||
};
|
||||
|
||||
class SweepstoreConcurrencyHeader {
|
||||
|
||||
private:
|
||||
SweepstoreFileHandle& file;
|
||||
|
||||
public:
|
||||
explicit SweepstoreConcurrencyHeader(SweepstoreFileHandle &fileStream) : file(fileStream) {}
|
||||
|
||||
// Offset 29 - 8 bytes
|
||||
uint64_t readMasterIdentifier();
|
||||
void writeMasterIdentifier(uint64_t identifier);
|
||||
|
||||
// Offset 37 - 4 bytes
|
||||
uint32_t readMasterHeartbeat();
|
||||
void writeMasterHeartbeat(uint32_t heartbeat);
|
||||
|
||||
// Offset 41 - 4 bytes
|
||||
uint32_t readNumberOfWorkers();
|
||||
void writeNumberOfWorkers(uint32_t numWorkers);
|
||||
|
||||
// Offset 45 - 1 byte
|
||||
bool readIsReadAllowed();
|
||||
void writeIsReadAllowed(bool isAllowed);
|
||||
|
||||
/**
|
||||
* Initialises the concurrency header with default values.
|
||||
*/
|
||||
void initialise(int concurrentWorkers);
|
||||
|
||||
SweepstoreWorkerTicket operator[](const uint32_t index) const {
|
||||
return SweepstoreWorkerTicket(index, file);
|
||||
}
|
||||
|
||||
};
|
||||
39
cpp/src/Public/sweepstore/structures.h
Normal file
39
cpp/src/Public/sweepstore/structures.h
Normal file
@@ -0,0 +1,39 @@
|
||||
#pragma once
|
||||
#include <cstdint>
|
||||
|
||||
class SweepstorePointer {
|
||||
|
||||
private:
|
||||
uint64_t address;
|
||||
|
||||
public:
|
||||
static const SweepstorePointer NULL_PTR;
|
||||
|
||||
SweepstorePointer(int64_t addr) : address(addr) {}
|
||||
|
||||
bool isNull() {
|
||||
return this->address == UINT64_MAX;
|
||||
}
|
||||
|
||||
bool operator==(const SweepstorePointer &p);
|
||||
|
||||
// Implicit conversion to uint64_t
|
||||
operator uint64_t() const {
|
||||
return address;
|
||||
}
|
||||
};
|
||||
|
||||
enum SweepstoreTicketState {
|
||||
FREE,
|
||||
WAITING,
|
||||
APPROVED,
|
||||
EXECUTING,
|
||||
COMPLETED,
|
||||
};
|
||||
|
||||
enum SweepstoreTicketOperation : int {
|
||||
NONE,
|
||||
READ,
|
||||
MODIFY,
|
||||
WRITE
|
||||
};
|
||||
67
cpp/src/Public/sweepstore/sweepstore.h
Normal file
67
cpp/src/Public/sweepstore/sweepstore.h
Normal file
@@ -0,0 +1,67 @@
|
||||
#pragma once
|
||||
|
||||
#include <iosfwd>
|
||||
|
||||
#include "concurrency.h"
|
||||
#include "header.h"
|
||||
#include "utils/helpers.h"
|
||||
#include "utils/file_handle.h"
|
||||
|
||||
class Sweepstore {
|
||||
|
||||
private:
|
||||
std::string filePath;
|
||||
SweepstoreFileHandle file;
|
||||
SweepstoreHeader* header;
|
||||
SweepstoreConcurrencyHeader* concurrencyHeader;
|
||||
|
||||
public:
|
||||
|
||||
Sweepstore(const std::string& filePath) : filePath(filePath), file(filePath, std::ios::binary | std::ios::in | std::ios::out | std::ios::trunc) {
|
||||
header = new SweepstoreHeader(file);
|
||||
concurrencyHeader = new SweepstoreConcurrencyHeader(file);
|
||||
}
|
||||
|
||||
~Sweepstore() {
|
||||
delete header;
|
||||
delete concurrencyHeader;
|
||||
file.close();
|
||||
}
|
||||
|
||||
void initialise(int concurrentWorkers = 4);
|
||||
|
||||
SweepstoreConcurrencyHeader* getConcurrencyHeader() {
|
||||
return concurrencyHeader;
|
||||
}
|
||||
|
||||
class Proxy {
|
||||
Sweepstore* sweepstore;
|
||||
std::string key;
|
||||
public:
|
||||
Proxy(Sweepstore* sweepstoreIn, const std::string& keyIn)
|
||||
: sweepstore(sweepstoreIn), key(keyIn) {}
|
||||
|
||||
template <typename T>
|
||||
operator T() {
|
||||
// Get value from sweepstore
|
||||
throw std::runtime_error("Not implemented: Reading values from Sweepstore by key is not implemented.");
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
void operator=(const T& value) {
|
||||
SweepstoreConcurrency::spawnTicket(&sweepstore->file,
|
||||
SweepstoreTicketOperation::WRITE,
|
||||
bt_hash(key),
|
||||
sizeof(T),
|
||||
[this, key = this->key, &value]() {
|
||||
|
||||
}
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
Proxy operator[](const std::string& key) {
|
||||
return Proxy(this, key);
|
||||
}
|
||||
|
||||
};
|
||||
113
cpp/src/Public/sweepstore/utils/file_handle.h
Normal file
113
cpp/src/Public/sweepstore/utils/file_handle.h
Normal file
@@ -0,0 +1,113 @@
|
||||
#pragma once
|
||||
|
||||
#include <fstream>
|
||||
#include <string>
|
||||
#include <memory>
|
||||
#include <iostream>
|
||||
#include <unordered_map>
|
||||
|
||||
#ifdef _WIN32
|
||||
#include <windows.h>
|
||||
#include <io.h>
|
||||
#else
|
||||
#include <unistd.h>
|
||||
#endif
|
||||
|
||||
#ifdef WITH_UNREAL
|
||||
#include "HAL/PlatformFileManager.h"
|
||||
#include "GenericPlatform/GenericPlatformFile.h"
|
||||
#endif
|
||||
|
||||
class SweepstoreFileHandle {
|
||||
private:
|
||||
std::string path;
|
||||
std::ios::openmode openMode;
|
||||
#ifdef WITH_UNREAL
|
||||
IFileHandle* unrealHandle;
|
||||
#else
|
||||
// Thread-local cache: each thread gets its own fstream per path
|
||||
static thread_local std::unordered_map<std::string, std::unique_ptr<std::fstream>> streamCache;
|
||||
|
||||
// Get or create the fstream for this thread
|
||||
std::fstream& getThreadStream();
|
||||
const std::fstream& getThreadStream() const;
|
||||
#endif
|
||||
|
||||
public:
|
||||
SweepstoreFileHandle(const std::string& p, std::ios::openmode mode = std::ios::in | std::ios::out | std::ios::binary);
|
||||
|
||||
const std::string& getPath() const { return path; }
|
||||
|
||||
#ifndef WITH_UNREAL
|
||||
std::fstream& getStream() { return getThreadStream(); }
|
||||
const std::fstream& getStream() const { return getThreadStream(); }
|
||||
|
||||
// Smart pointer-like interface
|
||||
std::fstream* operator->() { return &getThreadStream(); }
|
||||
const std::fstream* operator->() const { return &getThreadStream(); }
|
||||
|
||||
std::fstream& operator*() { return getThreadStream(); }
|
||||
const std::fstream& operator*() const { return getThreadStream(); }
|
||||
#endif
|
||||
|
||||
bool isOpen() const;
|
||||
void close();
|
||||
|
||||
// Windows-compatible I/O wrappers
|
||||
#if defined(_WIN32) || defined(WITH_UNREAL)
|
||||
void flush();
|
||||
void readSeek(std::streampos pos, std::ios::seekdir dir = std::ios::beg);
|
||||
void writeSeek(std::streampos pos, std::ios::seekdir dir = std::ios::beg);
|
||||
void readBytes(char* buffer, std::streamsize size);
|
||||
void writeBytes(const char* buffer, std::streamsize size);
|
||||
#else
|
||||
// Inline for non-Windows to avoid overhead
|
||||
inline void flush() {
|
||||
getThreadStream().flush();
|
||||
}
|
||||
inline void readSeek(std::streampos pos, std::ios::seekdir dir = std::ios::beg) {
|
||||
getThreadStream().seekg(pos, dir);
|
||||
}
|
||||
inline void writeSeek(std::streampos pos, std::ios::seekdir dir = std::ios::beg) {
|
||||
getThreadStream().seekp(pos, dir);
|
||||
}
|
||||
inline void readBytes(char* buffer, std::streamsize size) {
|
||||
getThreadStream().read(buffer, size);
|
||||
}
|
||||
inline void writeBytes(const char* buffer, std::streamsize size) {
|
||||
getThreadStream().write(buffer, size);
|
||||
}
|
||||
#endif
|
||||
|
||||
SweepstoreFileHandle(SweepstoreFileHandle&& other) noexcept
|
||||
: path(std::move(other.path))
|
||||
, openMode(other.openMode)
|
||||
#ifdef WITH_UNREAL
|
||||
, unrealHandle(other.unrealHandle)
|
||||
#endif
|
||||
{
|
||||
#ifdef WITH_UNREAL
|
||||
other.unrealHandle = nullptr;
|
||||
#endif
|
||||
}
|
||||
|
||||
SweepstoreFileHandle& operator=(SweepstoreFileHandle&& other) noexcept {
|
||||
if (this != &other) {
|
||||
close();
|
||||
path = std::move(other.path);
|
||||
openMode = other.openMode;
|
||||
#ifdef WITH_UNREAL
|
||||
unrealHandle = other.unrealHandle;
|
||||
other.unrealHandle = nullptr;
|
||||
#endif
|
||||
}
|
||||
return *this;
|
||||
}
|
||||
|
||||
SweepstoreFileHandle(const SweepstoreFileHandle&) = delete;
|
||||
SweepstoreFileHandle& operator=(const SweepstoreFileHandle&) = delete;
|
||||
|
||||
~SweepstoreFileHandle() {
|
||||
close();
|
||||
}
|
||||
};
|
||||
221
cpp/src/Public/sweepstore/utils/file_lock.h
Normal file
221
cpp/src/Public/sweepstore/utils/file_lock.h
Normal file
@@ -0,0 +1,221 @@
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
#include <cstdint>
|
||||
#include <unordered_map>
|
||||
#include <stdexcept>
|
||||
|
||||
#ifdef _WIN32
|
||||
#include <windows.h>
|
||||
#else
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
#include <sys/file.h>
|
||||
#endif
|
||||
|
||||
// Simple file lock using flock() with thread-local FD cache
|
||||
// Each thread has its own FD, flock() is per-FD, so threads don't conflict
|
||||
// Matches Dart's paradigm: each isolate has its own RandomAccessFile
|
||||
class SweepstoreFileLock {
|
||||
public:
|
||||
enum class Mode { Shared, Exclusive };
|
||||
|
||||
private:
|
||||
std::string filePath;
|
||||
uint64_t offset;
|
||||
uint64_t length;
|
||||
Mode mode;
|
||||
bool locked = false;
|
||||
|
||||
#ifdef _WIN32
|
||||
// Thread-local HANDLE cache for Windows
|
||||
static thread_local std::unordered_map<std::string, HANDLE> handleCache;
|
||||
|
||||
static HANDLE getOrOpenHandle(const std::string& path) {
|
||||
auto it = handleCache.find(path);
|
||||
if (it != handleCache.end()) {
|
||||
return it->second;
|
||||
}
|
||||
|
||||
HANDLE handle = CreateFileA(
|
||||
path.c_str(),
|
||||
GENERIC_READ | GENERIC_WRITE,
|
||||
FILE_SHARE_READ | FILE_SHARE_WRITE,
|
||||
NULL,
|
||||
OPEN_EXISTING,
|
||||
FILE_ATTRIBUTE_NORMAL,
|
||||
NULL
|
||||
);
|
||||
|
||||
if (handle == INVALID_HANDLE_VALUE) {
|
||||
throw std::runtime_error("Failed to open file for locking: " + path);
|
||||
}
|
||||
|
||||
handleCache[path] = handle;
|
||||
return handle;
|
||||
}
|
||||
|
||||
void acquire() {
|
||||
HANDLE handle = getOrOpenHandle(filePath);
|
||||
OVERLAPPED overlapped = {}; // Proper zero-initialization
|
||||
overlapped.Offset = static_cast<DWORD>(offset & 0xFFFFFFFF);
|
||||
overlapped.OffsetHigh = static_cast<DWORD>(offset >> 32);
|
||||
|
||||
DWORD length_low = static_cast<DWORD>(length & 0xFFFFFFFF);
|
||||
DWORD length_high = static_cast<DWORD>(length >> 32);
|
||||
DWORD flags = (mode == Mode::Exclusive) ? LOCKFILE_EXCLUSIVE_LOCK : 0;
|
||||
|
||||
if (!LockFileEx(handle, flags, 0, length_low, length_high, &overlapped)) {
|
||||
throw std::runtime_error("Failed to acquire file lock");
|
||||
}
|
||||
locked = true;
|
||||
}
|
||||
|
||||
void release() {
|
||||
if (locked) {
|
||||
HANDLE handle = getOrOpenHandle(filePath);
|
||||
OVERLAPPED overlapped = {};
|
||||
overlapped.Offset = static_cast<DWORD>(offset & 0xFFFFFFFF);
|
||||
overlapped.OffsetHigh = static_cast<DWORD>(offset >> 32);
|
||||
|
||||
DWORD length_low = static_cast<DWORD>(length & 0xFFFFFFFF);
|
||||
DWORD length_high = static_cast<DWORD>(length >> 32);
|
||||
|
||||
UnlockFileEx(handle, 0, length_low, length_high, &overlapped);
|
||||
locked = false;
|
||||
}
|
||||
}
|
||||
#else
|
||||
// Thread-local FD cache - each thread has its own FD per file
|
||||
static thread_local std::unordered_map<std::string, int> fdCache;
|
||||
|
||||
static int getOrOpenFD(const std::string& path) {
|
||||
auto it = fdCache.find(path);
|
||||
if (it != fdCache.end()) {
|
||||
return it->second;
|
||||
}
|
||||
|
||||
int fd = open(path.c_str(), O_RDWR);
|
||||
if (fd == -1) {
|
||||
throw std::runtime_error("Failed to open file for locking: " + path);
|
||||
}
|
||||
|
||||
fdCache[path] = fd;
|
||||
return fd;
|
||||
}
|
||||
|
||||
void acquire() {
|
||||
int fd = getOrOpenFD(filePath);
|
||||
|
||||
struct flock lock_info;
|
||||
lock_info.l_type = (mode == Mode::Exclusive) ? F_WRLCK : F_RDLCK;
|
||||
lock_info.l_whence = SEEK_SET;
|
||||
lock_info.l_start = offset;
|
||||
lock_info.l_len = length;
|
||||
lock_info.l_pid = 0;
|
||||
|
||||
if (fcntl(fd, F_SETLKW, &lock_info) == -1) {
|
||||
throw std::runtime_error("Failed to acquire file lock");
|
||||
}
|
||||
locked = true;
|
||||
}
|
||||
|
||||
void release() {
|
||||
if (locked) {
|
||||
int fd = getOrOpenFD(filePath);
|
||||
|
||||
struct flock lock_info;
|
||||
lock_info.l_type = F_UNLCK;
|
||||
lock_info.l_whence = SEEK_SET;
|
||||
lock_info.l_start = offset;
|
||||
lock_info.l_len = length;
|
||||
lock_info.l_pid = 0;
|
||||
|
||||
fcntl(fd, F_SETLK, &lock_info);
|
||||
locked = false;
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
public:
|
||||
// Constructor accepts offset/length for byte-range locking
|
||||
SweepstoreFileLock(const std::string& path, uint64_t off, uint64_t len, Mode m)
|
||||
: filePath(path), offset(off), length(len), mode(m) {}
|
||||
|
||||
~SweepstoreFileLock() { release(); }
|
||||
|
||||
void lock() {
|
||||
if (!locked) acquire();
|
||||
}
|
||||
|
||||
void unlock() {
|
||||
release();
|
||||
}
|
||||
|
||||
bool holdsLock() const {
|
||||
return locked;
|
||||
}
|
||||
|
||||
// Check if file is currently locked (non-blocking test)
|
||||
bool isLocked() {
|
||||
#ifdef _WIN32
|
||||
HANDLE handle = getOrOpenHandle(filePath);
|
||||
OVERLAPPED overlapped = {};
|
||||
overlapped.Offset = static_cast<DWORD>(offset & 0xFFFFFFFF);
|
||||
overlapped.OffsetHigh = static_cast<DWORD>(offset >> 32);
|
||||
|
||||
DWORD length_low = static_cast<DWORD>(length & 0xFFFFFFFF);
|
||||
DWORD length_high = static_cast<DWORD>(length >> 32);
|
||||
DWORD flags = (mode == Mode::Exclusive) ? LOCKFILE_EXCLUSIVE_LOCK : 0;
|
||||
flags |= LOCKFILE_FAIL_IMMEDIATELY;
|
||||
|
||||
// Try non-blocking lock
|
||||
if (!LockFileEx(handle, flags, 0, length_low, length_high, &overlapped)) {
|
||||
return true; // Already locked
|
||||
}
|
||||
|
||||
// Got the lock, release immediately
|
||||
UnlockFileEx(handle, 0, length_low, length_high, &overlapped);
|
||||
return false;
|
||||
#else
|
||||
int fd = getOrOpenFD(filePath);
|
||||
|
||||
struct flock lock_info;
|
||||
lock_info.l_type = (mode == Mode::Exclusive) ? F_WRLCK : F_RDLCK;
|
||||
lock_info.l_whence = SEEK_SET;
|
||||
lock_info.l_start = offset;
|
||||
lock_info.l_len = length;
|
||||
lock_info.l_pid = 0;
|
||||
|
||||
// Try non-blocking lock
|
||||
if (fcntl(fd, F_SETLK, &lock_info) == -1) {
|
||||
return true; // Already locked
|
||||
}
|
||||
|
||||
// Got the lock, release immediately
|
||||
lock_info.l_type = F_UNLCK;
|
||||
fcntl(fd, F_SETLK, &lock_info);
|
||||
return false;
|
||||
#endif
|
||||
}
|
||||
|
||||
// RAII helper for scoped locking
|
||||
class Scoped {
|
||||
SweepstoreFileLock& lock;
|
||||
public:
|
||||
Scoped(SweepstoreFileLock& l) : lock(l) {
|
||||
lock.lock();
|
||||
}
|
||||
|
||||
~Scoped() {
|
||||
lock.unlock();
|
||||
}
|
||||
|
||||
Scoped(const Scoped&) = delete;
|
||||
Scoped& operator=(const Scoped&) = delete;
|
||||
};
|
||||
|
||||
// Disable copying
|
||||
SweepstoreFileLock(const SweepstoreFileLock&) = delete;
|
||||
SweepstoreFileLock& operator=(const SweepstoreFileLock&) = delete;
|
||||
};
|
||||
383
cpp/src/Public/sweepstore/utils/helpers.h
Normal file
383
cpp/src/Public/sweepstore/utils/helpers.h
Normal file
@@ -0,0 +1,383 @@
|
||||
#pragma once
|
||||
|
||||
#include <iostream>
|
||||
#include <string>
|
||||
#include <sstream>
|
||||
#include <iomanip>
|
||||
#include <vector>
|
||||
#include <cstdint>
|
||||
#include <fstream>
|
||||
#include <chrono>
|
||||
#include <thread>
|
||||
#include <cstring>
|
||||
#include <algorithm>
|
||||
|
||||
// Toggleable debug printing via preprocessor
|
||||
#ifndef SWEEPSTORE_DEBUG
|
||||
#define SWEEPSTORE_DEBUG 0
|
||||
#endif
|
||||
|
||||
#if SWEEPSTORE_DEBUG
|
||||
#define debugPrint(msg) std::cout << msg << std::endl
|
||||
#else
|
||||
#define debugPrint(msg) ((void)0)
|
||||
#endif
|
||||
|
||||
|
||||
inline void print(const char* message) {
|
||||
// Print the message to the console
|
||||
std::cout << message << std::endl;
|
||||
}
|
||||
|
||||
inline std::string trim(const std::string& str) {
|
||||
size_t start = str.find_first_not_of(" \t\n\r");
|
||||
if (start == std::string::npos) return ""; // all whitespace
|
||||
|
||||
size_t end = str.find_last_not_of(" \t\n\r");
|
||||
return str.substr(start, end - start + 1);
|
||||
}
|
||||
|
||||
inline std::string binaryDump(const std::vector<uint8_t>& data) {
|
||||
std::ostringstream buffer;
|
||||
|
||||
for (size_t i = 0; i < data.size(); i += 16) {
|
||||
// Address
|
||||
buffer << "0x"
|
||||
<< std::setfill('0') << std::setw(4) << std::uppercase << std::hex << i
|
||||
<< " (" << std::dec << std::setw(4) << i << ") | ";
|
||||
|
||||
// Hex bytes
|
||||
for (size_t j = 0; j < 16; j++) {
|
||||
if (i + j < data.size()) {
|
||||
buffer << std::setfill('0') << std::setw(2) << std::uppercase << std::hex
|
||||
<< static_cast<int>(data[i + j]) << " ";
|
||||
} else {
|
||||
buffer << " ";
|
||||
}
|
||||
}
|
||||
|
||||
buffer << " | ";
|
||||
|
||||
// Integer representation
|
||||
for (size_t j = 0; j < 16; j++) {
|
||||
if (i + j < data.size()) {
|
||||
buffer << std::dec << std::setw(3) << static_cast<int>(data[i + j]) << " ";
|
||||
} else {
|
||||
buffer << " ";
|
||||
}
|
||||
}
|
||||
|
||||
buffer << " | ";
|
||||
|
||||
// ASCII representation
|
||||
for (size_t j = 0; j < 16; j++) {
|
||||
if (i + j < data.size()) {
|
||||
uint8_t byte = data[i + j];
|
||||
if (byte >= 32 && byte <= 126) {
|
||||
buffer << static_cast<char>(byte);
|
||||
} else {
|
||||
buffer << '.';
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
buffer << " | ";
|
||||
if (i + 16 < data.size()) buffer << '\n';
|
||||
}
|
||||
|
||||
return buffer.str();
|
||||
}
|
||||
|
||||
inline std::vector<uint8_t> loadFile(const std::string& filename) {
|
||||
std::ifstream file(filename, std::ios::binary | std::ios::ate);
|
||||
|
||||
if (!file) {
|
||||
throw std::runtime_error("Failed to open file: " + filename);
|
||||
}
|
||||
|
||||
// Get file size
|
||||
std::streamsize size = file.tellg();
|
||||
file.seekg(0, std::ios::beg);
|
||||
|
||||
// Pre-allocate vector and read
|
||||
std::vector<uint8_t> buffer(size);
|
||||
if (!file.read(reinterpret_cast<char*>(buffer.data()), size)) {
|
||||
throw std::runtime_error("Failed to read file: " + filename);
|
||||
}
|
||||
|
||||
return buffer;
|
||||
}
|
||||
|
||||
enum class Endian {
|
||||
Little,
|
||||
Big
|
||||
};
|
||||
|
||||
class RandomAccessMemory {
|
||||
private:
|
||||
std::vector<uint8_t> _buffer;
|
||||
size_t _position;
|
||||
|
||||
public:
|
||||
// Constructors
|
||||
RandomAccessMemory() : _position(0) {}
|
||||
|
||||
explicit RandomAccessMemory(const std::vector<uint8_t>& initialData)
|
||||
: _buffer(initialData), _position(0) {}
|
||||
|
||||
explicit RandomAccessMemory(const uint8_t* data, size_t size)
|
||||
: _buffer(data, data + size), _position(0) {}
|
||||
|
||||
// Position management
|
||||
size_t positionSync() const {
|
||||
return _position;
|
||||
}
|
||||
|
||||
void setPositionSync(size_t position) {
|
||||
_position = position;
|
||||
}
|
||||
|
||||
size_t length() const {
|
||||
return _buffer.size();
|
||||
}
|
||||
|
||||
// Read bytes
|
||||
std::vector<uint8_t> readSync(size_t count) {
|
||||
if (_position + count > _buffer.size()) {
|
||||
throw std::range_error("Not enough bytes to read");
|
||||
}
|
||||
|
||||
std::vector<uint8_t> result(_buffer.begin() + _position,
|
||||
_buffer.begin() + _position + count);
|
||||
_position += count;
|
||||
return result;
|
||||
}
|
||||
|
||||
// Write bytes
|
||||
void writeFromSync(const std::vector<uint8_t>& bytes) {
|
||||
for (size_t i = 0; i < bytes.size(); i++) {
|
||||
if (_position + i >= _buffer.size()) {
|
||||
_buffer.push_back(bytes[i]);
|
||||
} else {
|
||||
_buffer[_position + i] = bytes[i];
|
||||
}
|
||||
}
|
||||
_position += bytes.size();
|
||||
}
|
||||
|
||||
// Read/Write Int Dynamic
|
||||
int64_t readIntSync(int size = 4, Endian endianness = Endian::Little) {
|
||||
if (size < 1 || size > 8) {
|
||||
throw std::invalid_argument("Size must be between 1 and 8 bytes");
|
||||
}
|
||||
|
||||
std::vector<uint8_t> bytes = readSync(size);
|
||||
|
||||
// Build integer from bytes with proper endianness
|
||||
int64_t result = 0;
|
||||
if (endianness == Endian::Little) {
|
||||
for (int i = size - 1; i >= 0; i--) {
|
||||
result = (result << 8) | bytes[i];
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < size; i++) {
|
||||
result = (result << 8) | bytes[i];
|
||||
}
|
||||
}
|
||||
|
||||
// Sign extend if MSB is set
|
||||
int64_t signBit = 1LL << (size * 8 - 1);
|
||||
if (result & signBit) {
|
||||
result -= 1LL << (size * 8);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
uint64_t readUIntSync(int size = 4, Endian endianness = Endian::Little) {
|
||||
if (size < 1 || size > 8) {
|
||||
throw std::invalid_argument("Size must be between 1 and 8 bytes");
|
||||
}
|
||||
|
||||
std::vector<uint8_t> bytes = readSync(size);
|
||||
|
||||
// Build integer from bytes with proper endianness
|
||||
uint64_t result = 0;
|
||||
if (endianness == Endian::Little) {
|
||||
for (int i = size - 1; i >= 0; i--) {
|
||||
result = (result << 8) | bytes[i];
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < size; i++) {
|
||||
result = (result << 8) | bytes[i];
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
void writeIntSync(int64_t value, int size = 4, Endian endianness = Endian::Little) {
|
||||
if (size < 1 || size > 8) {
|
||||
throw std::invalid_argument("Size must be between 1 and 8 bytes");
|
||||
}
|
||||
|
||||
std::vector<uint8_t> bytes(size, 0);
|
||||
|
||||
// Extract bytes with proper endianness
|
||||
if (endianness == Endian::Little) {
|
||||
for (int i = 0; i < size; i++) {
|
||||
bytes[i] = (value >> (i * 8)) & 0xFF;
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < size; i++) {
|
||||
bytes[size - 1 - i] = (value >> (i * 8)) & 0xFF;
|
||||
}
|
||||
}
|
||||
|
||||
writeFromSync(bytes);
|
||||
}
|
||||
void writeUIntSync(uint64_t value, int size = 4, Endian endianness = Endian::Little) {
|
||||
if (size < 1 || size > 8) {
|
||||
throw std::invalid_argument("Size must be between 1 and 8 bytes");
|
||||
}
|
||||
|
||||
std::vector<uint8_t> bytes(size, 0);
|
||||
|
||||
// Extract bytes with proper endianness
|
||||
if (endianness == Endian::Little) {
|
||||
for (int i = 0; i < size; i++) {
|
||||
bytes[i] = (value >> (i * 8)) & 0xFF;
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < size; i++) {
|
||||
bytes[size - 1 - i] = (value >> (i * 8)) & 0xFF;
|
||||
}
|
||||
}
|
||||
|
||||
writeFromSync(bytes);
|
||||
}
|
||||
|
||||
// Read/Write Pointers (assuming POINTER size is 8 bytes)
|
||||
SweepstorePointer readPointerSync(int pointerSize = 8) {
|
||||
int64_t offset = readUIntSync(pointerSize);
|
||||
return SweepstorePointer(offset);
|
||||
}
|
||||
|
||||
void writePointerSync(const SweepstorePointer& pointer, int pointerSize = 8) {
|
||||
writeUIntSync(pointer, pointerSize);
|
||||
}
|
||||
|
||||
// Read/Write Float32
|
||||
float readFloat32Sync(Endian endianness = Endian::Little) {
|
||||
std::vector<uint8_t> bytes = readSync(4);
|
||||
float value;
|
||||
|
||||
if (endianness == Endian::Little) {
|
||||
std::memcpy(&value, bytes.data(), 4);
|
||||
} else {
|
||||
std::vector<uint8_t> reversed(bytes.rbegin(), bytes.rend());
|
||||
std::memcpy(&value, reversed.data(), 4);
|
||||
}
|
||||
|
||||
return value;
|
||||
}
|
||||
|
||||
void writeFloat32Sync(float value, Endian endianness = Endian::Little) {
|
||||
std::vector<uint8_t> bytes(4);
|
||||
std::memcpy(bytes.data(), &value, 4);
|
||||
|
||||
if (endianness == Endian::Big) {
|
||||
std::reverse(bytes.begin(), bytes.end());
|
||||
}
|
||||
|
||||
writeFromSync(bytes);
|
||||
}
|
||||
|
||||
// Read/Write Float64 (Double)
|
||||
double readFloat64Sync(Endian endianness = Endian::Little) {
|
||||
std::vector<uint8_t> bytes = readSync(8);
|
||||
double value;
|
||||
|
||||
if (endianness == Endian::Little) {
|
||||
std::memcpy(&value, bytes.data(), 8);
|
||||
} else {
|
||||
std::vector<uint8_t> reversed(bytes.rbegin(), bytes.rend());
|
||||
std::memcpy(&value, reversed.data(), 8);
|
||||
}
|
||||
|
||||
return value;
|
||||
}
|
||||
|
||||
void writeFloat64Sync(double value, Endian endianness = Endian::Little) {
|
||||
std::vector<uint8_t> bytes(8);
|
||||
std::memcpy(bytes.data(), &value, 8);
|
||||
|
||||
if (endianness == Endian::Big) {
|
||||
std::reverse(bytes.begin(), bytes.end());
|
||||
}
|
||||
|
||||
writeFromSync(bytes);
|
||||
}
|
||||
|
||||
// Conversion methods
|
||||
std::vector<uint8_t> toVector() const {
|
||||
return _buffer;
|
||||
}
|
||||
|
||||
const uint8_t* data() const {
|
||||
return _buffer.data();
|
||||
}
|
||||
|
||||
uint8_t* data() {
|
||||
return _buffer.data();
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
#ifdef _WIN32
|
||||
#include <windows.h>
|
||||
#endif
|
||||
|
||||
inline void preciseSleep(std::chrono::nanoseconds duration) {
|
||||
auto start = std::chrono::high_resolution_clock::now();
|
||||
|
||||
#ifdef _WIN32
|
||||
const auto windowsMinSleepTime = std::chrono::milliseconds(1);
|
||||
|
||||
if (duration < windowsMinSleepTime) {
|
||||
// Pure busy-wait with high-res timer
|
||||
while (std::chrono::high_resolution_clock::now() - start < duration) {
|
||||
// Optionally use _mm_pause() or YieldProcessor() to be nicer to hyperthreading
|
||||
}
|
||||
} else {
|
||||
// Hybrid: sleep most of it, busy-wait the remainder
|
||||
auto sleepDuration = duration - windowsMinSleepTime;
|
||||
std::this_thread::sleep_for(sleepDuration);
|
||||
while (std::chrono::high_resolution_clock::now() - start < duration) {}
|
||||
}
|
||||
#else
|
||||
std::this_thread::sleep_for(duration);
|
||||
#endif
|
||||
}
|
||||
|
||||
inline int32_t millisecondsSinceEpoch32() {
|
||||
auto now = std::chrono::system_clock::now();
|
||||
auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
|
||||
return static_cast<int32_t>((millis / 1000) & 0xFFFFFFFF);
|
||||
}
|
||||
|
||||
inline int64_t millisecondsSinceEpoch64() {
|
||||
auto now = std::chrono::system_clock::now();
|
||||
auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
|
||||
return millis;
|
||||
}
|
||||
|
||||
inline uint64_t bt_hash(const std::string& str) {
|
||||
uint64_t hash = 0xcbf29ce484222325ULL; // FNV offset basis
|
||||
|
||||
for (unsigned char byte : str) {
|
||||
hash ^= byte;
|
||||
hash *= 0x100000001b3ULL; // FNV prime
|
||||
}
|
||||
|
||||
return hash;
|
||||
}
|
||||
2
dart/CHANGELOG.md
Normal file
2
dart/CHANGELOG.md
Normal file
@@ -0,0 +1,2 @@
|
||||
|
||||
1.0.0 Initial Release
|
||||
232
dart/lib/concurrency.dart
Normal file
232
dart/lib/concurrency.dart
Normal file
@@ -0,0 +1,232 @@
|
||||
|
||||
|
||||
import 'dart:io';
|
||||
import 'dart:isolate';
|
||||
import 'dart:math';
|
||||
|
||||
import 'package:sweepstore/header.dart';
|
||||
import 'package:sweepstore/helpers.dart';
|
||||
import 'package:sweepstore/structures.dart';
|
||||
|
||||
// Stale Heartbeat threshold in milliseconds
|
||||
const int STALE_HEARTBEAT_THRESHOLD_MS = 5000; // 5 seconds
|
||||
|
||||
int _randomId() {
|
||||
// mix timestamp with random for better uniquness
|
||||
// keep it positive to avoid signed int issues when storing
|
||||
int time = DateTime.now().millisecondsSinceEpoch32();
|
||||
int random = Random().nextInt(0x80000000);
|
||||
return (time ^ random) & 0x7FFFFFFF;
|
||||
}
|
||||
|
||||
// Spawn a ticket for a worker to perform an operation
|
||||
void spawnTicket(RandomAccessFile file, {
|
||||
required SweepstoreTicketOperation operation,
|
||||
required int keyHash,
|
||||
required int writeSize,
|
||||
required void Function() onApproved,
|
||||
String? debugLabel,
|
||||
}) {
|
||||
|
||||
/*
|
||||
Useful Functions
|
||||
*/
|
||||
|
||||
/// Logging function
|
||||
void log(String message) {
|
||||
|
||||
String prefix = debugLabel != null ? "\x1B[38;5;208m[Ticket Spawner - $debugLabel]:\x1B[0m " : "\x1B[38;5;208m[Ticket Spawner]:\x1B[0m ";
|
||||
print("$prefix$message");
|
||||
|
||||
}
|
||||
|
||||
/// Sleep a bit - with variance - mainly used for heartbeats
|
||||
void tickSleep([int microsecondVariance = 10]) {
|
||||
preciseSleep(Duration(microseconds: 500 + Random().nextInt(microsecondVariance)));
|
||||
}
|
||||
|
||||
/// Exponential sleep function
|
||||
Map<String, int> expSleepTracker = {};
|
||||
void expSleep(String label) {
|
||||
int count = expSleepTracker[label] ?? 0;
|
||||
int sleepTime = (1 << count); // Exponential backoff
|
||||
// sleepTime = max(1, min(sleepTime, 1000)); // Clamp between 1ms and 1000ms
|
||||
preciseSleep(Duration(microseconds: sleepTime * 1000));
|
||||
expSleepTracker[label] = count + 1;
|
||||
}
|
||||
|
||||
// Get the header
|
||||
SweepstoreHeader header = SweepstoreHeader(file);
|
||||
SweepstoreConcurrencyHeader concurrencyHeader = SweepstoreConcurrencyHeader(header);
|
||||
|
||||
/*
|
||||
Ticket Acquisition
|
||||
*/
|
||||
SweepstoreWorkerTicket acquireTicket(int newIdentifier) {
|
||||
int? ticketIndex;
|
||||
|
||||
while (true) {
|
||||
|
||||
for (int i = 0; i < concurrencyHeader.numberOfWorkers; i++) {
|
||||
|
||||
SweepstoreWorkerTicket ticket = concurrencyHeader[i];
|
||||
|
||||
if (!ticket.writable()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
SweepstoreWorkerTicketSnapshot ticketSnapshot = ticket.snapshot();
|
||||
|
||||
int identifier = ticketSnapshot.identifier;
|
||||
|
||||
bool identifier_unassigned = identifier == 0;
|
||||
bool stale_heartbeat = (DateTime.now().millisecondsSinceEpoch32() - ticketSnapshot.workerHeartbeat) > STALE_HEARTBEAT_THRESHOLD_MS;
|
||||
bool is_free = ticketSnapshot.ticketState == SweepstoreTicketState.FREE;
|
||||
|
||||
if (identifier_unassigned && stale_heartbeat && is_free) {
|
||||
ticket.write(
|
||||
identifier: newIdentifier,
|
||||
workerHeartbeat: DateTime.now().millisecondsSinceEpoch32(),
|
||||
ticketState: SweepstoreTicketState.WAITING,
|
||||
);
|
||||
ticketIndex = i;
|
||||
// log("Acquired ticket $ticketIndex with identifier $newIdentifier.");
|
||||
break;
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
preciseSleep(Duration(milliseconds: 2));
|
||||
|
||||
// Ensure we still own the ticket - if not, reset and try again
|
||||
if (ticketIndex != null) {
|
||||
SweepstoreWorkerTicketSnapshot verifySnapshot = concurrencyHeader[ticketIndex].snapshot();
|
||||
|
||||
if (verifySnapshot.identifier != newIdentifier) {
|
||||
// log("Lost ticket $ticketIndex, retrying...");
|
||||
ticketIndex = null;
|
||||
} else {
|
||||
return concurrencyHeader[ticketIndex];
|
||||
}
|
||||
}
|
||||
|
||||
expSleep("acquire_loop");
|
||||
}
|
||||
|
||||
throw Exception("Failed to acquire ticket.");
|
||||
}
|
||||
|
||||
// Reduce the chance of race conditions by adding a small random delay
|
||||
tickSleep(500);
|
||||
|
||||
int myIdentifier = _randomId();
|
||||
|
||||
// We have a ticket, set it up
|
||||
SweepstoreWorkerTicket myTicket = acquireTicket(myIdentifier);
|
||||
myTicket.write(
|
||||
workerHeartbeat: DateTime.now().millisecondsSinceEpoch32(),
|
||||
ticketState: SweepstoreTicketState.WAITING,
|
||||
ticketOperation: operation,
|
||||
keyHash: keyHash,
|
||||
writeSize: writeSize,
|
||||
);
|
||||
|
||||
// Wait for approval - (Approval loop)
|
||||
while (true) {
|
||||
|
||||
SweepstoreWorkerTicketSnapshot snapshot = myTicket.snapshot();
|
||||
|
||||
// Check we still own the ticket
|
||||
if (snapshot.identifier != myIdentifier) {
|
||||
|
||||
preciseSleep(Duration(milliseconds: 10));
|
||||
|
||||
// Re-verify we lost the ticket
|
||||
SweepstoreWorkerTicketSnapshot recheckSnapshot = myTicket.snapshot();
|
||||
if (recheckSnapshot.identifier != myIdentifier) {
|
||||
String exceptionMessage = "CRITICAL: Lost ownership of ticket ${myTicket.ticketIndex}, was expecting identifier $myIdentifier but found ${snapshot.identifier}.";
|
||||
// throw Exception(exceptionMessage);
|
||||
log(exceptionMessage);
|
||||
return spawnTicket(file, operation: operation, keyHash: keyHash, writeSize: writeSize, onApproved: onApproved);
|
||||
}
|
||||
|
||||
// Nvm, false alarm
|
||||
log("False alarm, still own ticket ${myTicket.ticketIndex}.");
|
||||
}
|
||||
|
||||
if (snapshot.ticketState == SweepstoreTicketState.APPROVED) {
|
||||
myTicket.write(
|
||||
ticketState: SweepstoreTicketState.EXECUTING,
|
||||
);
|
||||
onApproved();
|
||||
myTicket.write(
|
||||
ticketState: SweepstoreTicketState.COMPLETED,
|
||||
);
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
// randomSleep(10);
|
||||
tickSleep();
|
||||
|
||||
// Update heartbeat
|
||||
int now = DateTime.now().millisecondsSinceEpoch32();
|
||||
if (now - snapshot.workerHeartbeat > 700) {
|
||||
myTicket.write(
|
||||
workerHeartbeat: now
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Master side
|
||||
void initialiseMasterListener(RandomAccessFile file) async {
|
||||
String filePath = file.path;
|
||||
Isolate.spawn((_) {
|
||||
|
||||
void log(String message) {
|
||||
// print("\x1B[38;5;82m[Master Listener]:\x1B[0m $message");
|
||||
}
|
||||
|
||||
RandomAccessFile file = File(filePath).openSync(mode: FileMode.append);
|
||||
|
||||
SweepstoreHeader header = SweepstoreHeader(file);
|
||||
SweepstoreConcurrencyHeader concurrencyHeader = SweepstoreConcurrencyHeader(header);
|
||||
|
||||
while (true) {
|
||||
|
||||
for (int i = 0; i < concurrencyHeader.numberOfWorkers; i++) {
|
||||
|
||||
SweepstoreWorkerTicket ticket = concurrencyHeader[i];
|
||||
SweepstoreWorkerTicketSnapshot snapshot = ticket.snapshot();
|
||||
|
||||
if (snapshot.ticketState == SweepstoreTicketState.WAITING) {
|
||||
log("Found waiting ticket $i (Key Hash: ${snapshot.keyHash})...");
|
||||
|
||||
// Approve the ticket
|
||||
ticket.write(
|
||||
ticketState: SweepstoreTicketState.APPROVED,
|
||||
);
|
||||
log("Approved ticket $i.");
|
||||
} else if (snapshot.ticketState == SweepstoreTicketState.COMPLETED) {
|
||||
log("Ticket $i completed. Resetting ticket...");
|
||||
// Reset the ticket
|
||||
ticket.write(
|
||||
identifier: 0,
|
||||
workerHeartbeat: 0,
|
||||
ticketState: SweepstoreTicketState.FREE,
|
||||
ticketOperation: SweepstoreTicketOperation.NONE,
|
||||
keyHash: 0,
|
||||
writeSize: 0,
|
||||
);
|
||||
log("Reset ticket $i.");
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
preciseSleep(Duration(milliseconds: 1));
|
||||
|
||||
}
|
||||
}, null);
|
||||
}
|
||||
50
dart/lib/debug.dart
Normal file
50
dart/lib/debug.dart
Normal file
@@ -0,0 +1,50 @@
|
||||
|
||||
import 'dart:typed_data';
|
||||
|
||||
String binaryDump(Uint8List data) {
|
||||
StringBuffer buffer = StringBuffer();
|
||||
|
||||
for (int i = 0; i < data.length; i += 16) {
|
||||
// Address
|
||||
buffer.write('0x${i.toRadixString(16).padLeft(4, '0').toUpperCase()} (${i.toString().padLeft(4)}) | ');
|
||||
|
||||
// Hex bytes
|
||||
for (int j = 0; j < 16; j++) {
|
||||
if (i + j < data.length) {
|
||||
buffer.write('${data[i + j].toRadixString(16).padLeft(2, '0').toUpperCase()} ');
|
||||
} else {
|
||||
buffer.write(' ');
|
||||
}
|
||||
}
|
||||
|
||||
buffer.write(' | ');
|
||||
|
||||
// Integer representation
|
||||
for (int j = 0; j < 16; j++) {
|
||||
if (i + j < data.length) {
|
||||
buffer.write('${data[i + j].toString().padLeft(3)} ');
|
||||
} else {
|
||||
buffer.write(' ');
|
||||
}
|
||||
}
|
||||
|
||||
buffer.write(' | ');
|
||||
|
||||
// ASCII representation
|
||||
for (int j = 0; j < 16; j++) {
|
||||
if (i + j < data.length) {
|
||||
int byte = data[i + j];
|
||||
if (byte >= 32 && byte <= 126) {
|
||||
buffer.write(String.fromCharCode(byte));
|
||||
} else {
|
||||
buffer.write('.');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
buffer.write(' | ');
|
||||
if (i + 16 < data.length) buffer.writeln();
|
||||
}
|
||||
|
||||
return buffer.toString();
|
||||
}
|
||||
31
dart/lib/dev_tools/watch_dump.dart
Normal file
31
dart/lib/dev_tools/watch_dump.dart
Normal file
@@ -0,0 +1,31 @@
|
||||
import 'dart:io';
|
||||
import 'dart:async';
|
||||
import 'package:sweepstore/debug.dart';
|
||||
|
||||
void main() async {
|
||||
int refreshCount = 0;
|
||||
while (true) {
|
||||
// Clear console
|
||||
if (Platform.isWindows) {
|
||||
print(Process.runSync("cls", [], runInShell: true).stdout);
|
||||
} else {
|
||||
print(Process.runSync("clear", [], runInShell: true).stdout);
|
||||
}
|
||||
|
||||
refreshCount++;
|
||||
|
||||
// Read example.bin
|
||||
final file = File('example.bin');
|
||||
|
||||
if (await file.exists()) {
|
||||
final data = await file.readAsBytes();
|
||||
print('Binary dump of example.bin (${data.length} bytes) - Refresh #$refreshCount\n');
|
||||
print(binaryDump(data));
|
||||
print('\n--- Refreshing in 1 seconds ---');
|
||||
} else {
|
||||
print('Error: example.bin not found - Refresh #$refreshCount');
|
||||
}
|
||||
|
||||
await Future.delayed(Duration(seconds: 1));
|
||||
}
|
||||
}
|
||||
95
dart/lib/dev_tools/watch_tickets.dart
Normal file
95
dart/lib/dev_tools/watch_tickets.dart
Normal file
@@ -0,0 +1,95 @@
|
||||
import 'dart:io';
|
||||
import 'dart:async';
|
||||
import 'package:sweepstore/header.dart';
|
||||
|
||||
|
||||
void main() async {
|
||||
int refreshCount = 0;
|
||||
int? previousMasterHeartbeat;
|
||||
Map<int, int> previousWorkerHeartbeats = {};
|
||||
|
||||
while (true) {
|
||||
// Clear console
|
||||
if (Platform.isWindows) {
|
||||
print(Process.runSync("cls", [], runInShell: true).stdout);
|
||||
} else {
|
||||
print(Process.runSync("clear", [], runInShell: true).stdout);
|
||||
}
|
||||
|
||||
refreshCount++;
|
||||
|
||||
// Read example.bin
|
||||
final file = File('example.bin');
|
||||
|
||||
if (await file.exists()) {
|
||||
// Check file size first
|
||||
int fileSize = await file.length();
|
||||
if (fileSize < 48) {
|
||||
print('Error: example.bin too small ($fileSize bytes) - Refresh #$refreshCount');
|
||||
await Future.delayed(Duration(seconds: 1));
|
||||
continue;
|
||||
}
|
||||
|
||||
final raf = await file.open(mode: FileMode.read);
|
||||
|
||||
try {
|
||||
final header = SweepstoreHeader(raf);
|
||||
final concurrency = header.concurrency;
|
||||
|
||||
int now32 = (DateTime.now().millisecondsSinceEpoch ~/ 1000) & 0xFFFFFFFF;
|
||||
|
||||
print('Sweepstore Tickets - Refresh #$refreshCount');
|
||||
print('Current Time (now32): $now32');
|
||||
print('Master ID: ${concurrency.masterIdentifier}');
|
||||
|
||||
int masterAge = now32 - concurrency.masterHeartbeat;
|
||||
String masterStatus = masterAge > 5 ? "(stale)" : "(active)";
|
||||
String masterPrevious = previousMasterHeartbeat != null ? "(previously $previousMasterHeartbeat)" : "";
|
||||
print('Master Heartbeat: ${concurrency.masterHeartbeat} $masterStatus $masterPrevious');
|
||||
|
||||
print('Workers: ${concurrency.numberOfWorkers}');
|
||||
print('Read Allowed: ${concurrency.isReadAllowed}');
|
||||
print('');
|
||||
|
||||
// display each ticket
|
||||
for (int i = 0; i < concurrency.numberOfWorkers; i++) {
|
||||
final ticket = concurrency[i];
|
||||
final snapshot = ticket.snapshot();
|
||||
|
||||
print('--- Ticket #$i ---');
|
||||
print(' Identifier: ${snapshot.identifier}');
|
||||
|
||||
int workerAge = now32 - snapshot.workerHeartbeat;
|
||||
String workerStatus = workerAge > 5 ? "(stale)" : "(active)";
|
||||
String workerPrevious = previousWorkerHeartbeats.containsKey(i) ? "(previously ${previousWorkerHeartbeats[i]})" : "";
|
||||
print(' Heartbeat: ${snapshot.workerHeartbeat} $workerStatus $workerPrevious');
|
||||
|
||||
print(' State: ${snapshot.ticketState.name}');
|
||||
print(' Operation: ${snapshot.ticketOperation.name}');
|
||||
print(' Key Hash: ${snapshot.keyHash}');
|
||||
print(' Write Ptr: ${snapshot.writePointer}');
|
||||
print(' Write Size: ${snapshot.writeSize} bytes');
|
||||
print('');
|
||||
|
||||
// update previous heartbeat
|
||||
previousWorkerHeartbeats[i] = snapshot.workerHeartbeat;
|
||||
|
||||
}
|
||||
|
||||
// updat previous master heartbeat
|
||||
previousMasterHeartbeat = concurrency.masterHeartbeat;
|
||||
|
||||
print('--- Refreshing in 1 second ---');
|
||||
} catch (e) {
|
||||
print('Error reading file: $e');
|
||||
print('File may be in inconsistent state, retrying...');
|
||||
} finally {
|
||||
await raf.close();
|
||||
}
|
||||
} else {
|
||||
print('Error: example.bin not found - Refresh #$refreshCount');
|
||||
}
|
||||
|
||||
await Future.delayed(Duration(seconds: 1));
|
||||
}
|
||||
}
|
||||
416
dart/lib/header.dart
Normal file
416
dart/lib/header.dart
Normal file
@@ -0,0 +1,416 @@
|
||||
|
||||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
import 'package:sweepstore/structures.dart';
|
||||
|
||||
import 'helpers.dart';
|
||||
|
||||
int roundToNearest16(int value) {
|
||||
int rounded = (value + 15) & ~15;
|
||||
return rounded;
|
||||
}
|
||||
|
||||
void initialiseSweepstoreHeader(RandomAccessFile file, {
|
||||
int concurrentWorkers = 4,
|
||||
}) {
|
||||
|
||||
SweepstoreHeaderWriter header = SweepstoreHeaderWriter(file);
|
||||
|
||||
if (header.magicNumber == 'SWPT') {
|
||||
throw ArgumentError('Sweepstore file is already initialised.');
|
||||
}
|
||||
|
||||
SweepstoreConcurrencyHeaderWriter concurrencyHeader = SweepstoreConcurrencyHeaderWriter(header);
|
||||
|
||||
header.magicNumber = 'SWPT';
|
||||
header.version = "undefined";
|
||||
header.addressTablePointer = SweepstorePointer.nullptr;
|
||||
header.freeListCount = 0;
|
||||
header.isFreeListLifted = false;
|
||||
concurrencyHeader.masterIdentifier = 0;
|
||||
concurrencyHeader.masterHeartbeat = 0;
|
||||
concurrencyHeader.numberOfWorkers = concurrentWorkers;
|
||||
concurrencyHeader.isReadAllowed = false;
|
||||
|
||||
for (int i = 0; i < concurrentWorkers; i++) {
|
||||
SweepstoreWorkerTicket ticket = concurrencyHeader[i];
|
||||
ticket.write(
|
||||
identifier: 0,
|
||||
workerHeartbeat: 0,
|
||||
ticketState: SweepstoreTicketState.FREE,
|
||||
ticketOperation: SweepstoreTicketOperation.NONE,
|
||||
keyHash: 0,
|
||||
writePointer: SweepstorePointer.nullptr,
|
||||
writeSize: 0,
|
||||
);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
class SweepstoreHeader {
|
||||
|
||||
final RandomAccessFile _file;
|
||||
|
||||
SweepstoreHeader(this._file);
|
||||
|
||||
// Offset 0 - 4 bytes
|
||||
String get magicNumber {
|
||||
_file.setPositionSync(0);
|
||||
final bytes = _file.readSync(4);
|
||||
return String.fromCharCodes(bytes);
|
||||
}
|
||||
|
||||
// Offset 4 - 12 bytes
|
||||
String get version {
|
||||
_file.setPositionSync(4);
|
||||
String version = utf8.decode(_file.readSync(12)).trim();
|
||||
return version;
|
||||
}
|
||||
|
||||
// Offset 16 - 8 bytes
|
||||
SweepstorePointer get addressTablePointer {
|
||||
_file.setPositionSync(16);
|
||||
final address = _file.readIntSync(8);
|
||||
return SweepstorePointer(address);
|
||||
}
|
||||
|
||||
// Offset 24 - 4 bytes
|
||||
int get freeListCount {
|
||||
_file.setPositionSync(24);
|
||||
int count = _file.readIntSync(4);
|
||||
return count;
|
||||
}
|
||||
|
||||
// Offset 28 - 1 byte
|
||||
bool get isFreeListLifted {
|
||||
_file.setPositionSync(28);
|
||||
int flag = _file.readIntSync(1);
|
||||
return flag != 0;
|
||||
}
|
||||
|
||||
SweepstoreConcurrencyHeader get concurrency => SweepstoreConcurrencyHeader(this);
|
||||
}
|
||||
|
||||
class SweepstoreHeaderWriter extends SweepstoreHeader {
|
||||
|
||||
SweepstoreHeaderWriter(RandomAccessFile file) : super(file);
|
||||
|
||||
// Offset 0 - 4 bytes
|
||||
void set magicNumber(String value) {
|
||||
if (value.length != 4) {
|
||||
throw ArgumentError('Magic number must be exactly 4 characters long');
|
||||
}
|
||||
_file.setPositionSync(0);
|
||||
_file.writeFromSync(value.codeUnits);
|
||||
}
|
||||
|
||||
// Offset 4 - 12 bytes
|
||||
void set version(String value) {
|
||||
if (value.length > 11) {
|
||||
throw ArgumentError('Version string must be at most 11 characters long');
|
||||
}
|
||||
_file.setPositionSync(4);
|
||||
_file.writeFromSync(utf8.encode(" " + value.padRight(11, ' ').substring(0, 11)));
|
||||
}
|
||||
|
||||
// Offset 16 - 8 bytes
|
||||
void set addressTablePointer(SweepstorePointer pointer) {
|
||||
_file.setPositionSync(16);
|
||||
_file.writeIntSync(pointer.address, 8);
|
||||
}
|
||||
|
||||
// Offset 24 - 4 bytes
|
||||
void set freeListCount(int value) {
|
||||
_file.setPositionSync(24);
|
||||
_file.writeIntSync(value, 4);
|
||||
}
|
||||
|
||||
// Offset 28 - 1 byte
|
||||
void set isFreeListLifted(bool lifted) {
|
||||
_file.setPositionSync(28);
|
||||
_file.writeIntSync(lifted ? 1 : 0, 1);
|
||||
}
|
||||
}
|
||||
|
||||
class SweepstoreConcurrencyHeader {
|
||||
|
||||
final SweepstoreHeader _header;
|
||||
|
||||
SweepstoreConcurrencyHeader(this._header);
|
||||
|
||||
// Offset 29 - 8 bytes
|
||||
int get masterIdentifier {
|
||||
_header._file.setPositionSync(29);
|
||||
int id = _header._file.readIntSync(8);
|
||||
return id;
|
||||
}
|
||||
|
||||
// Offset 37 - 4 bytes
|
||||
int get masterHeartbeat {
|
||||
_header._file.setPositionSync(37);
|
||||
int heartbeat = _header._file.readIntSync(4);
|
||||
return heartbeat;
|
||||
}
|
||||
|
||||
// Offset 41 - 4 bytes
|
||||
int get numberOfWorkers {
|
||||
_header._file.setPositionSync(41);
|
||||
int numWorkers = _header._file.readIntSync(4);
|
||||
return numWorkers;
|
||||
}
|
||||
|
||||
// Offset 45 - 1 byte
|
||||
bool get isReadAllowed {
|
||||
_header._file.setPositionSync(45);
|
||||
int flag = _header._file.readIntSync(1);
|
||||
return flag != 0;
|
||||
}
|
||||
|
||||
SweepstoreWorkerTicket operator [](int ticketIndex) {
|
||||
if (ticketIndex < 0 || ticketIndex >= numberOfWorkers) {
|
||||
throw RangeError.index(ticketIndex, this, 'ticketIndex', null, numberOfWorkers);
|
||||
}
|
||||
return SweepstoreWorkerTicket(ticketIndex, this);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
class SweepstoreConcurrencyHeaderWriter extends SweepstoreConcurrencyHeader {
|
||||
|
||||
SweepstoreConcurrencyHeaderWriter(SweepstoreHeader header) : super(header);
|
||||
|
||||
// Offset 29 - 8 bytes
|
||||
void set masterIdentifier(int id) {
|
||||
_header._file.setPositionSync(29);
|
||||
_header._file.writeIntSync(id, 8);
|
||||
}
|
||||
|
||||
// Offset 37 - 4 bytes
|
||||
void set masterHeartbeat(int heartbeat) {
|
||||
_header._file.setPositionSync(37);
|
||||
_header._file.writeIntSync(heartbeat, 4);
|
||||
}
|
||||
|
||||
// Offset 41 - 4 bytes
|
||||
void set numberOfWorkers(int numWorkers) {
|
||||
_header._file.setPositionSync(41);
|
||||
_header._file.writeIntSync(numWorkers, 4);
|
||||
}
|
||||
|
||||
// Offset 45 - 1 byte
|
||||
void set isReadAllowed(bool allowed) {
|
||||
_header._file.setPositionSync(45);
|
||||
_header._file.writeIntSync(allowed ? 1 : 0, 1);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
final int endOfStaticHeaderOffset = roundToNearest16(46);
|
||||
|
||||
class SweepstoreWorkerTicket {
|
||||
|
||||
static final int ticketSize = roundToNearest16(29);
|
||||
final SweepstoreConcurrencyHeader _concurrencyHeader;
|
||||
final int ticketIndex;
|
||||
|
||||
SweepstoreWorkerTicket(this.ticketIndex, this._concurrencyHeader);
|
||||
|
||||
// All offsets are relative to the start of the workers ticket
|
||||
int get _baseOffset => endOfStaticHeaderOffset + (ticketIndex * ticketSize);
|
||||
|
||||
// // Offset 0 - 4 bytes
|
||||
// int get identifier {
|
||||
// _concurrencyHeader._header._file.lockSync(FileLock.blockingShared, _baseOffset, _baseOffset + 4);
|
||||
// _concurrencyHeader._header._file.setPositionSync(_baseOffset);
|
||||
// int id = _concurrencyHeader._header._file.readIntSync(4);
|
||||
// _concurrencyHeader._header._file.unlockSync(_baseOffset, _baseOffset + 4);
|
||||
// return id;
|
||||
// }
|
||||
//
|
||||
// // Offset 4 - 4 bytes
|
||||
// int get workerHeartbeat {
|
||||
// _concurrencyHeader._header._file.lockSync(FileLock.blockingShared, _baseOffset + 4, _baseOffset + 8);
|
||||
// _concurrencyHeader._header._file.setPositionSync(_baseOffset + 4);
|
||||
// int heartbeat = _concurrencyHeader._header._file.readIntSync(4);
|
||||
// _concurrencyHeader._header._file.unlockSync(_baseOffset + 4, _baseOffset + 8);
|
||||
// return heartbeat;
|
||||
// }
|
||||
//
|
||||
// // Offset 8 - 1 byte
|
||||
// SweepstoreTicketState get ticketState {
|
||||
// _concurrencyHeader._header._file.lockSync(FileLock.blockingShared, _baseOffset + 8, _baseOffset + 9);
|
||||
// _concurrencyHeader._header._file.setPositionSync(_baseOffset + 8);
|
||||
// int stateValue = _concurrencyHeader._header._file.readIntSync(1);
|
||||
// _concurrencyHeader._header._file.unlockSync(_baseOffset + 8, _baseOffset + 9);
|
||||
// return SweepstoreTicketState.values[stateValue];
|
||||
// }
|
||||
//
|
||||
// // Offset 9 - 1 byte
|
||||
// SweepstoreTicketOperation get ticketOperation {
|
||||
// _concurrencyHeader._header._file.lockSync(FileLock.blockingShared, _baseOffset + 9, _baseOffset + 10);
|
||||
// _concurrencyHeader._header._file.setPositionSync(_baseOffset + 9);
|
||||
// int operationValue = _concurrencyHeader._header._file.readIntSync(1);
|
||||
// _concurrencyHeader._header._file.unlockSync(_baseOffset + 9, _baseOffset + 10);
|
||||
// return SweepstoreTicketOperation.values[operationValue];
|
||||
// }
|
||||
//
|
||||
// // Offset 10 - 8 bytes
|
||||
// int get keyHash {
|
||||
// _concurrencyHeader._header._file.lockSync(FileLock.blockingShared, _baseOffset + 10, _baseOffset + 18);
|
||||
// _concurrencyHeader._header._file.setPositionSync(_baseOffset + 10);
|
||||
// int hash = _concurrencyHeader._header._file.readIntSync(8);
|
||||
// _concurrencyHeader._header._file.unlockSync(_baseOffset + 10, _baseOffset + 18);
|
||||
// return hash;
|
||||
// }
|
||||
//
|
||||
// // Offset 18 - 8 bytes
|
||||
// SweepstorePointer get writePointer {
|
||||
// _concurrencyHeader._header._file.lockSync(FileLock.blockingShared, _baseOffset + 18, _baseOffset + 26);
|
||||
// _concurrencyHeader._header._file.setPositionSync(_baseOffset + 18);
|
||||
// int address = _concurrencyHeader._header._file.readIntSync(8);
|
||||
// _concurrencyHeader._header._file.unlockSync(_baseOffset + 18, _baseOffset + 26);
|
||||
// return SweepstorePointer(address);
|
||||
// }
|
||||
//
|
||||
// // Offset 26 - 4 bytes
|
||||
// int get writeSize {
|
||||
// _concurrencyHeader._header._file.lockSync(FileLock.blockingShared, _baseOffset + 26, _baseOffset + 30);
|
||||
// _concurrencyHeader._header._file.setPositionSync(_baseOffset + 26);
|
||||
// int size = _concurrencyHeader._header._file.readIntSync(4);
|
||||
// _concurrencyHeader._header._file.unlockSync(_baseOffset + 26, _baseOffset + 30);
|
||||
// return size;
|
||||
// }
|
||||
|
||||
// Writer
|
||||
void write({
|
||||
int? identifier,
|
||||
int? workerHeartbeat,
|
||||
SweepstoreTicketState? ticketState,
|
||||
SweepstoreTicketOperation? ticketOperation,
|
||||
int? keyHash,
|
||||
SweepstorePointer? writePointer,
|
||||
int? writeSize,
|
||||
}) {
|
||||
|
||||
try {
|
||||
|
||||
_concurrencyHeader._header._file.lockSync(FileLock.blockingExclusive, _baseOffset, _baseOffset + ticketSize);
|
||||
|
||||
_concurrencyHeader._header._file.setPositionSync(_baseOffset);
|
||||
List<int> existingBuffer = _concurrencyHeader._header._file.readSync(ticketSize);
|
||||
RandomAccessMemory buffer = RandomAccessMemory(existingBuffer);
|
||||
|
||||
if (identifier != null) {
|
||||
buffer.setPositionSync(0);
|
||||
buffer.writeIntSync(identifier, 4);
|
||||
}
|
||||
if (workerHeartbeat != null) {
|
||||
buffer.setPositionSync(4);
|
||||
buffer.writeIntSync(workerHeartbeat, 4);
|
||||
}
|
||||
if (ticketState != null) {
|
||||
buffer.setPositionSync(8);
|
||||
buffer.writeIntSync(ticketState.index, 1);
|
||||
}
|
||||
if (ticketOperation != null) {
|
||||
buffer.setPositionSync(9);
|
||||
buffer.writeIntSync(ticketOperation.index, 1);
|
||||
}
|
||||
if (keyHash != null) {
|
||||
buffer.setPositionSync(10);
|
||||
buffer.writeIntSync(keyHash, 8);
|
||||
}
|
||||
if (writePointer != null) {
|
||||
buffer.setPositionSync(18);
|
||||
buffer.writeIntSync(writePointer.address, 8);
|
||||
}
|
||||
if (writeSize != null) {
|
||||
buffer.setPositionSync(26);
|
||||
buffer.writeIntSync(writeSize, 4);
|
||||
}
|
||||
|
||||
// Pad the rest of the ticket with zeros if necessary
|
||||
buffer.setPositionSync(30);
|
||||
while (buffer.positionSync() < ticketSize) {
|
||||
buffer.writeIntSync(0, 1);
|
||||
}
|
||||
|
||||
_concurrencyHeader._header._file.setPositionSync(_baseOffset);
|
||||
_concurrencyHeader._header._file.writeFromSync(buffer.toUint8List());
|
||||
_concurrencyHeader._header._file.flushSync();
|
||||
|
||||
} finally {
|
||||
_concurrencyHeader._header._file.unlockSync(_baseOffset, _baseOffset + ticketSize);
|
||||
}
|
||||
}
|
||||
|
||||
bool writable() {
|
||||
try {
|
||||
_concurrencyHeader._header._file.lockSync(
|
||||
FileLock.blockingExclusive,
|
||||
_baseOffset,
|
||||
_baseOffset + ticketSize
|
||||
);
|
||||
// Successfully locked - immediately unlock and return true
|
||||
_concurrencyHeader._header._file.unlockSync(_baseOffset, _baseOffset + ticketSize);
|
||||
return true;
|
||||
} catch (e) {
|
||||
// Lock failed - already held by another process
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
SweepstoreWorkerTicketSnapshot snapshot() {
|
||||
|
||||
_concurrencyHeader._header._file.lockSync(FileLock.blockingShared, _baseOffset, _baseOffset + ticketSize);
|
||||
|
||||
_concurrencyHeader._header._file.setPositionSync(_baseOffset);
|
||||
List<int> existingBuffer = _concurrencyHeader._header._file.readSync(ticketSize);
|
||||
RandomAccessMemory buffer = RandomAccessMemory(existingBuffer);
|
||||
_concurrencyHeader._header._file.unlockSync(_baseOffset, _baseOffset + ticketSize);
|
||||
|
||||
buffer.setPositionSync(0);
|
||||
int identifier = buffer.readIntSync(4);
|
||||
int workerHeartbeat = buffer.readIntSync(4);
|
||||
SweepstoreTicketState ticketState = SweepstoreTicketState.values[buffer.readIntSync(1)];
|
||||
SweepstoreTicketOperation ticketOperation = SweepstoreTicketOperation.values[buffer.readIntSync(1)];
|
||||
int keyHash = buffer.readIntSync(8);
|
||||
SweepstorePointer writePointer = SweepstorePointer(buffer.readIntSync(8));
|
||||
int writeSize = buffer.readIntSync(4);
|
||||
return SweepstoreWorkerTicketSnapshot._(
|
||||
ticketIndex: ticketIndex,
|
||||
identifier: identifier,
|
||||
workerHeartbeat: workerHeartbeat,
|
||||
ticketState: ticketState,
|
||||
ticketOperation: ticketOperation,
|
||||
keyHash: keyHash,
|
||||
writePointer: writePointer,
|
||||
writeSize: writeSize,
|
||||
);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
class SweepstoreWorkerTicketSnapshot {
|
||||
|
||||
final int ticketIndex;
|
||||
final int identifier;
|
||||
final int workerHeartbeat;
|
||||
final SweepstoreTicketState ticketState;
|
||||
final SweepstoreTicketOperation ticketOperation;
|
||||
final int keyHash;
|
||||
final SweepstorePointer writePointer;
|
||||
final int writeSize;
|
||||
|
||||
SweepstoreWorkerTicketSnapshot._({
|
||||
required this.ticketIndex,
|
||||
required this.identifier,
|
||||
required this.workerHeartbeat,
|
||||
required this.ticketState,
|
||||
required this.ticketOperation,
|
||||
required this.keyHash,
|
||||
required this.writePointer,
|
||||
required this.writeSize,
|
||||
});
|
||||
|
||||
}
|
||||
251
dart/lib/helpers.dart
Normal file
251
dart/lib/helpers.dart
Normal file
@@ -0,0 +1,251 @@
|
||||
|
||||
import 'dart:io';
|
||||
import 'dart:math';
|
||||
import 'dart:typed_data';
|
||||
|
||||
import 'package:sweepstore/structures.dart';
|
||||
|
||||
|
||||
class RandomAccessMemory {
|
||||
List<int> _buffer;
|
||||
int _position = 0;
|
||||
|
||||
RandomAccessMemory([List<int>? initialData]) : _buffer = initialData != null ? List<int>.from(initialData) : [];
|
||||
|
||||
// Position management
|
||||
int positionSync() => _position;
|
||||
|
||||
void setPositionSync(int position) {
|
||||
_position = position;
|
||||
}
|
||||
|
||||
int length() => _buffer.length;
|
||||
|
||||
// Read bytes
|
||||
List<int> readSync(int count) {
|
||||
if (_position + count > _buffer.length) {
|
||||
throw RangeError('Not enough bytes to read');
|
||||
}
|
||||
List<int> result = _buffer.sublist(_position, _position + count);
|
||||
_position += count;
|
||||
return result;
|
||||
}
|
||||
|
||||
// Write bytes
|
||||
void writeFromSync(List<int> bytes) {
|
||||
for (int i = 0; i < bytes.length; i++) {
|
||||
if (_position + i >= _buffer.length) {
|
||||
_buffer.add(bytes[i]);
|
||||
} else {
|
||||
_buffer[_position + i] = bytes[i];
|
||||
}
|
||||
}
|
||||
_position += bytes.length;
|
||||
}
|
||||
|
||||
// Read/Write Int Dynamic
|
||||
int readIntSync([int size = 4, Endian endianness = Endian.little]) {
|
||||
if (size < 1 || size > 8) {
|
||||
throw ArgumentError('Size must be between 1 and 8 bytes');
|
||||
}
|
||||
|
||||
List<int> bytes = readSync(size);
|
||||
|
||||
// Build integer from bytes with proper endianness
|
||||
int result = 0;
|
||||
if (endianness == Endian.little) {
|
||||
for (int i = size - 1; i >= 0; i--) {
|
||||
result = (result << 8) | bytes[i];
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < size; i++) {
|
||||
result = (result << 8) | bytes[i];
|
||||
}
|
||||
}
|
||||
|
||||
// Sign extend if MSB is set
|
||||
int signBit = 1 << (size * 8 - 1);
|
||||
if (result & signBit != 0) {
|
||||
result -= 1 << (size * 8);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
void writeIntSync(int value, [int size = 4, Endian endianness = Endian.little]) {
|
||||
if (size < 1 || size > 8) {
|
||||
throw ArgumentError('Size must be between 1 and 8 bytes');
|
||||
}
|
||||
|
||||
List<int> bytes = List.filled(size, 0);
|
||||
|
||||
// Extract bytes with proper endianness
|
||||
if (endianness == Endian.little) {
|
||||
for (int i = 0; i < size; i++) {
|
||||
bytes[i] = (value >> (i * 8)) & 0xFF;
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < size; i++) {
|
||||
bytes[size - 1 - i] = (value >> (i * 8)) & 0xFF;
|
||||
}
|
||||
}
|
||||
|
||||
writeFromSync(bytes);
|
||||
}
|
||||
|
||||
// Read/Write Pointers
|
||||
SweepstorePointer readPointerSync() {
|
||||
int offset = readIntSync(SweepstorePrimitives.POINTER.size);
|
||||
return SweepstorePointer(offset);
|
||||
}
|
||||
|
||||
void writePointerSync(SweepstorePointer pointer) {
|
||||
writeIntSync(pointer.address, SweepstorePrimitives.POINTER.size);
|
||||
}
|
||||
|
||||
// Read/Write Float32
|
||||
double readFloat32Sync([Endian endianness = Endian.little]) {
|
||||
List<int> bytes = readSync(4);
|
||||
return ByteData.sublistView(Uint8List.fromList(bytes)).getFloat32(0, endianness);
|
||||
}
|
||||
|
||||
void writeFloat32Sync(double value, [Endian endianness = Endian.little]) {
|
||||
ByteData byteData = ByteData(4);
|
||||
byteData.setFloat32(0, value, endianness);
|
||||
writeFromSync(byteData.buffer.asUint8List());
|
||||
}
|
||||
|
||||
// Read/Write Float64 (Double)
|
||||
double readFloat64Sync([Endian endianness = Endian.little]) {
|
||||
List<int> bytes = readSync(8);
|
||||
return ByteData.sublistView(Uint8List.fromList(bytes)).getFloat64(0, endianness);
|
||||
}
|
||||
|
||||
void writeFloat64Sync(double value, [Endian endianness = Endian.little]) {
|
||||
ByteData byteData = ByteData(8);
|
||||
byteData.setFloat64(0, value, endianness);
|
||||
writeFromSync(byteData.buffer.asUint8List());
|
||||
}
|
||||
|
||||
// Conversion methods
|
||||
List<int> toList() => List<int>.from(_buffer);
|
||||
|
||||
Uint8List toUint8List() => Uint8List.fromList(_buffer);
|
||||
}
|
||||
|
||||
extension SweepstoreRandomAccessFileHelper on RandomAccessFile {
|
||||
|
||||
// Read/Write Int Dynamic - Can specify size in bytes, does not have to align to 1, 2, 4, or 8 bytes. Default is 4 bytes (Int32)
|
||||
int readIntSync([int size = 4, Endian endianness = Endian.little]) {
|
||||
if (size < 1 || size > 8) {
|
||||
throw ArgumentError('Size must be between 1 and 8 bytes');
|
||||
}
|
||||
|
||||
List<int> bytes = readSync(size);
|
||||
|
||||
// Build integer from bytes with proper endianness
|
||||
int result = 0;
|
||||
if (endianness == Endian.little) {
|
||||
for (int i = size - 1; i >= 0; i--) {
|
||||
result = (result << 8) | bytes[i];
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < size; i++) {
|
||||
result = (result << 8) | bytes[i];
|
||||
}
|
||||
}
|
||||
|
||||
// Sign extend if MSB is set
|
||||
int signBit = 1 << (size * 8 - 1);
|
||||
if (result & signBit != 0) {
|
||||
result -= 1 << (size * 8);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
void writeIntSync(int value, [int size = 4, Endian endianness = Endian.little]) {
|
||||
if (size < 1 || size > 8) {
|
||||
throw ArgumentError('Size must be between 1 and 8 bytes');
|
||||
}
|
||||
|
||||
List<int> bytes = List.filled(size, 0);
|
||||
|
||||
// Extract bytes with proper endianness
|
||||
if (endianness == Endian.little) {
|
||||
for (int i = 0; i < size; i++) {
|
||||
bytes[i] = (value >> (i * 8)) & 0xFF;
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < size; i++) {
|
||||
bytes[size - 1 - i] = (value >> (i * 8)) & 0xFF;
|
||||
}
|
||||
}
|
||||
|
||||
writeFromSync(bytes);
|
||||
}
|
||||
|
||||
// Read/Write Pointers
|
||||
SweepstorePointer readPointerSync() {
|
||||
int offset = readIntSync(SweepstorePrimitives.POINTER.size);
|
||||
return SweepstorePointer(offset);
|
||||
}
|
||||
void writePointerSync(SweepstorePointer pointer) {
|
||||
writeIntSync(pointer.address, SweepstorePrimitives.POINTER.size);
|
||||
}
|
||||
|
||||
// Read/Write Float32
|
||||
double readFloat32Sync([Endian endianness = Endian.little]) {
|
||||
List<int> bytes = readSync(4);
|
||||
return ByteData.sublistView(Uint8List.fromList(bytes)).getFloat32(0, endianness);
|
||||
}
|
||||
void writeFloat32Sync(double value, [Endian endianness = Endian.little]) {
|
||||
ByteData byteData = ByteData(4);
|
||||
byteData.setFloat32(0, value, endianness);
|
||||
writeFromSync(byteData.buffer.asUint8List());
|
||||
}
|
||||
|
||||
// Read/Write Float64 (Double)
|
||||
double readFloat64Sync([Endian endianness = Endian.little]) {
|
||||
List<int> bytes = readSync(8);
|
||||
return ByteData.sublistView(Uint8List.fromList(bytes)).getFloat64(0, endianness);
|
||||
}
|
||||
void writeFloat64Sync(double value, [Endian endianness = Endian.little]) {
|
||||
ByteData byteData = ByteData(8);
|
||||
byteData.setFloat64(0, value, endianness);
|
||||
writeFromSync(byteData.buffer.asUint8List());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
extension SweepstoreDateTimeHelper on DateTime {
|
||||
int millisecondsSinceEpoch32() {
|
||||
return (millisecondsSinceEpoch ~/ 1000) & 0xFFFFFFFF;
|
||||
}
|
||||
int millisecondsSinceEpoch64() {
|
||||
return millisecondsSinceEpoch;
|
||||
}
|
||||
}
|
||||
|
||||
const Duration _windowsMinSleepTime = Duration(milliseconds: 16);
|
||||
|
||||
void preciseSleep(Duration duration) {
|
||||
final stopwatch = Stopwatch()..start();
|
||||
|
||||
if (Platform.isWindows) {
|
||||
if (duration < _windowsMinSleepTime) {
|
||||
// Pure busy-wait with high-res timer
|
||||
while (stopwatch.elapsed < duration) {}
|
||||
} else {
|
||||
// Hybrid: sleep most of it, busy-wait the remainder
|
||||
final sleepDuration = duration - _windowsMinSleepTime;
|
||||
sleep(sleepDuration);
|
||||
while (stopwatch.elapsed < duration) {}
|
||||
}
|
||||
} else {
|
||||
sleep(duration);
|
||||
}
|
||||
|
||||
stopwatch.stop();
|
||||
// print('preciseSleep: requested ${duration.inMicroseconds}μs, actual ${stopwatch.elapsedMicroseconds}μs, diff ${stopwatch.elapsedMicroseconds - duration.inMicroseconds}μs');
|
||||
}
|
||||
48
dart/lib/structures.dart
Normal file
48
dart/lib/structures.dart
Normal file
@@ -0,0 +1,48 @@
|
||||
|
||||
enum SweepstorePrimitives {
|
||||
|
||||
POINTER (8),
|
||||
ADDRESS_TABLE (-1);
|
||||
|
||||
final int size;
|
||||
final bool arrayType;
|
||||
const SweepstorePrimitives(this.size, {
|
||||
this.arrayType = false
|
||||
});
|
||||
|
||||
}
|
||||
|
||||
class SweepstorePointer {
|
||||
|
||||
static const SweepstorePointer nullptr = SweepstorePointer(-1);
|
||||
|
||||
final int address;
|
||||
|
||||
const SweepstorePointer(this.address);
|
||||
|
||||
bool get isNull => address == -1;
|
||||
|
||||
operator ==(Object other) {
|
||||
if (identical(this, other)) return true;
|
||||
if (other is! SweepstorePointer) return false;
|
||||
return address == other.address;
|
||||
}
|
||||
|
||||
@override
|
||||
String toString() => '0x${address.toRadixString(16)} ($address)';
|
||||
}
|
||||
|
||||
enum SweepstoreTicketState {
|
||||
FREE,
|
||||
WAITING,
|
||||
APPROVED,
|
||||
EXECUTING,
|
||||
COMPLETED,
|
||||
}
|
||||
|
||||
enum SweepstoreTicketOperation {
|
||||
NONE,
|
||||
READ,
|
||||
MODIFY,
|
||||
WRITE,
|
||||
}
|
||||
119
dart/lib/sweepstore.dart
Normal file
119
dart/lib/sweepstore.dart
Normal file
@@ -0,0 +1,119 @@
|
||||
|
||||
import 'dart:io';
|
||||
import 'dart:isolate';
|
||||
import 'dart:math';
|
||||
import 'package:sweepstore/debug.dart';
|
||||
import 'package:sweepstore/header.dart';
|
||||
import 'package:sweepstore/structures.dart';
|
||||
import 'package:sweepstore/concurrency.dart';
|
||||
|
||||
class Sweepstore {
|
||||
|
||||
final RandomAccessFile _file;
|
||||
|
||||
Sweepstore(String filePath)
|
||||
: _file = File(filePath).openSync(mode: FileMode.append)
|
||||
{
|
||||
_header = SweepstoreHeaderWriter(_file);
|
||||
}
|
||||
|
||||
late final SweepstoreHeaderWriter _header;
|
||||
late final SweepstoreConcurrencyHeaderWriter _concurrencyHeader = SweepstoreConcurrencyHeaderWriter(_header);
|
||||
|
||||
void initialise({
|
||||
int concurrentWorkers = 4,
|
||||
}) {
|
||||
|
||||
initialiseSweepstoreHeader(_file,
|
||||
concurrentWorkers: concurrentWorkers,
|
||||
);
|
||||
|
||||
_header.version = "1.1.0.1";
|
||||
print("Version: ${_header.version}");
|
||||
|
||||
}
|
||||
|
||||
void operator []=(String key, dynamic value) {
|
||||
|
||||
spawnTicket(_file,
|
||||
operation: SweepstoreTicketOperation.WRITE,
|
||||
keyHash: key.hashCode,
|
||||
writeSize: 0, // Placeholder
|
||||
onApproved: () {
|
||||
// print("Writing key: $key with hash ${key.hashCode} and value: $value");
|
||||
},
|
||||
debugLabel: key
|
||||
);
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
Future<void> main() async {
|
||||
|
||||
String filePath = '../example.bin';
|
||||
|
||||
File file = File(filePath);
|
||||
if (file.existsSync()) {
|
||||
file.deleteSync();
|
||||
}
|
||||
file.createSync();
|
||||
|
||||
Sweepstore store = Sweepstore(filePath);
|
||||
store.initialise(
|
||||
concurrentWorkers: 32
|
||||
);
|
||||
initialiseMasterListener(file.openSync(mode: FileMode.append));
|
||||
|
||||
print(binaryDump(file.readAsBytesSync()));
|
||||
|
||||
int iteration = 0;
|
||||
int maxIterations = 16;
|
||||
|
||||
print("Concurrent Workers: ${store._concurrencyHeader.numberOfWorkers}");
|
||||
print("Stale Ticket Threshold: ${STALE_HEARTBEAT_THRESHOLD_MS}ms");
|
||||
|
||||
int concurrencyTest = 1;
|
||||
|
||||
|
||||
while (true) {
|
||||
final receivePort = ReceivePort();
|
||||
int completedJobs = 0;
|
||||
|
||||
if (iteration > maxIterations) {
|
||||
break;
|
||||
}
|
||||
|
||||
final stopwatch = Stopwatch()..start();
|
||||
|
||||
// print('\x1B[95mStarting iteration #$iteration with $concurrencyTest concurrent jobs...\x1B[0m');
|
||||
for (int i = 0; i < concurrencyTest; i++) {
|
||||
await Isolate.spawn((message) {
|
||||
final index = message['index'] as int;
|
||||
final sendPort = message['sendPort'] as SendPort;
|
||||
|
||||
Sweepstore store = Sweepstore(filePath);
|
||||
store['key_$index'] = 'value_$index';
|
||||
|
||||
sendPort.send('done');
|
||||
}, {'index': i, 'sendPort': receivePort.sendPort});
|
||||
}
|
||||
|
||||
|
||||
// wait for all jobs to finish
|
||||
await for (var msg in receivePort) {
|
||||
completedJobs++;
|
||||
if (completedJobs >= concurrencyTest) {
|
||||
receivePort.close();
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
stopwatch.stop();
|
||||
|
||||
print("[$iteration/$maxIterations] Completed $concurrencyTest operation in ${stopwatch.elapsedMilliseconds} ms");
|
||||
|
||||
iteration++;
|
||||
|
||||
concurrencyTest *= 2;
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
name: file_formats
|
||||
description: A sample command-line application with basic argument parsing.
|
||||
version: 0.0.1
|
||||
# repository: https://github.com/my_org/my_repo
|
||||
name: sweepstore
|
||||
description: SweepStore (formerly BinaryTable) A high-performance binary storage format for Dart applications with efficient memory management and random access capabilities.
|
||||
version: 1.0.0
|
||||
repository: https://github.com/ImBenji03/SweepStore
|
||||
|
||||
environment:
|
||||
sdk: ^3.0.0
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
© 2025-26 by Benjamin Watt of IMBENJI.NET LIMITED - All rights reserved.
|
||||
|
||||
Use of this source code is governed by a MIT license that can be found in the LICENSE file.
|
||||
Use of this source code is governed by the Business Source License 1.1 that can be found in the LICENSE file.
|
||||
|
||||
This file is part of the SweepStore (formerly Binary Table) package for Dart.
|
||||
|
||||
@@ -19,6 +19,8 @@ This file is part of the SweepStore (formerly Binary Table) package for Dart.
|
||||
|
||||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
import 'dart:isolate';
|
||||
import 'dart:math';
|
||||
import 'dart:typed_data';
|
||||
|
||||
enum BT_Type {
|
||||
@@ -313,7 +315,7 @@ class BT_UniformArray extends BT_Reference {
|
||||
|
||||
void addAll(Iterable<dynamic> values) {
|
||||
|
||||
_table.antiFreeListScope(() {
|
||||
_table._antiFreeListScope(() {
|
||||
// Determine the type of the array by reading the first items type
|
||||
BT_Type type = elementType ?? BT_Type.fromDynamic(values.first);
|
||||
|
||||
@@ -344,16 +346,15 @@ class BT_UniformArray extends BT_Reference {
|
||||
fullBuffer.replaceRange(1, 5, lengthBytes);
|
||||
|
||||
// Free the old array
|
||||
_table.free(_pointer, size);
|
||||
_table._free(_pointer, size);
|
||||
|
||||
// Allocate new space for the updated array
|
||||
BT_Pointer newPointer = _table.alloc(fullBuffer.length);
|
||||
BT_Pointer newPointer = _table._alloc(fullBuffer.length);
|
||||
|
||||
// Replace any references to the old pointer with the new one
|
||||
Map<int, BT_Pointer> addressTable = _table._addressTable;
|
||||
addressTable.updateAll((key, value) {
|
||||
if (value == _pointer) {
|
||||
print('Updating address table entry for key $key from $value to $newPointer');
|
||||
return newPointer;
|
||||
}
|
||||
return value;
|
||||
@@ -364,8 +365,6 @@ class BT_UniformArray extends BT_Reference {
|
||||
// Write the updated buffer to the new location
|
||||
_table._file.setPositionSync(newPointer.address);
|
||||
_table._file.writeFromSync(fullBuffer);
|
||||
|
||||
print('Array resized to new length $newLength at $newPointer');
|
||||
});
|
||||
|
||||
}
|
||||
@@ -468,7 +467,6 @@ extension FreeList on List<BT_FreeListEntry> {
|
||||
New encoding should reflect a "read from the end" approach.
|
||||
So it should look like:
|
||||
- Entries (variable)
|
||||
- Entry count (4 bytes)
|
||||
|
||||
Entry:
|
||||
- Pointer (8 bytes)
|
||||
@@ -485,9 +483,6 @@ extension FreeList on List<BT_FreeListEntry> {
|
||||
buffer.addAll((ByteData(4)..setInt32(0, entry.size, Endian.little)).buffer.asUint8List());
|
||||
}
|
||||
|
||||
// Entry count
|
||||
buffer.addAll((ByteData(4)..setInt32(0, length, Endian.little)).buffer.asUint8List());
|
||||
|
||||
return buffer;
|
||||
}
|
||||
|
||||
@@ -509,21 +504,239 @@ extension fnv1a on String {
|
||||
|
||||
}
|
||||
|
||||
// Convert double to float16
|
||||
extension on double {
|
||||
|
||||
List<int> _toFloat16Bytes() {
|
||||
ByteData float32Data = ByteData(4);
|
||||
float32Data.setFloat32(0, this, Endian.little);
|
||||
int f = float32Data.getUint32(0);
|
||||
|
||||
int sign = (f >> 31) & 0x1;
|
||||
int exponent = (f >> 23) & 0xff;
|
||||
int mantissa = f & 0x7fffff;
|
||||
|
||||
if (exponent == 0xff) { // Inf or NaN
|
||||
if (mantissa != 0) {
|
||||
return [(sign << 7) | 0x7e, 0x01]; // NaN
|
||||
} else {
|
||||
return [(sign << 7) | 0x7c, 0x00]; // Inf
|
||||
}
|
||||
}
|
||||
|
||||
if (exponent == 0) { // Zero or denormalized
|
||||
return [sign << 7, 0x00]; // Zero
|
||||
}
|
||||
|
||||
// Adjust exponent for float16 (bias: 127 for float32, 15 for float16)
|
||||
exponent = exponent - 127 + 15;
|
||||
|
||||
if (exponent >= 31) { // Overflow to infinity
|
||||
return [(sign << 7) | 0x7c, 0x00]; // Inf
|
||||
}
|
||||
|
||||
if (exponent <= 0) { // Underflow to zero
|
||||
return [sign << 7, 0x00]; // Zero
|
||||
}
|
||||
|
||||
// Extract the top 10 bits of mantissa for float16
|
||||
mantissa >>= 13;
|
||||
|
||||
int float16 = (sign << 15) | (exponent << 10) | mantissa;
|
||||
return [float16 & 0xff, (float16 >> 8) & 0xff]; // Little endian
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
enum BT_TicketState {
|
||||
IDLE,
|
||||
WAITING,
|
||||
APPROVED,
|
||||
EXECUTING,
|
||||
COMPLETED,
|
||||
}
|
||||
|
||||
enum BT_TicketOpcode {
|
||||
NONE,
|
||||
READ,
|
||||
MODIFY,
|
||||
WRITE,
|
||||
}
|
||||
|
||||
class BT_Ticket {
|
||||
|
||||
final int heartbeat;
|
||||
final BT_TicketState state;
|
||||
final BT_TicketOpcode opcode;
|
||||
final int keyHash;
|
||||
|
||||
// Defined by the slave
|
||||
final int writeSize;
|
||||
|
||||
// Defined by the master
|
||||
final BT_Pointer writePointer;
|
||||
|
||||
BT_Ticket({
|
||||
required this.heartbeat,
|
||||
required this.state,
|
||||
required this.opcode,
|
||||
required this.keyHash,
|
||||
|
||||
this.writeSize = 0,
|
||||
this.writePointer = BT_Null,
|
||||
});
|
||||
|
||||
BT_Ticket.fromIntList(List<int> data)
|
||||
: heartbeat = ByteData.sublistView(Uint8List.fromList(data.sublist(0, 4))).getInt32(0, Endian.little),
|
||||
state = BT_TicketState.values[data[4]],
|
||||
opcode = BT_TicketOpcode.values[data[5]],
|
||||
keyHash = ByteData.sublistView(Uint8List.fromList(data.sublist(6, 14))).getInt64(0, Endian.little),
|
||||
|
||||
writePointer = BT_Pointer(ByteData.sublistView(Uint8List.fromList(data.sublist(14, 22))).getInt64(0, Endian.little)),
|
||||
writeSize = ByteData.sublistView(Uint8List.fromList(data.sublist(22, 26))).getInt32(0, Endian.little);
|
||||
|
||||
List<int> toIntList() {
|
||||
List<int> data = [];
|
||||
|
||||
// Heartbeat (4 bytes)
|
||||
data.addAll((ByteData(4)..setInt32(0, heartbeat, Endian.little)).buffer.asUint8List());
|
||||
|
||||
// State (1 byte)
|
||||
data.add(state.index);
|
||||
|
||||
// Opcode (1 byte)
|
||||
data.add(opcode.index);
|
||||
|
||||
// Key Hash (8 bytes)
|
||||
data.addAll((ByteData(8)..setInt64(0, keyHash, Endian.little)).buffer.asUint8List());
|
||||
|
||||
// Write Pointer (8 bytes)
|
||||
data.addAll((ByteData(8)..setInt64(0, writePointer.address, Endian.little)).buffer.asUint8List());
|
||||
|
||||
// Write Size (4 bytes)
|
||||
data.addAll((ByteData(4)..setInt32(0, writeSize, Endian.little)).buffer.asUint8List());
|
||||
|
||||
return data;
|
||||
}
|
||||
|
||||
String toString() {
|
||||
return 'BT_Ticket(heartbeat: $heartbeat, state: $state, opcode: $opcode, keyHash: $keyHash, writePointer: $writePointer, writeSize: $writeSize)';
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
|
||||
|
||||
class BinaryTable {
|
||||
|
||||
late final int sessionId;
|
||||
|
||||
RandomAccessFile _file;
|
||||
|
||||
BinaryTable(String path) : _file = File(path).openSync(mode: FileMode.append);
|
||||
BinaryTable(String path) : _file = File(path).openSync(mode: FileMode.append) {
|
||||
|
||||
void initialise() {
|
||||
var nextSessionId = Random.secure();
|
||||
int high = nextSessionId.nextInt(1 << 32);
|
||||
int low = nextSessionId.nextInt(1 << 32);
|
||||
sessionId = (high << 32) | low;
|
||||
|
||||
// Check if the file is initialised
|
||||
_file.setPositionSync(0);
|
||||
_file.writePointerSync(BT_Null); // Address table pointer
|
||||
_file.writeIntSync(0, 4); // Free list entry count
|
||||
List<int> magicNumber = _file.readSync(4);
|
||||
bool isInitialised = magicNumber.length == 4 &&
|
||||
magicNumber[0] == 'S'.codeUnitAt(0) &&
|
||||
magicNumber[1] == 'W'.codeUnitAt(0) &&
|
||||
magicNumber[2] == 'P'.codeUnitAt(0) &&
|
||||
magicNumber[3] == 'S'.codeUnitAt(0);
|
||||
|
||||
if (isInitialised && !isMasterAlive) {
|
||||
|
||||
_initialiseMaster();
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
void initialise({
|
||||
int concurrentReaders = 4,
|
||||
})
|
||||
{
|
||||
_file.setPositionSync(0);
|
||||
|
||||
// Ensure the file hasnt already been initialised
|
||||
List<int> magicNumber = _file.readSync(4);
|
||||
bool isInitialised = magicNumber.length == 4 &&
|
||||
magicNumber[0] == 'S'.codeUnitAt(0) &&
|
||||
magicNumber[1] == 'W'.codeUnitAt(0) &&
|
||||
magicNumber[2] == 'P'.codeUnitAt(0) &&
|
||||
magicNumber[3] == 'S'.codeUnitAt(0);
|
||||
if (isInitialised) {
|
||||
throw Exception('Binary Table file is already initialised.');
|
||||
}
|
||||
|
||||
// Magic number "SWPS" (0/4 bytes)
|
||||
_file.writeFromSync("SWPS".codeUnits);
|
||||
|
||||
// Version (1.0 float16) (4/2 bytes)
|
||||
_file.writeFromSync(1.0._toFloat16Bytes());
|
||||
|
||||
// Address table pointer (null) (6/8 bytes)
|
||||
_file.writePointerSync(BT_Null);
|
||||
|
||||
// Free list count (0) (14/4 bytes)
|
||||
_file.writeIntSync(0, 4); // Free list entry count
|
||||
|
||||
/*
|
||||
The values below are for concurrency.
|
||||
*/
|
||||
|
||||
// Master Identifier (18/8 bytes)
|
||||
_file.writeIntSync(sessionId & 0xFFFFFFFF, 8);
|
||||
|
||||
// Master Heartbeat (26/4 bytes)
|
||||
int now = DateTime.now().millisecondsSinceEpoch;
|
||||
_file.writeIntSync(now, 4);
|
||||
|
||||
// Number of concurrent readers (30/4 bytes) // Cannot be changed after initialisation
|
||||
_file.writeIntSync(concurrentReaders, 4);
|
||||
|
||||
// Allow reads (34/1 bytes)
|
||||
_file.writeByteSync(1);
|
||||
|
||||
// Everything else is operator slots starting at 35 bytes.
|
||||
for (int i = 0; i < concurrentReaders; i++) {
|
||||
|
||||
// Slave Heartbeat (4 bytes)
|
||||
_file.writeIntSync(0, 4);
|
||||
|
||||
// Ticket state (1 byte)
|
||||
_file.writeByteSync(0);
|
||||
|
||||
// Ticket Opcode (1 byte)
|
||||
_file.writeByteSync(0);
|
||||
|
||||
// Key Hash (8 bytes)
|
||||
_file.writeIntSync(0, 8);
|
||||
|
||||
// Write Pointer (8 bytes)
|
||||
_file.writeIntSync(-1, 8);
|
||||
|
||||
// Write Size (4 bytes)
|
||||
_file.writeIntSync(0, 4);
|
||||
|
||||
}
|
||||
|
||||
// Run the master initialisation
|
||||
_initialiseMaster();
|
||||
}
|
||||
|
||||
/*
|
||||
Address Table
|
||||
*/
|
||||
|
||||
Map<int, BT_Pointer> get _addressTable {
|
||||
|
||||
_file.setPositionSync(0);
|
||||
_file.setPositionSync(6);
|
||||
BT_Reference tableRef = BT_Reference(this, _file.readPointerSync());
|
||||
|
||||
if (tableRef._pointer.isNull) {
|
||||
@@ -570,34 +783,37 @@ class BinaryTable {
|
||||
});
|
||||
|
||||
// Write new address table at end of file
|
||||
BT_Pointer tableAddress = alloc(buffer.length);
|
||||
BT_Pointer tableAddress = _alloc(buffer.length);
|
||||
_file.setPositionSync(tableAddress.address);
|
||||
_file.writeFromSync(buffer);
|
||||
|
||||
// Read old table pointer before updating
|
||||
_file.setPositionSync(0);
|
||||
_file.setPositionSync(6);
|
||||
BT_Reference oldTableRef = BT_Reference(this, _file.readPointerSync());
|
||||
|
||||
// Update header to point to new table
|
||||
_file.setPositionSync(0);
|
||||
_file.setPositionSync(6);
|
||||
_file.writePointerSync(tableAddress);
|
||||
|
||||
// Now free the old table if it exists and is not the same as the new one
|
||||
if (!oldTableRef._pointer.isNull && oldTableRef._pointer != tableAddress) {
|
||||
free(oldTableRef._pointer, oldTableRef.size);
|
||||
_free(oldTableRef._pointer, oldTableRef.size);
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
Free List
|
||||
*/
|
||||
|
||||
bool freeListLifted = false;
|
||||
List<BT_FreeListEntry>? _freeListCache;
|
||||
|
||||
List<BT_FreeListEntry> get _freeList {
|
||||
|
||||
if (freeListLifted) {
|
||||
return _freeListCache ?? [];
|
||||
}
|
||||
|
||||
_file.setPositionSync(_file.lengthSync() - 4);
|
||||
_file.setPositionSync(14);
|
||||
int entryCount = _file.readIntSync(4);
|
||||
if (entryCount == 0) {
|
||||
return [];
|
||||
@@ -605,7 +821,7 @@ class BinaryTable {
|
||||
|
||||
int entrySize = BT_Type.POINTER.size + 4; // Pointer + Size
|
||||
int freeListSize = entryCount * entrySize;
|
||||
_file.setPositionSync(_file.lengthSync() - 4 - freeListSize);
|
||||
_file.setPositionSync(_file.lengthSync() - freeListSize);
|
||||
List<int> buffer = _file.readSync(freeListSize);
|
||||
|
||||
List<BT_FreeListEntry> freeList = [];
|
||||
@@ -627,54 +843,77 @@ class BinaryTable {
|
||||
return freeList;
|
||||
}
|
||||
set _freeList(List<BT_FreeListEntry> list) {
|
||||
|
||||
if (freeListLifted) {
|
||||
_freeListCache = list;
|
||||
return;
|
||||
}
|
||||
|
||||
_file.setPositionSync(_file.lengthSync() - 4);
|
||||
// Read OLD count from header
|
||||
_file.setPositionSync(14);
|
||||
int oldEntryCount = _file.readIntSync(4);
|
||||
int oldListSize = (oldEntryCount * (BT_Type.POINTER.size + 4)) + 4; // Entries + Count
|
||||
_file.truncateSync(_file.lengthSync() - oldListSize);
|
||||
|
||||
List<int> buffer = list.bt_encode();
|
||||
_file.setPositionSync(_file.lengthSync());
|
||||
_file.writeFromSync(buffer);
|
||||
// Calculate old free list size (entries only, not count)
|
||||
int oldListSize = oldEntryCount * (BT_Type.POINTER.size + 4);
|
||||
|
||||
// Remove old free list entries from EOF
|
||||
if (oldEntryCount > 0) {
|
||||
int currentLength = _file.lengthSync();
|
||||
_file.truncateSync(currentLength - oldListSize);
|
||||
}
|
||||
|
||||
// Write NEW count to header
|
||||
_file.setPositionSync(14);
|
||||
_file.writeIntSync(list.length, 4);
|
||||
|
||||
// Write NEW entries to EOF (if any)
|
||||
if (list.isNotEmpty) {
|
||||
List<int> buffer = list.bt_encode();
|
||||
_file.setPositionSync(_file.lengthSync());
|
||||
_file.writeFromSync(buffer);
|
||||
}
|
||||
}
|
||||
|
||||
/// Caches the free list in memory, and removed it from the file.
|
||||
void liftFreeList() {
|
||||
void _liftFreeList() {
|
||||
if (freeListLifted) {
|
||||
throw StateError('Free list is already lifted');
|
||||
}
|
||||
|
||||
// Cache the free list
|
||||
_freeListCache = _freeList;
|
||||
|
||||
_file.setPositionSync(_file.lengthSync() - 4);
|
||||
// Read count from header
|
||||
_file.setPositionSync(14);
|
||||
int oldEntryCount = _file.readIntSync(4);
|
||||
int oldEntrySize = BT_Type.POINTER.size + 4; // Pointer + Size
|
||||
int oldFreeListSize = oldEntryCount * oldEntrySize + 4; // +4 for entry count
|
||||
_file.truncateSync(_file.lengthSync() - oldFreeListSize);
|
||||
|
||||
if (oldEntryCount > 0) {
|
||||
int oldEntrySize = BT_Type.POINTER.size + 4;
|
||||
int oldFreeListSize = oldEntryCount * oldEntrySize; // Just entries, no count
|
||||
|
||||
// Remove free list entries from EOF
|
||||
_file.truncateSync(_file.lengthSync() - oldFreeListSize);
|
||||
}
|
||||
|
||||
// Clear count in header
|
||||
_file.setPositionSync(14);
|
||||
_file.writeIntSync(0, 4);
|
||||
|
||||
freeListLifted = true;
|
||||
}
|
||||
|
||||
/// Appends the cached free list back to the file, and clears the cache.
|
||||
void dropFreeList() {
|
||||
if (!freeListLifted) {
|
||||
throw StateError('Free list is not lifted');
|
||||
}
|
||||
|
||||
_file.setPositionSync(_file.lengthSync());
|
||||
_file.writeIntSync(0, 4); // Placeholder for entry count
|
||||
|
||||
freeListLifted = false;
|
||||
_freeList = _freeListCache!;
|
||||
_freeList = _freeListCache!; // This now writes count to header and entries to EOF
|
||||
_freeListCache = null;
|
||||
}
|
||||
|
||||
void antiFreeListScope(void Function() fn) {
|
||||
liftFreeList();
|
||||
void _antiFreeListScope(void Function() fn) {
|
||||
_liftFreeList();
|
||||
try {
|
||||
fn();
|
||||
} finally {
|
||||
@@ -682,7 +921,7 @@ class BinaryTable {
|
||||
}
|
||||
}
|
||||
|
||||
void free(BT_Pointer pointer, int size) {
|
||||
void _free(BT_Pointer pointer, int size) {
|
||||
|
||||
if (!freeListLifted) {
|
||||
throw StateError('Free list must be lifted before freeing memory');
|
||||
@@ -734,7 +973,7 @@ class BinaryTable {
|
||||
// Update free list
|
||||
_freeList = freeList;
|
||||
}
|
||||
BT_Pointer alloc(int size) {
|
||||
BT_Pointer _alloc(int size) {
|
||||
|
||||
if (!freeListLifted) {
|
||||
throw StateError('Free list must be lifted before allocation');
|
||||
@@ -783,34 +1022,259 @@ class BinaryTable {
|
||||
}
|
||||
}
|
||||
|
||||
operator []=(String key, dynamic value) {
|
||||
/*
|
||||
Concurrency
|
||||
*/
|
||||
|
||||
antiFreeListScope(() {
|
||||
Map<int, BT_Pointer> addressTable = _addressTable;
|
||||
void _initialiseMaster() {
|
||||
Isolate.spawn((String filePath) {
|
||||
RandomAccessFile file = File(filePath).openSync(mode: FileMode.append);
|
||||
|
||||
int keyHash = key.bt_hash;
|
||||
|
||||
if (addressTable.containsKey(keyHash)) {
|
||||
throw Exception('Key already exists');
|
||||
// Print with yellow [MASTER] prefix - use ansi
|
||||
void _mstPrint(String message) {
|
||||
print('\x1B[33m[MASTER]\x1B[0m $message');
|
||||
}
|
||||
|
||||
late final int concurrentReaders;
|
||||
file.setPositionSync(30);
|
||||
concurrentReaders = file.readIntSync(4);
|
||||
|
||||
while (true) {
|
||||
|
||||
int now = DateTime.now().millisecondsSinceEpoch32();
|
||||
file.setPositionSync(26);
|
||||
file.writeIntSync(now, 4);
|
||||
|
||||
_mstPrint('Master heartbeat updated at $now');
|
||||
|
||||
file.setPositionSync(35);
|
||||
for (int i = 0; i < concurrentReaders; i++) {
|
||||
|
||||
// Read ticket
|
||||
List<int> ticketBuffer = file.readSync(26);
|
||||
BT_Ticket ticket = BT_Ticket.fromIntList(ticketBuffer);
|
||||
|
||||
// Check if ticket is alive and waiting
|
||||
int ticketHeartbeat = ticket.heartbeat;
|
||||
bool isTicketAlive = (now - ticket.heartbeat) <= 5000;
|
||||
BT_TicketState state = ticket.state;
|
||||
if (isTicketAlive && state == BT_TicketState.WAITING) {
|
||||
_mstPrint("Ticket ${i} is alive and waiting...");
|
||||
|
||||
// If reading is still allowed, we need to disallow it
|
||||
file.setPositionSync(34);
|
||||
if (file.readByteSync() == 1) {
|
||||
_mstPrint("Disallowing reads for ticket processing...");
|
||||
file.setPositionSync(34);
|
||||
file.writeByteSync(0);
|
||||
// _liftFreeList(); // Broken/Incorrect Logic. the isolate cant access the main instance's free list
|
||||
}
|
||||
|
||||
// We need to give it a write pointer and approve it
|
||||
// BT_Pointer allocation = _alloc(ticket.writeSize); // Broken/Incorrect Logic. the isolate cant access the main instance's free list
|
||||
|
||||
BT_Ticket approvedTicket = BT_Ticket(
|
||||
heartbeat: ticket.heartbeat,
|
||||
state: BT_TicketState.APPROVED,
|
||||
opcode: ticket.opcode,
|
||||
keyHash: ticket.keyHash,
|
||||
writeSize: ticket.writeSize,
|
||||
// writePointer: allocation, // Broken/Incorrect Logic. the isolate cant access the main instance's free list
|
||||
);
|
||||
|
||||
// Write approved ticket back to file
|
||||
file.setPositionSync(35 + i * 26);
|
||||
file.writeFromSync(approvedTicket.toIntList());
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// If reads are disallowed, we need to allow them again, and drop the free list
|
||||
file.setPositionSync(34);
|
||||
if (file.readByteSync() == 0) {
|
||||
_mstPrint("Allowing reads after ticket processing...");
|
||||
file.setPositionSync(34);
|
||||
file.writeByteSync(1);
|
||||
dropFreeList();
|
||||
}
|
||||
|
||||
sleep(Duration(seconds: 1));
|
||||
}
|
||||
}, _file.path);
|
||||
}
|
||||
|
||||
bool get isMasterAlive {
|
||||
_file.setPositionSync(18);
|
||||
int masterId = _file.readIntSync(8);
|
||||
|
||||
_file.setPositionSync(26);
|
||||
int masterHeartbeat = _file.readIntSync(4);
|
||||
|
||||
int now = DateTime.now().millisecondsSinceEpoch32();
|
||||
return (now - masterHeartbeat) <= 5000;
|
||||
}
|
||||
|
||||
int get concurrentReaders {
|
||||
_file.setPositionSync(30);
|
||||
return _file.readIntSync(4);
|
||||
}
|
||||
|
||||
bool get allowReads {
|
||||
_file.setPositionSync(34);
|
||||
int flag = _file.readByteSync();
|
||||
return flag == 1;
|
||||
}
|
||||
|
||||
// Add ticket
|
||||
void _ticket({
|
||||
required BT_TicketOpcode operation,
|
||||
required int keyHash,
|
||||
required int writeSize,
|
||||
required void Function(BT_Ticket ticket)? onApproved,
|
||||
}) {
|
||||
|
||||
// Reduce the chance of a race condition/deadlock by adding a small random delay
|
||||
sleep(Duration(milliseconds: Random().nextInt(10)));
|
||||
|
||||
int? ticketIndex;
|
||||
|
||||
// Were gonna iterate through all the tickets and find an empty slot
|
||||
while (ticketIndex == null) {
|
||||
|
||||
_file.setPositionSync(35);
|
||||
for (int i = 0; i < concurrentReaders; i++) {
|
||||
|
||||
List<int> ticketBuffer = _file.readSync(26); // Size of a ticket entry
|
||||
|
||||
BT_Ticket ticket = BT_Ticket.fromIntList(ticketBuffer);
|
||||
|
||||
// Check if the heartbeat is stale (older than 5 seconds)
|
||||
int now = DateTime.now().millisecondsSinceEpoch64();
|
||||
if (now - ticket.heartbeat > 5000) {
|
||||
|
||||
// We found a stale ticket, we can take this slot
|
||||
ticketIndex = i;
|
||||
break;
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
sleep(Duration(milliseconds: 500));
|
||||
}
|
||||
|
||||
if (ticketIndex == null) {
|
||||
throw Exception('Failed to acquire ticket');
|
||||
}
|
||||
|
||||
BT_Ticket newTicket = BT_Ticket(
|
||||
heartbeat: DateTime.now().millisecondsSinceEpoch32(),
|
||||
state: BT_TicketState.WAITING,
|
||||
opcode: operation,
|
||||
keyHash: keyHash,
|
||||
writeSize: writeSize,
|
||||
);
|
||||
|
||||
// Write the new ticket to the file
|
||||
_file.setPositionSync(35 + ticketIndex * 38);
|
||||
_file.writeFromSync(newTicket.toIntList());
|
||||
|
||||
// Wait for approval
|
||||
while (true) {
|
||||
|
||||
print('Waiting for ticket approval...');
|
||||
|
||||
_file.setPositionSync(35 + ticketIndex * 38);
|
||||
List<int> ticketBuffer = _file.readSync(38);
|
||||
BT_Ticket ticket = BT_Ticket.fromIntList(ticketBuffer);
|
||||
|
||||
if (ticket.state == BT_TicketState.APPROVED) {
|
||||
// Ticket approved
|
||||
onApproved?.call(ticket);
|
||||
break;
|
||||
}
|
||||
|
||||
if (!isMasterAlive) {
|
||||
print('Master is not alive, cannot proceed with ticket');
|
||||
}
|
||||
|
||||
sleep(Duration(milliseconds: 1000));
|
||||
|
||||
// Update heartbeat
|
||||
_file.setPositionSync(35 + ticketIndex * 38);
|
||||
int now = DateTime.now().millisecondsSinceEpoch32();
|
||||
_file.writeIntSync(now, 4);
|
||||
}
|
||||
|
||||
|
||||
List<int> valueBuffer = encodeValue(value);
|
||||
|
||||
// Write value to file
|
||||
BT_Pointer valueAddress = alloc(valueBuffer.length);
|
||||
|
||||
_file.setPositionSync(valueAddress.address);
|
||||
_file.writeFromSync(valueBuffer);
|
||||
|
||||
// Update address table
|
||||
addressTable[keyHash] = valueAddress;
|
||||
_addressTable = addressTable;
|
||||
});
|
||||
|
||||
}
|
||||
|
||||
/*
|
||||
Key-Value Operations
|
||||
*/
|
||||
|
||||
operator []=(String key, dynamic value) {
|
||||
|
||||
Map<int, BT_Pointer> addressTable = _addressTable;
|
||||
|
||||
// Check if key already exists
|
||||
int keyHash = key.bt_hash;
|
||||
if (addressTable.containsKey(keyHash)) {
|
||||
throw Exception('Key already exists'); // The pair should be deleted first
|
||||
}
|
||||
|
||||
List<int> valueBuffer = encodeValue(value);
|
||||
|
||||
_ticket(
|
||||
operation: BT_TicketOpcode.WRITE, // Modification operations will come later,
|
||||
keyHash: keyHash,
|
||||
writeSize: valueBuffer.length,
|
||||
onApproved: (BT_Ticket ticket) {
|
||||
// Write value to file
|
||||
_file.setPositionSync(ticket.writePointer.address);
|
||||
_file.writeFromSync(valueBuffer);
|
||||
}
|
||||
);
|
||||
|
||||
// v1.0.0 implementation
|
||||
// _antiFreeListScope(() {
|
||||
// Map<int, BT_Pointer> addressTable = _addressTable;
|
||||
//
|
||||
// int keyHash = key.bt_hash;
|
||||
//
|
||||
// // 1.1 Note: I cant remember why i did this check here.
|
||||
// if (addressTable.containsKey(keyHash)) {
|
||||
// throw Exception('Key already exists');
|
||||
// }
|
||||
//
|
||||
//
|
||||
//
|
||||
// List<int> valueBuffer = encodeValue(value);
|
||||
//
|
||||
// // Write value to file
|
||||
// BT_Pointer valueAddress = _alloc(valueBuffer.length);
|
||||
//
|
||||
// _file.setPositionSync(valueAddress.address);
|
||||
// _file.writeFromSync(valueBuffer);
|
||||
//
|
||||
// // Update address table
|
||||
// addressTable[keyHash] = valueAddress;
|
||||
// _addressTable = addressTable;
|
||||
// });
|
||||
}
|
||||
|
||||
|
||||
operator [](String key) {
|
||||
|
||||
while (!allowReads) {
|
||||
// Wait until reads are allowed
|
||||
sleep(Duration(milliseconds: 1));
|
||||
}
|
||||
|
||||
Map<int, BT_Pointer> addressTable = _addressTable;
|
||||
|
||||
int keyHash = key.bt_hash;
|
||||
@@ -827,7 +1291,7 @@ class BinaryTable {
|
||||
|
||||
void delete(String key) {
|
||||
|
||||
antiFreeListScope(() {
|
||||
_antiFreeListScope(() {
|
||||
Map<int, BT_Pointer> addressTable = _addressTable;
|
||||
|
||||
int keyHash = key.bt_hash;
|
||||
@@ -840,7 +1304,7 @@ class BinaryTable {
|
||||
BT_Reference valueRef = BT_Reference(this, valuePointer);
|
||||
|
||||
// Free the value
|
||||
free(valuePointer, valueRef.size);
|
||||
_free(valuePointer, valueRef.size);
|
||||
|
||||
// Remove from address table
|
||||
addressTable.remove(keyHash);
|
||||
@@ -851,7 +1315,7 @@ class BinaryTable {
|
||||
|
||||
void truncate() {
|
||||
|
||||
antiFreeListScope(() {
|
||||
_antiFreeListScope(() {
|
||||
// Relocate the address table if possible
|
||||
_addressTable = _addressTable;
|
||||
|
||||
@@ -1065,3 +1529,13 @@ String binaryDump(Uint8List data) {
|
||||
|
||||
return buffer.toString();
|
||||
}
|
||||
|
||||
// DateTime to 32-bit Unix timestamp (seconds since epoch)
|
||||
extension on DateTime {
|
||||
int millisecondsSinceEpoch32() {
|
||||
return (millisecondsSinceEpoch ~/ 1000) & 0xFFFFFFFF;
|
||||
}
|
||||
int millisecondsSinceEpoch64() {
|
||||
return millisecondsSinceEpoch;
|
||||
}
|
||||
}
|
||||
107
documentation/header.md
Normal file
107
documentation/header.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# Sweepstore Header Structure
|
||||
|
||||
The Sweepstore file format uses a structured header to manage file metadata and concurrency control. The header consists of three main parts: the static header, the concurrency header, and dynamic worker tickets.
|
||||
|
||||
## Static Header (Bytes 0-28)
|
||||
|
||||
The static header contains basic file information and pointers.
|
||||
|
||||
| Offset | Size | Field | Type | Description |
|
||||
|--------|------|-------|------|-------------|
|
||||
| 0 | 4 bytes | Magic Number | String | File identifier, must be "SWPT" |
|
||||
| 4 | 12 bytes | Version | String | Version string (UTF-8), max 11 chars (padded with spaces) |
|
||||
| 16 | 8 bytes | Address Table Pointer | int64 | Pointer to the address table location |
|
||||
| 24 | 4 bytes | Free List Count | int32 | Number of entries in the free list |
|
||||
| 28 | 1 byte | Is Free List Lifted | bool | Flag indicating if free list is lifted (0=false, 1=true) |
|
||||
|
||||
**Total Size:** 29 bytes
|
||||
|
||||
## Concurrency Header (Bytes 29-45)
|
||||
|
||||
The concurrency header manages multi-threaded access and coordination.
|
||||
|
||||
| Offset | Size | Field | Type | Description |
|
||||
|--------|------|-------|------|-------------|
|
||||
| 29 | 8 bytes | Master Identifier | int64 | Unique identifier for the master process |
|
||||
| 37 | 4 bytes | Master Heartbeat | int32 | Heartbeat counter for the master process |
|
||||
| 41 | 4 bytes | Number of Workers | int32 | Total number of concurrent worker tickets |
|
||||
| 45 | 1 byte | Is Read Allowed | bool | Flag indicating if read operations are allowed (0=false, 1=true) |
|
||||
|
||||
**Total Size:** 17 bytes
|
||||
|
||||
## Worker Tickets (Starting at Byte 46)
|
||||
|
||||
Worker tickets are dynamically sized based on the number of workers specified in the concurrency header. Each ticket is 30 bytes.
|
||||
|
||||
**Base Offset Calculation:** `46 + (ticketIndex * 30)`
|
||||
|
||||
### Single Ticket Structure
|
||||
|
||||
| Relative Offset | Size | Field | Type | Description |
|
||||
|-----------------|------|-------|------|-------------|
|
||||
| 0 | 4 bytes | Identifier | int32 | Unique identifier for this worker |
|
||||
| 4 | 4 bytes | Worker Heartbeat | int32 | Heartbeat counter for this worker |
|
||||
| 8 | 1 byte | Ticket State | byte (enum) | Current state of the ticket (see SweepstoreTicketState) |
|
||||
| 9 | 1 byte | Ticket Operation | byte (enum) | Current operation being performed (see SweepstoreTicketOperation) |
|
||||
| 10 | 8 bytes | Key Hash | int64 | Hash of the key being operated on |
|
||||
| 18 | 8 bytes | Write Pointer | int64 | Pointer to the write location |
|
||||
| 26 | 4 bytes | Write Size | int32 | Size of the write operation |
|
||||
|
||||
**Ticket Size:** 30 bytes
|
||||
|
||||
## Enumerations
|
||||
|
||||
Enum fields are stored as single-byte integers. The following tables show the integer values for each enum state:
|
||||
|
||||
### SweepstoreTicketState (1 byte)
|
||||
|
||||
| Value | Name | Description |
|
||||
|-------|------|-------------|
|
||||
| 0 | IDLE | Ticket is idle and not performing any work |
|
||||
| 1 | WAITING | Ticket is waiting for approval |
|
||||
| 2 | APPROVED | Ticket has been approved to proceed |
|
||||
| 3 | EXECUTING | Ticket is actively executing an operation |
|
||||
| 4 | COMPLETED | Ticket has completed its operation |
|
||||
|
||||
### SweepstoreTicketOperation (1 byte)
|
||||
|
||||
| Value | Name | Description |
|
||||
|-------|------|-------------|
|
||||
| 0 | NONE | No operation assigned |
|
||||
| 1 | READ | Read operation |
|
||||
| 2 | MODIFY | Modify operation |
|
||||
| 3 | WRITE | Write operation |
|
||||
|
||||
## Total Header Size Calculation
|
||||
|
||||
The total header size depends on the number of workers:
|
||||
|
||||
```
|
||||
Total Header Size = 46 + (numberOfWorkers * 30) bytes
|
||||
```
|
||||
|
||||
For example:
|
||||
- 4 workers: 46 + (4 <20> 30) = 166 bytes
|
||||
- 8 workers: 46 + (8 <20> 30) = 286 bytes
|
||||
|
||||
## Initialization
|
||||
|
||||
When initializing a new Sweepstore file using `initialiseSweepstoreHeader()`:
|
||||
- Magic number is set to "SWPT"
|
||||
- Version is set to "undefined"
|
||||
- Address table pointer is set to null pointer
|
||||
- Free list count is set to 0
|
||||
- Is free list lifted flag is set to false
|
||||
- Master identifier and heartbeat are set to 0
|
||||
- Number of workers is set according to the parameter (default: 4)
|
||||
- Read allowed flag is set to false
|
||||
- All worker tickets are initialized with identifier set to 0, heartbeat set to 0, IDLE state (0), and NONE operation (0)
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- All multi-byte integers are stored in little-endian byte order
|
||||
- The version string is padded with spaces and prefixed with a space character
|
||||
- Boolean values are stored as single bytes (0 or 1)
|
||||
- Enum values are stored as single-byte integers using their index values (0, 1, 2, etc.)
|
||||
- Pointers use int64 for addressing, with -1 representing a null pointer
|
||||
- The header is designed for concurrent access with heartbeat-based liveness detection
|
||||
BIN
example.bin
BIN
example.bin
Binary file not shown.
Reference in New Issue
Block a user