feat: let the generator read its own results, and measure it honestly
Nothing in the pipeline has ever fed an outcome back to the thing that makes predictions. Calibration reads autonomy_outcomes, but calibration only gates whether to act on a prediction, never what the prediction is. So the only thing that has ever changed this system's output is a human editing the prompt. score-replay-runs.js asks "compared to what". The answer is not flattering: over 2,114 scored replay predictions the system is right 50.05% of the time while answering "negative" to every one of the same bars scores 54.45%. It is 4.4 points below a constant, z=-4.06. The whole deficit is the prior. It says positive on 65% of calls when 45.5% of bars beat SPY, a 19 point skew. Its discrimination, P(up|positive) minus P(up|negative), is +3.2 points with p=0.15, so the direction it picks is weakly informative and completely buried by how often it defaults to positive. The first version of that script compared each direction group's accuracy to "always that direction" on the same rows, which is an identity and tests nothing. T3 replaces it with the two proportion test that actually asks whether the choice of direction carries information. build-feedback-brief.js turns a run's scored outcomes into a memo the next run reads before predicting. Generated from the data, not written by hand, or it is just me editing the prompt again with extra steps. Replay can now be pinned to an explicit article set, which is what makes two runs comparable at all. Comparing two calendar windows of one run compares two market regimes: the epochs in run 1 line up exactly with article vintage, E0 is late 2024 and E2 is 2026, so nothing could be attributed. A new run also inherits its parent's watermark instead of recomputing it from today, which silently guaranteed a different archive slice every time. prompt_version never moved across four material prompt changes, so every proposal on record claims to come from the first prompt. coordinator-2 and replay-coordinator-2. docs/replay-run-2-preregistration.md fixes the bar before the run exists, including which result counts as learning and which is only calibration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
This commit is contained in:
@@ -14,6 +14,10 @@ function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); }
|
||||
// bumped when the prompt changes in a way that changes what a prediction means.
|
||||
// autonomy-2 is the first version that can see the relationship graph.
|
||||
const STRATEGY_VERSION = 'autonomy-2';
|
||||
// This never moved across four prompt changes, so every proposal on record
|
||||
// claims to come from the same prompt as the very first one. Attribution was
|
||||
// impossible, which is why the only honest before/after we had was a timestamp.
|
||||
const PROMPT_VERSION = 'coordinator-2';
|
||||
|
||||
function loadConfig() {
|
||||
const configPath = path.resolve(process.env.DURIIN_CONFIG || path.join(__dirname, '..', 'config.json'));
|
||||
@@ -72,7 +76,7 @@ async function runCoordinatorWorker({ archivePath, intelligencePath, workerId =
|
||||
eventId: event.id,
|
||||
informationCutoff,
|
||||
model: config.openRouter.llmModel || 'unknown',
|
||||
promptVersion: 'coordinator-1',
|
||||
promptVersion: PROMPT_VERSION,
|
||||
strategyVersion: STRATEGY_VERSION,
|
||||
// only a genuine live lane job may ever feed learning
|
||||
origin: historical ? 'historical' : 'live',
|
||||
@@ -84,7 +88,7 @@ async function runCoordinatorWorker({ archivePath, intelligencePath, workerId =
|
||||
eventId: event.id,
|
||||
informationCutoff,
|
||||
model: config.openRouter.llmModel || 'unknown',
|
||||
promptVersion: 'coordinator-1',
|
||||
promptVersion: PROMPT_VERSION,
|
||||
origin: historical ? 'historical' : 'live',
|
||||
learningEligible: !historical,
|
||||
}, validationError.message);
|
||||
|
||||
Reference in New Issue
Block a user