fix: stop the coordinator copying its own prompt example

The JSON shape in both prompts used real values as placeholders, and the model
was reading them as the answer:

  instrument: 'NVDA'          -> 397 of 611 predictions are NVDA (65%),
                                 second place is LMT with 8
  horizon_days: 10            -> 606 of 611 are horizon 10 (99.2%), out of
                                 seven allowed horizons
  direction: 'positive|negative' -> 502 of 611 are positive (82.2%)
  event_type: 'stable_enum'   -> the enum was never listed, so the model
                                 invented one label per event, 201 distinct
                                 values across 611 predictions

replayWorker had its own copy of the same prompt with the same values, which is
why both lanes show the identical skew (replay is 147/147 horizon 10, 138/147
NVDA).

Every placeholder is now a description of the field rather than a usable value,
with an explicit line saying not to copy them. event_type is validated against
the same closed family list the cohort key uses, so a label cannot mean one
thing in the prompt and another in calibration. Off-enum labels are salvaged
through the existing mapper when they are placeable and rejected when they are
not, so 'other' does not quietly become the bin again.

This does not by itself create edge. It means the next batch of predictions
measures the model's judgement instead of its willingness to copy an example.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
This commit is contained in:
ImBenji
2026-08-29 23:29:11 +01:00
co-authored by Claude Opus 5
parent c4650fe45c
commit b8b3987e35
4 changed files with 61 additions and 8 deletions
+22
View File
@@ -272,3 +272,25 @@ test('order intents require the explicit instrument allowlist', () => {
assert.equal(intent.side, 'buy');
assert.equal(db.prepare('SELECT status FROM autonomy_order_intents').get().status, 'shadow');
});
test('event_type is held to the closed family enum', () => {
const base = { instrument: 'NVDA', direction: 'positive', horizon_days: 10, evidence_article_ids: [1] };
const typeOf = (eventType) => normalizeProposal({ predictions: [{ ...base, event_type: eventType }] }).predictions[0].eventType;
// the enum itself, in the shapes a model actually emits
assert.equal(typeOf('earnings'), 'earnings');
assert.equal(typeOf('Supply Chain'), 'supply_chain');
assert.equal(typeOf('M_AND_A'), 'm_and_a');
// "none of these fit" is a legitimate answer and has to survive
assert.equal(typeOf('other'), 'other');
// off-enum but placeable: salvaged onto the family calibration would have
// picked anyway, so we dont throw away a usable prediction over a label
assert.equal(typeOf('earnings_beat_q3'), 'earnings');
assert.equal(typeOf('ceo resignation'), 'leadership');
// unplaceable free text is the 201-distinct-values failure, and is rejected
assert.throws(() => typeOf('vibes_shifted'), /event_type must be one of/);
assert.throws(() => typeOf(''), /event_type must be one of/);
});