fix: stop the coordinator copying its own prompt example
The JSON shape in both prompts used real values as placeholders, and the model
was reading them as the answer:
instrument: 'NVDA' -> 397 of 611 predictions are NVDA (65%),
second place is LMT with 8
horizon_days: 10 -> 606 of 611 are horizon 10 (99.2%), out of
seven allowed horizons
direction: 'positive|negative' -> 502 of 611 are positive (82.2%)
event_type: 'stable_enum' -> the enum was never listed, so the model
invented one label per event, 201 distinct
values across 611 predictions
replayWorker had its own copy of the same prompt with the same values, which is
why both lanes show the identical skew (replay is 147/147 horizon 10, 138/147
NVDA).
Every placeholder is now a description of the field rather than a usable value,
with an explicit line saying not to copy them. event_type is validated against
the same closed family list the cohort key uses, so a label cannot mean one
thing in the prompt and another in calibration. Off-enum labels are salvaged
through the existing mapper when they are placeable and rejected when they are
not, so 'other' does not quietly become the bin again.
This does not by itself create edge. It means the next batch of predictions
measures the model's judgement instead of its willingness to copy an example.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
This commit is contained in:
@@ -1,5 +1,28 @@
|
||||
const { EVENT_FAMILY_NAMES, normalizeEventType } = require('./calibration');
|
||||
|
||||
const ALLOWED_DIRECTIONS = new Set(['positive', 'negative']);
|
||||
const ALLOWED_HORIZONS = new Set([1, 5, 10, 20, 30, 60, 90]);
|
||||
// The prompt used to say event_type was a 'stable_enum' and then never listed the
|
||||
// enum, so the model invented one label per event and production ended up with 201
|
||||
// distinct values. Same closed set the cohort key uses, so a label can never mean
|
||||
// one thing in the prompt and another in calibration.
|
||||
const ALLOWED_EVENT_TYPES = new Set(EVENT_FAMILY_NAMES);
|
||||
|
||||
// An exact family is what we want. If the model ignores the list we try to salvage
|
||||
// the label through the same mapper calibration uses, and only give up when it is
|
||||
// unplaceable -- an explicit 'other' is a legitimate answer, unplaceable free text
|
||||
// is not, and the difference is what stops 'other' quietly becoming the bin again.
|
||||
function normalizeProposedEventType(raw, instrument) {
|
||||
const value = String(raw || '').trim().toLowerCase().replace(/[\s-]+/g, '_');
|
||||
if (ALLOWED_EVENT_TYPES.has(value)) return value;
|
||||
|
||||
const salvaged = normalizeEventType(raw);
|
||||
if (salvaged !== 'other') {
|
||||
console.warn(`[coordinator] ${instrument} event_type "${raw}" is not in the enum, mapped to "${salvaged}"`);
|
||||
return salvaged;
|
||||
}
|
||||
throw new Error(`event_type must be one of ${EVENT_FAMILY_NAMES.join(', ')} (got "${raw}")`);
|
||||
}
|
||||
|
||||
function normalizeProposal(raw, { informationCutoff, model = 'unknown', promptVersion = 'unknown' } = {}) {
|
||||
if (!raw || typeof raw !== 'object') throw new Error('coordinator output must be an object');
|
||||
@@ -18,7 +41,7 @@ function normalizeProposal(raw, { informationCutoff, model = 'unknown', promptVe
|
||||
return {
|
||||
instrument,
|
||||
direction,
|
||||
eventType: String(item.event_type || 'unknown').trim().toLowerCase(),
|
||||
eventType: normalizeProposedEventType(item.event_type, instrument),
|
||||
causalChannel: item.causal_channel ? String(item.causal_channel).trim() : null,
|
||||
horizonDays,
|
||||
evidenceArticleIds: [...new Set(articleIds)],
|
||||
|
||||
Reference in New Issue
Block a user