fix: a recovered replay job can no longer hijack the active run
leaseNextJob hands back any pending replay_article job, it has no idea about runs, and the worker was attributing whatever came back to whichever run was active. One recovered dead letter from run 1 would have been stamped with run 2's id, given run 2's feedback brief, and dragged run 2's cursor to wherever that old article sits in the archive. A pinned run would then decide its set was finished after a couple of articles. There are 139 dead letters and they are built to recover, so this was not hypothetical. The idempotency key already says which run enqueued the job. Ask it. Also: refuse to inherit the parent's model label when starting a run. Inheriting is exactly how run 1 came to be labelled qwen for predictions deepseek made. The split moves to the replay container's actual restart time rather than the commit timestamp five minutes later. Verified the running container really does have the instrument rules, the de-anchoring and the enum before trusting it as the boundary. It makes no difference to the partition, there are no replay predictions at all between 15:57 and midnight that day, but the boundary should be the thing that actually changed the prompt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
This commit is contained in:
@@ -21,14 +21,27 @@ One variable. Run 2 answers exactly the articles run 1 answered under the
|
||||
current prompt and the current model, with the same model, same prompt, plus a
|
||||
feedback brief generated by `scripts/build-feedback-brief.js`.
|
||||
|
||||
- Split on `created_at >= "2026-09-04 19:30"`, the deploy that put the instrument
|
||||
rules into the replay prompt. Everything before it is TRAINING, everything at
|
||||
or after it is EVALUATION.
|
||||
- Split on `created_at >= "2026-09-04 19:17:43"`, when `duriin-api-replay-1`
|
||||
restarted onto the prompt it still runs today. That is the container start
|
||||
time, not the commit timestamp, which is five minutes later and would have put
|
||||
a handful of old-prompt predictions on the new-prompt side. Verified directly:
|
||||
the running container has the instrument rules, the de-anchored placeholders
|
||||
and the event_type enum in `/app/workers/replayWorker.js`.
|
||||
- There is no contamination to argue about. Replay was between daily budgets
|
||||
across the restart, so no replay prediction exists between 15:57 on 09-04 and
|
||||
00:00 on 09-05. Splits at 19:17:43, at 19:30 and at midnight all produce the
|
||||
identical partition, 1,142 training and 984 evaluation.
|
||||
- The brief is derived from the 1,133 training predictions only. No evaluation
|
||||
row contributes a single number to the text. Deriving the lesson and grading it
|
||||
on the same rows would measure nothing.
|
||||
- Evaluation set: 713 articles, 981 run-1 predictions, all
|
||||
`~deepseek/deepseek-v4-flash-latest`.
|
||||
- Evaluation set: 716 articles, 984 run-1 predictions, all
|
||||
`~deepseek/deepseek-v4-flash-latest`. 713 of those articles have a scored
|
||||
run-1 prediction and are the pairable set; the other three are replayed but
|
||||
cannot enter T4.
|
||||
- The 1,133 scored training predictions span two models, roughly 627 qwen and
|
||||
506 deepseek. So the brief describes the mistakes of the system as it has
|
||||
been, not of deepseek alone. Run 2 is deepseek throughout, as is the run-1
|
||||
half it is measured against.
|
||||
|
||||
## Run 1 on the evaluation slice, the bar
|
||||
|
||||
@@ -74,3 +87,12 @@ model the base rate will pull it toward the base rate. So:
|
||||
|
||||
Beating run 1 while still sitting below 57.39% is not a system worth trading.
|
||||
That distinction gets reported every time, not just when it is convenient.
|
||||
|
||||
## Housekeeping that is easy to forget
|
||||
|
||||
Run 2 stays `running` once it exhausts its 716 articles, and the replay lane
|
||||
just idles. That is intended, it keeps the spend at zero while the outcomes
|
||||
mature. Mark it `complete` when the results are read, otherwise it becomes the
|
||||
same stale metadata run 1 carried for a month. But do not mark it complete
|
||||
before reading, because `activeRun` would immediately create run 3 with no
|
||||
pinned set and no brief and start walking all 23k articles again.
|
||||
|
||||
Reference in New Issue
Block a user