Two things wrong with the fixed cap I added earlier.
It was guessed rather than measured. The largest proposal this has ever produced
was about 1,258 tokens carrying 12 predictions and the average is around 50, so
8000 was arbitrary, and worse, it was above what the key could afford by the
time it deployed. The ceiling openrouter will accept shrinks as the balance
depletes: 15,666 earlier today, 3,921 an hour later.
So the cap is now adaptive. A 402 names the ceiling, and we retry once just under
it, downwards only. A shrinking budget shortens the allowed answer instead of
stopping the pipeline dead. Worth being clear that removing the cap is not an
option on a limited key, an unbounded request is refused outright and produces
no output at all rather than a truncated one.
And truncation is no longer silent. finish_reason length now throws instead of
handing a half written response to extractJson, which could occasionally parse a
partial object and quietly drop predictions. That was the real risk in capping.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
signal, augor and consolidation had the same unbounded request as the
coordinator, so switching to a model with a 131k output window made all three
402 on every call while the coordinator itself was fine. Found them by grepping
for the endpoint rather than waiting for each one to surface in the logs.
Sized per worker rather than one global number, since these produce more than
the coordinator's small json object.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
Neither the coordinator nor the graph resolver set max_tokens, so openrouter
reserved the model's entire output window against the key's remaining budget and
returned 402 before running anything: 131k tokens reserved to produce a few
hundred. It never showed up with the old model because its output window is
small enough to fit under the limit.
Both are now bounded and overridable by env. The headroom is deliberate, the
newer reasoning models spend completion tokens thinking before they answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
Measured on the deployed console: the page shell and every module land in 39ms,
so the 1.4s was entirely /admin/api/ops/overview, fetched twice.
Twice because the sidebar and the overview view each called usePoll on the same
url, each with its own timer. usePoll now keeps one store per url, so any number
of subscribers share a single request and a single interval, and a request
already on the wire is joined rather than duplicated.
The endpoint itself was dominated by the jobs rollup: a full scan of
autonomy_jobs, 521k rows and growing about nine thousand a day. A covering index
on (job_type, lane, status, created_at) takes it from 548ms to 110ms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
Admin assets are no-store, so every load refetches all of them, and es module
imports are discovered one level at a time: main parses, then its imports are
found, then theirs. Preload hints turn that waterfall into one parallel burst.
The hrefs deliberately carry no version query because they have to byte-match
what the import specifiers resolve to, otherwise the browser fetches each module
twice instead of reusing the preload.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
htm passes props through untouched and react rejects a style string with error
#62, so every view rendered an empty page. Converting once in the createElement
wrapper keeps plain css in the templates instead of style objects everywhere.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
The old admin was five separate html pages, so every navigation was a full
reload and the operational picture was scattered across all of them. Worth
saying: the api was never the problem, every endpoint answers in under 200ms.
It felt slow because of the architecture, not the backend.
This is a single page console. React and htm from a cdn, no bundler and no
babel-in-the-browser, because a runtime transpiler on every load is exactly the
slowness we are trying to get rid of. Hash routing, polling that keeps the last
good payload on screen instead of flashing a spinner, and stale responses are
dropped so a slow request cannot overwrite a newer one.
/admin/api/ops/overview answers the whole dashboard in one call rather than
making the browser fan out and stitch. It carries the things that actually
matter and were not visible anywhere before: when live evidence matures, why a
cohort does or does not clear the trade gate, which stage of the pipeline has
gone quiet, and what is sitting in dead letters.
Controls, all of which change production and all of which ask twice:
- requeue dead letters, which only ever moves dead_letter back to pending
- execution mode and a kill switch, now read from autonomy_settings on every
poll instead of only from AUTONOMY_EXECUTION_MODE, so halting no longer needs
a redeploy first
- run the reaction analysis and read its output
The d3 graph is framed rather than ported. It works, and rewriting it would risk
something valuable for nothing the operator can see.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
The content backfill picker took 194 seconds per call. better-sqlite3 is
synchronous, so that blocked the whole ingest event loop, and with eight workers
each running it in a loop they serialised behind each other: about 26 minutes a
round. From outside it looked like a hang, and the gdelt loop went quiet at the
same time because it was stuck behind the same blocked loop.
Two causes. The picker tested `content IS NULL OR TRIM(content) = ''`, which
made sqlite read the content column, a 4GB blob, purely to decide which rows to
skip. content_status already records the same thing and agrees with the content
column on all 2.2M rows, so the test bought nothing. It also stopped any index
being usable.
Then there was no index matching the window function, so it built temp b-trees
over every unfetched row. idx_articles_pending_fetch is partial and column
ordered to match PARTITION BY source ORDER BY pub_date_effective DESC, id DESC.
The planner ignores it without stats, hence PRAGMA optimize.
Measured on production, same query, same 26k rows: 194.5s -> 1.03s.
Note for whoever reads this next: the playwright page slot leak fixed in 42fb929
was real but was not what froze the pipeline. This was.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
Three separate things had the pipeline frozen for 27 hours.
browserCrawler leaked page slots. context.newPage() sat outside the try, so a
throw or a hang there took the slot with it, and after maxConcurrentPages of
those every caller parked in acquirePageSlot forever. That is what it looked
like from outside: content workers alive, no logs, no progress, 13 chromium
renderers still up 10 hours after start. newPage is inside the try now, waiting
for a slot times out instead of blocking forever, and page.close() is raced so a
wedged renderer cant strand the slot on the way out either.
graphWorker had no backoff on quota failures. A blown OpenRouter monthly limit
returns an instant 403, so it retried as fast as the network allowed: 2356
failures in 20 minutes, drowning every other line in the log. Quota and auth
errors now pause resolution for 15 minutes and log once per window rather than
once per attempt.
The outcome worker retried unresolvable predictions forever. Yahoo writes class
shares with a dash, so BRK.B 404s every time, and a failed prediction stays open
and comes straight back on the next poll. Dots are translated to dashes, which
matters beyond this one name because the allowlist is full of dotted symbols,
and a prediction that fails five times is marked unresolvable instead of
spinning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
The JSON shape in both prompts used real values as placeholders, and the model
was reading them as the answer:
instrument: 'NVDA' -> 397 of 611 predictions are NVDA (65%),
second place is LMT with 8
horizon_days: 10 -> 606 of 611 are horizon 10 (99.2%), out of
seven allowed horizons
direction: 'positive|negative' -> 502 of 611 are positive (82.2%)
event_type: 'stable_enum' -> the enum was never listed, so the model
invented one label per event, 201 distinct
values across 611 predictions
replayWorker had its own copy of the same prompt with the same values, which is
why both lanes show the identical skew (replay is 147/147 horizon 10, 138/147
NVDA).
Every placeholder is now a description of the field rather than a usable value,
with an explicit line saying not to copy them. event_type is validated against
the same closed family list the cohort key uses, so a label cannot mean one
thing in the prompt and another in calibration. Off-enum labels are salvaged
through the existing mapper when they are placeable and rejected when they are
not, so 'other' does not quietly become the bin again.
This does not by itself create edge. It means the next batch of predictions
measures the model's judgement instead of its willingness to copy an example.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
The coordinator is scored on excess return vs SPY starting at the information
cutoff, so the announcement move sits outside the scored window. That move is
the best documented conditioner for post event drift and we were discarding it.
This measures whether keeping it would buy us anything, before any of it gets
wired into the cohort key or the prompt.
Reaction is measured from the last close before the event's first article up to
the outcome's own entry price, so the reaction and forward windows touch but
never overlap. Read only, and it caches price history so it can be re-run cheaply
as more outcomes mature.
T1 and T2 are pre-registered in the header because sweeping buckets over 611
outcomes that are half one ticker will always turn up something.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
Offline evidence can no longer authorise anything. createDecisions used to
prefer a live snapshot and fall back to the pooled historical one, so once the
live lane woke up a live prediction could have drawn a BUY off backfill data.
Backfill and replay are fine evidence that the pipeline works, they are not a
live track record.
No live snapshot now means ABSTAIN. The abstain says whether offline evidence
existed for that cohort, so "we have 60 offline samples but no live ones" stays
distinguishable from "we know nothing about this cohort".
Also split the health counter. It counted qualifying cohorts across every
source, which overstated how close we are to being able to trade now that only
live cohorts can authorise. It reports qualifying_live_cohorts and
qualifying_offline_cohorts separately, and applies the concentration cap it was
previously ignoring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
Archive ingestion had been dead since 2026-08-02 because nothing in the
compose stack actually ran it. Everything downstream starved from there.
- add ingest + enrichment services. server.js only starts the scheduler when
DURIIN_RUN_SCHEDULER is not "false", and workers/index.js was not running at
all, so articles never got event_id/content/has_embedding and the coordinator
had nothing to lease.
- pass an explicit origin from coordinatorWorker. it was never passed, so
acceptProposal defaulted to 'live' and 464 historical backfill predictions
were recorded as live. that also meant verifyEvidence got a null cutoff and
skipped its date check entirely.
- coarsen cohortKey to event families + horizon buckets. 201 free text event
types produced 221 cohorts averaging 2.76 samples, so the n>=30 gate could
never be reached and everything abstained for the wrong reason.
- gate on cohort diversity, not just sample count. one ticker was roughly half
of all resolved outcomes, so a pure count gate was measuring one company.
unknown diversity abstains rather than passing.
- resolve the admin archive db explicitly and probe it. it relied on a
Dockerfile symlink, and without it better-sqlite3 quietly creates an empty
file and serves a phantom archive.
- clamp implausible future publication dates at ingest.
- pin the db backend to sqlite by default. compose hardcoded postgres "true",
which would have overridden the operator's own .env on the next redeploy and
pointed everything at a stale snapshot.
scripts/repair-autonomy-labels.js relabels the affected rows. it is dry run by
default and has not been applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
