IMBENJI.NET/Duriin-API
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: IMBENJI.NET/Duriin-API:main
Branches Tags
IMBENJI.NET/Duriin-API:main IMBENJI.NET/Duriin-API:codex/duriin-autonomy-overhaul
...
compare: IMBENJI.NET/Duriin-API:b246bd9d4bfc392de2a7a4bc19d3c19633df99e9
Branches Tags
IMBENJI.NET/Duriin-API:codex/duriin-autonomy-overhaul IMBENJI.NET/Duriin-API:main
55 Commits
main ... b246bd9d4b
Author SHA1 Message Date
ImBenjiandClaude Opus 5 b246bd9d4b fix: a recovered replay job can no longer hijack the active run 
leaseNextJob hands back any pending replay_article job, it has no idea about
runs, and the worker was attributing whatever came back to whichever run was
active. One recovered dead letter from run 1 would have been stamped with run
2's id, given run 2's feedback brief, and dragged run 2's cursor to wherever
that old article sits in the archive. A pinned run would then decide its set was
finished after a couple of articles. There are 139 dead letters and they are
built to recover, so this was not hypothetical.

The idempotency key already says which run enqueued the job. Ask it.

Also: refuse to inherit the parent's model label when starting a run. Inheriting
is exactly how run 1 came to be labelled qwen for predictions deepseek made.

The split moves to the replay container's actual restart time rather than the
commit timestamp five minutes later. Verified the running container really does
have the instrument rules, the de-anchoring and the enum before trusting it as
the boundary. It makes no difference to the partition, there are no replay
predictions at all between 15:57 and midnight that day, but the boundary should
be the thing that actually changed the prompt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-08 01:01:36 +01:00
ImBenjiandClaude Opus 5 ec29e64e96 feat: let the generator read its own results, and measure it honestly 
Nothing in the pipeline has ever fed an outcome back to the thing that makes
predictions. Calibration reads autonomy_outcomes, but calibration only gates
whether to act on a prediction, never what the prediction is. So the only thing
that has ever changed this system's output is a human editing the prompt.

score-replay-runs.js asks "compared to what". The answer is not flattering:
over 2,114 scored replay predictions the system is right 50.05% of the time
while answering "negative" to every one of the same bars scores 54.45%. It is
4.4 points below a constant, z=-4.06. The whole deficit is the prior. It says
positive on 65% of calls when 45.5% of bars beat SPY, a 19 point skew. Its
discrimination, P(up|positive) minus P(up|negative), is +3.2 points with
p=0.15, so the direction it picks is weakly informative and completely buried
by how often it defaults to positive.

The first version of that script compared each direction group's accuracy to
"always that direction" on the same rows, which is an identity and tests
nothing. T3 replaces it with the two proportion test that actually asks whether
the choice of direction carries information.

build-feedback-brief.js turns a run's scored outcomes into a memo the next run
reads before predicting. Generated from the data, not written by hand, or it is
just me editing the prompt again with extra steps.

Replay can now be pinned to an explicit article set, which is what makes two
runs comparable at all. Comparing two calendar windows of one run compares two
market regimes: the epochs in run 1 line up exactly with article vintage, E0 is
late 2024 and E2 is 2026, so nothing could be attributed. A new run also
inherits its parent's watermark instead of recomputing it from today, which
silently guaranteed a different archive slice every time.

prompt_version never moved across four material prompt changes, so every
proposal on record claims to come from the first prompt. coordinator-2 and
replay-coordinator-2.

docs/replay-run-2-preregistration.md fixes the bar before the run exists,
including which result counts as learning and which is only calibration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-08 00:56:43 +01:00
ImBenjiandClaude Opus 5 028c07688d fix: bound the market data fetch so the outcome worker cannot wedge 
Live outcomes have been stuck at 4 while 35 matured live predictions sat
unscored, the oldest four days past its horizon. Running those three by hand
resolves them in 200-400ms each, so the data was there and the maths was fine.
The worker itself was wedged: up three days, last log 6 Sept, processing
nothing.

req.setTimeout only covers socket inactivity. A response that opens and then
stalls leaves the promise pending forever, and with it the entire loop, because
the fetch is awaited inline. There is now a hard bound around it.

This is the third time an unbounded await inside a long lived loop has silently
stopped a worker: the browser session in content, the content round itself, and
now market data. In every case the container stayed up, nothing threw, and
nothing was logged, which is the worst possible failure shape. So the worker also
announces what it is about to score, because an idle worker and a dead one
should not look identical from outside.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-08 00:25:52 +01:00
ImBenjiandClaude Opus 5 65a90820f6 fix: a wedged browser can no longer stop content fetching silently 
Content fetching stopped dead on 4 Sept at 17:06 and nobody noticed for three
days. In that window it managed about 328 articles and then produced nothing:
no error, no timeout, not a single line in the log. Meanwhile the live lane
starved, because an article needs content before it can be embedded, clustered
and handed to the coordinator, and 6596 articles arrived in 48 hours with zero
of them ready.

Every individual browser step already had a timeout. Acquiring the shared
session did not, and it is awaited while holding one of eight browser slots, so
a wedged chromium parks every slot permanently and nothing ever throws. The page
slot timeout added earlier never fired because it sits downstream of the thing
that was actually stuck. There is now an outer bound around the whole browser
path so the slot always comes back.

The workers also log the start and end of each round. The reason this took days
to find is that a healthy content worker and a completely wedged one looked
identical from outside, and that is worth fixing on its own.

Verified on the box first: outbound fetches return 200, the picker returns rows
in 4.8s, and fetchAndStoreContent stores a real article in 354ms. Every part
worked in isolation, which is what made the silence so misleading.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-07 23:10:54 +01:00
ImBenjiandClaude Opus 5 9e6ef20560 feat: let the coordinator see the relationship graph 
The graph has been built for months and fed nothing but a dashboard. Nothing in
the autonomy pipeline has ever read an edge: not the coordinator, not
calibration, not execution. Its only downstream consumer, trade_signals, last
produced anything in April. It was roughly 70% of the llm bill and informed no
prediction, decision or order.

Relationships are the one piece of context a per-event coordinator genuinely
cannot derive from its own articles, because "this company supplies that one" is
knowledge about companies rather than about this event. So the coordinator now
receives the relationships of the companies the event is about, and is told to
name the relationship in causal_channel when it reasons through one.

The cutoff filter is the part that matters. first_seen_at on a relationship is
derived from article dates rather than processing time, so a historical proposal
only sees what the world had actually revealed by its own cutoff. Without that
this feature would quietly reintroduce the lookahead the evidence check exists to
prevent, and it is pinned by a test rather than left to review.

Relationships are explicitly background rather than evidence: predictions still
have to cite the article ids the story came from, and an instrument the articles
give no reason to care about is still not a prediction.

strategy_version moves to autonomy-2, because a prompt change this material
changes what a prediction means and the two populations should be comparable
later rather than silently blended.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-04 22:46:26 +01:00
ImBenjiandClaude Opus 5 9b490c4d39 fix: honour OPEN_ROUTER_CHEAP_MODEL, it was set and never read 
The env var has been set to google/gemma-4-31b-it all along and nothing ever
mapped it onto openRouter.cheapModel, so graphWorker's fallback chain silently
used the main model instead. Graph entity resolution is the highest volume llm
call in the system and its entire job is to reply with the number of a match.

Measured on the live key, same prompt:

  deepseek v4 flash   7564 completion tokens, 7558 of them reasoning  $0.0013708
  gemma-4-31b-it        14 completion tokens, 0 reasoning             $0.0000121

113x. Gemma is actually the more expensive model per token, which is why this
was worth measuring rather than reasoning about prices: the cost is not the
price of the tokens, it is a reasoning model spending seven thousand tokens
thinking about a multiple choice question.

This also explains the reasoning tokens dominating the usage dashboard, and why
the daily spend roughly doubled today rather than yesterday. Removing the token
ceiling let a trivial prompt reason without bound. The ceiling was never the
right control for that, the model choice is.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-04 21:57:17 +01:00
ImBenjiandClaude Opus 5 f464c94708 fix: drop untradable predictions instead of the whole proposal 
One untradable ticker rejected everything alongside it. In a single day that was
93 proposals discarding 171 predictions, and 76 of those named something we could
trade perfectly well. They were lost because a sibling in the same response said
EURUSD.

Tradability is a filter, so it applies per prediction now. The untradable one is
dropped and logged, its siblings are kept, and the stored payload records what
was removed so the filtering is auditable rather than invisible.

Lookahead deliberately still rejects the entire proposal. Evidence that did not
exist at the proposal's own cutoff means the response is corrupt rather than
merely untradable, and keeping the rest of it would hide the one thing most worth
seeing. Both halves are pinned by tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-04 20:16:55 +01:00
ImBenjiandClaude Opus 5 75e183aaa3 feat: tell the coordinator which instruments it can actually trade 
96% of rejected proposals name something untradable: indices (SPX, DXY, ^TNX),
fx (EURUSD, XAU/USD), futures (CL=F, BZ=F) and home listings (VOW3.DE, RHM.DE,
1211.HK, 688169.SS). The analysis behind those is usually sound, it is the ticker
that cannot be used, and nothing in the prompt ever said so. We were paying for
the call and discarding the result at validation.

The rules point the model at what the allowlist actually holds: US listings and
ADRs for foreign companies, and US listed ETFs as the tradable expression of an
index, currency, rate or commodity. Every symbol named in the rules was checked
against the live allowlist first, so VWAGY, BABA, TM, SONY, SPY, QQQ, GLD, USO,
UUP and TLT all genuinely resolve. It also forbids predicting SPY itself, which
is the benchmark and whose excess return is zero by construction.

Shared between the coordinator and replay prompts rather than written twice,
since a rule that drifts between the two lanes is worse than no rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-04 19:22:48 +01:00
ImBenjiandClaude Opus 5 e8fb9e3b1c fix: no token ceiling unless the budget forces one 
Removing the caps rather than tuning them. Every number I picked was a number I
invented, and 6000 was already tight enough to truncate a real replay article,
which is the failure that was called out when the cap first went in.

The coordinator now sends no max_tokens at all. The 402 handler supplies one only
when openrouter says the budget cannot cover an open ended request, so the
ceiling exists exactly when it has to and never otherwise. Verified unbounded is
accepted against the live key before making this the default.

The caps on the signal, augor, consolidation and graph workers are gone too. They
were added to work around an empty account, not because any of them ever produced
too much, and unlike the coordinator none of them detect truncation, so an
invented ceiling there risked silently corrupting company facts. A budget failure
in those is at least loud.

The 220 token cap in crawlerClassifier is left alone, it predates this and bounds
a genuinely tiny classification.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-04 18:43:23 +01:00
ImBenjiandClaude Opus 5 b5d0cae4a7 fix: raise the coordinator token ceiling, 6000 truncated a real article 
A replay article dead-lettered with 'coordinator response was truncated by
max_tokens'. The cap was too tight, which is exactly the failure that was
predicted when it went in.

Raising it to 32000 costs nothing. The cap is there to satisfy openrouter's
affordability check, not to ration tokens, and billing is on tokens used rather
than reserved. The 402 handler already walks the ceiling down automatically when
the budget cannot cover the reservation, so a high default is effectively
uncapped while funded and degrades by itself when not.

Verified against the live key at 64000 before picking 32000, so there is real
headroom rather than a guess.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-04 02:54:30 +01:00
ImBenjiandClaude Opus 5 ca69ad0e73 fix: stop three loops that retry forever, and let budget dead letters recover 
The event outcome worker re-requested PSTG and GROQ on every poll for as long as
the process lived, because a fetch failure only logged and continued while the
prediction stayed pending. Ten requests every five minutes, indefinitely. Per
ticker backoff now doubles to an hour, so a symbol with no market data costs one
request an hour instead of one a minute. It also translates dotted tickers the
same way the autonomy worker does, which is why that helper moved into the
shared price module rather than being copied.

The gdelt loop had no pause on its error path at all, so once the api started
refusing connections it spun through failures continuously, burning cpu and
filling the log with the same stack. It has been doing that for days. Backs off
to half an hour now and resets on success.

isTransientCoordinatorFailure matched 408, 429 and 5xx but not a budget 402/403,
so the 380 jobs that dead-lettered during the exhausted quota window could never
come back on their own, including 55 live events. Budget failures are transient
in a way an ordinary auth failure is not, and a wrong key still dies permanently
because it says invalid or unauthorized rather than naming credits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-03 16:42:28 +01:00
ImBenjiandClaude Opus 5 859d0719b3 fix: stop discarding live predictions the moment they mature 
Six live predictions were marked unresolvable, including MSFT twice, WMT and
ITW. Re-running the calculation against yahoo resolves all six, so they were
never unresolvable, they were scored before the market data existed and then
thrown away permanently.

The due-check counted calendar days while calculateOutcome finds the exit bar by
trading days. A friday horizon-1 prediction therefore looked due on saturday,
when monday's close cannot exist. calculateOutcome returned null and the worker
treated null as permanently dead. This hit short horizons hardest, which is
exactly the cohort that produces the first live evidence.

The sql filter stays loose because it cannot know about weekends, and trading day
arithmetic now decides what is genuinely ready. A null result waits for the
horizon to be properly past before anything is retired, and says so when it
finally gives up.

Separately, at horizon 1 the entry and exit lookups could land on the same bar
and produce an excess return of exactly zero, which was recorded as a real
outcome and scored as a directional miss. ITW and WMT both did this. A horizon
that has not elapsed is no longer a measurement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-03 16:28:57 +01:00
ImBenjiandClaude Opus 5 73d09c943f fix: adapt to the budget ceiling and never parse truncated json 
Two things wrong with the fixed cap I added earlier.

It was guessed rather than measured. The largest proposal this has ever produced
was about 1,258 tokens carrying 12 predictions and the average is around 50, so
8000 was arbitrary, and worse, it was above what the key could afford by the
time it deployed. The ceiling openrouter will accept shrinks as the balance
depletes: 15,666 earlier today, 3,921 an hour later.

So the cap is now adaptive. A 402 names the ceiling, and we retry once just under
it, downwards only. A shrinking budget shortens the allowed answer instead of
stopping the pipeline dead. Worth being clear that removing the cap is not an
option on a limited key, an unbounded request is refused outright and produces
no output at all rather than a truncated one.

And truncation is no longer silent. finish_reason length now throws instead of
handing a half written response to extractJson, which could occasionally parse a
partial object and quietly drop predictions. That was the real risk in capping.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-01 21:54:34 +01:00
ImBenjiandClaude Opus 5 ce1ce4dd57 fix: bound max_tokens on the remaining llm callers 
signal, augor and consolidation had the same unbounded request as the
coordinator, so switching to a model with a 131k output window made all three
402 on every call while the coordinator itself was fine. Found them by grepping
for the endpoint rather than waiting for each one to surface in the logs.

Sized per worker rather than one global number, since these produce more than
the coordinator's small json object.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-01 21:36:13 +01:00
ImBenjiandClaude Opus 5 4718635c75 fix: bound max_tokens on the openrouter calls 
Neither the coordinator nor the graph resolver set max_tokens, so openrouter
reserved the model's entire output window against the key's remaining budget and
returned 402 before running anything: 131k tokens reserved to produce a few
hundred. It never showed up with the old model because its output window is
small enough to fit under the limit.

Both are now bounded and overridable by env. The headroom is deliberate, the
newer reasoning models spend completion tokens thinking before they answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-01 21:32:22 +01:00
ImBenjiandClaude Opus 5 9fc443579f perf: dedupe the overview poll and index the jobs rollup 
Measured on the deployed console: the page shell and every module land in 39ms,
so the 1.4s was entirely /admin/api/ops/overview, fetched twice.

Twice because the sidebar and the overview view each called usePoll on the same
url, each with its own timer. usePoll now keeps one store per url, so any number
of subscribers share a single request and a single interval, and a request
already on the wire is joined rather than duplicated.

The endpoint itself was dominated by the jobs rollup: a full scan of
autonomy_jobs, 521k rows and growing about nine thousand a day. A covering index
on (job_type, lane, status, created_at) takes it from 548ms to 110ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-01 20:34:55 +01:00
ImBenjiandClaude Opus 5 40f098129c perf: preload the ops module graph 
Admin assets are no-store, so every load refetches all of them, and es module
imports are discovered one level at a time: main parses, then its imports are
found, then theirs. Preload hints turn that waterfall into one parallel burst.
The hrefs deliberately carry no version query because they have to byte-match
what the import specifiers resolve to, otherwise the browser fetches each module
twice instead of reusing the preload.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-01 20:26:43 +01:00
ImBenjiandClaude Opus 5 93108a27fe fix: convert style strings for react in the ops console 
htm passes props through untouched and react rejects a style string with error
#62, so every view rendered an empty page. Converting once in the createElement
wrapper keeps plain css in the templates instead of style objects everywhere.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-01 20:20:27 +01:00
ImBenjiandClaude Opus 5 0a5b30ab63 feat: new ops console for the admin surface 
The old admin was five separate html pages, so every navigation was a full
reload and the operational picture was scattered across all of them. Worth
saying: the api was never the problem, every endpoint answers in under 200ms.
It felt slow because of the architecture, not the backend.

This is a single page console. React and htm from a cdn, no bundler and no
babel-in-the-browser, because a runtime transpiler on every load is exactly the
slowness we are trying to get rid of. Hash routing, polling that keeps the last
good payload on screen instead of flashing a spinner, and stale responses are
dropped so a slow request cannot overwrite a newer one.

/admin/api/ops/overview answers the whole dashboard in one call rather than
making the browser fan out and stitch. It carries the things that actually
matter and were not visible anywhere before: when live evidence matures, why a
cohort does or does not clear the trade gate, which stage of the pipeline has
gone quiet, and what is sitting in dead letters.

Controls, all of which change production and all of which ask twice:
- requeue dead letters, which only ever moves dead_letter back to pending
- execution mode and a kill switch, now read from autonomy_settings on every
  poll instead of only from AUTONOMY_EXECUTION_MODE, so halting no longer needs
  a redeploy first
- run the reaction analysis and read its output

The d3 graph is framed rather than ported. It works, and rewriting it would risk
something valuable for nothing the operator can see.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-01 20:17:36 +01:00
ImBenjiandClaude Opus 5 2d92759ae9 perf: stop the content picker reading 4GB of article bodies 
The content backfill picker took 194 seconds per call. better-sqlite3 is
synchronous, so that blocked the whole ingest event loop, and with eight workers
each running it in a loop they serialised behind each other: about 26 minutes a
round. From outside it looked like a hang, and the gdelt loop went quiet at the
same time because it was stuck behind the same blocked loop.

Two causes. The picker tested `content IS NULL OR TRIM(content) = ''`, which
made sqlite read the content column, a 4GB blob, purely to decide which rows to
skip. content_status already records the same thing and agrees with the content
column on all 2.2M rows, so the test bought nothing. It also stopped any index
being usable.

Then there was no index matching the window function, so it built temp b-trees
over every unfetched row. idx_articles_pending_fetch is partial and column
ordered to match PARTITION BY source ORDER BY pub_date_effective DESC, id DESC.
The planner ignores it without stats, hence PRAGMA optimize.

Measured on production, same query, same 26k rows: 194.5s -> 1.03s.

Note for whoever reads this next: the playwright page slot leak fixed in 42fb929
was real but was not what froze the pipeline. This was.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-09-01 15:03:03 +01:00
ImBenjiandClaude Opus 5 42fb9291b0 fix: unstick the content pipeline, the quota loop and the outcome retries 
Three separate things had the pipeline frozen for 27 hours.

browserCrawler leaked page slots. context.newPage() sat outside the try, so a
throw or a hang there took the slot with it, and after maxConcurrentPages of
those every caller parked in acquirePageSlot forever. That is what it looked
like from outside: content workers alive, no logs, no progress, 13 chromium
renderers still up 10 hours after start. newPage is inside the try now, waiting
for a slot times out instead of blocking forever, and page.close() is raced so a
wedged renderer cant strand the slot on the way out either.

graphWorker had no backoff on quota failures. A blown OpenRouter monthly limit
returns an instant 403, so it retried as fast as the network allowed: 2356
failures in 20 minutes, drowning every other line in the log. Quota and auth
errors now pause resolution for 15 minutes and log once per window rather than
once per attempt.

The outcome worker retried unresolvable predictions forever. Yahoo writes class
shares with a dash, so BRK.B 404s every time, and a failed prediction stays open
and comes straight back on the next poll. Dots are translated to dashes, which
matters beyond this one name because the allowlist is full of dotted symbols,
and a prediction that fails five times is marked unresolvable instead of
spinning.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-08-31 10:02:04 +01:00
ImBenjiandClaude Opus 5 b8b3987e35 fix: stop the coordinator copying its own prompt example 
The JSON shape in both prompts used real values as placeholders, and the model
was reading them as the answer:

  instrument: 'NVDA'          -> 397 of 611 predictions are NVDA (65%),
                                 second place is LMT with 8
  horizon_days: 10            -> 606 of 611 are horizon 10 (99.2%), out of
                                 seven allowed horizons
  direction: 'positive|negative' -> 502 of 611 are positive (82.2%)
  event_type: 'stable_enum'   -> the enum was never listed, so the model
                                 invented one label per event, 201 distinct
                                 values across 611 predictions

replayWorker had its own copy of the same prompt with the same values, which is
why both lanes show the identical skew (replay is 147/147 horizon 10, 138/147
NVDA).

Every placeholder is now a description of the field rather than a usable value,
with an explicit line saying not to copy them. event_type is validated against
the same closed family list the cohort key uses, so a label cannot mean one
thing in the prompt and another in calibration. Off-enum labels are salvaged
through the existing mapper when they are placeable and rejected when they are
not, so 'other' does not quietly become the bin again.

This does not by itself create edge. It means the next batch of predictions
measures the model's judgement instead of its willingness to copy an example.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-08-29 23:29:11 +01:00
ImBenjiandClaude Opus 5 c4650fe45c feat: measure whether the initial market reaction conditions anything 
The coordinator is scored on excess return vs SPY starting at the information
cutoff, so the announcement move sits outside the scored window. That move is
the best documented conditioner for post event drift and we were discarding it.
This measures whether keeping it would buy us anything, before any of it gets
wired into the cohort key or the prompt.

Reaction is measured from the last close before the event's first article up to
the outcome's own entry price, so the reaction and forward windows touch but
never overlap. Read only, and it caches price history so it can be re-run cheaply
as more outcomes mature.

T1 and T2 are pre-registered in the header because sweeping buckets over 611
outcomes that are half one ticker will always turn up something.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-08-29 23:02:06 +01:00
ImBenjiandClaude Opus 5 8f4d3b4ce9 fix: require live calibration to authorise a live order 
Offline evidence can no longer authorise anything. createDecisions used to
prefer a live snapshot and fall back to the pooled historical one, so once the
live lane woke up a live prediction could have drawn a BUY off backfill data.
Backfill and replay are fine evidence that the pipeline works, they are not a
live track record.

No live snapshot now means ABSTAIN. The abstain says whether offline evidence
existed for that cohort, so "we have 60 offline samples but no live ones" stays
distinguishable from "we know nothing about this cohort".

Also split the health counter. It counted qualifying cohorts across every
source, which overstated how close we are to being able to trade now that only
live cohorts can authorise. It reports qualifying_live_cohorts and
qualifying_offline_cohorts separately, and applies the concentration cap it was
previously ignoring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-08-29 22:22:36 +01:00
ImBenjiandClaude Opus 5 6f1d1eee2d fix: restart the stalled autonomy pipeline and make calibration honest 
Archive ingestion had been dead since 2026-08-02 because nothing in the
compose stack actually ran it. Everything downstream starved from there.

- add ingest + enrichment services. server.js only starts the scheduler when
  DURIIN_RUN_SCHEDULER is not "false", and workers/index.js was not running at
  all, so articles never got event_id/content/has_embedding and the coordinator
  had nothing to lease.
- pass an explicit origin from coordinatorWorker. it was never passed, so
  acceptProposal defaulted to 'live' and 464 historical backfill predictions
  were recorded as live. that also meant verifyEvidence got a null cutoff and
  skipped its date check entirely.
- coarsen cohortKey to event families + horizon buckets. 201 free text event
  types produced 221 cohorts averaging 2.76 samples, so the n>=30 gate could
  never be reached and everything abstained for the wrong reason.
- gate on cohort diversity, not just sample count. one ticker was roughly half
  of all resolved outcomes, so a pure count gate was measuring one company.
  unknown diversity abstains rather than passing.
- resolve the admin archive db explicitly and probe it. it relied on a
  Dockerfile symlink, and without it better-sqlite3 quietly creates an empty
  file and serves a phantom archive.
- clamp implausible future publication dates at ingest.
- pin the db backend to sqlite by default. compose hardcoded postgres "true",
  which would have overridden the operator's own .env on the next redeploy and
  pointed everything at a stale snapshot.

scripts/repair-autonomy-labels.js relabels the affected rows. it is dry run by
default and has not been applied.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
 2026-08-29 21:43:24 +01:00
ImBenji d778a02bfb fix: make postgres the default runtime service 2026-08-19 09:12:44 +01:00
ImBenji 9eb59443bd fix: support postgres pragma table info 2026-08-19 08:42:58 +01:00
ImBenji 8288bbc028 fix: stop eager admin page prefetch 2026-08-19 08:38:12 +01:00
ImBenji 7ff69ad99d fix: use async postgres for autonomy http routes 2026-08-17 14:10:29 +01:00
ImBenji aea0ed437e fix: isolate autonomy postgres url 2026-08-17 14:05:31 +01:00
ImBenji 51649fbf5f fix: pin postgres transaction clients 2026-08-17 14:02:59 +01:00
ImBenji 4c24b37af9 fix: keep postgres archive time lookup indexed 2026-08-17 13:59:13 +01:00
ImBenji e239a5d41e fix: cast postgres archive timestamps 2026-08-17 13:56:13 +01:00
ImBenji f0b598a3b8 feat: support postgres autonomy runtime 2026-08-17 13:43:50 +01:00
ImBenji 4aa2367971 fix: calibrate from replay outcomes 2026-08-13 19:10:59 +01:00
ImBenji c9c2d0c8ef fix: recover transient coordinator dead letters 2026-08-13 19:02:41 +01:00
ImBenji 2c023c8962 fix: let replay recover from dead letter jobs 2026-08-08 23:32:41 +01:00
ImBenji 0344d5ca97 Revert "style: reduce modal title scale" 
This reverts commit 2d5c0280de.
 2026-08-04 22:39:50 +01:00
ImBenji 2d5c0280de style: reduce modal title scale 2026-08-04 22:38:37 +01:00
ImBenji 0047adcb6d fix: scrub invalid nul bytes during postgres migration 2026-08-04 22:35:50 +01:00
ImBenji ffc85719ff feat: add bounded postgres data migration stack 2026-08-04 22:09:23 +01:00
ImBenji 5877783862 feat: add isolated historical replay calibration 2026-08-04 22:00:11 +01:00
ImBenji be18eb77f0 fix: cache background statistics response 2026-08-04 21:51:18 +01:00
ImBenji b3a55fac08 perf: source statistics from the catalog 2026-08-04 21:50:02 +01:00
ImBenji 6e666838ba perf: keep source aggregation off interactive requests 2026-08-04 21:48:44 +01:00
ImBenji d020fa3811 perf: avoid full archive scans in statistics 2026-08-04 21:46:56 +01:00
ImBenji f5f092cfb0 perf: compute archive stats off the API thread 2026-08-04 21:45:17 +01:00
ImBenji 002b7b320c perf: remove blocking archive counts from navigation 2026-08-04 21:42:23 +01:00
ImBenji 2020a0fde2 fix: keep archive indexing off API startup 2026-08-04 21:39:20 +01:00
ImBenji bd162606b6 perf: make admin navigation responsive 2026-08-04 21:36:39 +01:00
ImBenji 5680ee8a44 fix: prevent stale admin UI assets 2026-08-04 21:31:55 +01:00
ImBenji b9ee10a83a fix: serialize autonomy job leases 2026-08-04 21:07:42 +01:00
ImBenji 5037e1e192 feat: overhaul Duriin autonomy console 2026-08-04 21:05:47 +01:00
ImBenji 0724f5dc36 fix: run autonomy workers by default 2026-08-03 14:43:30 +01:00
ImBenji c4028cc394 feat: add autonomous paper-trading and calibration pipeline 2026-08-03 14:03:27 +01:00
103 changed files with 9279 additions and 640 deletions
Expand all files Collapse all files
.env.example
+8 -1
View File
@@ -12,9 +12,16 @@ ALPHA_VANTAGE_API_KEY=
FINNHUB_API_KEY=
OPEN_ROUTER_API_KEY=
OPEN_ROUTER_LLM_MODEL=qwen/qwen3-235b-a22b-2507
OPEN_ROUTER_LLM_MODEL=~deepseek/deepseek-v4-flash-latest
OPEN_ROUTER_EMBED_MODEL=qwen/qwen3-embedding-8b
# Paper execution is disabled unless AUTONOMY_EXECUTION_MODE=paper.
# These credentials are accepted only by the hard-coded Alpaca paper endpoint.
ALPACA_PAPER_KEY_ID=
ALPACA_PAPER_SECRET_KEY=
AUTONOMY_EXECUTION_MODE=shadow
AUTONOMY_DEFAULT_NOTIONAL=100
GDELT_BQ_PROJECT=
GDELT_BQ_KEY_FILE=./gdelt-credentials.json
README.md
+4
View File
@@ -16,6 +16,10 @@ Node.js Fastify server that ingests news articles from RSS, GDELT, SEC EDGAR 8-K
The server listens on the host and port defined in `config.json`.
The bounded autonomy runtime and paper-trading contracts are documented in
[`docs/autonomy.md`](docs/autonomy.md). It is opt-in and does not start with
the API-only Compose service.
## How the data pipeline works
On startup the server:
backtest/backtest.py
+26
View File
@@ -214,6 +214,32 @@ def main():
print(f" 20-day: {a20:.1f}% (n={n20})")
print()
# baselines — what would naive strategies have scored on the same set?
# this is the most important context for interpreting the model accuracy above
eval_df = df[df["correct_10d"].notna()].copy()
if len(eval_df) > 0:
# 1. always-positive baseline — predict every event as bullish
eval_df["always_pos_correct"] = eval_df["10d_return"].apply(lambda r: r > 0 if r is not None else None)
always_pos = eval_df["always_pos_correct"].mean() * 100
# 2. random baseline — flip a coin for each prediction (analytic expectation = 50%)
# we report the empirical positive rate of the underlying market over the test window
# since random would converge to that for a balanced dataset
market_up_rate = (eval_df["10d_return"] > 0).mean() * 100
# 3. always-negative baseline
always_neg = ((eval_df["10d_return"] < 0).sum() / len(eval_df)) * 100
print("BASELINES (10-day, same evaluation set)")
print(f" Always-positive: {always_pos:.1f}% (this is the bar to beat in a bull market)")
print(f" Always-negative: {always_neg:.1f}%")
print(f" Random (coin): 50.0% (analytic)")
print(f" Market up rate: {market_up_rate:.1f}% (% of events where stock rose 10d later)")
edge = a10 - always_pos
print(f" MODEL EDGE vs always-positive: {edge:+.1f} percentage points")
print()
# by magnitude
print("BY MAGNITUDE (10-day accuracy)")
for mag in sorted(df["magnitude"].dropna().unique()):
backtest/backtest_full.py
+2
View File
@@ -260,8 +260,10 @@ def main():
print()
sample = df[df["correct_10d"].notna()].head(30)
print("SAMPLE (30 most recent predictions)")
print(f"{'Ticker':<12} {'Date':<12} {'Dir':<10} {'Mag':<8} {'5d%':>7} {'10d%':>7} {'20d%':>7} @10d")
print("-" * 72)
for _, row in sample.iterrows():
r5s = f"{row['5d_return']:+.2f}" if row['5d_return'] is not None else "N/A"
backtest/backtest_results.csv
+88 -88
View File
@@ -92,8 +92,8 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
1240,HF,Hugging Face,2025-10-03,negative,medium,medium,OpenAI's valuation surge to $500 billion highlights its dominant market position and ability to attr,20.9692,20.8305,21.0089,20.9791,-0.6615,0.189,0.0472,True,False,False
1304,META,Meta,2025-10-02,positive,medium,medium,"By using AI interaction data to improve recommendation relevance, Meta can increase user engagement ",725.8361,732.2853,710.8811,665.3572,0.8885,-2.0604,-8.3323,True,False,False
1305,META,Meta,2025-10-02,positive,medium,medium,Enhanced personalization using deeper AI signals strengthens Meta's competitive edge in social media,725.8361,732.2853,710.8811,665.3572,0.8885,-2.0604,-8.3323,True,False,False
935,META,Meta,2025-10-01,positive,medium,short,AI-driven ad personalization strengthens Meta's competitive edge in digital advertising by improving,716.1423,716.6415,716.3519,750.4149,0.0697,0.0293,4.7857,True,True,True
937,META,Meta,2025-10-01,positive,medium,medium,"Enhanced ad targeting using AI interactions could increase advertiser ROI, driving higher ad spend s",716.1423,716.6415,716.3519,750.4149,0.0697,0.0293,4.7857,True,True,True
935,META,Meta,2025-10-01,positive,medium,short,AI-driven ad personalization strengthens Meta's competitive edge in digital advertising by improving,716.1423,716.6415,716.3519,750.415,0.0697,0.0293,4.7857,True,True,True
937,META,Meta,2025-10-01,positive,medium,medium,"Enhanced ad targeting using AI interactions could increase advertiser ROI, driving higher ad spend s",716.1423,716.6415,716.3519,750.415,0.0697,0.0293,4.7857,True,True,True
1302,SPOT,Spotify,2025-09-27,positive,medium,medium,"By adopting DDEX standards for AI transparency, Spotify positions itself as a responsible leader in ",728.47,680.5,685.29,645.78,-6.585,-5.9275,-11.3512,False,False,False
926,META,Meta,2025-09-18,positive,medium,medium,Launch of consumer-ready smartglasses with differentiated AI features may increase Meta's presence i,778.4218,747.6595,725.8361,710.8811,-3.9519,-6.7554,-8.6766,False,False,False
927,META,Meta,2025-09-18,positive,medium,short,Introduction of AI-integrated smartglasses with partnerships in fitness tech strengthens Meta's posi,778.4218,747.6595,725.8361,710.8811,-3.9519,-6.7554,-8.6766,False,False,False
@@ -110,11 +110,11 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
637,AZN,AstraZeneca,2025-09-13,negative,medium,short,"Pausing a major investment may signal reduced confidence in the UK market, potentially affecting inv",154.4894,150.9859,145.9979,167.3157,-2.2678,-5.4965,8.3024,True,True,False
638,AZN,AstraZeneca,2025-09-13,negative,medium,medium,Delaying expansion in a key life sciences hub like Cambridge could slow innovation and talent acquis,154.4894,150.9859,145.9979,167.3157,-2.2678,-5.4965,8.3024,True,True,False
628,ORCL,Oracle,2025-09-12,positive,medium,short,Bullish options activity suggests increased investor confidence in Oracle's near-term stock performa,289.8924,306.2434,281.2406,291.1707,5.6404,-2.9845,0.441,True,False,True
988,AAPL,Apple,2025-09-12,positive,medium,short,"Strong buy-side investor sentiment, particularly from Indian retail investors, combined with histori",233.6247,245.033,254.974,244.8034,4.8832,9.1383,4.7849,True,True,True
627,ORCL,Oracle,2025-09-11,negative,medium,short,"Shares retreated 6-7% after a record high due to concerns about overreliance on OpenAI for growth, d",305.4496,294.2976,289.049,295.1463,-3.651,-5.3693,-3.3732,True,True,True
1409,ORCL,Oracle,2025-09-11,positive,high,short,Oracle's stock surged 35.98% following strong cloud revenue forecasts and major AI-related contracts,305.4496,294.2976,289.049,295.1463,-3.651,-5.3693,-3.3732,False,False,False
1410,ORCL,Oracle,2025-09-11,positive,medium,medium,"With a cloud services backlog nearing $500 billion and major contracts in AI infrastructure, Oracle ",305.4496,294.2976,289.049,295.1463,-3.651,-5.3693,-3.3732,False,False,False
1411,ORCL,Oracle,2025-09-11,positive,high,medium,Oracle's strategic positioning as a key AI cloud infrastructure provider through partnerships with l,305.4496,294.2976,289.049,295.1463,-3.651,-5.3693,-3.3732,False,False,False
988,AAPL,Apple,2025-09-12,positive,medium,short,"Strong buy-side investor sentiment, particularly from Indian retail investors, combined with histori",233.6247,245.033,254.974,244.8034,4.8831,9.1383,4.7849,True,True,True
627,ORCL,Oracle,2025-09-11,negative,medium,short,"Shares retreated 6-7% after a record high due to concerns about overreliance on OpenAI for growth, d",305.4496,294.2976,289.049,295.1462,-3.651,-5.3693,-3.3732,True,True,True
1409,ORCL,Oracle,2025-09-11,positive,high,short,Oracle's stock surged 35.98% following strong cloud revenue forecasts and major AI-related contracts,305.4496,294.2976,289.049,295.1462,-3.651,-5.3693,-3.3732,False,False,False
1410,ORCL,Oracle,2025-09-11,positive,medium,medium,"With a cloud services backlog nearing $500 billion and major contracts in AI infrastructure, Oracle ",305.4496,294.2976,289.049,295.1462,-3.651,-5.3693,-3.3732,False,False,False
1411,ORCL,Oracle,2025-09-11,positive,high,medium,Oracle's strategic positioning as a key AI cloud infrastructure provider through partnerships with l,305.4496,294.2976,289.049,295.1462,-3.651,-5.3693,-3.3732,False,False,False
611,INTC,Intel,2025-08-28,positive,medium,short,"Direct government investment signals confidence and improves public perception, consistent with prio",24.93,24.61,24.61,33.99,-1.2836,-1.2836,36.3418,False,False,True
612,INTC,Intel,2025-08-28,positive,medium,medium,"Government funding and stake may enhance capital investment in manufacturing, supporting domestic pr",24.93,24.61,24.61,33.99,-1.2836,-1.2836,36.3418,False,False,True
608,NVDA,NVIDIA,2025-08-28,positive,high,short,"Extraordinary demand and full-speed ramp-up of Blackwell Ultra platform indicate strong adoption, re",180.1401,171.6315,177.1506,177.6705,-4.7233,-1.6595,-1.3709,False,False,False
@@ -135,24 +135,24 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
857,PLTR,Palantir,2025-08-10,positive,medium,medium,Recognition as 'the best story in all of software' and leadership in AI-driven government and enterp,182.68,177.17,158.74,153.11,-3.0162,-13.1049,-16.1868,False,False,False
1208,INTC,Intel,2025-08-08,negative,medium,short,Trump's public demand for CEO resignation and allegations of conflict due to ties with Chinese firms,19.95,24.56,24.8,24.49,23.1078,24.3108,22.7569,False,False,False
1209,INTC,Intel,2025-08-08,negative,low,short,"Leadership instability and political scrutiny may delay turnaround plans, giving competitors like Nv",19.95,24.56,24.8,24.49,23.1078,24.3108,22.7569,False,False,False
1210,META,Meta,2025-08-08,positive,high,medium,"The $29 billion financing enables accelerated AI infrastructure development, strengthening Meta's ca",767.4975,783.3901,753.0215,750.687,2.0707,-1.8861,-2.1903,True,False,False
1289,TSM,TSMC,2025-08-08,negative,medium,short,"The leak of trade secrets, even if not directly TSMC’s fault, could undermine confidence in its IP p",239.7449,236.8203,230.9811,241.3113,-1.2199,-3.6555,0.6534,True,True,False
1290,TSM,TSMC,2025-08-08,negative,low,short,"While the incident does not directly implicate TSMC in wrongdoing, associated supply chain instabili",239.7449,236.8203,230.9811,241.3113,-1.2199,-3.6555,0.6534,True,True,False
1218,META,Meta,2025-08-08,positive,high,long,"The $29 billion financing enables large-scale AI data center development, strengthening Meta's infra",767.4975,783.3901,753.0215,750.687,2.0707,-1.8861,-2.1903,True,False,False
1210,META,Meta,2025-08-08,positive,high,medium,"The $29 billion financing enables accelerated AI infrastructure development, strengthening Meta's ca",767.4976,783.3902,753.0215,750.6871,2.0707,-1.8861,-2.1903,True,False,False
1289,TSM,TSMC,2025-08-08,negative,medium,short,"The leak of trade secrets, even if not directly TSMC’s fault, could undermine confidence in its IP p",239.7449,236.8203,230.9811,241.3112,-1.2199,-3.6555,0.6533,True,True,False
1290,TSM,TSMC,2025-08-08,negative,low,short,"While the incident does not directly implicate TSMC in wrongdoing, associated supply chain instabili",239.7449,236.8203,230.9811,241.3112,-1.2199,-3.6555,0.6533,True,True,False
1218,META,Meta,2025-08-08,positive,high,long,"The $29 billion financing enables large-scale AI data center development, strengthening Meta's infra",767.4976,783.3902,753.0215,750.6871,2.0707,-1.8861,-2.1903,True,False,False
1511,SPOT,Spotify,2025-08-07,positive,medium,short,"The introduction of AI DJ, which provides personalized, context-rich music recommendations using gen",686.74,698.5,689.47,703.85,1.7124,0.3975,2.4915,True,True,True
1513,SPOT,Spotify,2025-08-07,positive,medium,medium,Investment in advanced AI features like AI DJ strengthens Spotify’s differentiation from competitors,686.74,698.5,689.47,703.85,1.7124,0.3975,2.4915,True,True,True
851,PLTR,Palantir,2025-08-07,negative,medium,medium,"CEO's remarks may deepen skepticism among educated public and academia, contributing to declining pu",182.2,181.02,156.18,156.14,-0.6476,-14.281,-14.303,True,True,True
852,PLTR,Palantir,2025-08-07,positive,medium,long,Positioning Palantir as a meritocratic alternative to elite education could strengthen employer bran,182.2,181.02,156.18,156.14,-0.6476,-14.281,-14.303,False,False,False
1288,TSM,TSMC,2025-08-07,negative,medium,short,Increased geopolitical risk and costly overseas expansion could weigh on investor sentiment in the n,240.5281,238.922,225.3699,233.182,-0.6677,-6.302,-3.0541,True,True,True
1509,TSM,TSMC,2025-08-07,positive,medium,short,Tariff exemption strengthens TSMC's competitive position relative to non-U.S.-based semiconductor ma,240.5281,238.922,225.3699,233.182,-0.6677,-6.302,-3.0541,False,False,False
1510,TSM,TSMC,2025-08-07,positive,low,short,"Exemption from high tariffs provides operational certainty and reduces near-term geopolitical risk, ",240.5281,238.922,225.3699,233.182,-0.6677,-6.302,-3.0541,False,False,False
1288,TSM,TSMC,2025-08-07,negative,medium,short,Increased geopolitical risk and costly overseas expansion could weigh on investor sentiment in the n,240.5281,238.922,225.3699,233.182,-0.6677,-6.302,-3.0542,True,True,True
1509,TSM,TSMC,2025-08-07,positive,medium,short,Tariff exemption strengthens TSMC's competitive position relative to non-U.S.-based semiconductor ma,240.5281,238.922,225.3699,233.182,-0.6677,-6.302,-3.0542,False,False,False
1510,TSM,TSMC,2025-08-07,positive,low,short,"Exemption from high tariffs provides operational certainty and reduces near-term geopolitical risk, ",240.5281,238.922,225.3699,233.182,-0.6677,-6.302,-3.0542,False,False,False
873,PLTR,Palantir,2025-08-05,positive,high,short,"Palantir's strong earnings, revenue growth of nearly 50%, net income up 144%, raised guidance, and i",173.27,186.97,157.75,157.09,7.9067,-8.9571,-9.338,True,False,False
874,PLTR,Palantir,2025-08-05,positive,medium,medium,Investor confidence in Palantir's AI-powered efficiency and scalability may accelerate adoption in g,173.27,186.97,157.75,157.09,7.9067,-8.9571,-9.338,True,False,False
875,PLTR,Palantir,2025-08-05,positive,medium,short,Palantir's demonstrated ability to grow revenue while reducing workforce through AI gives it a perce,173.27,186.97,157.75,157.09,7.9067,-8.9571,-9.338,True,False,False
1463,META,Meta,2025-08-01,positive,high,short,Meta shares jump after strong third-quarter sales forecast and better-than-expected financial perfor,748.2527,767.4975,783.3901,736.9692,2.572,4.6959,-1.508,True,True,False
1307,MSFT,Microsoft,2025-08-01,positive,high,short,"Microsoft's stock rose 3.9% on July 31, 2025, following record valuation and strong quarterly result",521.0829,519.0249,517.1657,504.5917,-0.395,-0.7517,-3.1648,False,False,False
1308,MSFT,Microsoft,2025-08-01,positive,medium,medium,Azure's 39% growth and increasing AI-driven cloud revenue suggest Microsoft is gaining cloud market ,521.0829,519.0249,517.1657,504.5917,-0.395,-0.7517,-3.1648,False,False,False
1309,MSFT,Microsoft,2025-08-01,positive,high,long,"Surpassing four trillion dollars in market cap, second only to Nvidia, reinforces Microsoft's elite ",521.0829,519.0249,517.1657,504.5917,-0.395,-0.7517,-3.1648,False,False,False
1463,META,Meta,2025-08-01,positive,high,short,Meta shares jump after strong third-quarter sales forecast and better-than-expected financial perfor,748.2527,767.4976,783.3902,736.9692,2.572,4.6959,-1.508,True,True,False
1307,MSFT,Microsoft,2025-08-01,positive,high,short,"Microsoft's stock rose 3.9% on July 31, 2025, following record valuation and strong quarterly result",521.0829,519.025,517.1657,504.5917,-0.3949,-0.7517,-3.1648,False,False,False
1308,MSFT,Microsoft,2025-08-01,positive,medium,medium,Azure's 39% growth and increasing AI-driven cloud revenue suggest Microsoft is gaining cloud market ,521.0829,519.025,517.1657,504.5917,-0.3949,-0.7517,-3.1648,False,False,False
1309,MSFT,Microsoft,2025-08-01,positive,high,long,"Surpassing four trillion dollars in market cap, second only to Nvidia, reinforces Microsoft's elite ",521.0829,519.025,517.1657,504.5917,-0.3949,-0.7517,-3.1648,False,False,False
1400,ARM,Arm Holdings,2025-07-31,negative,high,short,Arm's shares dropped nearly 13% following disappointing guidance and a strategic shift that risks cu,141.375,135.57,140.55,142.55,-4.1061,-0.5836,0.8311,True,True,False
1401,ARM,Arm Holdings,2025-07-31,negative,medium,medium,Developing full chip solutions may lead to conflicts of interest with key customers like Nvidia and ,141.375,135.57,140.55,142.55,-4.1061,-0.5836,0.8311,True,True,False
1402,ARM,Arm Holdings,2025-07-31,negative,low,long,Long-term market share could be affected if customers reduce reliance on Arm's IP due to competitive,141.375,135.57,140.55,142.55,-4.1061,-0.5836,0.8311,True,True,False
@@ -163,10 +163,10 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
1222,META,Meta,2025-07-31,positive,medium,medium,"Aggressive investment in AI talent and infrastructure positions Meta to better compete with OpenAI, ",771.6278,760.045,780.2974,749.3502,-1.5011,1.1235,-2.8871,False,True,False
1223,META,Meta,2025-07-31,positive,medium,medium,Strong ad revenue growth and AI-driven product enhancements may increase user engagement and ad mark,771.6278,760.045,780.2974,749.3502,-1.5011,1.1235,-2.8871,False,True,False
1279,UBS,UBS,2025-07-30,negative,medium,medium,Proposed capital requirements could impair competitiveness if implemented; CEO warns 42 billion doll,36.9761,37.0834,38.6529,39.15,0.29,4.5347,5.8793,False,False,False
893,META,Meta,2025-07-26,positive,high,medium,Hiring a foundational OpenAI researcher and formalizing leadership under a strong AI executive team ,715.9486,748.2527,767.4975,753.0215,4.5121,7.2001,5.1781,True,True,True
894,META,Meta,2025-07-26,positive,medium,long,"Strengthening AI talent base supports future AI product development, potentially increasing market s",715.9486,748.2527,767.4975,753.0215,4.5121,7.2001,5.1781,True,True,True
1558,VZ,Verizon,2025-07-24,negative,low,short,"The $175 million penalty represents a one-time cost that may slightly pressure earnings, but given V",41.0128,40.7082,40.8891,42.8693,-0.7428,-0.3018,4.5264,True,True,False
1559,VZ,Verizon,2025-07-24,negative,medium,short,Increased legal risk from patent litigation could affect investor sentiment and raise concerns about,41.0128,40.7082,40.8891,42.8693,-0.7428,-0.3018,4.5264,True,True,False
893,META,Meta,2025-07-26,positive,high,medium,Hiring a foundational OpenAI researcher and formalizing leadership under a strong AI executive team ,715.9486,748.2527,767.4976,753.0215,4.5121,7.2001,5.1781,True,True,True
894,META,Meta,2025-07-26,positive,medium,long,"Strengthening AI talent base supports future AI product development, potentially increasing market s",715.9486,748.2527,767.4976,753.0215,4.5121,7.2001,5.1781,True,True,True
1558,VZ,Verizon,2025-07-24,negative,low,short,"The $175 million penalty represents a one-time cost that may slightly pressure earnings, but given V",41.0128,40.7082,40.8891,42.8693,-0.7428,-0.3018,4.5265,True,True,False
1559,VZ,Verizon,2025-07-24,negative,medium,short,Increased legal risk from patent litigation could affect investor sentiment and raise concerns about,41.0128,40.7082,40.8891,42.8693,-0.7428,-0.3018,4.5265,True,True,False
799,META,Meta,2025-07-17,negative,medium,short,Public perception may be negatively influenced by the association of top executives with privacy vio,699.7665,713.1252,771.6278,780.2974,1.909,10.2693,11.5083,False,False,False
1109,NVDA,NVIDIA,2025-07-10,positive,high,short,Nvidia's stock has surged 69% since early April and analysts project an additional 17% rise to $190 ,164.0727,172.9713,173.7111,180.74,5.4235,5.8745,10.1584,True,True,True
1110,NVDA,NVIDIA,2025-07-10,positive,medium,medium,Ongoing demand from big tech and AI innovators for high-performance computing chips reinforces Nvidi,164.0727,172.9713,173.7111,180.74,5.4235,5.8745,10.1584,True,True,True
@@ -182,8 +182,8 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
1084,MU,Micron,2025-07-03,positive,low,long,Sustained revenue growth from rising HBM demand and expanded production capacity could support long-,121.998,122.9316,113.0959,108.9819,0.7653,-7.2969,-10.6691,True,False,False
1506,BP,BP,2025-07-02,positive,medium,short,"Takeover speculation and activist involvement typically create short-term stock price momentum, even",30.038,30.0093,30.633,30.9497,-0.0958,1.9808,3.0351,False,True,True
1507,BP,BP,2025-07-02,negative,medium,medium,Persistent speculation that BP could be acquired may weaken its strategic autonomy and bargaining po,30.038,30.0093,30.633,30.9497,-0.0958,1.9808,3.0351,True,False,False
902,META,Meta,2025-06-13,positive,high,medium,Acquiring key AI talent and investing heavily in AI infrastructure through Scale AI strengthens Meta,680.7462,680.7512,731.9111,715.8289,0.0007,7.516,5.1536,True,True,True
903,META,Meta,2025-06-13,positive,medium,short,Major strategic investment in AI and high-profile talent acquisition may be viewed favorably by inve,680.7462,680.7512,731.9111,715.8289,0.0007,7.516,5.1536,True,True,True
902,META,Meta,2025-06-13,positive,high,medium,Acquiring key AI talent and investing heavily in AI infrastructure through Scale AI strengthens Meta,680.7463,680.7512,731.9111,715.8289,0.0007,7.516,5.1535,True,True,True
903,META,Meta,2025-06-13,positive,medium,short,Major strategic investment in AI and high-profile talent acquisition may be viewed favorably by inve,680.7463,680.7512,731.9111,715.8289,0.0007,7.516,5.1535,True,True,True
1476,NVS,Novartis,2025-06-13,positive,medium,short,"Novo Nordisk's leadership turmoil and stock decline may weaken its market positioning, creating oppo",115.9217,112.3504,116.4652,117.4453,-3.0808,0.4688,1.3144,False,True,True
1477,NVS,Novartis,2025-06-13,positive,low,short,"While Novartis is not directly affected by Novo Nordisk's CEO dismissal, reduced competitive pressur",115.9217,112.3504,116.4652,117.4453,-3.0808,0.4688,1.3144,False,True,True
1085,AMD,AMD,2025-06-13,positive,high,medium,AMD’s launch of the Helios server and partnership with OpenAI strengthen its position in the AI chip,116.16,128.24,143.81,146.42,10.3995,23.8034,26.0503,True,True,True
@@ -195,32 +195,32 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
1397,UBS,UBS,2025-06-08,positive,high,medium,"The new draft law significantly increases regulatory obligations, including capitalization of foreig",32.1019,31.1758,29.655,33.3107,-2.8849,-7.6222,3.7656,False,False,True
1556,UBS,UBS,2025-06-06,negative,medium,short,"Stricter capital requirements may constrain UBS's ability to deploy capital efficiently, increasing ",32.7745,31.1758,29.655,33.3107,-4.878,-9.5181,1.6359,True,True,False
1557,UBS,UBS,2025-06-06,negative,low,short,"Announcement of higher capital requirements could lead to margin compression concerns, negatively im",32.7745,31.1758,29.655,33.3107,-4.878,-9.5181,1.6359,True,True,False
1393,AVGO,Broadcom,2025-06-06,negative,medium,short,"Broadcom's stock fell 2% in extended trading after the forecast missed the loftiest expectations, de",244.9453,246.7011,248.5644,272.6165,0.7168,1.4775,11.2969,False,False,False
913,META,Meta,2025-06-04,positive,medium,long,"Securing long-term, stable nuclear power enhances Meta's ability to scale AI infrastructure, improvi",685.8104,691.9812,694.1398,711.8981,0.8998,1.2145,3.8039,True,True,True
914,META,Meta,2025-06-04,positive,high,long,The 20-year power purchase agreement with Constellation Energy directly supports Meta's growing AI w,685.8104,691.9812,694.1398,711.8981,0.8998,1.2145,3.8039,True,True,True
1106,AVGO,Broadcom,2025-06-02,negative,medium,short,"Dissatisfied partners may drive customers toward competing private cloud solutions, increasing migra",246.711,242.3166,250.0738,274.0781,-1.7812,1.363,11.0927,True,False,False
1107,AVGO,Broadcom,2025-06-02,negative,medium,medium,"Reducing the number of channel partners, especially long-standing ones, could erode VMware's market ",246.711,242.3166,250.0738,274.0781,-1.7812,1.363,11.0927,True,False,False
1393,AVGO,Broadcom,2025-06-06,negative,medium,short,"Broadcom's stock fell 2% in extended trading after the forecast missed the loftiest expectations, de",244.9453,246.7011,248.5644,272.6164,0.7168,1.4775,11.2969,False,False,False
913,META,Meta,2025-06-04,positive,medium,long,"Securing long-term, stable nuclear power enhances Meta's ability to scale AI infrastructure, improvi",685.8105,691.9812,694.1398,711.8981,0.8998,1.2145,3.8039,True,True,True
914,META,Meta,2025-06-04,positive,high,long,The 20-year power purchase agreement with Constellation Energy directly supports Meta's growing AI w,685.8105,691.9812,694.1398,711.8981,0.8998,1.2145,3.8039,True,True,True
1106,AVGO,Broadcom,2025-06-02,negative,medium,short,"Dissatisfied partners may drive customers toward competing private cloud solutions, increasing migra",246.711,242.3166,250.0738,274.0781,-1.7812,1.363,11.0928,True,False,False
1107,AVGO,Broadcom,2025-06-02,negative,medium,medium,"Reducing the number of channel partners, especially long-standing ones, could erode VMware's market ",246.711,242.3166,250.0738,274.0781,-1.7812,1.363,11.0928,True,False,False
1196,NVDA,NVIDIA,2025-05-29,positive,high,short,Nvidia's revenue surpassing $44bn despite China sales restrictions indicates robust global demand fo,139.1572,139.957,144.9759,154.9942,0.5748,4.1814,11.3807,True,True,True
1197,NVDA,NVIDIA,2025-05-29,positive,medium,medium,Ability to achieve record revenue under geopolitical constraints reinforces Nvidia's competitive adv,139.1572,139.957,144.9759,154.9942,0.5748,4.1814,11.3807,True,True,True
1198,NVDA,NVIDIA,2025-05-29,positive,medium,short,Strong revenue performance despite headwinds aligns with analyst sentiment supporting stock valuatio,139.1572,139.957,144.9759,154.9942,0.5748,4.1814,11.3807,True,True,True
1203,LHX,L3Harris Technologies,2025-05-29,positive,medium,short,Technical breakout above $232 resistance and inclusion in a model portfolio suggest near-term upward,239.3204,239.1635,247.3939,243.8468,-0.0655,3.3735,1.8914,False,True,True
1503,META,Meta,2025-05-29,positive,medium,medium,"One billion monthly users of Meta AI strengthens Meta's position in the generative AI race, though i",643.0439,682.4907,691.2036,724.3888,6.1344,7.4893,12.65,True,True,True
1504,META,Meta,2025-05-29,positive,low,short,"High user engagement with Meta AI may lead to incremental gains in AI assistant market share, but Go",643.0439,682.4907,691.2036,724.3888,6.1344,7.4893,12.65,True,True,True
1503,META,Meta,2025-05-29,positive,medium,medium,"One billion monthly users of Meta AI strengthens Meta's position in the generative AI race, though i",643.0439,682.4907,691.2036,724.3887,6.1344,7.4893,12.65,True,True,True
1504,META,Meta,2025-05-29,positive,low,short,"High user engagement with Meta AI may lead to incremental gains in AI assistant market share, but Go",643.0439,682.4907,691.2036,724.3887,6.1344,7.4893,12.65,True,True,True
1271,NVS,Novartis,2025-05-29,positive,medium,short,"As a competitor in the metabolic and pharmaceutical space, Novartis may benefit from Novo Nordisk's ",109.2546,114.3108,117.2027,116.766,4.6278,7.2748,6.8751,True,True,True
1272,NVS,Novartis,2025-05-29,positive,low,short,"While not directly mentioned, the known positive impact of production investment and divestment on N",109.2546,114.3108,117.2027,116.766,4.6278,7.2748,6.8751,True,True,True
1273,META,Meta,2025-05-29,positive,medium,medium,"Increased AI integration in advertising and user content, combined with high advertiser adoption, po",643.0439,682.4907,691.2036,724.3888,6.1344,7.4893,12.65,True,True,True
1274,META,Meta,2025-05-29,positive,high,medium,"Meta's massive AI investment, global user base, and early lead in AI-powered advertising tools stren",643.0439,682.4907,691.2036,724.3888,6.1344,7.4893,12.65,True,True,True
1199,META,Meta,2025-05-25,positive,medium,medium,"Expanding AI training with user data may improve Meta AI and Llama models, enhancing competitiveness",640.3224,645.4763,695.5401,680.7512,0.8049,8.6234,6.3138,True,True,True
1200,META,Meta,2025-05-25,negative,high,short,"Noyb's challenge and opt-out model may lead to GDPR enforcement actions, increasing regulatory and l",640.3224,645.4763,695.5401,680.7512,0.8049,8.6234,6.3138,False,False,False
1201,META,Meta,2025-05-25,negative,medium,short,"Public backlash over data usage for AI without explicit consent could harm user trust, especially in",640.3224,645.4763,695.5401,680.7512,0.8049,8.6234,6.3138,False,False,False
1202,META,Meta,2025-05-25,negative,low,short,Short-term investor concerns may arise from increased regulatory scrutiny and reputational risk tied,640.3224,645.4763,695.5401,680.7512,0.8049,8.6234,6.3138,False,False,False
1273,META,Meta,2025-05-29,positive,medium,medium,"Increased AI integration in advertising and user content, combined with high advertiser adoption, po",643.0439,682.4907,691.2036,724.3887,6.1344,7.4893,12.65,True,True,True
1274,META,Meta,2025-05-29,positive,high,medium,"Meta's massive AI investment, global user base, and early lead in AI-powered advertising tools stren",643.0439,682.4907,691.2036,724.3887,6.1344,7.4893,12.65,True,True,True
1199,META,Meta,2025-05-25,positive,medium,medium,"Expanding AI training with user data may improve Meta AI and Llama models, enhancing competitiveness",640.3223,645.4763,695.5401,680.7512,0.8049,8.6234,6.3138,True,True,True
1200,META,Meta,2025-05-25,negative,high,short,"Noyb's challenge and opt-out model may lead to GDPR enforcement actions, increasing regulatory and l",640.3223,645.4763,695.5401,680.7512,0.8049,8.6234,6.3138,False,False,False
1201,META,Meta,2025-05-25,negative,medium,short,"Public backlash over data usage for AI without explicit consent could harm user trust, especially in",640.3223,645.4763,695.5401,680.7512,0.8049,8.6234,6.3138,False,False,False
1202,META,Meta,2025-05-25,negative,low,short,Short-term investor concerns may arise from increased regulatory scrutiny and reputational risk tied,640.3223,645.4763,695.5401,680.7512,0.8049,8.6234,6.3138,False,False,False
1259,TSLA,Tesla,2025-05-23,negative,medium,short,Being outsold in Europe suggests Tesla is losing market share to a strong competitor in a key region,339.34,346.46,295.14,322.16,2.0982,-13.0253,-5.0628,False,True,True
1260,TSLA,Tesla,2025-05-23,negative,medium,short,A competitor labeled as a 'Tesla-killer' gaining sales leadership in Europe undermines Tesla's compe,339.34,346.46,295.14,322.16,2.0982,-13.0253,-5.0628,False,True,True
1261,TSLA,Tesla,2025-05-23,negative,low,short,"Missing sales expectations in a major market can negatively impact investor sentiment, especially am",339.34,346.46,295.14,322.16,2.0982,-13.0253,-5.0628,False,True,True
1464,SHOP,Shopify,2025-05-23,positive,medium,short,"The launch of the AI Store Builder enhances Shopify's product offering, differentiating its platform",101.51,107.22,111.41,106.4,5.6251,9.7527,4.8173,True,True,True
1189,WBD,Warner Bros Discovery,2025-05-14,positive,medium,medium,"By refocusing on HBO's premium brand and distinct adult-oriented content, WBD aims to carve out a cl",9.21,8.95,10.02,10.51,-2.823,8.7948,14.1151,False,True,True
1282,META,Meta,2025-05-08,negative,medium,short,Public exposure of widespread scam activity leveraging Meta's platforms may weaken user trust and in,596.1501,641.8775,634.5903,682.4907,7.6704,6.4481,14.483,False,False,False
1283,META,Meta,2025-05-08,negative,low,short,"While the takedown demonstrates proactive moderation, the underlying prevalence of sophisticated sca",596.1501,641.8775,634.5903,682.4907,7.6704,6.4481,14.483,False,False,False
1282,META,Meta,2025-05-08,negative,medium,short,Public exposure of widespread scam activity leveraging Meta's platforms may weaken user trust and in,596.1502,641.8775,634.5902,682.4907,7.6704,6.448,14.483,False,False,False
1283,META,Meta,2025-05-08,negative,low,short,"While the takedown demonstrates proactive moderation, the underlying prevalence of sophisticated sca",596.1502,641.8775,634.5902,682.4907,7.6704,6.448,14.483,False,False,False
1473,V,Visa,2025-05-08,positive,high,long,"By enabling AI agents to use its payment network, Visa positions itself as a foundational player in ",348.6606,360.2059,355.9009,364.6501,3.3113,2.0766,4.586,True,True,True
1474,V,Visa,2025-05-08,positive,medium,medium,Opening its network to AI developers and expanding in key markets like Europe by 2025 could increase,348.6606,360.2059,355.9009,364.6501,3.3113,2.0766,4.586,True,True,True
1475,V,Visa,2025-05-08,positive,low,short,"The announcement of a forward-looking strategic initiative may generate investor interest, though im",348.6606,360.2059,355.9009,364.6501,3.3113,2.0766,4.586,True,True,True
@@ -235,22 +235,22 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
1022,SPOT,Spotify,2025-05-04,positive,low,long,"While Backstage is gaining traction in the internal developer portal space, its market share impact ",637.65,648.25,656.3,665.14,1.6624,2.9248,4.3111,True,True,True
1067,AMZN,Amazon,2025-05-02,positive,medium,short,"Amazon is positioned to gain market share during tariff-related uncertainty, as it did during the pa",189.98,193.06,205.59,205.01,1.6212,8.2167,7.9114,True,True,True
1068,AMZN,Amazon,2025-05-02,positive,medium,short,"Diverse seller base and proactive inventory management reduce the risk of price increases, strengthe",189.98,193.06,205.59,205.01,1.6212,8.2167,7.9114,True,True,True
1375,META,Meta,2025-05-02,positive,medium,medium,"Launching a premium AI service positions Meta to compete more effectively with OpenAI, Google, and M",595.1632,590.6473,638.3485,645.4763,-0.7588,7.256,8.4537,False,True,True
1376,META,Meta,2025-05-02,positive,medium,medium,With nearly a billion users already on Meta AI and a potential premium tier offering enhanced featur,595.1632,590.6473,638.3485,645.4763,-0.7588,7.256,8.4537,False,True,True
1037,META,Meta,2025-04-29,positive,medium,short,Launching a standalone AI app enhances Meta's visibility and positioning in the competitive AI assis,552.7156,585.4835,653.9897,640.3223,5.9285,18.323,15.8502,True,True,True
1038,META,Meta,2025-04-29,positive,low,medium,"The app leverages Meta's extensive user data for personalization, which could attract users over tim",552.7156,585.4835,653.9897,640.3223,5.9285,18.323,15.8502,True,True,True
1375,META,Meta,2025-05-02,positive,medium,medium,"Launching a premium AI service positions Meta to compete more effectively with OpenAI, Google, and M",595.1632,590.6473,638.3484,645.4763,-0.7588,7.256,8.4537,False,True,True
1376,META,Meta,2025-05-02,positive,medium,medium,With nearly a billion users already on Meta AI and a potential premium tier offering enhanced featur,595.1632,590.6473,638.3484,645.4763,-0.7588,7.256,8.4537,False,True,True
1037,META,Meta,2025-04-29,positive,medium,short,Launching a standalone AI app enhances Meta's visibility and positioning in the competitive AI assis,552.7156,585.4834,653.9897,640.3223,5.9285,18.323,15.8502,True,True,True
1038,META,Meta,2025-04-29,positive,low,medium,"The app leverages Meta's extensive user data for personalization, which could attract users over tim",552.7156,585.4834,653.9897,640.3223,5.9285,18.323,15.8502,True,True,True
1039,PL,Planet Labs,2025-04-29,negative,medium,medium,"The $20M funding enables Near Space Labs to scale its stratospheric imaging operations, increasing c",3.43,3.5,3.78,3.97,2.0408,10.2041,15.7434,False,False,False
1291,META,Meta,2025-04-26,negative,medium,short,"High-profile public protest by grieving parents increases reputational damage and litigation risk, w",548.0302,595.1632,590.6473,625.1098,8.6004,7.7764,14.0648,False,False,False
1292,META,Meta,2025-04-26,negative,medium,medium,Growing civil society and regulatory pressure could force Meta to implement stricter safety measures,548.0302,595.1632,590.6473,625.1098,8.6004,7.7764,14.0648,False,False,False
1293,META,Meta,2025-04-26,positive,high,short,"The protest is a direct indicator of escalating civil society pressure, which is likely to attract f",548.0302,595.1632,590.6473,625.1098,8.6004,7.7764,14.0648,True,True,True
1291,META,Meta,2025-04-26,negative,medium,short,"High-profile public protest by grieving parents increases reputational damage and litigation risk, w",548.0302,595.1632,590.6473,625.1097,8.6004,7.7764,14.0648,False,False,False
1292,META,Meta,2025-04-26,negative,medium,medium,Growing civil society and regulatory pressure could force Meta to implement stricter safety measures,548.0302,595.1632,590.6473,625.1097,8.6004,7.7764,14.0648,False,False,False
1293,META,Meta,2025-04-26,positive,high,short,"The protest is a direct indicator of escalating civil society pressure, which is likely to attract f",548.0302,595.1632,590.6473,625.1097,8.6004,7.7764,14.0648,True,True,True
1362,GOOGL,Alphabet,2025-04-24,positive,medium,short,Alphabet exceeded earnings expectations and stock jumped over 7% in after-hours trading despite macr,158.7296,160.7426,153.7469,170.2796,1.2682,-3.1391,7.2765,True,False,True
1047,NFLX,Netflix,2025-04-23,positive,medium,long,"Articulation of a $1 trillion market cap goal by co-CEO suggests strong long-term confidence, which ",104.959,113.172,115.541,119.463,7.825,10.082,13.8187,True,True,True
1048,NFLX,Netflix,2025-04-23,positive,medium,long,Ambitious growth targets and diversification into new ventures like theater and retail may strengthe,104.959,113.172,115.541,119.463,7.825,10.082,13.8187,True,True,True
1052,META,Meta,2025-04-23,positive,medium,medium,"Expanding ads globally increases Threads' competitiveness against X, leveraging high user engagement",518.652,547.2925,594.9539,633.5235,5.5221,14.7116,22.1481,True,True,True
1053,META,Meta,2025-04-23,positive,high,short,Opening ad inventory to global advertisers directly expands monetization opportunities across a user,518.652,547.2925,594.9539,633.5235,5.5221,14.7116,22.1481,True,True,True
1054,META,Meta,2025-04-23,positive,medium,medium,"Meta positions Threads as more advertiser-friendly than X, using Instagram's network effects to stre",518.652,547.2925,594.9539,633.5235,5.5221,14.7116,22.1481,True,True,True
1045,META,Meta,2025-04-20,negative,medium,long,Ongoing struggles with Facebook's cultural relevance may weaken Meta's competitive position over tim,483.1527,545.568,595.1632,638.3485,12.9183,23.1833,32.1215,False,False,False
1046,META,Meta,2025-04-20,negative,medium,long,Declining cultural relevance of Facebook could lead to erosion in user engagement and time spent on ,483.1527,545.568,595.1632,638.3485,12.9183,23.1833,32.1215,False,False,False
1052,META,Meta,2025-04-23,positive,medium,medium,"Expanding ads globally increases Threads' competitiveness against X, leveraging high user engagement",518.652,547.2925,594.9539,633.5236,5.5221,14.7116,22.1481,True,True,True
1053,META,Meta,2025-04-23,positive,high,short,Opening ad inventory to global advertisers directly expands monetization opportunities across a user,518.652,547.2925,594.9539,633.5236,5.5221,14.7116,22.1481,True,True,True
1054,META,Meta,2025-04-23,positive,medium,medium,"Meta positions Threads as more advertiser-friendly than X, using Instagram's network effects to stre",518.652,547.2925,594.9539,633.5236,5.5221,14.7116,22.1481,True,True,True
1045,META,Meta,2025-04-20,negative,medium,long,Ongoing struggles with Facebook's cultural relevance may weaken Meta's competitive position over tim,483.1527,545.568,595.1632,638.3484,12.9183,23.1833,32.1215,False,False,False
1046,META,Meta,2025-04-20,negative,medium,long,Declining cultural relevance of Facebook could lead to erosion in user engagement and time spent on ,483.1527,545.568,595.1632,638.3484,12.9183,23.1833,32.1215,False,False,False
1040,META,Meta,2025-04-17,negative,medium,short,TikTok's continued dominance in short-form video has already caused a dramatic slowdown in Meta's gr,499.9204,531.4919,570.4304,641.8775,6.3153,14.1043,28.3959,False,False,False
1041,META,Meta,2025-04-17,negative,medium,short,Zuckerberg's admission that TikTok directly slowed Meta's growth implies a loss of user engagement a,499.9204,531.4919,570.4304,641.8775,6.3153,14.1043,28.3959,False,False,False
1058,RIVN,Rivian,2025-04-16,positive,medium,medium,"HelloFresh’s adoption of 70 Rivian vans marks the first major commercial customer beyond Amazon, sig",11.49,11.8,13.66,14.82,2.698,18.886,28.9817,True,True,True
@@ -261,61 +261,61 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
1207,NVS,Novartis,2025-04-11,positive,medium,short,The announcement of a major investment in U.S. production facilities led to a 1.6% increase in Novar,104.3441,107.2749,108.8276,105.4892,2.8088,4.2969,1.0975,True,True,True
1372,HF,Hugging Face,2025-04-11,positive,medium,medium,Increased public legal conflict between OpenAI and Elon Musk may divert focus and resources from Ope,19.9983,20.0924,19.9884,19.9785,0.4707,-0.0495,-0.0991,True,False,False
1108,SHOP,Shopify,2025-04-10,negative,medium,short,"Growing traction of TikTok Shop in social commerce, especially among Gen Z and legacy brands, may er",84.63,83.65,95.12,94.0,-1.158,12.3951,11.0717,True,False,False
1478,META,Meta,2025-04-10,negative,medium,short,Whistleblower testimony before Congress alleging collusion with the Chinese government on censorship,544.591,499.9204,531.4919,596.1501,-8.2026,-2.4053,9.4675,True,True,False
1479,META,Meta,2025-04-10,negative,medium,medium,"Allegations of aiding Chinese AI development via Llama could damage partnerships, restrict future AI",544.591,499.9204,531.4919,596.1501,-8.2026,-2.4053,9.4675,True,True,False
1480,META,Meta,2025-04-10,positive,high,short,Senate testimony alleging Meta’s cooperation with the Chinese Communist Party on censorship and data,544.591,499.9204,531.4919,596.1501,-8.2026,-2.4053,9.4675,False,False,True
1373,META,Meta,2025-04-10,negative,medium,short,"Allegations of collaboration with the Chinese Communist Party on censorship tools, aired during a U.",544.591,499.9204,531.4919,596.1501,-8.2026,-2.4053,9.4675,True,True,False
1374,META,Meta,2025-04-10,negative,medium,medium,Increased reputational damage and regulatory scrutiny from U.S. lawmakers may weaken Meta's public t,544.591,499.9204,531.4919,596.1501,-8.2026,-2.4053,9.4675,True,True,False
1175,ORCL,Oracle,2025-03-31,negative,medium,short,"Public criticism over handling of security incidents, particularly involving sensitive patient data,",137.9258,125.4463,133.3026,138.7479,-9.048,-3.3519,0.5961,True,True,False
1176,ORCL,Oracle,2025-03-31,negative,medium,medium,"Security concerns, especially in healthcare and legacy infrastructure, may weaken trust in Oracle's ",137.9258,125.4463,133.3026,138.7479,-9.048,-3.3519,0.5961,True,True,False
1478,META,Meta,2025-04-10,negative,medium,short,Whistleblower testimony before Congress alleging collusion with the Chinese government on censorship,544.591,499.9204,531.4919,596.1502,-8.2026,-2.4053,9.4675,True,True,False
1479,META,Meta,2025-04-10,negative,medium,medium,"Allegations of aiding Chinese AI development via Llama could damage partnerships, restrict future AI",544.591,499.9204,531.4919,596.1502,-8.2026,-2.4053,9.4675,True,True,False
1480,META,Meta,2025-04-10,positive,high,short,Senate testimony alleging Meta’s cooperation with the Chinese Communist Party on censorship and data,544.591,499.9204,531.4919,596.1502,-8.2026,-2.4053,9.4675,False,False,True
1373,META,Meta,2025-04-10,negative,medium,short,"Allegations of collaboration with the Chinese Communist Party on censorship tools, aired during a U.",544.591,499.9204,531.4919,596.1502,-8.2026,-2.4053,9.4675,True,True,False
1374,META,Meta,2025-04-10,negative,medium,medium,Increased reputational damage and regulatory scrutiny from U.S. lawmakers may weaken Meta's public t,544.591,499.9204,531.4919,596.1502,-8.2026,-2.4053,9.4675,True,True,False
1175,ORCL,Oracle,2025-03-31,negative,medium,short,"Public criticism over handling of security incidents, particularly involving sensitive patient data,",137.9258,125.4463,133.3026,138.748,-9.048,-3.3519,0.5961,True,True,False
1176,ORCL,Oracle,2025-03-31,negative,medium,medium,"Security concerns, especially in healthcare and legacy infrastructure, may weaken trust in Oracle's ",137.9258,125.4463,133.3026,138.748,-9.048,-3.3519,0.5961,True,True,False
1481,TSLA,Tesla,2025-03-27,positive,medium,short,Tesla's 100% US production insulates it from the 25% import tariffs that will burden competitors lik,273.13,267.28,252.4,259.51,-2.1418,-7.5898,-4.9866,False,False,False
1482,TSLA,Tesla,2025-03-27,positive,low,medium,"With competitors facing higher costs due to tariffs, Tesla may gain slight market share in the US, t",273.13,267.28,252.4,259.51,-2.1418,-7.5898,-4.9866,False,False,False
1554,RIVN,Rivian,2025-03-26,positive,medium,medium,"By expanding into micromobility through Also, Rivian strengthens its technological brand and diversi",12.1,12.49,11.77,11.8,3.2231,-2.7273,-2.4793,True,False,False
1494,TSM,TSMC,2025-03-25,negative,medium,medium,The substantial capital expenditure with expected margin pressure from U.S. operations may weigh on ,178.6869,166.5769,139.6405,149.5478,-6.7772,-21.8518,-16.3074,True,True,True
1181,META,Meta,2025-03-24,negative,medium,short,"Meta's failed acquisition of FuriosaAI, an AI chip startup developing competitive chips for reasonin",616.9253,574.5675,514.6444,483.1527,-6.866,-16.5791,-21.6838,True,True,True
1182,META,Meta,2025-03-24,negative,low,medium,"While FuriosaAI remains independent and may expand its partnerships (e.g., with LG AI Research), Met",616.9253,574.5675,514.6444,483.1527,-6.866,-16.5791,-21.6838,True,True,True
1494,TSM,TSMC,2025-03-25,negative,medium,medium,The substantial capital expenditure with expected margin pressure from U.S. operations may weigh on ,178.6869,166.5769,139.6405,149.5478,-6.7772,-21.8519,-16.3073,True,True,True
1181,META,Meta,2025-03-24,negative,medium,short,"Meta's failed acquisition of FuriosaAI, an AI chip startup developing competitive chips for reasonin",616.9254,574.5674,514.6444,483.1526,-6.866,-16.5791,-21.6838,True,True,True
1182,META,Meta,2025-03-24,negative,low,medium,"While FuriosaAI remains independent and may expand its partnerships (e.g., with LG AI Research), Met",616.9254,574.5674,514.6444,483.1526,-6.866,-16.5791,-21.6838,True,True,True
1183,FTNT,Fortinet,2025-03-17,negative,medium,short,The public disclosure of active exploitation of Fortinet vulnerabilities by a LockBit-linked group m,96.67,99.79,96.26,96.85,3.2275,-0.4241,0.1862,False,True,False
1184,FTNT,Fortinet,2025-03-17,negative,medium,medium,Repeated security breaches linked to Fortinet products could weaken its market standing versus compe,96.67,99.79,96.26,96.85,3.2275,-0.4241,0.1862,False,True,False
1359,RIVN,Rivian,2025-03-05,positive,high,long,Providing core software and architecture to a major automaker like Volkswagen enhances Rivian’s stra,11.42,11.06,11.36,12.49,-3.1524,-0.5254,9.3695,False,False,True
1360,RIVN,Rivian,2025-03-05,positive,medium,short,The influx of capital and validation of Rivian’s technology by Volkswagen may boost investor confide,11.42,11.06,11.36,12.49,-3.1524,-0.5254,9.3695,False,False,True
1354,META,Meta,2025-03-05,positive,medium,medium,Expanding facial recognition tools in regulated markets like the UK and EU strengthens Meta's positi,653.8467,617.084,582.2435,582.1139,-5.6225,-10.9511,-10.9709,False,False,False
1355,META,Meta,2025-03-05,positive,low,medium,"Optional anti-fraud and verification tools may improve user trust and retention, particularly among ",653.8467,617.084,582.2435,582.1139,-5.6225,-10.9511,-10.9709,False,False,False
1560,BLK,BlackRock,2025-03-02,positive,medium,short,"Record inflows suggest investor confidence remains strong despite the ESG reversal, potentially supp",941.8132,927.7991,909.947,927.5837,-1.488,-3.3835,-1.5109,False,False,False
1562,BLK,BlackRock,2025-03-02,positive,medium,short,"By aligning with conservative political forces and avoiding regulatory scrutiny, BlackRock may gain ",941.8132,927.7991,909.947,927.5837,-1.488,-3.3835,-1.5109,False,False,False
1350,META,Meta,2025-02-27,positive,low,short,"Terminating leakers may strengthen internal discipline and protect strategic information, slightly i",655.6096,625.4207,588.2797,600.706,-4.6047,-10.2698,-8.3744,False,False,False
1351,META,Meta,2025-02-27,negative,medium,short,Public disclosure of internal leaks and subsequent firings may amplify perceptions of internal disse,655.6096,625.4207,588.2797,600.706,-4.6047,-10.2698,-8.3744,True,True,True
1354,META,Meta,2025-03-05,positive,medium,medium,Expanding facial recognition tools in regulated markets like the UK and EU strengthens Meta's positi,653.8466,617.0841,582.2435,582.114,-5.6225,-10.9511,-10.9709,False,False,False
1355,META,Meta,2025-03-05,positive,low,medium,"Optional anti-fraud and verification tools may improve user trust and retention, particularly among ",653.8466,617.0841,582.2435,582.114,-5.6225,-10.9511,-10.9709,False,False,False
1560,BLK,BlackRock,2025-03-02,positive,medium,short,"Record inflows suggest investor confidence remains strong despite the ESG reversal, potentially supp",941.8132,927.7991,909.947,927.5836,-1.488,-3.3835,-1.5109,False,False,False
1562,BLK,BlackRock,2025-03-02,positive,medium,short,"By aligning with conservative political forces and avoiding regulatory scrutiny, BlackRock may gain ",941.8132,927.7991,909.947,927.5836,-1.488,-3.3835,-1.5109,False,False,False
1350,META,Meta,2025-02-27,positive,low,short,"Terminating leakers may strengthen internal discipline and protect strategic information, slightly i",655.6095,625.4207,588.2797,600.706,-4.6047,-10.2698,-8.3744,False,False,False
1351,META,Meta,2025-02-27,negative,medium,short,Public disclosure of internal leaks and subsequent firings may amplify perceptions of internal disse,655.6095,625.4207,588.2797,600.706,-4.6047,-10.2698,-8.3744,True,True,True
1142,SAP,SAP,2025-02-27,negative,low,short,"The stock is continuing a downward trend, underperforming the Dax, and trading volume has decreased,",272.1395,276.837,252.9034,265.7473,1.7261,-7.0685,-2.3489,False,True,True
1114,PLTR,Palantir,2025-02-08,positive,high,short,"Retail investor enthusiasm, dollar-cost averaging, and a significant rise in stock price in 2024 and",116.65,119.16,101.35,84.91,2.1517,-13.1162,-27.2096,True,False,False
1065,AMD,AMD,2025-02-04,positive,medium,medium,"Powering 100 million game consoles indicates strong market penetration in the gaming segment, likely",119.5,111.1,114.28,100.75,-7.0293,-4.3682,-15.6904,False,False,False
1066,AMD,AMD,2025-02-04,positive,medium,short,Success in securing design wins across major console platforms strengthens AMD's position against co,119.5,111.1,114.28,100.75,-7.0293,-4.3682,-15.6904,False,False,False
1125,NVDA,NVIDIA,2025-01-30,negative,medium,short,"Technical indicators suggest further downside toward $110, with momentum shifting negative in the in",124.6092,128.6379,135.2457,120.1106,3.2331,8.5359,-3.6101,False,False,True
1244,NVDA,NVIDIA,2025-01-30,positive,medium,short,"The stock rebounded nearly 9% following a sharp decline, indicating strong investor resilience and c",124.6092,128.6379,135.2457,120.1106,3.2331,8.5359,-3.6101,True,True,False
1243,MS,Morgan Stanley,2025-01-24,positive,medium,short,"Morgan Stanley is highlighted as one of the top stocks to buy for strong Q4 earnings, with positive ",133.331,134.8122,136.3217,128.2484,1.1109,2.2431,-3.812,True,True,False
1125,NVDA,NVIDIA,2025-01-30,negative,medium,short,"Technical indicators suggest further downside toward $110, with momentum shifting negative in the in",124.6092,128.6378,135.2457,120.1106,3.233,8.5359,-3.6101,False,False,True
1244,NVDA,NVIDIA,2025-01-30,positive,medium,short,"The stock rebounded nearly 9% following a sharp decline, indicating strong investor resilience and c",124.6092,128.6378,135.2457,120.1106,3.233,8.5359,-3.6101,True,True,False
1243,MS,Morgan Stanley,2025-01-24,positive,medium,short,"Morgan Stanley is highlighted as one of the top stocks to buy for strong Q4 earnings, with positive ",132.6183,134.0916,135.593,127.5628,1.1109,2.2431,-3.812,True,True,False
1248,STX,Seagate,2025-01-23,positive,medium,short,Solid 2Q25 performance driven by cloud sector recovery and increasing AI storage demand support upwa,106.1709,96.2413,94.5374,100.5108,-9.3525,-10.9574,-5.3311,False,False,False
1130,META,Meta,2025-01-21,positive,low,short,"Enhanced cross-platform integration improves user engagement and data sharing across Meta's apps, sl",613.9966,671.6353,701.3759,713.5073,9.3875,14.2312,16.207,True,True,True
1130,META,Meta,2025-01-21,positive,low,short,"Enhanced cross-platform integration improves user engagement and data sharing across Meta's apps, sl",613.9966,671.6353,701.3759,713.5073,9.3875,14.2312,16.2071,True,True,True
1466,TSM,TSMC,2025-01-16,positive,high,short,Strong profit growth driven by high AI chip demand supports a positive short-term stock price moveme,211.3388,221.0109,204.8055,198.5871,4.5766,-3.0914,-6.0338,True,False,False
1467,TSM,TSMC,2025-01-16,positive,medium,medium,Continued leadership in advanced semiconductor manufacturing and strong customer relationships reinf,211.3388,221.0109,204.8055,198.5871,4.5766,-3.0914,-6.0338,True,False,False
1468,TSM,TSMC,2025-01-16,negative,medium,medium,Geopolitical uncertainties and export restrictions may negatively affect future business operations ,211.3388,221.0109,204.8055,198.5871,4.5766,-3.0914,-6.0338,False,True,True
1312,META,Meta,2025-01-10,negative,high,short,"Widespread condemnation from 71 fact-checking organizations, including a public open letter, signals",613.3989,610.3214,644.9025,711.6646,-0.5017,5.1359,16.0199,True,False,False
1313,META,Meta,2025-01-10,positive,medium,medium,Strong pushback from civil society groups may prompt increased scrutiny from regulators concerned wi,613.3989,610.3214,644.9025,711.6646,-0.5017,5.1359,16.0199,False,True,True
1312,META,Meta,2025-01-10,negative,high,short,"Widespread condemnation from 71 fact-checking organizations, including a public open letter, signals",613.3988,610.3212,644.9025,711.6646,-0.5017,5.1359,16.0199,True,False,False
1313,META,Meta,2025-01-10,positive,medium,medium,Strong pushback from civil society groups may prompt increased scrutiny from regulators concerned wi,613.3988,610.3212,644.9025,711.6646,-0.5017,5.1359,16.0199,False,True,True
1159,NVDA,NVIDIA,2025-01-07,positive,medium,short,Analyst reaffirmation of Nvidia as a top pick following a major executive keynote typically boosts i,140.0941,131.7168,140.7839,118.6111,-5.9797,0.4924,-15.3347,False,True,False
1160,NVDA,NVIDIA,2025-01-07,positive,low,medium,Positive analyst sentiment following a strategic keynote may reinforce market perception of Nvidia's,140.0941,131.7168,140.7839,118.6111,-5.9797,0.4924,-15.3347,False,True,False
1157,RIVN,Rivian,2025-01-03,positive,high,short,"Rivian's stock surged 24.5%, its largest daily increase since going public, after meeting revised pr",16.49,13.85,14.21,12.56,-16.0097,-13.8266,-23.8326,False,False,False
1158,RIVN,Rivian,2025-01-03,positive,medium,medium,Resolving production constraints and delivering above analyst expectations strengthens Rivian's posi,16.49,13.85,14.21,12.56,-16.0097,-13.8266,-23.8326,False,False,False
1162,AVGO,Broadcom,2024-12-31,positive,high,short,Broadcom's stock price rose 111% in 2024 due to strong AI-related demand and market outperformance r,229.2828,226.1181,222.2215,205.0728,-1.3803,-3.0797,-10.559,False,False,False
1163,AVGO,Broadcom,2024-12-31,positive,medium,medium,"Broadcom is gaining ground in the AI chip and networking space despite Nvidia's dominance, positioni",229.2828,226.1181,222.2215,205.0728,-1.3803,-3.0797,-10.559,False,False,False
1162,AVGO,Broadcom,2024-12-31,positive,high,short,Broadcom's stock price rose 111% in 2024 due to strong AI-related demand and market outperformance r,229.2828,226.1181,222.2216,205.0728,-1.3803,-3.0797,-10.559,False,False,False
1163,AVGO,Broadcom,2024-12-31,positive,medium,medium,"Broadcom is gaining ground in the AI chip and networking space despite Nvidia's dominance, positioni",229.2828,226.1181,222.2216,205.0728,-1.3803,-3.0797,-10.559,False,False,False
1161,AAPL,Apple,2024-12-26,positive,high,medium,"Apple's stock soars to a record high, and JPMorgan has a positive outlook for the company in 2025, s",257.6127,242.5252,235.5632,222.4449,-5.8567,-8.5592,-13.6515,False,False,False
1317,LLY,Eli Lilly,2024-12-23,positive,medium,short,Eli Lilly's potential to extend a winning streak over the broad market for 6 years suggests continue,789.0677,766.8309,758.17,735.6262,-2.8181,-3.9157,-6.7727,False,False,False
1317,LLY,Eli Lilly,2024-12-23,positive,medium,short,Eli Lilly's potential to extend a winning streak over the broad market for 6 years suggests continue,789.0677,766.8309,758.1701,735.6262,-2.8181,-3.9157,-6.7727,False,False,False
1427,SMCI,Super Micro Computer,2024-12-16,negative,medium,short,"Removal from Nasdaq 100 triggers passive fund selling, combined with ongoing governance and complian",33.44,32.4,30.68,31.08,-3.11,-8.2536,-7.0574,True,True,True
1428,SMCI,Super Micro Computer,2024-12-16,negative,low,medium,Reduced market visibility and investor confidence may impair access to capital and strategic partner,33.44,32.4,30.68,31.08,-3.11,-8.2536,-7.0574,True,True,True
1254,META,Meta,2024-12-14,positive,medium,medium,"By challenging OpenAI’s for-profit transition, Meta aims to constrain a key AI competitor’s flexibil",621.7454,582.9113,597.4131,613.3989,-6.246,-3.9135,-1.3424,False,False,False
1255,META,Meta,2024-12-14,positive,high,short,"Meta’s public call for regulatory intervention increases scrutiny on OpenAI, amplifying broader regu",621.7454,582.9113,597.4131,613.3989,-6.246,-3.9135,-1.3424,False,False,False
1424,AAPL,Apple,2024-12-13,positive,medium,medium,"Morgan Stanley naming Apple a top pick for 2025 signals strong institutional confidence, supporting ",246.7819,253.1074,254.2014,235.5632,2.5632,3.0065,-4.546,True,True,False
1254,META,Meta,2024-12-14,positive,medium,medium,"By challenging OpenAI’s for-profit transition, Meta aims to constrain a key AI competitor’s flexibil",621.7454,582.9113,597.4131,613.3988,-6.246,-3.9136,-1.3424,False,False,False
1255,META,Meta,2024-12-14,positive,high,short,"Meta’s public call for regulatory intervention increases scrutiny on OpenAI, amplifying broader regu",621.7454,582.9113,597.4131,613.3988,-6.246,-3.9136,-1.3424,False,False,False
1424,AAPL,Apple,2024-12-13,positive,medium,medium,"Morgan Stanley naming Apple a top pick for 2025 signals strong institutional confidence, supporting ",246.7819,253.1073,254.2014,235.5632,2.5632,3.0065,-4.546,True,True,False
1441,SPOT,Spotify,2024-12-11,positive,medium,short,Public discussion around faking Spotify Wrapped indicates high user interest and emotional investmen,476.91,448.65,457.98,479.73,-5.9256,-3.9693,0.5913,False,False,True
1439,RIVN,Rivian,2024-12-09,positive,medium,medium,Rivian's superior charging experience and renewable energy partnerships enhance its differentiation ,14.45,15.34,13.75,15.715,6.1592,-4.8443,8.7543,True,False,True
1440,RIVN,Rivian,2024-12-09,positive,low,long,Expanded charging infrastructure accessible to all EV users increases Rivian's visibility and brand ,14.45,15.34,13.75,15.715,6.1592,-4.8443,8.7543,True,False,True
1431,CVX,Chevron,2024-12-08,positive,medium,short,"Goldman Sachs reiterated a buy rating with a raised price target, citing strong shareholder returns ",148.6666,145.6285,135.1987,139.9309,-2.0435,-9.0591,-5.876,False,False,False
1432,CVX,Chevron,2024-12-08,positive,medium,medium,Recognition by top Wall Street analysts as a top dividend stock enhances Chevron's profile among ene,148.6666,145.6285,135.1987,139.9309,-2.0435,-9.0591,-5.876,False,False,False
1431,CVX,Chevron,2024-12-08,positive,medium,short,"Goldman Sachs reiterated a buy rating with a raised price target, citing strong shareholder returns ",148.6666,145.6285,135.1987,139.931,-2.0435,-9.0591,-5.876,False,False,False
1432,CVX,Chevron,2024-12-08,positive,medium,medium,Recognition by top Wall Street analysts as a top dividend stock enhances Chevron's profile among ene,148.6666,145.6285,135.1987,139.931,-2.0435,-9.0591,-5.876,False,False,False
1429,SHOP,Shopify,2024-12-06,positive,medium,short,"Analyst upgrade typically leads to improved market sentiment and short-term stock price momentum, es",118.37,114.63,108.95,109.25,-3.1596,-7.9581,-7.7047,False,False,False
1430,SHOP,Shopify,2024-12-06,positive,low,medium,Increased recognition of AI capabilities may enhance Shopify's positioning against competitors over ,118.37,114.63,108.95,109.25,-3.1596,-7.9581,-7.7047,False,False,False
1437,AMD,AMD,2024-12-04,positive,medium,short,"The article suggests a potential catch-up trade based on technical charts, indicating upward momentu",143.99,130.15,121.41,120.63,-9.6118,-15.6816,-16.2234,False,False,False
@@ -324,15 +324,15 @@ id,ticker,name,event_date,direction,magnitude,timeframe,rationale,price_0,price_
1433,NOW,ServiceNow,2024-12-01,positive,high,medium,"Analyst upgraded price target due to strong financials, AI tailwinds, and confidence in near- and me",209.686,224.868,224.22,216.292,7.2403,6.9313,3.1504,True,True,True
1434,NOW,ServiceNow,2024-12-01,positive,medium,long,"New Workflow Data Fabric product expected to power new workflows and AI agents, enhancing differenti",209.686,224.868,224.22,216.292,7.2403,6.9313,3.1504,True,True,True
1435,NOW,ServiceNow,2024-12-01,positive,high,long,"Product innovation expected to double total addressable market to $500 billion, enabling greater mar",209.686,224.868,224.22,216.292,7.2403,6.9313,3.1504,True,True,True
1442,META,Meta,2024-11-29,positive,high,long,"By building a private, globally spanning subsea cable, Meta gains greater control over data transmis",571.5638,620.7766,617.373,597.4131,8.6102,8.0147,4.5226,True,True,True
1443,META,Meta,2024-11-29,negative,medium,short,"The $10 billion upfront investment may raise investor concerns about near-term profitability, especi",571.5638,620.7766,617.373,597.4131,8.6102,8.0147,4.5226,False,False,False
1444,META,Meta,2024-11-29,positive,medium,long,Exclusive control over high-capacity global data infrastructure will enable Meta to scale AI-driven ,571.5638,620.7766,617.373,597.4131,8.6102,8.0147,4.5226,True,True,True
1446,META,Meta,2024-11-21,positive,low,medium,"Proactive measures against scams may improve platform trustworthiness, slightly enhancing Meta's com",560.3878,571.5639,606.0079,593.1899,1.9943,8.1408,5.8535,True,True,True
1442,META,Meta,2024-11-29,positive,high,long,"By building a private, globally spanning subsea cable, Meta gains greater control over data transmis",571.5638,620.7766,617.3729,597.4131,8.6102,8.0147,4.5225,True,True,True
1443,META,Meta,2024-11-29,negative,medium,short,"The $10 billion upfront investment may raise investor concerns about near-term profitability, especi",571.5638,620.7766,617.3729,597.4131,8.6102,8.0147,4.5225,False,False,False
1444,META,Meta,2024-11-29,positive,medium,long,Exclusive control over high-capacity global data infrastructure will enable Meta to scale AI-driven ,571.5638,620.7766,617.3729,597.4131,8.6102,8.0147,4.5225,True,True,True
1446,META,Meta,2024-11-21,positive,low,medium,"Proactive measures against scams may improve platform trustworthiness, slightly enhancing Meta's com",560.3878,571.564,606.0078,593.19,1.9944,8.1408,5.8535,True,True,True
1453,CMCSA,Comcast,2024-11-20,positive,medium,short,"A spin-off of the cable business could unlock shareholder value and streamline operations, potential",37.9328,37.5534,37.5446,33.4063,-1.0002,-1.0235,-11.933,False,False,False
1454,CMCSA,Comcast,2024-11-20,positive,medium,long,Separating the cable business may allow Comcast to focus on growth areas like streaming and broadban,37.9328,37.5534,37.5446,33.4063,-1.0002,-1.0235,-11.933,False,False,False
1448,META,Meta,2024-11-14,negative,medium,short,The $840 million fine represents a significant financial penalty and reinforces investor concerns ab,574.3903,560.3878,571.5639,627.7628,-2.4378,-0.4921,9.292,True,True,False
1449,META,Meta,2024-11-14,negative,medium,medium,"The EU ruling may force Meta to alter how Marketplace is integrated into Facebook, potentially reduc",574.3903,560.3878,571.5639,627.7628,-2.4378,-0.4921,9.292,True,True,False
1450,META,Meta,2024-11-14,positive,high,long,"This fine adds to a pattern of EU enforcement actions, signaling sustained and increasing regulatory",574.3903,560.3878,571.5639,627.7628,-2.4378,-0.4921,9.292,False,False,True
1448,META,Meta,2024-11-14,negative,medium,short,The $840 million fine represents a significant financial penalty and reinforces investor concerns ab,574.3902,560.3878,571.564,627.7629,-2.4378,-0.492,9.2921,True,True,False
1449,META,Meta,2024-11-14,negative,medium,medium,"The EU ruling may force Meta to alter how Marketplace is integrated into Facebook, potentially reduc",574.3902,560.3878,571.564,627.7629,-2.4378,-0.492,9.2921,True,True,False
1450,META,Meta,2024-11-14,positive,high,long,"This fine adds to a pattern of EU enforcement actions, signaling sustained and increasing regulatory",574.3902,560.3878,571.564,627.7629,-2.4378,-0.492,9.2921,False,False,True
1527,TSLA,Tesla,2024-11-08,positive,high,short,"Tesla's stock surged 29% in one week following Trump's election, driven by investor optimism over re",321.22,320.72,352.56,389.22,-0.1557,9.7566,21.1693,False,True,True
1528,TSLA,Tesla,2024-11-08,positive,medium,medium,Potential higher tariffs on Chinese EVs like BYD could reduce competitive pressure in the U.S. marke,321.22,320.72,352.56,389.22,-0.1557,9.7566,21.1693,False,True,True
1552,RIVN,Rivian,2024-10-17,positive,low,short,"The novelty of themed updates may attract media attention and increase short-term consumer interest,",10.12,10.43,10.1,10.31,3.0632,-0.1976,1.8775,True,False,True
1 id ticker name event_date direction magnitude timeframe rationale price_0 price_5d price_10d price_20d 5d_return 10d_return 20d_return correct_5d correct_10d correct_20d
92 1240 HF Hugging Face 2025-10-03 negative medium medium OpenAI's valuation surge to $500 billion highlights its dominant market position and ability to attr 20.9692 20.8305 21.0089 20.9791 -0.6615 0.189 0.0472 True False False
93 1304 META Meta 2025-10-02 positive medium medium By using AI interaction data to improve recommendation relevance, Meta can increase user engagement 725.8361 732.2853 710.8811 665.3572 0.8885 -2.0604 -8.3323 True False False
94 1305 META Meta 2025-10-02 positive medium medium Enhanced personalization using deeper AI signals strengthens Meta's competitive edge in social media 725.8361 732.2853 710.8811 665.3572 0.8885 -2.0604 -8.3323 True False False
95 935 META Meta 2025-10-01 positive medium short AI-driven ad personalization strengthens Meta's competitive edge in digital advertising by improving 716.1423 716.6415 716.3519 750.4149 750.415 0.0697 0.0293 4.7857 True True True
96 937 META Meta 2025-10-01 positive medium medium Enhanced ad targeting using AI interactions could increase advertiser ROI, driving higher ad spend s 716.1423 716.6415 716.3519 750.4149 750.415 0.0697 0.0293 4.7857 True True True
97 1302 SPOT Spotify 2025-09-27 positive medium medium By adopting DDEX standards for AI transparency, Spotify positions itself as a responsible leader in 728.47 680.5 685.29 645.78 -6.585 -5.9275 -11.3512 False False False
98 926 META Meta 2025-09-18 positive medium medium Launch of consumer-ready smartglasses with differentiated AI features may increase Meta's presence i 778.4218 747.6595 725.8361 710.8811 -3.9519 -6.7554 -8.6766 False False False
99 927 META Meta 2025-09-18 positive medium short Introduction of AI-integrated smartglasses with partnerships in fitness tech strengthens Meta's posi 778.4218 747.6595 725.8361 710.8811 -3.9519 -6.7554 -8.6766 False False False
110 637 AZN AstraZeneca 2025-09-13 negative medium short Pausing a major investment may signal reduced confidence in the UK market, potentially affecting inv 154.4894 150.9859 145.9979 167.3157 -2.2678 -5.4965 8.3024 True True False
111 638 AZN AstraZeneca 2025-09-13 negative medium medium Delaying expansion in a key life sciences hub like Cambridge could slow innovation and talent acquis 154.4894 150.9859 145.9979 167.3157 -2.2678 -5.4965 8.3024 True True False
112 628 ORCL Oracle 2025-09-12 positive medium short Bullish options activity suggests increased investor confidence in Oracle's near-term stock performa 289.8924 306.2434 281.2406 291.1707 5.6404 -2.9845 0.441 True False True
113 988 AAPL Apple 2025-09-12 positive medium short Strong buy-side investor sentiment, particularly from Indian retail investors, combined with histori 233.6247 245.033 254.974 244.8034 4.8832 4.8831 9.1383 4.7849 True True True
114 627 ORCL Oracle 2025-09-11 negative medium short Shares retreated 6-7% after a record high due to concerns about overreliance on OpenAI for growth, d 305.4496 294.2976 289.049 295.1463 295.1462 -3.651 -5.3693 -3.3732 True True True
115 1409 ORCL Oracle 2025-09-11 positive high short Oracle's stock surged 35.98% following strong cloud revenue forecasts and major AI-related contracts 305.4496 294.2976 289.049 295.1463 295.1462 -3.651 -5.3693 -3.3732 False False False
116 1410 ORCL Oracle 2025-09-11 positive medium medium With a cloud services backlog nearing $500 billion and major contracts in AI infrastructure, Oracle 305.4496 294.2976 289.049 295.1463 295.1462 -3.651 -5.3693 -3.3732 False False False
117 1411 ORCL Oracle 2025-09-11 positive high medium Oracle's strategic positioning as a key AI cloud infrastructure provider through partnerships with l 305.4496 294.2976 289.049 295.1463 295.1462 -3.651 -5.3693 -3.3732 False False False
118 611 INTC Intel 2025-08-28 positive medium short Direct government investment signals confidence and improves public perception, consistent with prio 24.93 24.61 24.61 33.99 -1.2836 -1.2836 36.3418 False False True
119 612 INTC Intel 2025-08-28 positive medium medium Government funding and stake may enhance capital investment in manufacturing, supporting domestic pr 24.93 24.61 24.61 33.99 -1.2836 -1.2836 36.3418 False False True
120 608 NVDA NVIDIA 2025-08-28 positive high short Extraordinary demand and full-speed ramp-up of Blackwell Ultra platform indicate strong adoption, re 180.1401 171.6315 177.1506 177.6705 -4.7233 -1.6595 -1.3709 False False False
135 857 PLTR Palantir 2025-08-10 positive medium medium Recognition as 'the best story in all of software' and leadership in AI-driven government and enterp 182.68 177.17 158.74 153.11 -3.0162 -13.1049 -16.1868 False False False
136 1208 INTC Intel 2025-08-08 negative medium short Trump's public demand for CEO resignation and allegations of conflict due to ties with Chinese firms 19.95 24.56 24.8 24.49 23.1078 24.3108 22.7569 False False False
137 1209 INTC Intel 2025-08-08 negative low short Leadership instability and political scrutiny may delay turnaround plans, giving competitors like Nv 19.95 24.56 24.8 24.49 23.1078 24.3108 22.7569 False False False
138 1210 META Meta 2025-08-08 positive high medium The $29 billion financing enables accelerated AI infrastructure development, strengthening Meta's ca 767.4975 767.4976 783.3901 783.3902 753.0215 750.687 750.6871 2.0707 -1.8861 -2.1903 True False False
139 1289 TSM TSMC 2025-08-08 negative medium short The leak of trade secrets, even if not directly TSMC’s fault, could undermine confidence in its IP p 239.7449 236.8203 230.9811 241.3113 241.3112 -1.2199 -3.6555 0.6534 0.6533 True True False
140 1290 TSM TSMC 2025-08-08 negative low short While the incident does not directly implicate TSMC in wrongdoing, associated supply chain instabili 239.7449 236.8203 230.9811 241.3113 241.3112 -1.2199 -3.6555 0.6534 0.6533 True True False
141 1218 META Meta 2025-08-08 positive high long The $29 billion financing enables large-scale AI data center development, strengthening Meta's infra 767.4975 767.4976 783.3901 783.3902 753.0215 750.687 750.6871 2.0707 -1.8861 -2.1903 True False False
142 1511 SPOT Spotify 2025-08-07 positive medium short The introduction of AI DJ, which provides personalized, context-rich music recommendations using gen 686.74 698.5 689.47 703.85 1.7124 0.3975 2.4915 True True True
143 1513 SPOT Spotify 2025-08-07 positive medium medium Investment in advanced AI features like AI DJ strengthens Spotify’s differentiation from competitors 686.74 698.5 689.47 703.85 1.7124 0.3975 2.4915 True True True
144 851 PLTR Palantir 2025-08-07 negative medium medium CEO's remarks may deepen skepticism among educated public and academia, contributing to declining pu 182.2 181.02 156.18 156.14 -0.6476 -14.281 -14.303 True True True
145 852 PLTR Palantir 2025-08-07 positive medium long Positioning Palantir as a meritocratic alternative to elite education could strengthen employer bran 182.2 181.02 156.18 156.14 -0.6476 -14.281 -14.303 False False False
146 1288 TSM TSMC 2025-08-07 negative medium short Increased geopolitical risk and costly overseas expansion could weigh on investor sentiment in the n 240.5281 238.922 225.3699 233.182 -0.6677 -6.302 -3.0541 -3.0542 True True True
147 1509 TSM TSMC 2025-08-07 positive medium short Tariff exemption strengthens TSMC's competitive position relative to non-U.S.-based semiconductor ma 240.5281 238.922 225.3699 233.182 -0.6677 -6.302 -3.0541 -3.0542 False False False
148 1510 TSM TSMC 2025-08-07 positive low short Exemption from high tariffs provides operational certainty and reduces near-term geopolitical risk, 240.5281 238.922 225.3699 233.182 -0.6677 -6.302 -3.0541 -3.0542 False False False
149 873 PLTR Palantir 2025-08-05 positive high short Palantir's strong earnings, revenue growth of nearly 50%, net income up 144%, raised guidance, and i 173.27 186.97 157.75 157.09 7.9067 -8.9571 -9.338 True False False
150 874 PLTR Palantir 2025-08-05 positive medium medium Investor confidence in Palantir's AI-powered efficiency and scalability may accelerate adoption in g 173.27 186.97 157.75 157.09 7.9067 -8.9571 -9.338 True False False
151 875 PLTR Palantir 2025-08-05 positive medium short Palantir's demonstrated ability to grow revenue while reducing workforce through AI gives it a perce 173.27 186.97 157.75 157.09 7.9067 -8.9571 -9.338 True False False
152 1463 META Meta 2025-08-01 positive high short Meta shares jump after strong third-quarter sales forecast and better-than-expected financial perfor 748.2527 767.4975 767.4976 783.3901 783.3902 736.9692 2.572 4.6959 -1.508 True True False
153 1307 MSFT Microsoft 2025-08-01 positive high short Microsoft's stock rose 3.9% on July 31, 2025, following record valuation and strong quarterly result 521.0829 519.0249 519.025 517.1657 504.5917 -0.395 -0.3949 -0.7517 -3.1648 False False False
154 1308 MSFT Microsoft 2025-08-01 positive medium medium Azure's 39% growth and increasing AI-driven cloud revenue suggest Microsoft is gaining cloud market 521.0829 519.0249 519.025 517.1657 504.5917 -0.395 -0.3949 -0.7517 -3.1648 False False False
155 1309 MSFT Microsoft 2025-08-01 positive high long Surpassing four trillion dollars in market cap, second only to Nvidia, reinforces Microsoft's elite 521.0829 519.0249 519.025 517.1657 504.5917 -0.395 -0.3949 -0.7517 -3.1648 False False False
156 1400 ARM Arm Holdings 2025-07-31 negative high short Arm's shares dropped nearly 13% following disappointing guidance and a strategic shift that risks cu 141.375 135.57 140.55 142.55 -4.1061 -0.5836 0.8311 True True False
157 1401 ARM Arm Holdings 2025-07-31 negative medium medium Developing full chip solutions may lead to conflicts of interest with key customers like Nvidia and 141.375 135.57 140.55 142.55 -4.1061 -0.5836 0.8311 True True False
158 1402 ARM Arm Holdings 2025-07-31 negative low long Long-term market share could be affected if customers reduce reliance on Arm's IP due to competitive 141.375 135.57 140.55 142.55 -4.1061 -0.5836 0.8311 True True False
163 1222 META Meta 2025-07-31 positive medium medium Aggressive investment in AI talent and infrastructure positions Meta to better compete with OpenAI, 771.6278 760.045 780.2974 749.3502 -1.5011 1.1235 -2.8871 False True False
164 1223 META Meta 2025-07-31 positive medium medium Strong ad revenue growth and AI-driven product enhancements may increase user engagement and ad mark 771.6278 760.045 780.2974 749.3502 -1.5011 1.1235 -2.8871 False True False
165 1279 UBS UBS 2025-07-30 negative medium medium Proposed capital requirements could impair competitiveness if implemented; CEO warns 42 billion doll 36.9761 37.0834 38.6529 39.15 0.29 4.5347 5.8793 False False False
166 893 META Meta 2025-07-26 positive high medium Hiring a foundational OpenAI researcher and formalizing leadership under a strong AI executive team 715.9486 748.2527 767.4975 767.4976 753.0215 4.5121 7.2001 5.1781 True True True
167 894 META Meta 2025-07-26 positive medium long Strengthening AI talent base supports future AI product development, potentially increasing market s 715.9486 748.2527 767.4975 767.4976 753.0215 4.5121 7.2001 5.1781 True True True
168 1558 VZ Verizon 2025-07-24 negative low short The $175 million penalty represents a one-time cost that may slightly pressure earnings, but given V 41.0128 40.7082 40.8891 42.8693 -0.7428 -0.3018 4.5264 4.5265 True True False
169 1559 VZ Verizon 2025-07-24 negative medium short Increased legal risk from patent litigation could affect investor sentiment and raise concerns about 41.0128 40.7082 40.8891 42.8693 -0.7428 -0.3018 4.5264 4.5265 True True False
170 799 META Meta 2025-07-17 negative medium short Public perception may be negatively influenced by the association of top executives with privacy vio 699.7665 713.1252 771.6278 780.2974 1.909 10.2693 11.5083 False False False
171 1109 NVDA NVIDIA 2025-07-10 positive high short Nvidia's stock has surged 69% since early April and analysts project an additional 17% rise to $190 164.0727 172.9713 173.7111 180.74 5.4235 5.8745 10.1584 True True True
172 1110 NVDA NVIDIA 2025-07-10 positive medium medium Ongoing demand from big tech and AI innovators for high-performance computing chips reinforces Nvidi 164.0727 172.9713 173.7111 180.74 5.4235 5.8745 10.1584 True True True
182 1084 MU Micron 2025-07-03 positive low long Sustained revenue growth from rising HBM demand and expanded production capacity could support long- 121.998 122.9316 113.0959 108.9819 0.7653 -7.2969 -10.6691 True False False
183 1506 BP BP 2025-07-02 positive medium short Takeover speculation and activist involvement typically create short-term stock price momentum, even 30.038 30.0093 30.633 30.9497 -0.0958 1.9808 3.0351 False True True
184 1507 BP BP 2025-07-02 negative medium medium Persistent speculation that BP could be acquired may weaken its strategic autonomy and bargaining po 30.038 30.0093 30.633 30.9497 -0.0958 1.9808 3.0351 True False False
185 902 META Meta 2025-06-13 positive high medium Acquiring key AI talent and investing heavily in AI infrastructure through Scale AI strengthens Meta 680.7462 680.7463 680.7512 731.9111 715.8289 0.0007 7.516 5.1536 5.1535 True True True
186 903 META Meta 2025-06-13 positive medium short Major strategic investment in AI and high-profile talent acquisition may be viewed favorably by inve 680.7462 680.7463 680.7512 731.9111 715.8289 0.0007 7.516 5.1536 5.1535 True True True
187 1476 NVS Novartis 2025-06-13 positive medium short Novo Nordisk's leadership turmoil and stock decline may weaken its market positioning, creating oppo 115.9217 112.3504 116.4652 117.4453 -3.0808 0.4688 1.3144 False True True
188 1477 NVS Novartis 2025-06-13 positive low short While Novartis is not directly affected by Novo Nordisk's CEO dismissal, reduced competitive pressur 115.9217 112.3504 116.4652 117.4453 -3.0808 0.4688 1.3144 False True True
189 1085 AMD AMD 2025-06-13 positive high medium AMD’s launch of the Helios server and partnership with OpenAI strengthen its position in the AI chip 116.16 128.24 143.81 146.42 10.3995 23.8034 26.0503 True True True
195 1397 UBS UBS 2025-06-08 positive high medium The new draft law significantly increases regulatory obligations, including capitalization of foreig 32.1019 31.1758 29.655 33.3107 -2.8849 -7.6222 3.7656 False False True
196 1556 UBS UBS 2025-06-06 negative medium short Stricter capital requirements may constrain UBS's ability to deploy capital efficiently, increasing 32.7745 31.1758 29.655 33.3107 -4.878 -9.5181 1.6359 True True False
197 1557 UBS UBS 2025-06-06 negative low short Announcement of higher capital requirements could lead to margin compression concerns, negatively im 32.7745 31.1758 29.655 33.3107 -4.878 -9.5181 1.6359 True True False
198 1393 AVGO Broadcom 2025-06-06 negative medium short Broadcom's stock fell 2% in extended trading after the forecast missed the loftiest expectations, de 244.9453 246.7011 248.5644 272.6165 272.6164 0.7168 1.4775 11.2969 False False False
199 913 META Meta 2025-06-04 positive medium long Securing long-term, stable nuclear power enhances Meta's ability to scale AI infrastructure, improvi 685.8104 685.8105 691.9812 694.1398 711.8981 0.8998 1.2145 3.8039 True True True
200 914 META Meta 2025-06-04 positive high long The 20-year power purchase agreement with Constellation Energy directly supports Meta's growing AI w 685.8104 685.8105 691.9812 694.1398 711.8981 0.8998 1.2145 3.8039 True True True
201 1106 AVGO Broadcom 2025-06-02 negative medium short Dissatisfied partners may drive customers toward competing private cloud solutions, increasing migra 246.711 242.3166 250.0738 274.0781 -1.7812 1.363 11.0927 11.0928 True False False
202 1107 AVGO Broadcom 2025-06-02 negative medium medium Reducing the number of channel partners, especially long-standing ones, could erode VMware's market 246.711 242.3166 250.0738 274.0781 -1.7812 1.363 11.0927 11.0928 True False False
203 1196 NVDA NVIDIA 2025-05-29 positive high short Nvidia's revenue surpassing $44bn despite China sales restrictions indicates robust global demand fo 139.1572 139.957 144.9759 154.9942 0.5748 4.1814 11.3807 True True True
204 1197 NVDA NVIDIA 2025-05-29 positive medium medium Ability to achieve record revenue under geopolitical constraints reinforces Nvidia's competitive adv 139.1572 139.957 144.9759 154.9942 0.5748 4.1814 11.3807 True True True
205 1198 NVDA NVIDIA 2025-05-29 positive medium short Strong revenue performance despite headwinds aligns with analyst sentiment supporting stock valuatio 139.1572 139.957 144.9759 154.9942 0.5748 4.1814 11.3807 True True True
206 1203 LHX L3Harris Technologies 2025-05-29 positive medium short Technical breakout above $232 resistance and inclusion in a model portfolio suggest near-term upward 239.3204 239.1635 247.3939 243.8468 -0.0655 3.3735 1.8914 False True True
207 1503 META Meta 2025-05-29 positive medium medium One billion monthly users of Meta AI strengthens Meta's position in the generative AI race, though i 643.0439 682.4907 691.2036 724.3888 724.3887 6.1344 7.4893 12.65 True True True
208 1504 META Meta 2025-05-29 positive low short High user engagement with Meta AI may lead to incremental gains in AI assistant market share, but Go 643.0439 682.4907 691.2036 724.3888 724.3887 6.1344 7.4893 12.65 True True True
209 1271 NVS Novartis 2025-05-29 positive medium short As a competitor in the metabolic and pharmaceutical space, Novartis may benefit from Novo Nordisk's 109.2546 114.3108 117.2027 116.766 4.6278 7.2748 6.8751 True True True
210 1272 NVS Novartis 2025-05-29 positive low short While not directly mentioned, the known positive impact of production investment and divestment on N 109.2546 114.3108 117.2027 116.766 4.6278 7.2748 6.8751 True True True
211 1273 META Meta 2025-05-29 positive medium medium Increased AI integration in advertising and user content, combined with high advertiser adoption, po 643.0439 682.4907 691.2036 724.3888 724.3887 6.1344 7.4893 12.65 True True True
212 1274 META Meta 2025-05-29 positive high medium Meta's massive AI investment, global user base, and early lead in AI-powered advertising tools stren 643.0439 682.4907 691.2036 724.3888 724.3887 6.1344 7.4893 12.65 True True True
213 1199 META Meta 2025-05-25 positive medium medium Expanding AI training with user data may improve Meta AI and Llama models, enhancing competitiveness 640.3224 640.3223 645.4763 695.5401 680.7512 0.8049 8.6234 6.3138 True True True
214 1200 META Meta 2025-05-25 negative high short Noyb's challenge and opt-out model may lead to GDPR enforcement actions, increasing regulatory and l 640.3224 640.3223 645.4763 695.5401 680.7512 0.8049 8.6234 6.3138 False False False
215 1201 META Meta 2025-05-25 negative medium short Public backlash over data usage for AI without explicit consent could harm user trust, especially in 640.3224 640.3223 645.4763 695.5401 680.7512 0.8049 8.6234 6.3138 False False False
216 1202 META Meta 2025-05-25 negative low short Short-term investor concerns may arise from increased regulatory scrutiny and reputational risk tied 640.3224 640.3223 645.4763 695.5401 680.7512 0.8049 8.6234 6.3138 False False False
217 1259 TSLA Tesla 2025-05-23 negative medium short Being outsold in Europe suggests Tesla is losing market share to a strong competitor in a key region 339.34 346.46 295.14 322.16 2.0982 -13.0253 -5.0628 False True True
218 1260 TSLA Tesla 2025-05-23 negative medium short A competitor labeled as a 'Tesla-killer' gaining sales leadership in Europe undermines Tesla's compe 339.34 346.46 295.14 322.16 2.0982 -13.0253 -5.0628 False True True
219 1261 TSLA Tesla 2025-05-23 negative low short Missing sales expectations in a major market can negatively impact investor sentiment, especially am 339.34 346.46 295.14 322.16 2.0982 -13.0253 -5.0628 False True True
220 1464 SHOP Shopify 2025-05-23 positive medium short The launch of the AI Store Builder enhances Shopify's product offering, differentiating its platform 101.51 107.22 111.41 106.4 5.6251 9.7527 4.8173 True True True
221 1189 WBD Warner Bros Discovery 2025-05-14 positive medium medium By refocusing on HBO's premium brand and distinct adult-oriented content, WBD aims to carve out a cl 9.21 8.95 10.02 10.51 -2.823 8.7948 14.1151 False True True
222 1282 META Meta 2025-05-08 negative medium short Public exposure of widespread scam activity leveraging Meta's platforms may weaken user trust and in 596.1501 596.1502 641.8775 634.5903 634.5902 682.4907 7.6704 6.4481 6.448 14.483 False False False
223 1283 META Meta 2025-05-08 negative low short While the takedown demonstrates proactive moderation, the underlying prevalence of sophisticated sca 596.1501 596.1502 641.8775 634.5903 634.5902 682.4907 7.6704 6.4481 6.448 14.483 False False False
224 1473 V Visa 2025-05-08 positive high long By enabling AI agents to use its payment network, Visa positions itself as a foundational player in 348.6606 360.2059 355.9009 364.6501 3.3113 2.0766 4.586 True True True
225 1474 V Visa 2025-05-08 positive medium medium Opening its network to AI developers and expanding in key markets like Europe by 2025 could increase 348.6606 360.2059 355.9009 364.6501 3.3113 2.0766 4.586 True True True
226 1475 V Visa 2025-05-08 positive low short The announcement of a forward-looking strategic initiative may generate investor interest, though im 348.6606 360.2059 355.9009 364.6501 3.3113 2.0766 4.586 True True True
235 1022 SPOT Spotify 2025-05-04 positive low long While Backstage is gaining traction in the internal developer portal space, its market share impact 637.65 648.25 656.3 665.14 1.6624 2.9248 4.3111 True True True
236 1067 AMZN Amazon 2025-05-02 positive medium short Amazon is positioned to gain market share during tariff-related uncertainty, as it did during the pa 189.98 193.06 205.59 205.01 1.6212 8.2167 7.9114 True True True
237 1068 AMZN Amazon 2025-05-02 positive medium short Diverse seller base and proactive inventory management reduce the risk of price increases, strengthe 189.98 193.06 205.59 205.01 1.6212 8.2167 7.9114 True True True
238 1375 META Meta 2025-05-02 positive medium medium Launching a premium AI service positions Meta to compete more effectively with OpenAI, Google, and M 595.1632 590.6473 638.3485 638.3484 645.4763 -0.7588 7.256 8.4537 False True True
239 1376 META Meta 2025-05-02 positive medium medium With nearly a billion users already on Meta AI and a potential premium tier offering enhanced featur 595.1632 590.6473 638.3485 638.3484 645.4763 -0.7588 7.256 8.4537 False True True
240 1037 META Meta 2025-04-29 positive medium short Launching a standalone AI app enhances Meta's visibility and positioning in the competitive AI assis 552.7156 585.4835 585.4834 653.9897 640.3223 5.9285 18.323 15.8502 True True True
241 1038 META Meta 2025-04-29 positive low medium The app leverages Meta's extensive user data for personalization, which could attract users over tim 552.7156 585.4835 585.4834 653.9897 640.3223 5.9285 18.323 15.8502 True True True
242 1039 PL Planet Labs 2025-04-29 negative medium medium The $20M funding enables Near Space Labs to scale its stratospheric imaging operations, increasing c 3.43 3.5 3.78 3.97 2.0408 10.2041 15.7434 False False False
243 1291 META Meta 2025-04-26 negative medium short High-profile public protest by grieving parents increases reputational damage and litigation risk, w 548.0302 595.1632 590.6473 625.1098 625.1097 8.6004 7.7764 14.0648 False False False
244 1292 META Meta 2025-04-26 negative medium medium Growing civil society and regulatory pressure could force Meta to implement stricter safety measures 548.0302 595.1632 590.6473 625.1098 625.1097 8.6004 7.7764 14.0648 False False False
245 1293 META Meta 2025-04-26 positive high short The protest is a direct indicator of escalating civil society pressure, which is likely to attract f 548.0302 595.1632 590.6473 625.1098 625.1097 8.6004 7.7764 14.0648 True True True
246 1362 GOOGL Alphabet 2025-04-24 positive medium short Alphabet exceeded earnings expectations and stock jumped over 7% in after-hours trading despite macr 158.7296 160.7426 153.7469 170.2796 1.2682 -3.1391 7.2765 True False True
247 1047 NFLX Netflix 2025-04-23 positive medium long Articulation of a $1 trillion market cap goal by co-CEO suggests strong long-term confidence, which 104.959 113.172 115.541 119.463 7.825 10.082 13.8187 True True True
248 1048 NFLX Netflix 2025-04-23 positive medium long Ambitious growth targets and diversification into new ventures like theater and retail may strengthe 104.959 113.172 115.541 119.463 7.825 10.082 13.8187 True True True
249 1052 META Meta 2025-04-23 positive medium medium Expanding ads globally increases Threads' competitiveness against X, leveraging high user engagement 518.652 547.2925 594.9539 633.5235 633.5236 5.5221 14.7116 22.1481 True True True
250 1053 META Meta 2025-04-23 positive high short Opening ad inventory to global advertisers directly expands monetization opportunities across a user 518.652 547.2925 594.9539 633.5235 633.5236 5.5221 14.7116 22.1481 True True True
251 1054 META Meta 2025-04-23 positive medium medium Meta positions Threads as more advertiser-friendly than X, using Instagram's network effects to stre 518.652 547.2925 594.9539 633.5235 633.5236 5.5221 14.7116 22.1481 True True True
252 1045 META Meta 2025-04-20 negative medium long Ongoing struggles with Facebook's cultural relevance may weaken Meta's competitive position over tim 483.1527 545.568 595.1632 638.3485 638.3484 12.9183 23.1833 32.1215 False False False
253 1046 META Meta 2025-04-20 negative medium long Declining cultural relevance of Facebook could lead to erosion in user engagement and time spent on 483.1527 545.568 595.1632 638.3485 638.3484 12.9183 23.1833 32.1215 False False False
254 1040 META Meta 2025-04-17 negative medium short TikTok's continued dominance in short-form video has already caused a dramatic slowdown in Meta's gr 499.9204 531.4919 570.4304 641.8775 6.3153 14.1043 28.3959 False False False
255 1041 META Meta 2025-04-17 negative medium short Zuckerberg's admission that TikTok directly slowed Meta's growth implies a loss of user engagement a 499.9204 531.4919 570.4304 641.8775 6.3153 14.1043 28.3959 False False False
256 1058 RIVN Rivian 2025-04-16 positive medium medium HelloFresh’s adoption of 70 Rivian vans marks the first major commercial customer beyond Amazon, sig 11.49 11.8 13.66 14.82 2.698 18.886 28.9817 True True True
261 1207 NVS Novartis 2025-04-11 positive medium short The announcement of a major investment in U.S. production facilities led to a 1.6% increase in Novar 104.3441 107.2749 108.8276 105.4892 2.8088 4.2969 1.0975 True True True
262 1372 HF Hugging Face 2025-04-11 positive medium medium Increased public legal conflict between OpenAI and Elon Musk may divert focus and resources from Ope 19.9983 20.0924 19.9884 19.9785 0.4707 -0.0495 -0.0991 True False False
263 1108 SHOP Shopify 2025-04-10 negative medium short Growing traction of TikTok Shop in social commerce, especially among Gen Z and legacy brands, may er 84.63 83.65 95.12 94.0 -1.158 12.3951 11.0717 True False False
264 1478 META Meta 2025-04-10 negative medium short Whistleblower testimony before Congress alleging collusion with the Chinese government on censorship 544.591 499.9204 531.4919 596.1501 596.1502 -8.2026 -2.4053 9.4675 True True False
265 1479 META Meta 2025-04-10 negative medium medium Allegations of aiding Chinese AI development via Llama could damage partnerships, restrict future AI 544.591 499.9204 531.4919 596.1501 596.1502 -8.2026 -2.4053 9.4675 True True False
266 1480 META Meta 2025-04-10 positive high short Senate testimony alleging Meta’s cooperation with the Chinese Communist Party on censorship and data 544.591 499.9204 531.4919 596.1501 596.1502 -8.2026 -2.4053 9.4675 False False True
267 1373 META Meta 2025-04-10 negative medium short Allegations of collaboration with the Chinese Communist Party on censorship tools, aired during a U. 544.591 499.9204 531.4919 596.1501 596.1502 -8.2026 -2.4053 9.4675 True True False
268 1374 META Meta 2025-04-10 negative medium medium Increased reputational damage and regulatory scrutiny from U.S. lawmakers may weaken Meta's public t 544.591 499.9204 531.4919 596.1501 596.1502 -8.2026 -2.4053 9.4675 True True False
269 1175 ORCL Oracle 2025-03-31 negative medium short Public criticism over handling of security incidents, particularly involving sensitive patient data, 137.9258 125.4463 133.3026 138.7479 138.748 -9.048 -3.3519 0.5961 True True False
270 1176 ORCL Oracle 2025-03-31 negative medium medium Security concerns, especially in healthcare and legacy infrastructure, may weaken trust in Oracle's 137.9258 125.4463 133.3026 138.7479 138.748 -9.048 -3.3519 0.5961 True True False
271 1481 TSLA Tesla 2025-03-27 positive medium short Tesla's 100% US production insulates it from the 25% import tariffs that will burden competitors lik 273.13 267.28 252.4 259.51 -2.1418 -7.5898 -4.9866 False False False
272 1482 TSLA Tesla 2025-03-27 positive low medium With competitors facing higher costs due to tariffs, Tesla may gain slight market share in the US, t 273.13 267.28 252.4 259.51 -2.1418 -7.5898 -4.9866 False False False
273 1554 RIVN Rivian 2025-03-26 positive medium medium By expanding into micromobility through Also, Rivian strengthens its technological brand and diversi 12.1 12.49 11.77 11.8 3.2231 -2.7273 -2.4793 True False False
274 1494 TSM TSMC 2025-03-25 negative medium medium The substantial capital expenditure with expected margin pressure from U.S. operations may weigh on 178.6869 166.5769 139.6405 149.5478 -6.7772 -21.8518 -21.8519 -16.3074 -16.3073 True True True
275 1181 META Meta 2025-03-24 negative medium short Meta's failed acquisition of FuriosaAI, an AI chip startup developing competitive chips for reasonin 616.9253 616.9254 574.5675 574.5674 514.6444 483.1527 483.1526 -6.866 -16.5791 -21.6838 True True True
276 1182 META Meta 2025-03-24 negative low medium While FuriosaAI remains independent and may expand its partnerships (e.g., with LG AI Research), Met 616.9253 616.9254 574.5675 574.5674 514.6444 483.1527 483.1526 -6.866 -16.5791 -21.6838 True True True
277 1183 FTNT Fortinet 2025-03-17 negative medium short The public disclosure of active exploitation of Fortinet vulnerabilities by a LockBit-linked group m 96.67 99.79 96.26 96.85 3.2275 -0.4241 0.1862 False True False
278 1184 FTNT Fortinet 2025-03-17 negative medium medium Repeated security breaches linked to Fortinet products could weaken its market standing versus compe 96.67 99.79 96.26 96.85 3.2275 -0.4241 0.1862 False True False
279 1359 RIVN Rivian 2025-03-05 positive high long Providing core software and architecture to a major automaker like Volkswagen enhances Rivian’s stra 11.42 11.06 11.36 12.49 -3.1524 -0.5254 9.3695 False False True
280 1360 RIVN Rivian 2025-03-05 positive medium short The influx of capital and validation of Rivian’s technology by Volkswagen may boost investor confide 11.42 11.06 11.36 12.49 -3.1524 -0.5254 9.3695 False False True
281 1354 META Meta 2025-03-05 positive medium medium Expanding facial recognition tools in regulated markets like the UK and EU strengthens Meta's positi 653.8467 653.8466 617.084 617.0841 582.2435 582.1139 582.114 -5.6225 -10.9511 -10.9709 False False False
282 1355 META Meta 2025-03-05 positive low medium Optional anti-fraud and verification tools may improve user trust and retention, particularly among 653.8467 653.8466 617.084 617.0841 582.2435 582.1139 582.114 -5.6225 -10.9511 -10.9709 False False False
283 1560 BLK BlackRock 2025-03-02 positive medium short Record inflows suggest investor confidence remains strong despite the ESG reversal, potentially supp 941.8132 927.7991 909.947 927.5837 927.5836 -1.488 -3.3835 -1.5109 False False False
284 1562 BLK BlackRock 2025-03-02 positive medium short By aligning with conservative political forces and avoiding regulatory scrutiny, BlackRock may gain 941.8132 927.7991 909.947 927.5837 927.5836 -1.488 -3.3835 -1.5109 False False False
285 1350 META Meta 2025-02-27 positive low short Terminating leakers may strengthen internal discipline and protect strategic information, slightly i 655.6096 655.6095 625.4207 588.2797 600.706 -4.6047 -10.2698 -8.3744 False False False
286 1351 META Meta 2025-02-27 negative medium short Public disclosure of internal leaks and subsequent firings may amplify perceptions of internal disse 655.6096 655.6095 625.4207 588.2797 600.706 -4.6047 -10.2698 -8.3744 True True True
287 1142 SAP SAP 2025-02-27 negative low short The stock is continuing a downward trend, underperforming the Dax, and trading volume has decreased, 272.1395 276.837 252.9034 265.7473 1.7261 -7.0685 -2.3489 False True True
288 1114 PLTR Palantir 2025-02-08 positive high short Retail investor enthusiasm, dollar-cost averaging, and a significant rise in stock price in 2024 and 116.65 119.16 101.35 84.91 2.1517 -13.1162 -27.2096 True False False
289 1065 AMD AMD 2025-02-04 positive medium medium Powering 100 million game consoles indicates strong market penetration in the gaming segment, likely 119.5 111.1 114.28 100.75 -7.0293 -4.3682 -15.6904 False False False
290 1066 AMD AMD 2025-02-04 positive medium short Success in securing design wins across major console platforms strengthens AMD's position against co 119.5 111.1 114.28 100.75 -7.0293 -4.3682 -15.6904 False False False
291 1125 NVDA NVIDIA 2025-01-30 negative medium short Technical indicators suggest further downside toward $110, with momentum shifting negative in the in 124.6092 128.6379 128.6378 135.2457 120.1106 3.2331 3.233 8.5359 -3.6101 False False True
292 1244 NVDA NVIDIA 2025-01-30 positive medium short The stock rebounded nearly 9% following a sharp decline, indicating strong investor resilience and c 124.6092 128.6379 128.6378 135.2457 120.1106 3.2331 3.233 8.5359 -3.6101 True True False
293 1243 MS Morgan Stanley 2025-01-24 positive medium short Morgan Stanley is highlighted as one of the top stocks to buy for strong Q4 earnings, with positive 133.331 132.6183 134.8122 134.0916 136.3217 135.593 128.2484 127.5628 1.1109 2.2431 -3.812 True True False
294 1248 STX Seagate 2025-01-23 positive medium short Solid 2Q25 performance driven by cloud sector recovery and increasing AI storage demand support upwa 106.1709 96.2413 94.5374 100.5108 -9.3525 -10.9574 -5.3311 False False False
295 1130 META Meta 2025-01-21 positive low short Enhanced cross-platform integration improves user engagement and data sharing across Meta's apps, sl 613.9966 671.6353 701.3759 713.5073 9.3875 14.2312 16.207 16.2071 True True True
296 1466 TSM TSMC 2025-01-16 positive high short Strong profit growth driven by high AI chip demand supports a positive short-term stock price moveme 211.3388 221.0109 204.8055 198.5871 4.5766 -3.0914 -6.0338 True False False
297 1467 TSM TSMC 2025-01-16 positive medium medium Continued leadership in advanced semiconductor manufacturing and strong customer relationships reinf 211.3388 221.0109 204.8055 198.5871 4.5766 -3.0914 -6.0338 True False False
298 1468 TSM TSMC 2025-01-16 negative medium medium Geopolitical uncertainties and export restrictions may negatively affect future business operations 211.3388 221.0109 204.8055 198.5871 4.5766 -3.0914 -6.0338 False True True
299 1312 META Meta 2025-01-10 negative high short Widespread condemnation from 71 fact-checking organizations, including a public open letter, signals 613.3989 613.3988 610.3214 610.3212 644.9025 711.6646 -0.5017 5.1359 16.0199 True False False
300 1313 META Meta 2025-01-10 positive medium medium Strong pushback from civil society groups may prompt increased scrutiny from regulators concerned wi 613.3989 613.3988 610.3214 610.3212 644.9025 711.6646 -0.5017 5.1359 16.0199 False True True
301 1159 NVDA NVIDIA 2025-01-07 positive medium short Analyst reaffirmation of Nvidia as a top pick following a major executive keynote typically boosts i 140.0941 131.7168 140.7839 118.6111 -5.9797 0.4924 -15.3347 False True False
302 1160 NVDA NVIDIA 2025-01-07 positive low medium Positive analyst sentiment following a strategic keynote may reinforce market perception of Nvidia's 140.0941 131.7168 140.7839 118.6111 -5.9797 0.4924 -15.3347 False True False
303 1157 RIVN Rivian 2025-01-03 positive high short Rivian's stock surged 24.5%, its largest daily increase since going public, after meeting revised pr 16.49 13.85 14.21 12.56 -16.0097 -13.8266 -23.8326 False False False
304 1158 RIVN Rivian 2025-01-03 positive medium medium Resolving production constraints and delivering above analyst expectations strengthens Rivian's posi 16.49 13.85 14.21 12.56 -16.0097 -13.8266 -23.8326 False False False
305 1162 AVGO Broadcom 2024-12-31 positive high short Broadcom's stock price rose 111% in 2024 due to strong AI-related demand and market outperformance r 229.2828 226.1181 222.2215 222.2216 205.0728 -1.3803 -3.0797 -10.559 False False False
306 1163 AVGO Broadcom 2024-12-31 positive medium medium Broadcom is gaining ground in the AI chip and networking space despite Nvidia's dominance, positioni 229.2828 226.1181 222.2215 222.2216 205.0728 -1.3803 -3.0797 -10.559 False False False
307 1161 AAPL Apple 2024-12-26 positive high medium Apple's stock soars to a record high, and JPMorgan has a positive outlook for the company in 2025, s 257.6127 242.5252 235.5632 222.4449 -5.8567 -8.5592 -13.6515 False False False
308 1317 LLY Eli Lilly 2024-12-23 positive medium short Eli Lilly's potential to extend a winning streak over the broad market for 6 years suggests continue 789.0677 766.8309 758.17 758.1701 735.6262 -2.8181 -3.9157 -6.7727 False False False
309 1427 SMCI Super Micro Computer 2024-12-16 negative medium short Removal from Nasdaq 100 triggers passive fund selling, combined with ongoing governance and complian 33.44 32.4 30.68 31.08 -3.11 -8.2536 -7.0574 True True True
310 1428 SMCI Super Micro Computer 2024-12-16 negative low medium Reduced market visibility and investor confidence may impair access to capital and strategic partner 33.44 32.4 30.68 31.08 -3.11 -8.2536 -7.0574 True True True
311 1254 META Meta 2024-12-14 positive medium medium By challenging OpenAI’s for-profit transition, Meta aims to constrain a key AI competitor’s flexibil 621.7454 582.9113 597.4131 613.3989 613.3988 -6.246 -3.9135 -3.9136 -1.3424 False False False
312 1255 META Meta 2024-12-14 positive high short Meta’s public call for regulatory intervention increases scrutiny on OpenAI, amplifying broader regu 621.7454 582.9113 597.4131 613.3989 613.3988 -6.246 -3.9135 -3.9136 -1.3424 False False False
313 1424 AAPL Apple 2024-12-13 positive medium medium Morgan Stanley naming Apple a top pick for 2025 signals strong institutional confidence, supporting 246.7819 253.1074 253.1073 254.2014 235.5632 2.5632 3.0065 -4.546 True True False
314 1441 SPOT Spotify 2024-12-11 positive medium short Public discussion around faking Spotify Wrapped indicates high user interest and emotional investmen 476.91 448.65 457.98 479.73 -5.9256 -3.9693 0.5913 False False True
315 1439 RIVN Rivian 2024-12-09 positive medium medium Rivian's superior charging experience and renewable energy partnerships enhance its differentiation 14.45 15.34 13.75 15.715 6.1592 -4.8443 8.7543 True False True
316 1440 RIVN Rivian 2024-12-09 positive low long Expanded charging infrastructure accessible to all EV users increases Rivian's visibility and brand 14.45 15.34 13.75 15.715 6.1592 -4.8443 8.7543 True False True
317 1431 CVX Chevron 2024-12-08 positive medium short Goldman Sachs reiterated a buy rating with a raised price target, citing strong shareholder returns 148.6666 145.6285 135.1987 139.9309 139.931 -2.0435 -9.0591 -5.876 False False False
318 1432 CVX Chevron 2024-12-08 positive medium medium Recognition by top Wall Street analysts as a top dividend stock enhances Chevron's profile among ene 148.6666 145.6285 135.1987 139.9309 139.931 -2.0435 -9.0591 -5.876 False False False
319 1429 SHOP Shopify 2024-12-06 positive medium short Analyst upgrade typically leads to improved market sentiment and short-term stock price momentum, es 118.37 114.63 108.95 109.25 -3.1596 -7.9581 -7.7047 False False False
320 1430 SHOP Shopify 2024-12-06 positive low medium Increased recognition of AI capabilities may enhance Shopify's positioning against competitors over 118.37 114.63 108.95 109.25 -3.1596 -7.9581 -7.7047 False False False
321 1437 AMD AMD 2024-12-04 positive medium short The article suggests a potential catch-up trade based on technical charts, indicating upward momentu 143.99 130.15 121.41 120.63 -9.6118 -15.6816 -16.2234 False False False
324 1433 NOW ServiceNow 2024-12-01 positive high medium Analyst upgraded price target due to strong financials, AI tailwinds, and confidence in near- and me 209.686 224.868 224.22 216.292 7.2403 6.9313 3.1504 True True True
325 1434 NOW ServiceNow 2024-12-01 positive medium long New Workflow Data Fabric product expected to power new workflows and AI agents, enhancing differenti 209.686 224.868 224.22 216.292 7.2403 6.9313 3.1504 True True True
326 1435 NOW ServiceNow 2024-12-01 positive high long Product innovation expected to double total addressable market to $500 billion, enabling greater mar 209.686 224.868 224.22 216.292 7.2403 6.9313 3.1504 True True True
327 1442 META Meta 2024-11-29 positive high long By building a private, globally spanning subsea cable, Meta gains greater control over data transmis 571.5638 620.7766 617.373 617.3729 597.4131 8.6102 8.0147 4.5226 4.5225 True True True
328 1443 META Meta 2024-11-29 negative medium short The $10 billion upfront investment may raise investor concerns about near-term profitability, especi 571.5638 620.7766 617.373 617.3729 597.4131 8.6102 8.0147 4.5226 4.5225 False False False
329 1444 META Meta 2024-11-29 positive medium long Exclusive control over high-capacity global data infrastructure will enable Meta to scale AI-driven 571.5638 620.7766 617.373 617.3729 597.4131 8.6102 8.0147 4.5226 4.5225 True True True
330 1446 META Meta 2024-11-21 positive low medium Proactive measures against scams may improve platform trustworthiness, slightly enhancing Meta's com 560.3878 571.5639 571.564 606.0079 606.0078 593.1899 593.19 1.9943 1.9944 8.1408 5.8535 True True True
331 1453 CMCSA Comcast 2024-11-20 positive medium short A spin-off of the cable business could unlock shareholder value and streamline operations, potential 37.9328 37.5534 37.5446 33.4063 -1.0002 -1.0235 -11.933 False False False
332 1454 CMCSA Comcast 2024-11-20 positive medium long Separating the cable business may allow Comcast to focus on growth areas like streaming and broadban 37.9328 37.5534 37.5446 33.4063 -1.0002 -1.0235 -11.933 False False False
333 1448 META Meta 2024-11-14 negative medium short The $840 million fine represents a significant financial penalty and reinforces investor concerns ab 574.3903 574.3902 560.3878 571.5639 571.564 627.7628 627.7629 -2.4378 -0.4921 -0.492 9.292 9.2921 True True False
334 1449 META Meta 2024-11-14 negative medium medium The EU ruling may force Meta to alter how Marketplace is integrated into Facebook, potentially reduc 574.3903 574.3902 560.3878 571.5639 571.564 627.7628 627.7629 -2.4378 -0.4921 -0.492 9.292 9.2921 True True False
335 1450 META Meta 2024-11-14 positive high long This fine adds to a pattern of EU enforcement actions, signaling sustained and increasing regulatory 574.3903 574.3902 560.3878 571.5639 571.564 627.7628 627.7629 -2.4378 -0.4921 -0.492 9.292 9.2921 False False True
336 1527 TSLA Tesla 2024-11-08 positive high short Tesla's stock surged 29% in one week following Trump's election, driven by investor optimism over re 321.22 320.72 352.56 389.22 -0.1557 9.7566 21.1693 False True True
337 1528 TSLA Tesla 2024-11-08 positive medium medium Potential higher tariffs on Chinese EVs like BYD could reduce competitive pressure in the U.S. marke 321.22 320.72 352.56 389.22 -0.1557 9.7566 21.1693 False True True
338 1552 RIVN Rivian 2024-10-17 positive low short The novelty of themed updates may attract media attention and increase short-term consumer interest, 10.12 10.43 10.1 10.31 3.0632 -0.1976 1.8775 True False True
docker-compose.yml
+275
View File
@@ -1,4 +1,60 @@
services:
postgres:
image: pgvector/pgvector:pg16
environment:
POSTGRES_DB: "${POSTGRES_DB:-duriin}"
POSTGRES_USER: "${POSTGRES_USER:-duriin}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env before starting PostgreSQL}"
command:
- postgres
- -c
- shared_buffers=128MB
- -c
- work_mem=4MB
- -c
- maintenance_work_mem=64MB
- -c
- effective_cache_size=256MB
- -c
- max_connections=40
volumes:
- postgres_data:/var/lib/postgresql/data
mem_limit: 512m
cpus: "0.50"
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
interval: 10s
timeout: 5s
retries: 12
networks:
- nginx_proxy_manager_default
postgres-migrate:
profiles: [migration]
build:
context: .
provenance: false
command: node scripts/migrate-sqlite-to-postgres.js
env_file: .env
volumes:
- ./config.json:/app/config.json:ro
- ./data:/data:ro
environment:
DATABASE_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
SQLITE_ARCHIVE_PATH: /data/archive.sqlite
SQLITE_INTELLIGENCE_PATH: /data/intelligence.sqlite
depends_on:
postgres:
condition: service_healthy
networks:
- nginx_proxy_manager_default
# DB backend is one switch for the whole stack and it defaults to sqlite on purpose.
# the postgres copy is a stale snapshot (predictions stop around 2026-08-17) and the
# migrate script only appends, it never replays UPDATEs, so a `compose up` must not
# quietly flip us over. set DURIIN_DB_BACKEND=postgres + DURIIN_USE_POSTGRES=true in
# .env only after a fresh intelligence migration has been run and verifyed.
api:
build:
context: .
@@ -10,11 +66,73 @@ services:
environment:
NODE_ENV: production
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
DURIIN_RUN_SCHEDULER: "false"
AUTONOMY_EXECUTION_MODE: "${AUTONOMY_EXECUTION_MODE:-shadow}"
depends_on:
postgres:
condition: service_healthy
restart: unless-stopped
networks:
- nginx_proxy_manager_default
# same image as api, but this one actually runs the cron scheduler
# (rss / gdelt / edgar / alphavantage / finnhub). it also boots fastify on
# 3001 but nothing proxies to it, so its effectivly ingestion only.
ingest:
build:
context: .
provenance: false
env_file: .env
volumes:
- ./config.json:/app/config.json:ro
- ./data:/data
environment:
NODE_ENV: production
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
DURIIN_RUN_SCHEDULER: "true"
AUTONOMY_EXECUTION_MODE: "${AUTONOMY_EXECUTION_MODE:-shadow}"
depends_on:
postgres:
condition: service_healthy
restart: unless-stopped
networks:
- nginx_proxy_manager_default
# enrichment chain: queue feeder -> augor -> consolidation -> graph -> signal -> outcome.
# this is what fills event_id / content / has_embedding, which the autonomy
# reconcilers require before they enqueue anything.
enrichment:
build:
context: .
provenance: false
command: node workers/index.js
env_file: .env
volumes:
- ./config.json:/app/config.json:ro
- ./data:/data
environment:
NODE_ENV: production
DURIIN_DB: /data/archive.sqlite
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
depends_on:
postgres:
condition: service_healthy
restart: unless-stopped
networks:
- nginx_proxy_manager_default
intelligence:
# superseded by the "enrichment" service above, kept behind a profile
profiles: [legacy]
build:
context: .
provenance: false
@@ -31,6 +149,163 @@ services:
networks:
- nginx_proxy_manager_default
autonomy:
build:
context: .
provenance: false
command: node workers/autonomy-entrypoint.js
env_file: .env
volumes:
- ./config.json:/app/config.json:ro
- ./data:/data
environment:
NODE_ENV: production
DURIIN_DB: /data/archive.sqlite
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
AUTONOMY_POLL_MS: "${AUTONOMY_POLL_MS:-5000}"
depends_on:
postgres:
condition: service_healthy
cpus: "0.50"
mem_limit: 512m
restart: unless-stopped
networks:
- nginx_proxy_manager_default
coordinator:
build:
context: .
provenance: false
command: node workers/coordinator-entrypoint.js
env_file: .env
volumes:
- ./config.json:/app/config.json:ro
- ./data:/data
environment:
NODE_ENV: production
DURIIN_DB: /data/archive.sqlite
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
AUTONOMY_POLL_MS: "${AUTONOMY_COORDINATOR_POLL_MS:-5000}"
depends_on:
postgres:
condition: service_healthy
cpus: "0.75"
mem_limit: 768m
restart: unless-stopped
networks:
- nginx_proxy_manager_default
autonomy-outcomes:
build:
context: .
provenance: false
command: node workers/outcome-autonomy-entrypoint.js
env_file: .env
volumes:
- ./data:/data
environment:
NODE_ENV: production
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
AUTONOMY_OUTCOME_POLL_MS: "${AUTONOMY_OUTCOME_POLL_MS:-60000}"
depends_on:
postgres:
condition: service_healthy
cpus: "0.25"
mem_limit: 384m
restart: unless-stopped
networks:
- nginx_proxy_manager_default
replay:
build:
context: .
provenance: false
command: node workers/replay-entrypoint.js
env_file: .env
volumes:
- ./config.json:/app/config.json:ro
- ./data:/data
environment:
NODE_ENV: production
DURIIN_DB: /data/archive.sqlite
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
AUTONOMY_REPLAY_POLL_MS: "${AUTONOMY_REPLAY_POLL_MS:-15000}"
AUTONOMY_REPLAY_DAILY_BUDGET: "${AUTONOMY_REPLAY_DAILY_BUDGET:-100}"
AUTONOMY_REPLAY_WATERMARK_DAYS: "${AUTONOMY_REPLAY_WATERMARK_DAYS:-7}"
depends_on:
postgres:
condition: service_healthy
cpus: "0.50"
mem_limit: 768m
restart: unless-stopped
networks:
- nginx_proxy_manager_default
calibration:
build:
context: .
provenance: false
command: node workers/calibration-entrypoint.js
env_file: .env
volumes:
- ./data:/data
environment:
NODE_ENV: production
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
AUTONOMY_CALIBRATION_POLL_MS: "${AUTONOMY_CALIBRATION_POLL_MS:-60000}"
depends_on:
postgres:
condition: service_healthy
cpus: "0.25"
mem_limit: 384m
restart: unless-stopped
networks:
- nginx_proxy_manager_default
execution:
build:
context: .
provenance: false
command: node workers/execution-entrypoint.js
env_file: .env
volumes:
- ./data:/data
environment:
NODE_ENV: production
INTELLIGENCE_DB: /data/intelligence.sqlite
DURIIN_DB_BACKEND: "${DURIIN_DB_BACKEND:-sqlite}"
DURIIN_USE_POSTGRES: "${DURIIN_USE_POSTGRES:-false}"
DURIIN_POSTGRES_URL: "postgresql://${POSTGRES_USER:-duriin}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-duriin}"
AUTONOMY_EXECUTION_MODE: "${AUTONOMY_EXECUTION_MODE:-shadow}"
AUTONOMY_DEFAULT_NOTIONAL: "${AUTONOMY_DEFAULT_NOTIONAL:-100}"
AUTONOMY_EXECUTION_POLL_MS: "${AUTONOMY_EXECUTION_POLL_MS:-10000}"
depends_on:
postgres:
condition: service_healthy
cpus: "0.25"
mem_limit: 384m
restart: unless-stopped
networks:
- nginx_proxy_manager_default
networks:
nginx_proxy_manager_default:
external: true
volumes:
postgres_data:
docs/autonomy.md
+46
View File
@@ -0,0 +1,46 @@
# Duriin autonomy runtime
The autonomy runtime is additive to the existing archive and intelligence
tables. Its workers are part of the default runtime and restart automatically.
## Services
- `api`: HTTP only; no background scheduler when `DURIIN_RUN_SCHEDULER=false`.
- `autonomy`: bounded archive reconciliation and live/historical job creation.
- `coordinator`: LLM proposal extraction only. It cannot create an order.
- `autonomy-outcomes`: resolves matured predictions against market and benchmark returns.
- `calibration`: creates empirical calibration snapshots and deterministic decisions.
- `intelligence`: legacy worker, available only under the `legacy` Compose profile.
Start the API and autonomy runtime:
```bash
docker compose up -d
```
## Authority rules
The coordinator may return categorical, evidence-backed proposals. It may not
return probabilities, returns, position sizes or trade actions. Proposals must
reference existing archive article IDs and instruments in
`autonomy_instruments` with `active=1` and `tradable=1`.
Calibration is computed from resolved market-relative outcomes. The policy
engine emits `BUY`, `SELL`, `HOLD` or `ABSTAIN`; an LLM is not in this path.
Paper order intents use deterministic client IDs and are bounded by the
execution validator. A broker adapter must be explicitly configured; the
included simulator is the default test adapter.
## Database initialization
The autonomy schema is initialized additively by the autonomy workers and
maintenance scripts in the intelligence database. To create the initial
reconciliation job without starting workers:
```bash
npm run autonomy:init
```
Legacy predictions remain legacy records and are not silently included in
calibration. New prospective predictions are the trusted learning set.
docs/postgres-migration.md
+23
View File
@@ -0,0 +1,23 @@
# PostgreSQL migration
The Compose stack includes PostgreSQL with a hard 512 MB container limit and
conservative in-server memory settings. Set `POSTGRES_PASSWORD` in the server
`.env`, then start it with `docker compose up -d postgres`.
Run the resumable logical copy with:
```sh
docker compose --profile migration run --rm postgres-migrate
```
It copies archive data to the `archive` schema and autonomy/intelligence data
to the `intelligence` schema. Each table is count-verified and progress is
persisted in each schema's `_migration_progress` table, so restarting resumes
at the last committed batch. SQLite vec0 internal tables are derived indexes;
their canonical embedding data is migrated from `article_embedding_store`.
The application is intentionally not switched by this migration alone: it has
SQLite-specific synchronous query and transaction code in the autonomy workers
and admin routes. Pointing those processes at PostgreSQL before that code is
ported would make the live system unreliable. Keep SQLite mounted until a
separate verified runtime cutover is complete.
docs/replay-run-2-preregistration.md
+98
View File
@@ -0,0 +1,98 @@
# Replay run 2: pre-registration
Written 2026-09-08, before run 2 exists. The numbers below are run 1's, measured
on the evaluation slice only. They are fixed. If the analysis after run 2 uses a
different bar than the one written here, the analysis is wrong, not the bar.
## What is being tested
Whether feeding the generator its own scored results changes what it predicts,
and whether the change is an improvement.
Nothing in the pipeline has ever read `autonomy_outcomes` back into the thing
that makes predictions. Calibration reads outcomes, but calibration only decides
whether to ACT on a prediction, never what the prediction is. So the only thing
that has ever altered this system's output is a human editing the prompt. Run 2
is the first time the system is told about its own mistakes.
## Design
One variable. Run 2 answers exactly the articles run 1 answered under the
current prompt and the current model, with the same model, same prompt, plus a
feedback brief generated by `scripts/build-feedback-brief.js`.
- Split on `created_at >= "2026-09-04 19:17:43"`, when `duriin-api-replay-1`
restarted onto the prompt it still runs today. That is the container start
time, not the commit timestamp, which is five minutes later and would have put
a handful of old-prompt predictions on the new-prompt side. Verified directly:
the running container has the instrument rules, the de-anchored placeholders
and the event_type enum in `/app/workers/replayWorker.js`.
- There is no contamination to argue about. Replay was between daily budgets
across the restart, so no replay prediction exists between 15:57 on 09-04 and
00:00 on 09-05. Splits at 19:17:43, at 19:30 and at midnight all produce the
identical partition, 1,142 training and 984 evaluation.
- The brief is derived from the 1,133 training predictions only. No evaluation
row contributes a single number to the text. Deriving the lesson and grading it
on the same rows would measure nothing.
- Evaluation set: 716 articles, 984 run-1 predictions, all
`~deepseek/deepseek-v4-flash-latest`. 713 of those articles have a scored
run-1 prediction and are the pairable set; the other three are replayed but
cannot enter T4.
- The 1,133 scored training predictions span two models, roughly 627 qwen and
506 deepseek. So the brief describes the mistakes of the system as it has
been, not of deepseek alone. Run 2 is deepseek throughout, as is the run-1
half it is measured against.
## Run 1 on the evaluation slice, the bar
| metric | run 1 |
| --- | --- |
| predictions scored | 981 over 713 articles |
| accuracy | 50.56% |
| always_negative on the same bars | 57.39% |
| edge over the constant | **-6.83 points** |
| signed excess, system | 0.679% |
| signed excess, always_negative | 1.264% |
| discrimination P(up given positive) | 44.35% (n=593) |
| discrimination P(up given negative) | 39.95% (n=388) |
| discrimination spread | **+4.40 points**, z=1.363, p=0.173 |
| share of calls that were positive | 60.45%, against 42.61% of bars up |
## Tests, declared now
- **Primary, T4.** Paired per-article accuracy, run 2 minus run 1, over the
shared articles. Two sided paired t. Per article, not per prediction, so one
article that produced eleven calls does not outvote one that produced a single
call.
- **Secondary, T3.** Discrimination spread. Run 1 is +4.40 points.
- **Absolute, T1.** Run 2 accuracy against always_negative, 57.39%.
## What each outcome means, declared now
The brief tells the model its positive share is 17.8 points too high. Telling a
model the base rate will pull it toward the base rate. So:
- **Accuracy up, discrimination spread flat.** The expected result. This is
calibration, not skill. The system learned its prior was wrong, which is worth
having and is genuinely the loop working, but it is not evidence that it reads
news any better. Do not report it as new skill.
- **Accuracy up AND discrimination spread up, T3 significant.** Genuine
learning. The feedback changed which way it calls things, not just how often.
This is the only result that justifies building the loop into the workers.
- **T4 flat.** The feedback changed nothing. Either the brief is too weak to move
the model or the model cannot use this kind of instruction. Either way the
answer to "should the loop be automated" is no, and the structural
alternatives become the next move.
- **T4 negative.** The feedback made it worse. Report it as such and stop.
Beating run 1 while still sitting below 57.39% is not a system worth trading.
That distinction gets reported every time, not just when it is convenient.
## Housekeeping that is easy to forget
Run 2 stays `running` once it exhausts its 716 articles, and the replay lane
just idles. That is intended, it keeps the spend at zero while the outcomes
mature. Mark it `complete` when the results are read, otherwise it becomes the
same stale metadata run 1 carried for a month. But do not mark it complete
before reading, because `activeRun` would immediately create run 3 with no
pinned set and no brief and start walking all 23k articles again.
package.json
+7 -1
View File
@@ -5,7 +5,13 @@
"main": "server.js",
"scripts": {
"start": "node server.js",
"workers": "node workers/index.js"
"workers": "node workers/index.js",
"test": "node --test test/**/*.test.js",
"autonomy:init": "node scripts/initialize-autonomy.js",
"autonomy:allowlist": "node scripts/set-autonomy-instrument.js",
"autonomy:sync-assets": "node scripts/sync-paper-assets.js",
"autonomy:import-legacy": "node scripts/import-legacy-intelligence.js",
"autonomy:replay": "node workers/replay-entrypoint.js"
},
"keywords": [],
"author": "",
public/admin/app.html
+38
View File
@@ -0,0 +1,38 @@
<!doctype html>
<html lang="en" data-theme="dark">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Duriin Ops</title>
<link rel="stylesheet" href="/admin/assets/css/ops.css?v=20260901-1">
<link rel="preconnect" href="https://cdnjs.cloudflare.com">
<!-- Admin assets are served no-store so an operator can never run stale UI against
a fresh api. That means every module is refetched every load, and left alone
they arrive in a waterfall: main is parsed, then its imports are discovered,
then theirs. Preloading flattens that into one parallel burst. -->
<link rel="modulepreload" href="/admin/assets/js/ops/core.js">
<link rel="modulepreload" href="/admin/assets/js/ops/ui.js">
<link rel="modulepreload" href="/admin/assets/js/ops/overview.js">
<link rel="modulepreload" href="/admin/assets/js/ops/controls.js">
<link rel="modulepreload" href="/admin/assets/js/ops/explore.js">
<link rel="modulepreload" href="/admin/assets/js/ops/sql.js">
</head>
<body>
<div id="root">
<div class="boot">
<div class="boot-mark">DURIIN</div>
<div class="boot-note">loading console…</div>
</div>
</div>
<!-- UMD builds, no bundler and no in-browser transpiler. htm gives us JSX-ish
tagged templates for ~700 bytes, which is the whole reason we can skip Babel
standalone -- that thing compiles on every page load and is exactly the kind
of slow we are trying to get away from. -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/react/18.3.1/umd/react.production.min.js" crossorigin></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react-dom/18.3.1/umd/react-dom.production.min.js" crossorigin></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/htm/3.1.1/htm.js" crossorigin></script>
<script type="module" src="/admin/assets/js/ops/main.js"></script>
</body>
</html>
public/admin/assets/css/autonomy.css
+106
View File
@@ -0,0 +1,106 @@
.page-autonomy { background-image: radial-gradient(circle at 78% 0, rgba(223,255,79,.07), transparent 31rem); }
.autonomy-shell { width: min(1500px, 100%); margin: 0 auto; padding: 38px 38px 70px; }
.autonomy-hero { min-height: 238px; padding: 22px 4px 34px; display: flex; align-items: flex-end; justify-content: space-between; gap: 40px; border-bottom: 1px solid var(--border); }
.hero-copy { max-width: 780px; }
.eyebrow { margin-bottom: 23px; display: flex; align-items: center; gap: 9px; color: var(--accent); font-family: var(--mono); font-size: 9px; font-weight: 700; letter-spacing: .13em; text-transform: uppercase; }
.pulse { width: 7px; height: 7px; border-radius: 50%; background: var(--accent); box-shadow: 0 0 0 0 rgba(223,255,79,.5); animation: pulse 2.4s infinite; }
@keyframes pulse { 60% { box-shadow: 0 0 0 8px rgba(223,255,79,0); } 100% { box-shadow: 0 0 0 0 rgba(223,255,79,0); } }
.hero-copy h2 { max-width: 760px; font-size: clamp(42px, 6vw, 82px); font-weight: 560; letter-spacing: -.068em; line-height: .94; }
.hero-copy p { max-width: 640px; margin-top: 22px; color: var(--muted); font-size: 16px; line-height: 1.6; }
.hero-mode { min-width: 220px; padding: 17px 0 3px 22px; display: flex; flex-direction: column; gap: 5px; border-left: 1px solid var(--border); }
.mode-label { color: var(--muted-dark); font-family: var(--mono); font-size: 9px; font-weight: 700; letter-spacing: .1em; text-transform: uppercase; }
.hero-mode strong { color: var(--accent); font-family: var(--mono); font-size: 25px; letter-spacing: -.04em; }
.hero-mode > span:last-child { color: var(--muted); font-size: 11px; }
.metric-strip { display: grid; grid-template-columns: repeat(4, 1fr); border-bottom: 1px solid var(--border); }
.metric-block { min-height: 132px; padding: 26px 24px; display: flex; flex-direction: column; border-right: 1px solid var(--border-light); }
.metric-block:first-child { padding-left: 4px; }
.metric-block:last-child { border-right: 0; }
.metric-kicker { color: var(--muted-dark); font-family: var(--mono); font-size: 9px; font-weight: 700; letter-spacing: .09em; text-transform: uppercase; }
.metric-block strong { margin-top: auto; color: var(--foreground); font-size: 34px; font-weight: 580; letter-spacing: -.05em; line-height: 1; }
.metric-block small { margin-top: 7px; color: #636b61; font-size: 10px; }
.panel { background: rgba(16,19,16,.74); border: 1px solid var(--border); border-radius: 8px; }
.section-head { min-height: 58px; padding: 0 18px; display: flex; align-items: center; justify-content: space-between; gap: 18px; border-bottom: 1px solid var(--border-light); }
.section-head > div { display: flex; align-items: center; gap: 11px; }
.section-head h3 { font-size: 13px; font-weight: 650; letter-spacing: -.01em; }
.section-index { color: #596157; font-family: var(--mono); font-size: 9px; }
.freshness, .text-link { color: var(--muted-dark); font-family: var(--mono); font-size: 9px; letter-spacing: .04em; text-decoration: none; }
.text-link:hover { color: var(--accent); }
.pipeline-section { margin-top: 26px; }
.pipeline { min-height: 158px; padding: 26px 28px; display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto 1fr auto 1fr; align-items: center; }
.pipeline-step { min-width: 0; display: grid; grid-template-columns: 22px 1fr; grid-template-rows: auto auto auto; align-items: center; }
.step-number { grid-row: 1 / span 3; align-self: start; color: #4f574e; font-family: var(--mono); font-size: 9px; }
.pipeline-step strong { font-size: 13px; }
.pipeline-step small { margin-top: 3px; color: var(--muted-dark); font-size: 10px; }
.pipeline-step b { margin-top: 12px; color: var(--accent); font-family: var(--mono); font-size: 18px; font-weight: 550; }
.pipeline-arrow { padding: 0 10px; color: #3c433b; font-family: var(--mono); }
.autonomy-grid { margin-top: 16px; display: grid; grid-template-columns: minmax(0, 1.8fr) minmax(280px, .7fr); gap: 16px; }
.hypothesis-list { min-height: 320px; }
.hypothesis-row { padding: 18px; display: grid; grid-template-columns: 82px minmax(100px, .7fr) minmax(220px, 1.8fr) 100px; gap: 18px; align-items: center; border-bottom: 1px solid var(--border-light); }
.hypothesis-row:last-child { border-bottom: 0; }
.hypothesis-symbol { font-family: var(--mono); font-size: 15px; font-weight: 750; }
.direction-tag { width: fit-content; margin-top: 5px; color: var(--muted-dark); font-family: var(--mono); font-size: 8px; text-transform: uppercase; }
.direction-tag.positive { color: var(--positive); }
.direction-tag.negative { color: var(--negative); }
.hypothesis-type { color: #d4d7d0; font-size: 12px; font-weight: 600; }
.hypothesis-channel { margin-top: 4px; color: var(--muted); font-size: 11px; line-height: 1.5; }
.hypothesis-meta { color: var(--muted-dark); font-family: var(--mono); font-size: 9px; line-height: 1.7; }
.decision-chip { justify-self: end; padding: 5px 8px; color: var(--warning); background: rgba(243,201,105,.06); border: 1px solid rgba(243,201,105,.16); border-radius: 3px; font-family: var(--mono); font-size: 9px; font-weight: 750; }
.decision-chip.buy { color: var(--positive); border-color: rgba(142,230,168,.18); background: rgba(142,230,168,.06); }
.decision-chip.sell { color: var(--negative); border-color: rgba(255,141,125,.18); background: rgba(255,141,125,.06); }
.decision-chip.qualifies { color: var(--positive); border-color: rgba(142,230,168,.18); background: rgba(142,230,168,.06); }
.performance-panel { padding-bottom: 18px; }
.accuracy-orbit { --accuracy: 0deg; width: 166px; height: 166px; margin: 28px auto 24px; padding: 1px; display: grid; place-items: center; border-radius: 50%; background: conic-gradient(var(--accent) var(--accuracy), #252b24 0); }
.accuracy-orbit::before { content: ""; grid-area: 1/1; width: 138px; height: 138px; border-radius: 50%; background: #111411; }
.accuracy-orbit > div { z-index: 1; grid-area: 1/1; display: flex; flex-direction: column; align-items: center; }
.accuracy-orbit strong { font-size: 31px; font-weight: 570; letter-spacing: -.055em; }
.accuracy-orbit span { color: var(--muted-dark); font-family: var(--mono); font-size: 8px; letter-spacing: .1em; text-transform: uppercase; }
.performance-facts { margin: 0 18px; border-top: 1px solid var(--border-light); }
.performance-facts > div { padding: 11px 0; display: flex; justify-content: space-between; color: var(--muted); border-bottom: 1px solid var(--border-light); font-size: 11px; }
.performance-facts strong { color: var(--foreground); font-family: var(--mono); font-size: 11px; }
.performance-note { margin: 17px 18px 0; color: var(--muted-dark); font-size: 10px; line-height: 1.55; }
.lower-grid { grid-template-columns: minmax(0, 1.8fr) minmax(280px, .7fr); }
.ledger-table { border: 0; border-radius: 0 0 8px 8px; }
.mode-pill { padding: 4px 7px; color: var(--warning); border: 1px solid rgba(243,201,105,.2); border-radius: 20px; font-family: var(--mono); font-size: 8px; font-weight: 700; text-transform: uppercase; }
.runtime-list { padding: 7px 18px; }
.runtime-list > div { min-height: 41px; display: flex; align-items: center; justify-content: space-between; color: var(--muted); border-bottom: 1px solid var(--border-light); font-size: 10px; }
.runtime-list span { display: flex; align-items: center; gap: 8px; }
.runtime-list strong { color: var(--foreground); font-family: var(--mono); font-size: 9px; font-weight: 550; }
.status-light { width: 6px; height: 6px; border-radius: 50%; background: var(--warning); }
.status-light.ok { background: var(--positive); box-shadow: 0 0 8px rgba(142,230,168,.35); }
.account-card { margin: 10px 18px 18px; padding: 15px; display: flex; flex-direction: column; background: #0d100e; border: 1px solid var(--border-light); border-radius: 5px; }
.account-card span { color: var(--muted-dark); font-family: var(--mono); font-size: 8px; text-transform: uppercase; letter-spacing: .1em; }
.account-card strong { margin-top: 9px; font-size: 15px; font-weight: 580; }
.account-card small { margin-top: 5px; color: var(--muted-dark); font-size: 9px; }
.loading-line { width: 42%; height: 1px; margin: 100px auto; background: linear-gradient(90deg, transparent, var(--accent), transparent); animation: loading 1.2s infinite; }
@keyframes loading { from { transform: translateX(-40%); opacity: .25; } 50% { opacity: 1; } to { transform: translateX(40%); opacity: .25; } }
@media (max-width: 1100px) {
.metric-strip { grid-template-columns: repeat(2, 1fr); }
.metric-block:nth-child(2) { border-right: 0; }
.autonomy-grid, .lower-grid { grid-template-columns: 1fr; }
.pipeline { grid-template-columns: repeat(5, 1fr); gap: 10px; }
.pipeline-arrow { display: none; }
}
@media (max-width: 720px) {
.autonomy-shell { padding: 22px 16px 50px; }
.autonomy-hero { min-height: 0; padding-top: 16px; flex-direction: column; align-items: stretch; }
.hero-copy h2 { font-size: 44px; }
.hero-mode { padding-left: 0; border-left: 0; }
.metric-block, .metric-block:first-child { min-height: 112px; padding: 18px 12px; }
.metric-block strong { font-size: 28px; }
.pipeline { grid-template-columns: 1fr; padding: 16px 18px; gap: 18px; }
.pipeline-step { grid-template-columns: 24px 1fr auto; grid-template-rows: auto auto; }
.pipeline-step b { grid-column: 3; grid-row: 1 / span 2; margin: 0; }
.hypothesis-row { grid-template-columns: 72px 1fr auto; gap: 12px; }
.hypothesis-meta { display: none; }
.decision-chip { grid-column: 3; }
}
public/admin/assets/css/base.css
+109 -104
View File
@@ -1,117 +1,122 @@
/* design tokens + resets + element defaults shared across every admin page */
:root {
--bg: #020817;
--bg-card: #0f172a;
--bg-subtle: #0b1120;
--border: #1e293b;
--border-light: #162032;
--foreground: #f8fafc;
--muted: #94a3b8;
--muted-dark: #475569;
--primary: #f8fafc;
--primary-bg: #1e293b;
--accent: #3b82f6;
--accent-hover: #2563eb;
--destructive: #7f1d1d;
--destructive-fg: #fca5a5;
--radius: 6px;
--radius-lg: 10px;
--bg: #090b0a;
--bg-card: #111411;
--bg-elevated: #171a17;
--bg-subtle: #0d100e;
--border: #293029;
--border-light: #1d231e;
--foreground: #f2f0e8;
--muted: #9ca398;
--muted-dark: #687066;
--primary: #dfff4f;
--primary-bg: #dfff4f;
--accent: #dfff4f;
--accent-hover: #edff93;
--positive: #8ee6a8;
--negative: #ff8d7d;
--warning: #f3c969;
--destructive: #8a3029;
--destructive-fg: #ffaaa0;
--radius: 4px;
--radius-lg: 8px;
--rail: 224px;
--sans: "Avenir Next", "Helvetica Neue", Helvetica, Arial, sans-serif;
--mono: "SFMono-Regular", Consolas, "Liberation Mono", monospace;
}
* { box-sizing: border-box; margin: 0; padding: 0; }
html { background: var(--bg); }
body {
font-family: -apple-system, BlinkMacSystemFont, "Inter", "Segoe UI", sans-serif;
background: var(--bg);
font-family: var(--sans);
background:
radial-gradient(circle at 85% -10%, rgba(223, 255, 79, .055), transparent 34rem),
var(--bg);
color: var(--foreground);
font-size: 14px;
line-height: 1.45;
min-height: 100vh;
padding-left: var(--rail);
-webkit-font-smoothing: antialiased;
}
/* ── inputs / selects ── */
input[type="text"], input[type="date"], select {
background: var(--bg-subtle);
border: 1px solid var(--border);
color: var(--foreground);
padding: 7px 10px;
border-radius: var(--radius);
font-size: 13px;
outline: none;
min-width: 140px;
transition: border-color .15s, box-shadow .15s;
}
input[type="text"]:focus, input[type="date"]:focus, select:focus {
border-color: var(--accent);
box-shadow: 0 0 0 3px rgba(59, 130, 246, .15);
}
select option { background: #0f172a; }
/* ── buttons ── */
button {
background: var(--primary-bg);
border: 1px solid var(--border);
color: var(--foreground);
padding: 7px 14px;
border-radius: var(--radius);
cursor: pointer;
font-size: 13px;
font-weight: 500;
transition: background .15s, opacity .1s;
line-height: 1;
}
button:hover { background: #263347; }
button.primary {
background: var(--foreground);
color: #0f172a;
border-color: transparent;
font-weight: 600;
}
button.primary:hover { background: #e2e8f0; }
button.danger {
background: transparent;
border-color: var(--destructive);
color: var(--destructive-fg);
}
button.danger:hover { background: rgba(127, 29, 29, .3); }
button:disabled { opacity: .4; cursor: not-allowed; }
/* textarea — shared across article modal and sql console */
textarea {
background: var(--bg-subtle);
border: 1px solid var(--border);
color: var(--foreground);
padding: 8px 10px;
border-radius: var(--radius);
font-size: 13px;
resize: vertical;
font-family: inherit;
outline: none;
min-height: 120px;
transition: border-color .15s, box-shadow .15s;
}
textarea:focus {
border-color: var(--accent);
box-shadow: 0 0 0 3px rgba(59, 130, 246, .15);
}
.url-link { color: #60a5fa; text-decoration: none; }
.url-link:hover { text-decoration: underline; }
::selection { background: var(--accent); color: #0b0d0b; }
a { color: inherit; }
button, input, select, textarea { font: inherit; }
input[type="text"], input[type="date"], input[type="search"], input[type="number"], select {
min-width: 140px;
height: 38px;
padding: 0 11px;
color: var(--foreground);
background: #0b0e0c;
border: 1px solid var(--border);
border-radius: var(--radius);
outline: none;
transition: border-color 140ms, box-shadow 140ms, background 140ms;
}
input::placeholder { color: #596057; }
select option { background: #111411; }
input:focus, select:focus, textarea:focus {
border-color: rgba(223,255,79,.75);
box-shadow: 0 0 0 3px rgba(223,255,79,.09);
background: #0f130f;
}
button {
min-height: 36px;
padding: 0 14px;
color: var(--foreground);
background: var(--bg-elevated);
border: 1px solid var(--border);
border-radius: var(--radius);
cursor: pointer;
font-size: 12px;
font-weight: 650;
letter-spacing: .02em;
transition: transform 100ms, background 140ms, border-color 140ms;
}
button:hover { background: #20251f; border-color: #414a40; }
button:active { transform: translateY(1px); }
button.primary {
color: #0c0e0c;
background: var(--primary-bg);
border-color: var(--primary-bg);
}
button.primary:hover { background: var(--accent-hover); border-color: var(--accent-hover); }
button.danger { background: transparent; border-color: #69332e; color: var(--destructive-fg); }
button.danger:hover { background: rgba(138,48,41,.22); }
button:disabled { opacity: .38; cursor: not-allowed; transform: none; }
textarea {
min-height: 120px;
padding: 10px 11px;
resize: vertical;
color: var(--foreground);
background: var(--bg-subtle);
border: 1px solid var(--border);
border-radius: var(--radius);
outline: none;
}
.url-link { color: var(--accent); text-decoration: none; }
.url-link:hover { text-decoration: underline; text-underline-offset: 3px; }
.mono { font-family: var(--mono); }
.positive { color: var(--positive); }
.negative { color: var(--negative); }
.muted { color: var(--muted); }
@media (max-width: 820px) {
:root { --rail: 0px; }
body { padding-left: 0; padding-top: 68px; }
}
public/admin/assets/css/components.css
+83 -142
View File
@@ -1,168 +1,109 @@
/* reusable building blocks: tables, badges, pagination, dialogs, toasts */
/* ── table ── */
.table-wrap {
background: var(--bg-card);
overflow: auto;
background: rgba(17,20,17,.78);
border: 1px solid var(--border);
border-radius: var(--radius-lg);
overflow: hidden;
}
table { width: 100%; border-collapse: collapse; }
th {
text-align: left;
padding: 10px 14px;
padding: 12px 14px;
color: #697167;
background: #0e110f;
border-bottom: 1px solid var(--border);
color: var(--muted-dark);
font-size: 11px;
text-align: left;
font-family: var(--mono);
font-size: 9px;
font-weight: 700;
letter-spacing: .1em;
text-transform: uppercase;
letter-spacing: .06em;
font-weight: 600;
background: var(--bg-subtle);
}
td {
padding: 10px 14px;
border-bottom: 1px solid var(--border-light);
vertical-align: middle;
}
tr:last-child td { border-bottom: none; }
tr:hover td { background: rgba(255,255,255,.02); }
/* ── truncate ── */
.truncate {
max-width: 280px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
display: block;
}
td { padding: 13px 14px; color: #c9cdc5; border-bottom: 1px solid var(--border-light); vertical-align: middle; font-size: 12px; }
tr:last-child td { border-bottom: 0; }
tbody tr { transition: background 120ms; }
tbody tr:hover { background: rgba(255,255,255,.022); }
/* ── badges ── */
.truncate { max-width: 320px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; display: block; }
.badge {
display: inline-flex;
align-items: center;
padding: 2px 8px;
border-radius: 4px;
font-size: 11px;
font-weight: 600;
letter-spacing: .03em;
gap: 5px;
padding: 3px 7px;
border: 1px solid var(--border);
border-radius: 3px;
font-family: var(--mono);
font-size: 9px;
font-weight: 700;
letter-spacing: .05em;
text-transform: uppercase;
}
.badge.ok { background: rgba(20, 83, 45, .5); color: #86efac; border: 1px solid rgba(134,239,172,.15); }
.badge.err { background: rgba(127, 29, 29, .5); color: #fca5a5; border: 1px solid rgba(252,165,165,.15); }
.badge.pending { background: rgba(30, 58, 95, .5); color: #93c5fd; border: 1px solid rgba(147,197,253,.15); }
.badge.null { background: rgba(30, 41, 59, .7); color: #64748b; border: 1px solid var(--border); }
.badge.ok { color: var(--positive); background: rgba(74,143,94,.1); border-color: rgba(142,230,168,.2); }
.badge.err { color: var(--negative); background: rgba(165,67,54,.1); border-color: rgba(255,141,125,.2); }
.badge.pending { color: var(--warning); background: rgba(180,137,47,.1); border-color: rgba(243,201,105,.2); }
.badge.null { color: var(--muted-dark); background: rgba(100,110,100,.08); }
.pagination { display: flex; align-items: center; gap: 10px; margin-top: 14px; color: var(--muted-dark); font-family: var(--mono); font-size: 10px; }
.pagination button { min-height: 32px; padding: 0 11px; }
/* ── pagination ── */
.pagination {
display: flex;
align-items: center;
gap: 10px;
margin-top: 14px;
color: var(--muted-dark);
font-size: 12px;
font-weight: 500;
}
.pagination button { font-size: 12px; padding: 5px 12px; }
/* ── overlay / dialog ── */
.overlay {
display: none;
position: fixed;
inset: 0;
background: rgba(2, 8, 23, .75);
backdrop-filter: blur(4px);
z-index: 100;
align-items: center;
justify-content: center;
}
.overlay { display: none; position: fixed; inset: 0; z-index: 100; align-items: center; justify-content: center; padding: 24px; background: rgba(3,5,4,.8); backdrop-filter: blur(10px); }
.overlay.open { display: flex; }
.modal {
background: var(--bg-card);
border: 1px solid var(--border);
border-radius: var(--radius-lg);
padding: 28px;
width: 680px;
max-width: 95vw;
max-height: 90vh;
overflow-y: auto;
box-shadow: 0 25px 50px -12px rgba(0,0,0,.5);
}
.modal h2 {
font-size: 16px;
font-weight: 600;
margin-bottom: 6px;
letter-spacing: -.01em;
}
.modal-divider {
height: 1px;
background: var(--border);
margin: 16px -28px;
}
.field { margin-bottom: 14px; display: flex; flex-direction: column; gap: 5px; }
.field label {
font-size: 11px;
font-weight: 600;
color: var(--muted);
text-transform: uppercase;
letter-spacing: .05em;
}
.field input[type="text"],
.field textarea,
.field select { width: 100%; min-width: unset; }
.modal-footer {
display: flex;
justify-content: flex-end;
gap: 8px;
margin-top: 20px;
padding-top: 16px;
border-top: 1px solid var(--border);
}
/* ── toast ── */
#toast {
position: fixed;
bottom: 24px;
right: 24px;
background: var(--bg-card);
border: 1px solid var(--border);
color: var(--foreground);
padding: 10px 16px;
border-radius: var(--radius);
font-size: 13px;
font-weight: 500;
display: none;
z-index: 200;
box-shadow: 0 8px 24px rgba(0,0,0,.4);
gap: 8px;
align-items: center;
}
.modal { width: 680px; max-width: 100%; max-height: 90vh; overflow-y: auto; padding: 26px; background: #121512; border: 1px solid #343c34; border-radius: 8px; box-shadow: 0 28px 90px rgba(0,0,0,.55); }
.modal h2 { font-size: 18px; font-weight: 650; letter-spacing: -.025em; }
.modal-divider { height: 1px; margin: 18px -26px; background: var(--border); }
.field { margin-bottom: 14px; display: flex; flex-direction: column; gap: 6px; }
.field label { color: var(--muted-dark); font-family: var(--mono); font-size: 9px; font-weight: 700; letter-spacing: .09em; text-transform: uppercase; }
.field input, .field textarea, .field select { width: 100%; min-width: 0; }
.modal-footer { display: flex; justify-content: flex-end; gap: 8px; margin-top: 22px; padding-top: 16px; border-top: 1px solid var(--border); }
#toast { position: fixed; right: 24px; bottom: 24px; z-index: 200; display: none; align-items: center; gap: 9px; padding: 11px 14px; color: var(--foreground); background: #171b17; border: 1px solid #374037; border-radius: 5px; box-shadow: 0 14px 40px rgba(0,0,0,.45); font-size: 12px; }
#toast.show { display: flex; }
#toast .toast-dot { width: 7px; height: 7px; border-radius: 50%; background: #22c55e; flex-shrink: 0; }
#toast.error .toast-dot { background: #ef4444; }
#toast .toast-dot { width: 7px; height: 7px; flex: 0 0 auto; border-radius: 50%; background: var(--positive); }
#toast.error .toast-dot { background: var(--negative); }
.empty-state { padding: 42px 24px; color: var(--muted-dark); text-align: center; }
.skeleton-row:hover { background: transparent; }
.skeleton-row td { height: 205px; padding: 22px 18px; vertical-align: top; }
.skeleton-row td span {
width: min(680px, 82%);
height: 11px;
margin-bottom: 20px;
display: block;
border-radius: 2px;
background: linear-gradient(90deg, #171b17 20%, #252b24 42%, #171b17 64%);
background-size: 300% 100%;
animation: skeleton-wave 1.6s ease infinite;
}
.skeleton-row td span:nth-child(2) { width: 58%; }
.skeleton-row td span:nth-child(3) { width: 72%; }
.skeleton-row td span:nth-child(4) { width: 45%; }
.content-loading { min-height: 160px; position: relative; }
.content-loading::after { content: "Loading"; position: absolute; inset: 0; display: grid; place-items: center; color: var(--muted-dark); font-family: var(--mono); font-size: 9px; letter-spacing: .1em; text-transform: uppercase; }
@keyframes skeleton-wave { from { background-position: 100% 0; } to { background-position: 0 0; } }
html::after {
content: "";
position: fixed;
z-index: 500;
top: 0;
left: var(--rail);
width: 0;
height: 2px;
opacity: 0;
background: var(--accent);
box-shadow: 0 0 12px rgba(223,255,79,.45);
transition: width 220ms ease, opacity 120ms;
}
html.navigating::after { width: calc(100% - var(--rail)); opacity: 1; }
@media (max-width: 640px) {
.table-wrap { border-radius: 0; margin-inline: -16px; border-left: 0; border-right: 0; }
th, td { padding-inline: 11px; }
.modal { padding: 20px; }
}
public/admin/assets/css/intel.css
+14
View File
@@ -58,6 +58,20 @@
#intel-stats-row .intel-stat-card:first-child { padding-left: 24px; }
#intel-stats-row .intel-stat-card:last-child { border-right: none; }
/* The autonomy-era shell owns the stat strip layout. Keep the legacy
intelligence data but present it in the same visual system. */
#intel-stats-row {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
}
#intel-stats-row .intel-stat-card,
#intel-stats-row .intel-stat-card:first-child {
min-width: 0;
padding: 20px 26px;
border-right: 1px solid var(--border-light);
}
/* ── intel detail body (modal) ── */
public/admin/assets/css/layout.css
+115 -120
View File
@@ -1,178 +1,173 @@
/* page chrome — header, tabs, subnav, stats bar, content shell */
/* ── header ── */
header.app-header {
background: var(--bg-card);
border-bottom: 1px solid var(--border);
padding: 0 24px;
position: fixed;
inset: 0 auto 0 0;
z-index: 80;
width: 224px;
height: 100vh;
padding: 26px 18px 18px;
display: flex;
align-items: center;
gap: 24px;
height: 52px;
flex-direction: column;
align-items: stretch;
background: rgba(13, 16, 14, .96);
border-right: 1px solid var(--border);
backdrop-filter: blur(18px);
}
header.app-header h1 {
font-size: 15px;
font-weight: 600;
display: flex;
align-items: center;
gap: 11px;
margin: 0 8px 34px;
color: var(--foreground);
letter-spacing: -.01em;
font-size: 15px;
font-weight: 700;
letter-spacing: -.02em;
}
header.app-header h1 span {
color: var(--muted);
font-weight: 400;
.brand-mark {
width: 30px;
height: 30px;
display: grid;
place-items: center;
color: #0b0d0b;
background: var(--accent);
border-radius: 3px 11px 3px 3px;
font-family: var(--mono);
font-size: 13px;
font-weight: 900;
}
/* ── primary tabs (underline style) ── */
.brand-copy { display: flex; flex-direction: column; line-height: 1.05; }
.brand-copy span { margin-top: 5px; color: var(--muted-dark); font-family: var(--mono); font-size: 9px; font-weight: 600; letter-spacing: .14em; text-transform: uppercase; }
.tabs {
display: flex;
gap: 0;
margin-left: auto;
height: 100%;
align-items: stretch;
flex-direction: column;
gap: 3px;
margin: 0;
height: auto;
}
.tabs::before {
content: "Workspace";
margin: 0 10px 10px;
color: #5d655b;
font-family: var(--mono);
font-size: 9px;
font-weight: 700;
letter-spacing: .14em;
text-transform: uppercase;
}
.tabs a {
background: none;
border: none;
border-bottom: 2px solid transparent;
color: var(--muted);
padding: 0 14px;
cursor: pointer;
font-size: 13px;
font-weight: 500;
transition: color .15s, border-color .15s;
height: 100%;
position: relative;
height: 40px;
padding: 0 11px;
display: flex;
align-items: center;
color: #899087;
border: 1px solid transparent;
border-radius: 5px;
text-decoration: none;
font-size: 12px;
font-weight: 600;
transition: color 140ms, background 140ms, border-color 140ms;
}
.tabs a:hover { color: var(--foreground); }
.tabs a:hover { color: var(--foreground); background: #151915; }
.tabs a.active { color: var(--foreground); background: #1a1f1a; border-color: #293129; }
.tabs a.active::after { content: ""; position: absolute; right: 10px; width: 5px; height: 5px; border-radius: 50%; background: var(--accent); box-shadow: 0 0 12px rgba(223,255,79,.6); }
.tabs a.active {
color: var(--foreground);
border-bottom-color: var(--foreground);
.rail-status {
margin-top: auto;
padding: 13px 12px;
background: #101310;
border: 1px solid var(--border-light);
border-radius: 6px;
}
/* ── subnav (intelligence sub-sections) ── */
.rail-status-label { display: flex; align-items: center; gap: 7px; color: var(--muted); font-size: 11px; font-weight: 600; }
.rail-status-dot { width: 7px; height: 7px; border-radius: 50%; background: var(--warning); box-shadow: 0 0 10px rgba(243,201,105,.35); }
.rail-status-dot.live { background: var(--positive); box-shadow: 0 0 10px rgba(142,230,168,.4); }
.rail-status-meta { margin-top: 7px; color: #5e665d; font-family: var(--mono); font-size: 9px; text-transform: uppercase; letter-spacing: .08em; }
.subnav {
background: var(--bg-card);
border-bottom: 1px solid var(--border);
padding: 0 24px;
min-height: 48px;
padding: 0 32px;
display: flex;
align-items: stretch;
gap: 0;
height: 38px;
gap: 22px;
background: rgba(9,11,10,.88);
border-bottom: 1px solid var(--border-light);
backdrop-filter: blur(14px);
}
.subnav a {
color: var(--muted-dark);
font-size: 12px;
font-weight: 500;
text-decoration: none;
padding: 0 14px;
display: flex;
align-items: center;
border-bottom: 2px solid transparent;
transition: color .15s, border-color .15s;
color: var(--muted-dark);
border-bottom: 1px solid transparent;
text-decoration: none;
font-family: var(--mono);
font-size: 10px;
font-weight: 650;
letter-spacing: .09em;
text-transform: uppercase;
letter-spacing: .05em;
}
.subnav a:hover { color: var(--foreground); }
.subnav a.active { color: var(--accent); border-bottom-color: var(--accent); }
.subnav a.active {
color: var(--foreground);
border-bottom-color: var(--accent);
.stats-bar, #intel-stats-row {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(160px, 1fr));
background: #0c0f0d;
border-bottom: 1px solid var(--border-light);
}
/* ── stats bar ── */
/* each .stat owns its own horizontal padding so the vertical separator
renders edge-to-edge; no margin gap between cells. */
.stats-bar {
display: flex;
background: var(--bg-card);
border-bottom: 1px solid var(--border);
flex-wrap: wrap;
}
.stat {
.stat, #intel-stats-row .intel-stat-card {
min-width: 0;
padding: 20px 26px;
display: flex;
flex-direction: column;
justify-content: center;
gap: 4px;
padding: 16px 32px;
border-right: 1px solid var(--border);
flex: 0 0 auto;
gap: 7px;
background: transparent;
border: 0;
border-right: 1px solid var(--border-light);
border-radius: 0;
}
.stat:first-child { padding-left: 24px; }
.stat:last-child { border-right: none; }
.stat:last-child, #intel-stats-row .intel-stat-card:last-child { border-right: 0; }
.stat .label, #intel-stats-row .label { color: var(--muted-dark); font-family: var(--mono); font-size: 9px; font-weight: 650; letter-spacing: .11em; text-transform: uppercase; }
.stat .value, #intel-stats-row .value { color: var(--foreground); font-size: 25px; font-weight: 630; letter-spacing: -.045em; line-height: 1; }
.stat .label {
color: var(--muted-dark);
font-size: 11px;
text-transform: uppercase;
letter-spacing: .06em;
font-weight: 500;
}
.stat .value {
font-size: 22px;
font-weight: 700;
color: var(--foreground);
letter-spacing: -.02em;
line-height: 1;
}
/* ── content ── */
.content { padding: 24px; }
/* ── filters ── */
.content { width: min(1480px, 100%); margin: 0 auto; padding: 32px; }
.filters {
display: flex;
gap: 10px;
margin-bottom: 18px;
flex-wrap: wrap;
padding: 13px;
align-items: flex-end;
padding: 14px;
background: var(--bg-card);
flex-wrap: wrap;
background: rgba(17,20,17,.72);
border: 1px solid var(--border);
border-radius: var(--radius-lg);
}
.filters label {
display: flex;
flex-direction: column;
gap: 5px;
font-size: 11px;
font-weight: 500;
color: var(--muted);
text-transform: uppercase;
letter-spacing: .05em;
}
.filters label { display: flex; flex-direction: column; gap: 6px; color: var(--muted-dark); font-family: var(--mono); font-size: 9px; font-weight: 650; letter-spacing: .09em; text-transform: uppercase; }
.section-heading { margin-bottom: 12px; color: var(--muted-dark); font-family: var(--mono); font-size: 10px; font-weight: 650; letter-spacing: .11em; text-transform: uppercase; }
/* ── section heading ── */
.section-heading {
font-size: 12px;
color: var(--muted-dark);
font-weight: 600;
text-transform: uppercase;
letter-spacing: .06em;
margin-bottom: 12px;
@media (max-width: 820px) {
header.app-header { inset: 0 0 auto 0; width: 100%; height: 68px; padding: 12px 16px; flex-direction: row; align-items: center; overflow-x: auto; border-right: 0; border-bottom: 1px solid var(--border); }
header.app-header h1 { margin: 0 10px 0 0; flex: 0 0 auto; }
.brand-copy span, .tabs::before, .rail-status { display: none; }
.tabs { flex-direction: row; gap: 2px; }
.tabs a { height: 38px; padding: 0 10px; white-space: nowrap; }
.tabs a.active::after { display: none; }
.subnav { padding: 0 18px; overflow-x: auto; }
.content { padding: 20px 16px 32px; }
.stats-bar, #intel-stats-row { grid-template-columns: repeat(2, 1fr); }
.stat, #intel-stats-row .intel-stat-card { padding: 16px; }
}
public/admin/assets/css/ops.css
+197
View File
@@ -0,0 +1,197 @@
/* Ops console. Dark by default because this is a thing you stare at during an
incident, but the tokens are defined so a light theme is a one line change. */
:root {
--bg: #0b0d10;
--panel: #12161b;
--panel-2: #171c22;
--line: #232a33;
--ink: #e6edf3;
--ink-dim: #9aa7b4;
--ink-faint: #6b7885;
--accent: #4c8dff;
--ok: #3fb950;
--warn: #d29922;
--bad: #f85149;
--radius: 10px;
--mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;
--sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Inter, Roboto, sans-serif;
}
* { box-sizing: border-box; }
body {
margin: 0;
background: var(--bg);
color: var(--ink);
font: 14px/1.5 var(--sans);
-webkit-font-smoothing: antialiased;
}
.boot { display: grid; place-content: center; gap: 8px; height: 100vh; text-align: center; }
.boot-mark { font: 600 20px var(--mono); letter-spacing: .3em; }
.boot-note { color: var(--ink-faint); font-size: 13px; }
/* ---------- shell ---------- */
.shell { display: grid; grid-template-columns: 208px 1fr; min-height: 100vh; }
.side {
border-right: 1px solid var(--line);
background: var(--panel);
padding: 18px 12px;
position: sticky; top: 0; height: 100vh;
display: flex; flex-direction: column; gap: 4px;
}
.brand { font: 600 14px var(--mono); letter-spacing: .28em; padding: 6px 10px 16px; }
.nav-item {
display: flex; align-items: center; justify-content: space-between; gap: 8px;
padding: 8px 10px; border-radius: 8px; cursor: pointer;
color: var(--ink-dim); text-decoration: none; font-size: 13.5px;
border: 1px solid transparent;
}
.nav-item:hover { background: var(--panel-2); color: var(--ink); }
.nav-item.active { background: var(--panel-2); color: var(--ink); border-color: var(--line); }
.nav-spacer { flex: 1; }
.main { padding: 20px 24px 60px; min-width: 0; }
.topbar {
display: flex; align-items: center; gap: 12px; flex-wrap: wrap;
margin-bottom: 18px;
}
.topbar h1 { font-size: 17px; margin: 0; font-weight: 600; }
.topbar .grow { flex: 1; }
/* ---------- primitives ---------- */
.grid { display: grid; gap: 14px; }
.cols-2 { grid-template-columns: repeat(auto-fit, minmax(340px, 1fr)); }
.cols-3 { grid-template-columns: repeat(auto-fit, minmax(230px, 1fr)); }
.card {
background: var(--panel);
border: 1px solid var(--line);
border-radius: var(--radius);
padding: 14px 16px;
min-width: 0;
}
.card h2 {
font-size: 11px; text-transform: uppercase; letter-spacing: .12em;
color: var(--ink-faint); margin: 0 0 12px; font-weight: 600;
}
.stat { font: 600 24px/1.15 var(--mono); }
.stat-sub { color: var(--ink-faint); font-size: 12px; margin-top: 2px; }
.dot { width: 8px; height: 8px; border-radius: 50%; display: inline-block; flex: none; }
.dot.ok { background: var(--ok); }
.dot.warn { background: var(--warn); }
.dot.bad { background: var(--bad); }
.dot.idle { background: var(--ink-faint); }
.dot.live { box-shadow: 0 0 0 0 rgba(63,185,80,.6); animation: pulse 2.4s infinite; }
@keyframes pulse {
70% { box-shadow: 0 0 0 7px rgba(63,185,80,0); }
100% { box-shadow: 0 0 0 0 rgba(63,185,80,0); }
}
.row { display: flex; align-items: center; gap: 10px; }
.row + .row { margin-top: 8px; }
.spread { justify-content: space-between; }
.muted { color: var(--ink-faint); }
.mono { font-family: var(--mono); }
.small { font-size: 12px; }
.nowrap { white-space: nowrap; }
table { width: 100%; border-collapse: collapse; font-size: 13px; }
th {
text-align: left; font-weight: 500; color: var(--ink-faint);
font-size: 11px; text-transform: uppercase; letter-spacing: .08em;
padding: 0 10px 8px 0; border-bottom: 1px solid var(--line);
}
td { padding: 8px 10px 8px 0; border-bottom: 1px solid var(--line); vertical-align: top; }
tr:last-child td { border-bottom: 0; }
.num { text-align: right; font-family: var(--mono); }
.table-scroll { overflow-x: auto; }
.pill {
display: inline-flex; align-items: center; gap: 6px;
padding: 2px 8px; border-radius: 999px; font-size: 11.5px;
border: 1px solid var(--line); background: var(--panel-2); color: var(--ink-dim);
font-family: var(--mono);
}
.pill.ok { color: var(--ok); border-color: rgba(63,185,80,.35); }
.pill.warn { color: var(--warn); border-color: rgba(210,153,34,.35); }
.pill.bad { color: var(--bad); border-color: rgba(248,81,73,.35); }
button {
font: inherit; color: var(--ink); background: var(--panel-2);
border: 1px solid var(--line); border-radius: 8px;
padding: 7px 12px; cursor: pointer;
}
button:hover:not(:disabled) { border-color: var(--accent); }
button:disabled { opacity: .45; cursor: default; }
button.primary { background: var(--accent); border-color: var(--accent); color: #06101f; font-weight: 600; }
button.danger { color: var(--bad); border-color: rgba(248,81,73,.4); }
button.danger:hover:not(:disabled) { background: rgba(248,81,73,.12); border-color: var(--bad); }
button.sm { padding: 4px 9px; font-size: 12px; }
input, select, textarea {
font: inherit; color: var(--ink); background: var(--bg);
border: 1px solid var(--line); border-radius: 8px; padding: 7px 10px;
}
textarea { font-family: var(--mono); font-size: 12.5px; width: 100%; resize: vertical; }
input:focus, select:focus, textarea:focus { outline: none; border-color: var(--accent); }
pre {
margin: 0; padding: 12px; background: var(--bg);
border: 1px solid var(--line); border-radius: 8px;
overflow: auto; max-height: 460px;
font: 12px/1.5 var(--mono); color: var(--ink-dim);
white-space: pre;
}
/* ---------- bars ---------- */
.bar { height: 6px; border-radius: 3px; background: var(--panel-2); overflow: hidden; }
.bar > span { display: block; height: 100%; background: var(--accent); }
.bar.ok > span { background: var(--ok); }
.bar.warn > span { background: var(--warn); }
.bar.bad > span { background: var(--bad); }
.gate { display: grid; gap: 9px; }
.gate-row { display: grid; grid-template-columns: 1fr auto; gap: 4px 10px; align-items: center; }
.gate-label { font-size: 12.5px; color: var(--ink-dim); }
.banner {
display: flex; align-items: center; gap: 12px; flex-wrap: wrap;
border: 1px solid rgba(248,81,73,.4); background: rgba(248,81,73,.08);
border-radius: var(--radius); padding: 12px 14px; margin-bottom: 14px;
}
.banner.warn { border-color: rgba(210,153,34,.4); background: rgba(210,153,34,.08); }
.toast {
position: fixed; right: 18px; bottom: 18px; z-index: 50;
background: var(--panel); border: 1px solid var(--line);
border-radius: 10px; padding: 11px 14px; max-width: 380px;
box-shadow: 0 10px 30px rgba(0,0,0,.45);
}
.toast.bad { border-color: rgba(248,81,73,.5); }
.toast.ok { border-color: rgba(63,185,80,.45); }
.empty { color: var(--ink-faint); font-size: 13px; padding: 8px 0; }
.skel {
background: linear-gradient(90deg, var(--panel-2) 25%, #1d232b 37%, var(--panel-2) 63%);
background-size: 400% 100%;
animation: shimmer 1.3s ease infinite;
border-radius: 6px; height: 14px;
}
@keyframes shimmer { 0% { background-position: 100% 0; } 100% { background-position: -100% 0; } }
@media (max-width: 820px) {
.shell { grid-template-columns: 1fr; }
.side {
position: static; height: auto; flex-direction: row; overflow-x: auto;
border-right: 0; border-bottom: 1px solid var(--line);
}
.brand { padding: 6px 10px; }
.nav-spacer { display: none; }
.main { padding: 16px; }
}
public/admin/assets/js/app.js
+113 -2
View File
@@ -34,6 +34,72 @@ function escapeHtml(s) {
}
function formatNumber(value, compact = false) {
const number = Number(value || 0);
return new Intl.NumberFormat("en-GB", compact ? { notation: "compact", maximumFractionDigits: 1 } : {}).format(number);
}
function formatPercent(value, digits = 1) {
if (value == null || !Number.isFinite(Number(value))) return "—";
return `${(Number(value) * 100).toFixed(digits)}%`;
}
function formatRelative(value) {
if (!value) return "—";
const timestamp = new Date(value.endsWith && value.endsWith("Z") ? value : `${value}Z`).getTime();
if (!Number.isFinite(timestamp)) return value;
const seconds = Math.round((timestamp - Date.now()) / 1000);
const units = [[86400, "day"], [3600, "hour"], [60, "minute"]];
const formatter = new Intl.RelativeTimeFormat("en", { numeric: "auto" });
for (const [size, name] of units) {
if (Math.abs(seconds) >= size) return formatter.format(Math.round(seconds / size), name);
}
return "just now";
}
function installChrome() {
const header = document.querySelector("header.app-header");
if (!header) return;
const title = header.querySelector("h1");
if (title) title.innerHTML = `<span class="brand-mark">D</span><span class="brand-copy">Duriin<span>Autonomous intelligence</span></span>`;
const nav = header.querySelector(".tabs");
if (nav && !nav.querySelector('[href="/admin/autonomy"]')) {
nav.insertAdjacentHTML("afterbegin", '<a href="/admin/autonomy">Autonomy</a>');
}
if (nav) {
const path = location.pathname;
nav.querySelectorAll("a").forEach(link => {
const href = link.getAttribute("href");
const active = href === "/admin/autonomy"
? path === href
: href === "/admin/ingest"
? path.startsWith("/admin/ingest")
: href === "/admin/intelligence"
? path.startsWith("/admin/intelligence")
: path === href;
link.classList.toggle("active", active);
});
}
const status = document.createElement("div");
status.className = "rail-status";
status.innerHTML = `<div class="rail-status-label"><span class="rail-status-dot"></span><span id="rail-mode">Connecting</span></div><div class="rail-status-meta" id="rail-meta">Runtime status</div>`;
header.appendChild(status);
api("/admin/api/autonomy/overview").then(data => {
const mode = String(data.mode || "offline").toUpperCase();
document.getElementById("rail-mode").textContent = data.enabled ? `${mode} runtime` : "Runtime unavailable";
document.getElementById("rail-meta").textContent = data.broker?.configured ? `${data.broker.name} · connected` : "Broker not configured";
document.querySelector(".rail-status-dot")?.classList.toggle("live", data.enabled);
}).catch(() => {
document.getElementById("rail-mode").textContent = "Runtime unavailable";
});
}
// ── url query-param helpers ────────────────────────────────────────────────
//
// filters and sort state live in the url so reloads and shared links keep
@@ -82,7 +148,7 @@ async function loadGlobalStats() {
if (!bar) return;
try {
const data = await api("/admin/api/stats");
const data = await api("/admin/api/stats/summary");
const t = document.getElementById("s-total");
if (t) t.textContent = data.total.toLocaleString();
const c = document.getElementById("s-content");
@@ -95,6 +161,46 @@ async function loadGlobalStats() {
}
function installLoadingStates() {
document.querySelectorAll("tbody:empty").forEach(tbody => {
const columns = Math.max(1, tbody.closest("table")?.querySelectorAll("thead th").length || 5);
tbody.setAttribute("aria-busy", "true");
tbody.innerHTML = `<tr class="skeleton-row"><td colspan="${columns}"><span></span><span></span><span></span><span></span></td></tr>`;
const observer = new MutationObserver(() => {
if (!tbody.querySelector(".skeleton-row")) {
tbody.removeAttribute("aria-busy");
observer.disconnect();
}
});
observer.observe(tbody, { childList: true });
});
["intel-signals-grid", "sourceTable", "statusTable"].forEach(id => {
const node = document.getElementById(id);
if (node && !node.children.length) node.classList.add("content-loading");
});
}
function installNavigationPrefetch() {
const seen = new Set();
const prefetch = link => {
const href = link?.href;
if (!href || seen.has(href) || link.origin !== location.origin) return;
seen.add(href);
const hint = document.createElement("link");
hint.rel = "prefetch";
hint.href = href;
hint.as = "document";
document.head.appendChild(hint);
};
document.querySelectorAll(".tabs a, .subnav a").forEach(link => {
link.addEventListener("pointerenter", () => prefetch(link), { once: true });
link.addEventListener("focus", () => prefetch(link), { once: true });
link.addEventListener("click", () => document.documentElement.classList.add("navigating"));
});
}
// common overlay close-on-backdrop wiring
function wireOverlays() {
document.querySelectorAll(".overlay").forEach(ov => {
@@ -106,6 +212,11 @@ function wireOverlays() {
document.addEventListener("DOMContentLoaded", () => {
installChrome();
installLoadingStates();
installNavigationPrefetch();
wireOverlays();
loadGlobalStats();
const loadStats = () => loadGlobalStats();
if ("requestIdleCallback" in window) requestIdleCallback(loadStats, { timeout: 1800 });
else setTimeout(loadStats, 500);
});
public/admin/assets/js/articles.js
+6 -3
View File
@@ -74,11 +74,14 @@ async function loadArticles() {
`).join("");
const total = data.total;
document.getElementById("pageInfo").textContent =
`${articleOffset + 1}–${Math.min(articleOffset + PAGE, total)} of ${total.toLocaleString()}`;
const start = data.rows.length ? articleOffset + 1 : 0;
const end = articleOffset + data.rows.length;
document.getElementById("pageInfo").textContent = total == null
? `${start}–${end} · filtered results`
: `${start}–${end} of approximately ${total.toLocaleString()}`;
document.getElementById("prevBtn").disabled = articleOffset === 0;
document.getElementById("nextBtn").disabled = articleOffset + PAGE >= total;
document.getElementById("nextBtn").disabled = !data.hasMore;
}
public/admin/assets/js/autonomy.js
+185
View File
@@ -0,0 +1,185 @@
(function () {
const byId = id => document.getElementById(id);
const count = (rows, key, value) => Number((rows || []).find(row => row[key] === value)?.count || 0);
const sum = (rows, predicate) => (rows || []).filter(predicate).reduce((total, row) => total + Number(row.count || 0), 0);
function renderHypotheses(rows) {
const host = byId("hypothesis-list");
if (!rows?.length) {
host.innerHTML = '<div class="empty-state">No autonomy hypotheses yet. The coordinator is working through the evidence queue.</div>';
return;
}
host.innerHTML = rows.slice(0, 7).map(row => {
const action = row.action || (row.status === "resolved" ? "MEASURED" : "OPEN");
const direction = row.direction === "positive" ? "positive" : "negative";
return `<article class="hypothesis-row">
<div><div class="hypothesis-symbol">${escapeHtml(row.instrument)}</div><div class="direction-tag ${direction}">${escapeHtml(row.direction)}</div></div>
<div><div class="hypothesis-type">${escapeHtml(String(row.event_type || "event").replaceAll("_", " "))}</div><div class="hypothesis-channel">${escapeHtml(row.causal_channel || "Evidence-backed market hypothesis")}</div></div>
<div class="hypothesis-meta">${row.evidence_count || 0} evidence source${row.evidence_count === 1 ? "" : "s"}<br>${row.horizon_days} trading-day horizon<br>${formatRelative(row.created_at)}</div>
<span class="decision-chip ${String(action).toLowerCase()}">${escapeHtml(action)}</span>
</article>`;
}).join("");
}
function renderLedger(rows) {
const host = byId("decision-ledger");
const decisions = (rows || []).filter(row => row.action);
if (!decisions.length) return;
host.innerHTML = decisions.map(row => `<tr>
<td class="mono">${escapeHtml(row.instrument)}</td>
<td><span class="decision-chip ${String(row.action).toLowerCase()}">${escapeHtml(row.action)}</span></td>
<td class="${row.direction === "positive" ? "positive" : "negative"}">${escapeHtml(row.direction)}</td>
<td class="mono">${formatPercent(row.calibrated_probability)}</td>
<td>${row.horizon_days}d</td>
<td class="muted">${formatRelative(row.created_at)}</td>
</tr>`).join("");
}
const ORIGIN_LABELS = {
live: "Live",
historical: "Historical backfill",
replay: "Walk-forward replay",
};
// live, historical and replay are deliberately never blended. only the live
// row is an edge claim, the other two are how the model was taught.
function renderOriginSplit(byOrigin, livePredictions) {
const host = byId("origin-split");
if (!host) return;
const rows = (byOrigin || []).filter(row => row.origin !== "live");
if (!rows.length) {
host.innerHTML = "";
return;
}
host.innerHTML = rows.map(row => {
const total = Number(row.total || 0);
const label = ORIGIN_LABELS[row.origin] || row.origin;
const accuracy = total ? formatPercent(Number(row.correct || 0) / total) : "—";
return `<div><span>${escapeHtml(label)}</span><strong>${formatNumber(total)} measured · ${accuracy}</strong></div>`;
}).join("");
byId("origin-note").textContent = livePredictions
? "Historical backfill and walk-forward replay are listed separately. Neither counts toward live edge."
: "No live predictions exist yet, so the numbers above are training and replay only — not evidence of live edge.";
}
function cohortCell(check, format) {
if (!check || !check.known) return '<td class="mono muted">unknown</td>';
return `<td class="mono ${check.ok ? "positive" : "negative"}">${format(check.value)} / ${format(check.threshold)}</td>`;
}
function renderCohorts(rows) {
const host = byId("cohort-list");
if (!host) return;
if (!rows?.length) {
host.innerHTML = '<tr><td colspan="6" class="empty-state">No calibration snapshots yet.</td></tr>';
return;
}
host.innerHTML = rows.map(row => {
const checks = row.qualification?.checks || {};
const qualified = Boolean(row.qualification?.qualified);
const reasons = row.qualification?.reasons || [];
const status = qualified
? '<span class="decision-chip qualifies">QUALIFIES</span>'
: `<span class="decision-chip abstain">ABSTAIN</span><div class="hypothesis-channel">${escapeHtml(reasons.join(" · ") || "does not qualify")}</div>`;
const key = row.legacy_cohort_key
? `${escapeHtml(row.cohort_key || "—")}<div class="hypothesis-channel">legacy key, not comparable to current cohorts</div>`
: escapeHtml(row.cohort_key || "—");
return `<tr>
<td class="mono">${key}</td>
<td class="muted">${escapeHtml(row.source || "unknown")}</td>
${cohortCell(checks.sample_size, value => formatNumber(value))}
${cohortCell(checks.distinct_instruments, value => formatNumber(value))}
${cohortCell(checks.top_instrument_share, value => formatPercent(value, 0))}
<td>${status}</td>
</tr>`;
}).join("");
}
function render(data) {
if (!data.enabled) throw new Error(data.reason || "Autonomy is unavailable");
const mode = String(data.mode || "shadow").toUpperCase();
const open = count(data.predictionCounts, "status", "open");
const resolved = count(data.predictionCounts, "status", "resolved");
const byOrigin = data.outcomesByOrigin || [];
const livePredictions = (data.predictionsByOrigin || []).filter(row => row.origin === "live")
.reduce((total, row) => total + Number(row.count || 0), 0);
const outcomes = Number(data.outcomes?.total || 0);
const correct = Number(data.outcomes?.correct || 0);
const accuracy = outcomes ? correct / outcomes : null;
const pendingHistorical = count((data.jobs || []).filter(row => row.lane === "historical"), "status", "pending");
const completedJobs = sum(data.jobs, row => row.status === "complete");
const proposals = sum(data.proposalCounts, row => row.status === "accepted");
const decisions = sum(data.decisionCounts, () => true);
byId("execution-mode").textContent = mode;
byId("ledger-mode").textContent = mode;
byId("runtime-execution").textContent = mode;
byId("broker-state").textContent = data.broker?.configured ? `${data.broker.name} connected` : "Broker credentials unavailable";
byId("runtime-state").textContent = `${mode} runtime active`;
byId("hero-title").textContent = mode === "PAPER" ? "Duriin is trading in simulation." : "Duriin is learning before it acts.";
byId("hero-description").textContent = mode === "PAPER"
? "Every order is backed by measured evidence, empirical calibration and deterministic risk policy. No model has direct execution authority."
: "It is turning evidence into hypotheses, waiting for outcomes, and calibrating its judgment without placing broker orders.";
byId("metric-open").textContent = formatNumber(open);
byId("metric-resolved").textContent = `${formatNumber(resolved)} resolved`;
byId("metric-accuracy").textContent = formatPercent(accuracy);
byId("metric-sample").textContent = outcomes
? `${formatNumber(outcomes)} measured live outcomes`
: (livePredictions ? "Live predictions have not matured yet" : "No live predictions yet");
byId("metric-alpha").textContent = formatPercent(data.outcomes?.average_excess_return, 2);
byId("metric-universe").textContent = formatNumber(data.allowlistedInstruments, true);
byId("pipe-observe").textContent = formatNumber(completedJobs, true);
byId("pipe-propose").textContent = formatNumber(proposals, true);
byId("pipe-measure").textContent = formatNumber(outcomes, true);
byId("pipe-calibrate").textContent = formatNumber(data.calibration?.length || 0);
byId("pipe-act").textContent = formatNumber(decisions);
byId("freshness").textContent = `Updated ${formatRelative(data.generatedAt)}`;
byId("orbit-value").textContent = formatPercent(accuracy, 0);
byId("accuracy-orbit").style.setProperty("--accuracy", `${(accuracy || 0) * 360}deg`);
byId("perf-resolved").textContent = formatNumber(outcomes);
byId("perf-correct").textContent = formatNumber(correct);
byId("perf-cohorts").textContent = formatNumber(data.calibration?.length || 0);
if (outcomes) {
byId("performance-note").textContent = `Measured on ${outcomes} matured live predictions. Results remain descriptive until the sample is large enough for stable calibration.`;
} else if (livePredictions) {
byId("performance-note").textContent = `No live prediction has matured yet — ${formatNumber(livePredictions)} are still open. Nothing here is a live track record.`;
} else {
byId("performance-note").textContent = "No live predictions yet. Everything measured so far is historical backfill or replay, which is training, not a live track record.";
}
renderOriginSplit(byOrigin, livePredictions);
renderCohorts(data.calibration);
const replay = data.replay;
byId("replay-status").textContent = replay ? replay.status : "Not started";
byId("replay-articles").textContent = replay ? formatNumber(replay.processed_articles || 0) : "—";
byId("replay-evaluations").textContent = replay ? formatNumber(replay.evaluations || 0) : "—";
byId("replay-watermark").textContent = replay?.watermark_at ? formatRelative(replay.watermark_at) : "—";
byId("runtime-queue").textContent = `${formatNumber(pendingHistorical, true)} pending`;
byId("runtime-coordinator").textContent = pendingHistorical ? "Processing" : "Watching";
if (data.account) {
byId("account-equity").textContent = new Intl.NumberFormat("en-US", { style: "currency", currency: "USD", maximumFractionDigits: 0 }).format(data.account.equity || 0);
byId("account-meta").textContent = `${data.account.broker} · sampled ${formatRelative(data.account.captured_at)}`;
}
renderHypotheses(data.latestPredictions);
renderLedger(data.latestPredictions);
}
api("/admin/api/autonomy/overview").then(render).catch(error => {
byId("hero-title").textContent = "Duriin cannot read its autonomy state.";
byId("hero-description").textContent = error.message;
byId("hypothesis-list").innerHTML = '<div class="empty-state">Runtime data is unavailable.</div>';
toast("Autonomy overview unavailable", true);
});
})();
public/admin/assets/js/events.js
+6 -3
View File
@@ -51,10 +51,13 @@ async function loadEvents() {
`).join("");
const total = data.total;
document.getElementById("ePageInfo").textContent =
`${eventOffset + 1}–${Math.min(eventOffset + PAGE, total)} of ${total.toLocaleString()}`;
const start = data.rows.length ? eventOffset + 1 : 0;
const end = eventOffset + data.rows.length;
document.getElementById("ePageInfo").textContent = total == null
? `${start}–${end} · filtered results`
: `${start}–${end} of approximately ${total.toLocaleString()}`;
document.getElementById("ePrevBtn").disabled = eventOffset === 0;
document.getElementById("eNextBtn").disabled = eventOffset + PAGE >= total;
document.getElementById("eNextBtn").disabled = !data.hasMore;
}
public/admin/assets/js/intel-signals.js
+1
View File
@@ -13,6 +13,7 @@ const signalsById = new Map();
function renderSignals(data) {
const grid = document.getElementById("intel-signals-grid");
const empty = document.getElementById("intel-signals-empty");
grid.classList.remove("content-loading");
if (!data || data.length === 0) {
grid.innerHTML = "";
public/admin/assets/js/ops/controls.js
Vendored
+154
View File
@@ -0,0 +1,154 @@
import { html, usePoll, api, useState, num } from './core.js';
import { Card, Dot, Pill, Confirm, Empty, Skeleton } from './ui.js';
export function Controls({ notify }) {
const { data, error, loading, refresh } = usePoll('/admin/api/ops/overview', 8000);
const [busy, setBusy] = useState(null);
const [analysis, setAnalysis] = useState(null);
const act = async (key, run, describe) => {
setBusy(key);
try {
const result = await run();
notify({ tone: 'ok', message: describe(result) });
await refresh();
return result;
} catch (err) {
console.error('[ops] control failed:', key, err.message);
notify({ tone: 'bad', message: `${key} failed: ${err.message}`, sticky: true });
return null;
} finally {
setBusy(null);
}
};
if (loading && !data) return html`<section class="card"><${Skeleton} rows=${4} /></section>`;
const controls = (data && data.controls) || {};
const dead = (data && data.deadLetters) || [];
const deadTotal = dead.reduce((sum, row) => sum + Number(row.n || 0), 0);
const killed = Boolean(controls.killSwitch);
return html`
<div class="grid" style="gap:14px">
${error && html`<div class="banner warn"><${Dot} tone="warn" /><span>${error}</span></div>`}
<${Card} title="Execution"
right=${html`<${Pill} tone=${killed ? 'bad' : (controls.mode === 'paper' ? 'warn' : 'ok')}>
${killed ? 'halted' : (controls.mode || 'unknown')}
<//>`}>
<div class="grid" style="gap:14px">
<div class="row spread" style="align-items:flex-start">
<div style="max-width:52ch">
<div class="row" style="gap:8px">
<${Dot} tone=${killed ? 'bad' : 'ok'} />
<strong>Kill switch ${killed ? 'engaged' : 'released'}</strong>
</div>
<div class="muted small" style="margin-top:5px">
While engaged no order intents are created. Broker reconciliation keeps
running so the account stays visible. Takes effect within one poll,
no redeploy.
</div>
</div>
<${Confirm}
label=${killed ? 'release' : 'engage kill switch'}
confirmLabel=${killed ? 'yes, release' : 'yes, halt trading'}
danger=${!killed}
busy=${busy === 'kill'}
onConfirm=${() => act('kill',
() => api('/admin/api/ops/settings', { method: 'POST', body: { killSwitch: !killed } }),
(r) => `Kill switch ${r.controls.killSwitch ? 'engaged' : 'released'}`)} />
</div>
<div class="row spread" style="align-items:flex-start; border-top:1px solid var(--line); padding-top:14px">
<div style="max-width:52ch">
<strong>Mode</strong>
<div class="muted small" style="margin-top:5px">
<span class="mono">shadow</span> records intents without touching the broker.
<span class="mono">paper</span> submits them to Alpaca paper.
Currently from ${controls.modeSource === 'settings' ? 'this dashboard' : 'the environment'}.
</div>
</div>
<${Confirm}
label=${controls.mode === 'paper' ? 'switch to shadow' : 'switch to paper'}
confirmLabel=${controls.mode === 'paper' ? 'yes, shadow' : 'yes, submit to broker'}
danger=${controls.mode !== 'paper'}
busy=${busy === 'mode'}
onConfirm=${() => act('mode',
() => api('/admin/api/ops/settings', {
method: 'POST',
body: { mode: controls.mode === 'paper' ? 'shadow' : 'paper' },
}),
(r) => `Execution mode is now ${r.controls.mode}`)} />
</div>
</div>
<//>
<${Card} title="Dead letters"
right=${deadTotal > 0 && html`<${Pill} tone="bad">${num(deadTotal)} stuck<//>`}>
${!deadTotal
? html`<${Empty}>Nothing dead-lettered. Good.<//>`
: html`
<div class="grid" style="gap:12px">
<div class="muted small">
Requeue moves rows from <span class="mono">dead_letter</span> back to
<span class="mono">pending</span>. It never deletes and never edits a payload,
so the worst case is repeated work.
</div>
<div class="table-scroll">
<table>
<thead><tr><th>job</th><th>lane</th><th class="num">n</th><th>last error</th><th></th></tr></thead>
<tbody>
${dead.map((row) => html`
<tr key=${`${row.job_type}:${row.lane}`}>
<td class="mono">${row.job_type}</td>
<td><${Pill}>${row.lane}<//></td>
<td class="num">${num(row.n)}</td>
<td class="muted small" style="max-width:40ch">${row.sample_error || '—'}</td>
<td>
<${Confirm} label="requeue"
busy=${busy === `dl:${row.job_type}:${row.lane}`}
confirmLabel=${`requeue ${num(row.n)}`}
onConfirm=${() => act(`dl:${row.job_type}:${row.lane}`,
() => api('/admin/api/ops/dead-letters/requeue', {
method: 'POST', body: { jobType: row.job_type, lane: row.lane },
}),
(r) => `Requeued ${num(r.requeued)} ${row.job_type} jobs`)} />
</td>
</tr>`)}
</tbody>
</table>
</div>
</div>`}
<//>
<${Card} title="Analyses">
<div class="grid" style="gap:12px">
<div class="row spread">
<div style="max-width:52ch">
<strong>Reaction conditioning</strong>
<div class="muted small" style="margin-top:5px">
Tests whether the initial market reaction predicts anything, on every
matured outcome. Read only, takes a few minutes, caches price history.
</div>
</div>
<button class="sm" disabled=${busy === 'analysis'}
onClick=${async () => {
setAnalysis(null);
const result = await act('analysis',
() => api('/admin/api/ops/analysis/reaction', { method: 'POST' }),
(r) => (r.ok ? 'Analysis finished' : 'Analysis exited with an error'));
if (result) setAnalysis(result);
}}>
${busy === 'analysis' ? 'running…' : 'run'}
</button>
</div>
${analysis && html`
<div class="grid" style="gap:8px">
${analysis.error && html`<div class="muted small" style="color:var(--bad)">${analysis.error}</div>`}
<pre>${analysis.output || '(no output)'}</pre>
</div>`}
</div>
<//>
</div>`;
}
public/admin/assets/js/ops/core.js
+161
View File
@@ -0,0 +1,161 @@
// Shared plumbing: htm binding, hash router, polling fetch, formatters.
const { createElement, useState, useEffect, useRef, useCallback } = React;
// htm hands props straight through, and React refuses a style string (error #62).
// Writing style objects everywhere in tagged templates is miserable, so convert
// once here instead and let the views keep using plain css.
function cssToObject(css) {
const style = {};
for (const declaration of String(css).split(';')) {
const split = declaration.indexOf(':');
if (split === -1) continue;
const property = declaration.slice(0, split).trim();
const value = declaration.slice(split + 1).trim();
if (!property || !value) continue;
// custom properties have to stay verbatim, the rest camelCase
style[property.startsWith('--') ? property : property.replace(/-([a-z])/g, (_, c) => c.toUpperCase())] = value;
}
return style;
}
function h(type, props, ...children) {
if (props && typeof props.style === 'string') {
return createElement(type, { ...props, style: cssToObject(props.style) }, ...children);
}
return createElement(type, props, ...children);
}
export const html = htm.bind(h);
export { useState, useEffect, useRef, useCallback };
export async function api(path, options = {}) {
const response = await fetch(path, {
...options,
headers: { 'Content-Type': 'application/json', ...(options.headers || {}) },
body: options.body ? JSON.stringify(options.body) : undefined,
});
const text = await response.text();
let parsed = null;
try { parsed = text ? JSON.parse(text) : null; } catch (error) {
// A non JSON body from an API that always speaks JSON means something upstream
// failed, so surface the raw text rather than a parse error nobody can act on.
console.error('[ops] non-JSON response from', path, error.message);
throw new Error(text.slice(0, 200) || `HTTP ${response.status}`);
}
if (!response.ok) throw new Error((parsed && parsed.error) || `HTTP ${response.status}`);
return parsed;
}
// One store per url, shared by every component asking for it. The sidebar and the
// overview both want /ops/overview, and without this they each opened their own
// request on their own timer -- the same payload fetched twice, forever.
const stores = new Map();
function storeFor(path) {
if (stores.has(path)) return stores.get(path);
const store = {
state: { data: null, error: null, loading: true, at: null },
listeners: new Set(),
inflight: null,
timer: null,
emit() { for (const fn of this.listeners) fn(this.state); },
async load() {
// a request already on the wire is shared rather than duplicated
if (this.inflight) return this.inflight;
this.inflight = api(path)
.then((data) => { this.state = { data, error: null, loading: false, at: Date.now() }; })
.catch((error) => {
console.error('[ops] poll failed for', path, error.message);
this.state = { ...this.state, error: error.message, loading: false };
})
.finally(() => { this.inflight = null; this.emit(); });
return this.inflight;
},
};
stores.set(path, store);
return store;
}
export function usePoll(path, intervalMs = 5000) {
const store = storeFor(path);
const [state, setState] = useState(store.state);
useEffect(() => {
const listener = (next) => setState(next);
store.listeners.add(listener);
setState(store.state);
store.load();
// the interval belongs to the store, not the component, so N subscribers still
// produce exactly one request per tick
if (!store.timer) {
store.timer = setInterval(() => { if (!document.hidden) store.load(); }, intervalMs);
}
const onVisible = () => { if (!document.hidden) store.load(); };
document.addEventListener('visibilitychange', onVisible);
return () => {
store.listeners.delete(listener);
document.removeEventListener('visibilitychange', onVisible);
if (!store.listeners.size && store.timer) {
clearInterval(store.timer);
store.timer = null;
}
};
}, [store, intervalMs]);
return { ...state, refresh: () => store.load() };
}
export function useHashRoute(fallback) {
const read = () => (location.hash || '').replace(/^#\/?/, '') || fallback;
const [route, setRoute] = useState(read);
useEffect(() => {
const onHash = () => setRoute(read());
addEventListener('hashchange', onHash);
return () => removeEventListener('hashchange', onHash);
}, []);
return route;
}
/* ---------- formatting ---------- */
export const num = (value) => (value === null || value === undefined || Number.isNaN(Number(value))
? '—' : Number(value).toLocaleString());
export function compact(value) {
const n = Number(value);
if (!Number.isFinite(n)) return '—';
if (Math.abs(n) >= 1e9) return (n / 1e9).toFixed(2) + 'B';
if (Math.abs(n) >= 1e6) return (n / 1e6).toFixed(2) + 'M';
if (Math.abs(n) >= 1e3) return (n / 1e3).toFixed(1) + 'k';
return String(n);
}
export const pct = (value, digits = 1) => (Number.isFinite(Number(value))
? `${(Number(value) * 100).toFixed(digits)}%` : '—');
export function ago(minutes) {
if (minutes === null || minutes === undefined || !Number.isFinite(Number(minutes))) return 'unknown';
const m = Number(minutes);
if (m < 1) return 'just now';
if (m < 60) return `${Math.round(m)}m ago`;
if (m < 60 * 24) return `${(m / 60).toFixed(1)}h ago`;
return `${(m / 1440).toFixed(1)}d ago`;
}
export function whenDate(value) {
if (!value) return '—';
const stamp = String(value).slice(0, 10);
const days = Math.round((Date.parse(`${stamp}T00:00:00Z`) - Date.now()) / 86400000);
if (!Number.isFinite(days)) return stamp;
if (days === 0) return `${stamp} (today)`;
return days > 0 ? `${stamp} (in ${days}d)` : `${stamp} (${-days}d ago)`;
}
export function health(minutes, threshold) {
if (minutes === null || minutes === undefined) return 'idle';
if (!Number.isFinite(Number(threshold))) return 'ok';
const m = Number(minutes);
if (m <= threshold) return 'ok';
return m <= threshold * 3 ? 'warn' : 'bad';
}
public/admin/assets/js/ops/explore.js
+156
View File
@@ -0,0 +1,156 @@
import { html, usePoll, useState, num, compact, ago } from './core.js';
import { Card, Stat, Pill, Skeleton, Empty } from './ui.js';
const PAGE = 50;
function shortUrl(url) {
try { return new URL(url).hostname.replace(/^www\./, ''); }
catch (error) { return String(url || '').slice(0, 40); }
}
function statusTone(status) {
if (status === 'ready') return 'ok';
if (status === 'failed') return 'bad';
if (status === 'pending') return 'warn';
return '';
}
// One list component behind both Articles and Events. They differ only in columns
// and endpoint, and having two near-identical files is how they drift apart.
function ListView({ title, path, columns, searchable }) {
const [offset, setOffset] = useState(0);
const [query, setQuery] = useState('');
const [applied, setApplied] = useState('');
const url = `${path}?limit=${PAGE}&offset=${offset}${applied ? `&q=${encodeURIComponent(applied)}` : ''}`;
const { data, error, loading } = usePoll(url, 20000);
const rows = (data && data.rows) || [];
const total = data && data.total;
return html`
<div class="grid" style="gap:14px">
<${Card} title=${title}
right=${html`<span class="muted small">${total === undefined ? '' : `${num(total)} total`}</span>`}>
${searchable && html`
<form class="row" style="gap:8px; margin-bottom:12px"
onSubmit=${(e) => { e.preventDefault(); setOffset(0); setApplied(query.trim()); }}>
<input style="flex:1" placeholder="search title…" value=${query}
onInput=${(e) => setQuery(e.target.value)} />
<button class="sm" type="submit">search</button>
${applied && html`<button class="sm" type="button"
onClick=${() => { setQuery(''); setApplied(''); setOffset(0); }}>clear</button>`}
</form>`}
${error && html`<div class="muted small" style="color:var(--bad); margin-bottom:10px">${error}</div>`}
${loading && !data
? html`<${Skeleton} rows=${6} />`
: !rows.length
? html`<${Empty}>Nothing here.<//>`
: html`
<div class="table-scroll">
<table>
<thead><tr>${columns.map((c) => html`
<th key=${c.key} class=${c.num ? 'num' : ''}>${c.label}</th>`)}</tr></thead>
<tbody>
${rows.map((row) => html`
<tr key=${row.id}>
${columns.map((c) => html`
<td key=${c.key} class=${c.num ? 'num' : ''}>${c.render(row)}</td>`)}
</tr>`)}
</tbody>
</table>
</div>`}
<div class="row spread" style="margin-top:12px">
<span class="muted small mono">
${rows.length ? `${num(offset + 1)}–${num(offset + rows.length)}` : '—'}
</span>
<span class="row" style="gap:8px">
<button class="sm" disabled=${offset === 0}
onClick=${() => setOffset(Math.max(0, offset - PAGE))}>prev</button>
<button class="sm" disabled=${!(data && data.hasMore)}
onClick=${() => setOffset(offset + PAGE)}>next</button>
</span>
</div>
<//>
</div>`;
}
export const Articles = () => html`<${ListView}
title="Articles" path="/admin/api/articles" searchable
columns=${[
{ key: 'title', label: 'title', render: (r) => html`
<a href=${r.url} target="_blank" rel="noopener noreferrer"
style="color:var(--ink); text-decoration:none">${r.title || '(untitled)'}</a>
<div class="muted small mono" style="margin-top:3px">${shortUrl(r.url)}</div>` },
{ key: 'source', label: 'source', render: (r) => html`<span class="small">${r.source}</span>` },
{ key: 'status', label: 'content', render: (r) => html`
<${Pill} tone=${statusTone(r.content_status)}>${r.content_status || 'unfetched'}<//>` },
{ key: 'pub', label: 'published', render: (r) => html`
<span class="mono small nowrap">${String(r.pub_date_effective || r.pub_date || '').slice(0, 16)}</span>` },
]} />`;
export const Events = () => html`<${ListView}
title="Events" path="/admin/api/events"
columns=${[
{ key: 'title', label: 'title', render: (r) => r.title || '(untitled)' },
{ key: 'articles', label: 'articles', num: true, render: (r) => num(r.article_count) },
{ key: 'created', label: 'created', render: (r) => html`
<span class="mono small nowrap">${String(r.created_at || '').slice(0, 16)}</span>` },
]} />`;
export function Intelligence() {
const summary = usePoll('/admin/api/stats/summary', 15000);
const intel = usePoll('/admin/api/intelligence/stats', 15000);
const s = summary.data || {};
const i = intel.data || {};
return html`
<div class="grid" style="gap:14px">
<div class="grid cols-3">
<${Stat} label="Articles" value=${compact(s.total)}
sub=${`${compact(s.withContent)} with content · ${compact(s.withEmbedding)} embedded`} />
<${Stat} label="Events" value=${compact(s.eventCount)} sub="clustered" />
<${Stat} label="Knowledge rows" value=${compact(i.knowledge)}
sub=${`${compact(i.companies)} companies tracked`} />
</div>
<div class="grid cols-2">
<${Card} title="Worker rates" right=${html`<span class="muted small">rows written</span>`}>
${!(i.workerRates || []).length
? html`<${Empty}>No worker activity recorded.<//>`
: html`
<table>
<thead><tr><th>worker</th><th class="num">last 1m</th><th class="num">last 5m</th></tr></thead>
<tbody>
${i.workerRates.map((w) => html`
<tr key=${w.worker}>
<td class="mono">${w.worker}</td>
<td class="num">${num(w.n1m)}</td>
<td class="num">${num(w.n5m)}</td>
</tr>`)}
</tbody>
</table>`}
<//>
<${Card} title="Article queue">
${!(i.queue || []).length
? html`<${Empty}>Queue is empty.<//>`
: html`
<table>
<thead><tr><th>status</th><th class="num">n</th></tr></thead>
<tbody>
${i.queue.map((q) => html`
<tr key=${q.status}>
<td><${Pill} tone=${statusTone(q.status)}>${q.status}<//></td>
<td class="num">${num(q.n)}</td>
</tr>`)}
</tbody>
</table>`}
<//>
</div>
</div>`;
}
public/admin/assets/js/ops/main.js
+87
View File
@@ -0,0 +1,87 @@
import { html, useHashRoute, useState, usePoll, num, ago } from './core.js';
import { Toast, Dot } from './ui.js';
import { Overview } from './overview.js';
import { Controls } from './controls.js';
import { Articles, Events, Intelligence } from './explore.js';
import { Sql } from './sql.js';
// The d3 force graph is a big specialised visualisation that already works. It is
// mounted in a frame rather than rewritten, because porting it would risk breaking
// something valuable to gain nothing the operator can see.
const Graph = () => html`
<section class="card" style="padding:0; overflow:hidden">
<iframe src="/admin/intelligence/graph" title="Intelligence graph"
style="width:100%; height:calc(100vh - 150px); border:0; display:block"></iframe>
</section>`;
const ROUTES = [
{ id: 'overview', label: 'Overview', view: Overview },
{ id: 'controls', label: 'Controls', view: Controls },
{ id: 'articles', label: 'Articles', view: Articles },
{ id: 'events', label: 'Events', view: Events },
{ id: 'intelligence', label: 'Intelligence', view: Intelligence },
{ id: 'graph', label: 'Graph', view: Graph },
{ id: 'sql', label: 'SQL', view: Sql },
];
function Nav({ route, health }) {
return html`
<nav class="side">
<div class="brand">DURIIN</div>
${ROUTES.map((r) => html`
<a key=${r.id} class=${`nav-item ${route === r.id ? 'active' : ''}`} href=${`#/${r.id}`}>
<span>${r.label}</span>
${r.id === 'controls' && health.deadTotal > 0
&& html`<span class="pill bad">${num(health.deadTotal)}</span>`}
${r.id === 'overview' && health.halted && html`<${Dot} tone="bad" />`}
</a>`)}
<div class="nav-spacer"></div>
<div class="muted small mono" style="padding:8px 10px; line-height:1.7">
<div class="row" style="gap:7px">
<${Dot} tone=${health.tone} live />
<span>${health.label}</span>
</div>
<div>ingest ${ago(health.ingestMinutes)}</div>
</div>
</nav>`;
}
function App() {
const route = useHashRoute('overview');
const [toast, setToast] = useState(null);
// A single cheap poll drives the sidebar so every view does not need its own.
const { data } = usePoll('/admin/api/ops/overview', 10000);
const controls = (data && data.controls) || {};
const deadTotal = ((data && data.deadLetters) || [])
.reduce((sum, row) => sum + Number(row.n || 0), 0);
const ingestMinutes = data && data.freshness ? data.freshness.ingestMinutes : null;
const halted = Boolean(controls.killSwitch);
const health = {
deadTotal,
halted,
ingestMinutes,
tone: halted ? 'bad' : (deadTotal > 0 ? 'warn' : 'ok'),
label: halted ? 'trading halted' : (controls.mode ? `${controls.mode} mode` : 'connecting…'),
};
const active = ROUTES.find((r) => r.id === route) || ROUTES[0];
const View = active.view;
return html`
<div class="shell">
<${Nav} route=${active.id} health=${health} />
<main class="main">
<header class="topbar">
<h1>${active.label}</h1>
<span class="grow"></span>
${halted && html`<span class="pill bad">kill switch engaged</span>`}
</header>
<${View} notify=${setToast} />
</main>
<${Toast} toast=${toast} />
</div>`;
}
ReactDOM.createRoot(document.getElementById('root')).render(html`<${App} />`);
public/admin/assets/js/ops/overview.js
+216
View File
@@ -0,0 +1,216 @@
import { html, usePoll, num, compact, pct, ago, whenDate, health } from './core.js';
import { Card, Stat, Dot, Pill, Bar, Skeleton, Empty } from './ui.js';
const GATE = { minSample: 30, minInstruments: 5, maxConcentration: 0.5 };
const ORIGIN_NOTE = {
live: 'genuine real-time work, the only thing that can authorise an order',
historical: 'coordinator backfill, training evidence only',
replay: 'walk-forward replay, training evidence only',
};
function PipelineRow({ label, tone, detail, note }) {
return html`
<div class="row spread">
<span class="row" style="gap:9px">
<${Dot} tone=${tone} live />
<span>${label}</span>
</span>
<span class="row" style="gap:10px">
${note && html`<span class="muted small nowrap">${note}</span>`}
<span class="mono small nowrap">${detail}</span>
</span>
</div>`;
}
function LiveEvidence({ maturity, liveOpen, liveResolved }) {
const total = (maturity || []).reduce((sum, row) => sum + Number(row.n || 0), 0);
const next = (maturity || [])
.map((row) => row.first_matures).filter(Boolean).sort()[0];
if (!total && !liveResolved) {
return html`<${Empty}>No live predictions yet.<//>`;
}
return html`
<div class="grid" style="gap:12px">
<div class="row spread">
<span class="stat">${num(liveResolved)}</span>
<span class="muted small">matured of ${num(liveOpen + liveResolved)} live</span>
</div>
<${Bar} value=${liveResolved} max=${liveOpen + liveResolved}
tone=${liveResolved > 0 ? 'ok' : 'warn'} />
<div class="muted small">
${liveResolved > 0
? 'Live evidence is accumulating.'
: html`Nothing has matured yet. First outcome ${html`<strong>${whenDate(next)}</strong>`}.`}
</div>
<div class="table-scroll">
<table>
<thead><tr><th>horizon</th><th class="num">open</th><th>first matures</th></tr></thead>
<tbody>
${(maturity || []).map((row) => html`
<tr key=${row.horizon_days}>
<td class="mono">${row.horizon_days}d</td>
<td class="num">${num(row.n)}</td>
<td class="mono small">${whenDate(row.first_matures)}</td>
</tr>`)}
</tbody>
</table>
</div>
</div>`;
}
function GateCard({ cohorts }) {
const live = (cohorts || []).filter((c) => c.source === 'live');
const best = live.slice().sort((a, b) => Number(b.sample_size) - Number(a.sample_size))[0];
const qualifying = live.filter((c) => Number(c.sample_size) >= GATE.minSample
&& Number(c.distinct_instruments) >= GATE.minInstruments
&& Number(c.top_instrument_share) <= GATE.maxConcentration);
const rows = [
{ label: 'samples', value: best ? Number(best.sample_size) : 0, need: GATE.minSample,
fmt: (v) => num(v) },
{ label: 'distinct tickers', value: best ? Number(best.distinct_instruments) : 0, need: GATE.minInstruments,
fmt: (v) => num(v) },
{ label: 'top ticker share', value: best ? Number(best.top_instrument_share) : 1,
need: GATE.maxConcentration, invert: true, fmt: (v) => pct(v, 0) },
];
return html`
<div class="grid" style="gap:14px">
<div class="row spread">
<span class="stat" style=${qualifying.length ? 'color:var(--ok)' : null}>${qualifying.length}</span>
<span class="muted small">live cohorts clear the gate</span>
</div>
${!live.length
? html`<${Empty}>No live calibration exists yet, so every decision abstains. Offline cohorts cannot authorise orders.<//>`
: html`
<div class="gate">
${rows.map((row) => {
const passed = row.invert ? row.value <= row.need : row.value >= row.need;
return html`
<div class="gate-row" key=${row.label}>
<span class="gate-label">${row.label}</span>
<span class="mono small" style=${`color:var(--${passed ? 'ok' : 'ink-dim'})`}>
${row.fmt(row.value)} ${passed ? '✓' : `/ ${row.fmt(row.need)}`}
</span>
<div style="grid-column:1/-1">
<${Bar} value=${row.invert ? Math.max(0, 1 - row.value) : row.value}
max=${row.invert ? 1 : row.need} tone=${passed ? 'ok' : 'warn'} />
</div>
</div>`;
})}
</div>
<div class="muted small">Best-populated live cohort shown.</div>`}
</div>`;
}
export function Overview({ notify }) {
const { data, error, loading, at } = usePoll('/admin/api/ops/overview', 5000);
if (loading && !data) {
return html`<div class="grid cols-3">
${[0, 1, 2].map((i) => html`<section class="card" key=${i}><${Skeleton} rows=${2} /></section>`)}
</div>`;
}
if (error && !data) return html`<${Card} title="Overview"><div class="muted">${error}</div><//>`;
const p = data.predictions || {};
const f = data.freshness || {};
const thresholds = f.thresholds || {};
const deadTotal = (data.deadLetters || []).reduce((s, r) => s + Number(r.n || 0), 0);
const abstain = (data.decisions || []).find((d) => d.action === 'ABSTAIN');
const acting = (data.decisions || []).filter((d) => d.action === 'BUY' || d.action === 'SELL')
.reduce((s, d) => s + Number(d.n || 0), 0);
return html`
<div class="grid" style="gap:14px">
${error && html`<div class="banner warn">
<${Dot} tone="warn" /><span>Live updates interrupted: ${error}. Showing the last good snapshot.</span>
</div>`}
${deadTotal > 0 && html`
<div class="banner">
<${Dot} tone="bad" />
<span><strong>${num(deadTotal)}</strong> dead-lettered jobs are not being retried.</span>
<span class="grow"></span>
<a class="nav-item" href="#/controls" style="padding:4px 10px">Go to controls →</a>
</div>`}
<div class="grid cols-3">
<${Stat} label="Live predictions open" value=${num(p.live_open)}
sub=${`${num(p.live_resolved)} matured · ${num(p.total)} total`} />
<${Stat} label="Order intents" value=${num(data.orderIntents)}
tone=${Number(data.orderIntents) > 0 ? 'warn' : null}
sub=${acting ? `${num(acting)} actionable decisions` : 'every decision has abstained'} />
<${Stat} label="Archive" value=${compact(data.archive && data.archive.max_id)}
sub=${`last ingest ${ago(f.ingestMinutes)}`} />
</div>
<div class="grid cols-2">
<${Card} title="Pipeline">
<div class="grid" style="gap:9px">
<${PipelineRow} label="Ingest" note="articles"
tone=${health(f.ingestMinutes, thresholds.ingest)}
detail=${ago(f.ingestMinutes)} />
<${PipelineRow} label="Coordinator" note="predictions"
tone=${health(f.predictionMinutes, thresholds.prediction)}
detail=${ago(f.predictionMinutes)} />
<${PipelineRow} label="Outcomes" note="scored"
tone=${health(f.outcomeMinutes, thresholds.outcome)}
detail=${ago(f.outcomeMinutes)} />
<${PipelineRow} label="Execution"
note=${data.controls ? (data.controls.killSwitch ? 'kill switch on' : data.controls.mode) : ''}
tone=${data.controls && data.controls.killSwitch ? 'bad' : 'ok'}
detail=${`${num(data.orderIntents)} intents`} />
</div>
<//>
<${Card} title="Can it trade yet?">
<${GateCard} cohorts=${data.cohorts} />
<//>
<${Card} title="Live evidence">
<${LiveEvidence} maturity=${data.maturity}
liveOpen=${Number(p.live_open || 0)} liveResolved=${Number(p.live_resolved || 0)} />
<//>
<${Card} title="Accuracy by origin"
right=${html`<span class="muted small">only live counts as a track record</span>`}>
${!(data.byOrigin || []).length
? html`<${Empty}>No scored outcomes yet.<//>`
: html`
<div class="table-scroll">
<table>
<thead><tr><th>origin</th><th class="num">n</th><th class="num">accuracy</th><th class="num">mean excess</th></tr></thead>
<tbody>
${data.byOrigin.map((row) => {
const acc = Number(row.total) ? Number(row.correct) / Number(row.total) : null;
return html`
<tr key=${row.origin}>
<td>
<div class="row" style="gap:7px">
<${Pill} tone=${row.origin === 'live' ? 'ok' : ''}>${row.origin}<//>
</div>
<div class="muted small" style="margin-top:3px">${ORIGIN_NOTE[row.origin] || ''}</div>
</td>
<td class="num">${num(row.total)}</td>
<td class="num">${pct(acc)}</td>
<td class="num" style=${`color:var(--${Number(row.mean_excess) >= 0 ? 'ok' : 'bad'})`}>
${pct(row.mean_excess, 2)}
</td>
</tr>`;
})}
</tbody>
</table>
</div>`}
<//>
</div>
<div class="muted small mono">
${abstain ? `${num(abstain.n)} decisions, all ABSTAIN · ` : ''}
updated ${at ? new Date(at).toLocaleTimeString() : '—'}
</div>
</div>`;
}
public/admin/assets/js/ops/sql.js
+106
View File
@@ -0,0 +1,106 @@
import { html, api, useState, num } from './core.js';
import { Card, Pill, Empty } from './ui.js';
const SAMPLES = [
{ label: 'live prediction maturity', db: 'intelligence', sql:
`SELECT horizon_days, COUNT(*) AS n,
MIN(date(information_cutoff, '+' || horizon_days || ' days')) AS first_matures
FROM autonomy_predictions
WHERE origin='live' AND status='open'
GROUP BY horizon_days ORDER BY horizon_days` },
{ label: 'accuracy by origin', db: 'intelligence', sql:
`SELECT p.origin, COUNT(*) AS n,
ROUND(100.0*AVG(o.direction_correct),1) AS acc_pct,
ROUND(100.0*AVG(o.excess_return),2) AS mean_excess_pct
FROM autonomy_predictions p JOIN autonomy_outcomes o ON o.prediction_id=p.id
GROUP BY p.origin` },
{ label: 'dead letters', db: 'intelligence', sql:
`SELECT job_type, lane, COUNT(*) AS n, substr(MAX(last_error),1,80) AS err
FROM autonomy_jobs WHERE status='dead_letter' GROUP BY job_type, lane` },
{ label: 'content backlog', db: 'archive', sql:
`SELECT COALESCE(content_status,'unfetched') AS status, COUNT(*) AS n
FROM articles GROUP BY status ORDER BY n DESC` },
];
export function Sql({ notify }) {
const [sql, setSql] = useState(SAMPLES[0].sql);
const [database, setDatabase] = useState('intelligence');
const [result, setResult] = useState(null);
const [busy, setBusy] = useState(false);
const run = async () => {
setBusy(true);
const started = performance.now();
try {
const response = await api('/admin/api/sql', { method: 'POST', body: { sql, database } });
setResult({ ...response, clientMs: Math.round(performance.now() - started) });
} catch (error) {
console.error('[ops] sql failed:', error.message);
notify({ tone: 'bad', message: error.message, sticky: true });
setResult(null);
} finally {
setBusy(false);
}
};
const first = result && result.results && result.results[0];
const rows = (first && first.rows) || [];
const columns = rows.length ? Object.keys(rows[0]) : [];
return html`
<div class="grid" style="gap:14px">
<${Card} title="Query"
right=${html`
<div class="row" style="gap:8px">
<select value=${database} onChange=${(e) => setDatabase(e.target.value)}>
<option value="intelligence">intelligence</option>
<option value="archive">archive</option>
</select>
<button class="primary sm" disabled=${busy} onClick=${run}>
${busy ? 'running…' : 'run'}
</button>
</div>`}>
<textarea rows="9" spellcheck="false" value=${sql}
onInput=${(e) => setSql(e.target.value)}
onKeyDown=${(e) => { if ((e.metaKey || e.ctrlKey) && e.key === 'Enter') run(); }}></textarea>
<div class="row" style="gap:8px; margin-top:10px; flex-wrap:wrap">
<span class="muted small">samples:</span>
${SAMPLES.map((s) => html`
<button class="sm" key=${s.label}
onClick=${() => { setSql(s.sql); setDatabase(s.db); }}>${s.label}</button>`)}
<span class="grow"></span>
<span class="muted small mono">⌘/ctrl + enter</span>
</div>
<//>
${result && html`
<${Card} title="Result"
right=${html`<span class="row" style="gap:8px">
<${Pill}>${num(rows.length)} rows<//>
<${Pill}>${num(result.elapsed)}ms server<//>
<${Pill}>${num(result.clientMs)}ms total<//>
</span>`}>
${first && first.error
? html`<div class="muted small" style="color:var(--bad)">${first.error}</div>`
: !rows.length
? html`<${Empty}>No rows.<//>`
: html`
<div class="table-scroll">
<table>
<thead><tr>${columns.map((c) => html`<th key=${c}>${c}</th>`)}</tr></thead>
<tbody>
${rows.slice(0, 500).map((row, i) => html`
<tr key=${i}>
${columns.map((c) => html`
<td key=${c} class="mono small">${row[c] === null ? '—' : String(row[c])}</td>`)}
</tr>`)}
</tbody>
</table>
</div>
${rows.length > 500 && html`
<div class="muted small" style="margin-top:10px">
Showing the first 500 of ${num(rows.length)} rows.
</div>`}`}
<//>`}
</div>`;
}
public/admin/assets/js/ops/ui.js
+71
View File
@@ -0,0 +1,71 @@
import { html, useState, useEffect } from './core.js';
export const Card = ({ title, right, children }) => html`
<section class="card">
${title && html`
<div class="row spread" style="margin-bottom:12px">
<h2 style="margin:0">${title}</h2>
${right}
</div>`}
${children}
</section>`;
export const Stat = ({ label, value, sub, tone }) => html`
<section class="card">
<h2>${label}</h2>
<div class="stat" style=${tone ? `color:var(--${tone})` : null}>${value}</div>
${sub && html`<div class="stat-sub">${sub}</div>`}
</section>`;
export const Dot = ({ tone = 'idle', live }) => html`
<span class=${`dot ${tone}${live && tone === 'ok' ? ' live' : ''}`}></span>`;
export const Pill = ({ tone, children }) => html`<span class=${`pill ${tone || ''}`}>${children}</span>`;
export const Bar = ({ value, max, tone }) => {
const width = max > 0 ? Math.min(100, (Number(value) / Number(max)) * 100) : 0;
return html`<div class=${`bar ${tone || ''}`}><span style=${`width:${width}%`}></span></div>`;
};
export const Skeleton = ({ rows = 3 }) => html`
<div class="grid" style="gap:8px">
${Array.from({ length: rows }, (_, i) => html`
<div key=${i} class="skel" style=${`width:${90 - i * 12}%`}></div>`)}
</div>`;
export const Empty = ({ children }) => html`<div class="empty">${children}</div>`;
export function Toast({ toast }) {
const [shown, setShown] = useState(toast);
useEffect(() => {
setShown(toast);
if (!toast) return undefined;
const timer = setTimeout(() => setShown(null), toast.sticky ? 12000 : 4500);
return () => clearTimeout(timer);
}, [toast]);
if (!shown) return null;
return html`<div class=${`toast ${shown.tone || ''}`}>${shown.message}</div>`;
}
// Anything that changes production asks twice. The second click is a different
// button label so muscle memory cannot carry you through both.
export function Confirm({ label, confirmLabel, onConfirm, danger, disabled, busy }) {
const [armed, setArmed] = useState(false);
useEffect(() => {
if (!armed) return undefined;
const timer = setTimeout(() => setArmed(false), 5000);
return () => clearTimeout(timer);
}, [armed]);
if (busy) return html`<button class="sm" disabled>working…</button>`;
if (!armed) {
return html`<button class=${`sm ${danger ? 'danger' : ''}`} disabled=${disabled}
onClick=${() => setArmed(true)}>${label}</button>`;
}
return html`
<span class="row" style="gap:6px">
<button class=${`sm ${danger ? 'danger' : 'primary'}`}
onClick=${() => { setArmed(false); onConfirm(); }}>${confirmLabel || 'confirm'}</button>
<button class="sm" onClick=${() => setArmed(false)}>cancel</button>
</span>`;
}
public/admin/assets/js/stats.js
+5 -2
View File
@@ -4,11 +4,14 @@
async function loadStatsPage() {
const data = await api("/admin/api/stats");
document.getElementById("sourceTable").classList.remove("content-loading");
document.getElementById("statusTable").classList.remove("content-loading");
document.getElementById("sourceTable").innerHTML = data.bySource
.map(r => `<tr><td>${escapeHtml(r.source)}</td><td style="text-align:right; padding-left:24px">${r.n.toLocaleString()}</td></tr>`).join("");
.map(r => `<tr><td>${escapeHtml(r.source)}</td><td style="text-align:right; padding-left:24px"><span class="badge ok">indexed</span></td></tr>`).join("");
document.getElementById("statusTable").innerHTML = data.byStatus
.map(r => `<tr><td>${badgeHtml(r.status === "null" ? null : r.status)}</td><td style="text-align:right; padding-left:24px">${r.n.toLocaleString()}</td></tr>`).join("");
.map(r => `<tr><td><span class="badge ${r.status === "ready" ? "ok" : "pending"}">${escapeHtml(r.status)}</span></td><td style="text-align:right; padding-left:24px">${r.n.toLocaleString()}</td></tr>`).join("");
document.getElementById("rate-ingested").textContent = (data.ingestedPerHour || 0).toLocaleString();
document.getElementById("rate-content").textContent = (data.contentPerHour || 0).toLocaleString();
public/admin/pages/autonomy.html
+159
View File
@@ -0,0 +1,159 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="theme-color" content="#090b0a">
<title>Duriin — Autonomy</title>
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/autonomy.css?v=20260829-1">
</head>
<body class="page-autonomy">
<header class="app-header">
<h1>Duriin</h1>
<nav class="tabs">
<a href="/admin/autonomy" class="active">Autonomy</a>
<a href="/admin/ingest">Ingest</a>
<a href="/admin/intelligence">Intelligence</a>
<a href="/admin/stats">Stats</a>
<a href="/admin/sql">SQL</a>
</nav>
</header>
<main class="autonomy-shell">
<section class="autonomy-hero">
<div class="hero-copy">
<div class="eyebrow"><span class="pulse"></span><span id="runtime-state">Autonomy runtime</span></div>
<h2 id="hero-title">Duriin is observing.</h2>
<p id="hero-description">Building evidence-backed hypotheses from the archive and measuring them against the market.</p>
</div>
<div class="hero-mode">
<span class="mode-label">Execution authority</span>
<strong id="execution-mode">—</strong>
<span id="broker-state">Connecting to broker state…</span>
</div>
</section>
<section class="metric-strip" aria-label="Autonomy summary">
<article class="metric-block">
<span class="metric-kicker">Open hypotheses</span>
<strong id="metric-open">—</strong>
<small id="metric-resolved">— resolved</small>
</article>
<article class="metric-block">
<span class="metric-kicker">Directional accuracy</span>
<strong id="metric-accuracy">—</strong>
<small id="metric-sample">Waiting for outcomes</small>
</article>
<article class="metric-block">
<span class="metric-kicker">Average excess return</span>
<strong id="metric-alpha">—</strong>
<small>Against SPY benchmark</small>
</article>
<article class="metric-block">
<span class="metric-kicker">Tradable universe</span>
<strong id="metric-universe">—</strong>
<small>Verified Alpaca instruments</small>
</article>
</section>
<section class="pipeline-section panel">
<div class="section-head">
<div><span class="section-index">01</span><h3>Learning loop</h3></div>
<span class="freshness" id="freshness">Updating…</span>
</div>
<div class="pipeline" id="pipeline">
<div class="pipeline-step"><span class="step-number">01</span><strong>Observe</strong><small>Archive events</small><b id="pipe-observe">—</b></div>
<div class="pipeline-arrow">→</div>
<div class="pipeline-step"><span class="step-number">02</span><strong>Propose</strong><small>Evidence hypotheses</small><b id="pipe-propose">—</b></div>
<div class="pipeline-arrow">→</div>
<div class="pipeline-step"><span class="step-number">03</span><strong>Measure</strong><small>Market outcomes</small><b id="pipe-measure">—</b></div>
<div class="pipeline-arrow">→</div>
<div class="pipeline-step"><span class="step-number">04</span><strong>Calibrate</strong><small>Empirical confidence</small><b id="pipe-calibrate">—</b></div>
<div class="pipeline-arrow">→</div>
<div class="pipeline-step"><span class="step-number">05</span><strong>Act</strong><small>Policy decisions</small><b id="pipe-act">—</b></div>
</div>
</section>
<div class="autonomy-grid">
<section class="panel hypotheses-panel">
<div class="section-head">
<div><span class="section-index">02</span><h3>Latest hypotheses</h3></div>
<a href="/admin/intelligence/predictions" class="text-link">Legacy intelligence ↗</a>
</div>
<div id="hypothesis-list" class="hypothesis-list"><div class="loading-line"></div></div>
</section>
<aside class="panel performance-panel">
<div class="section-head"><div><span class="section-index">03</span><h3>Measured performance</h3></div></div>
<div class="accuracy-orbit" id="accuracy-orbit">
<div><strong id="orbit-value">—</strong><span>correct</span></div>
</div>
<div class="performance-facts">
<div><span>Resolved</span><strong id="perf-resolved">0</strong></div>
<div><span>Correct</span><strong id="perf-correct">0</strong></div>
<div><span>Calibration cohorts</span><strong id="perf-cohorts">0</strong></div>
</div>
<p class="performance-note" id="performance-note">Duriin will only claim an edge after predictions mature and are measured out of sample.</p>
<div class="performance-facts" id="origin-split"></div>
<p class="performance-note" id="origin-note">Historical backfill and walk-forward replay are shown separately. Neither is evidence of live edge.</p>
</aside>
</div>
<div class="autonomy-grid lower-grid">
<section class="panel ledger-panel">
<div class="section-head"><div><span class="section-index">04</span><h3>Decision ledger</h3></div><span class="mode-pill" id="ledger-mode">Shadow</span></div>
<div class="table-wrap ledger-table">
<table>
<thead><tr><th>Instrument</th><th>Decision</th><th>Direction</th><th>Calibration</th><th>Horizon</th><th>Created</th></tr></thead>
<tbody id="decision-ledger"><tr><td colspan="6" class="empty-state">Waiting for calibrated decisions.</td></tr></tbody>
</table>
</div>
</section>
<aside class="panel runtime-panel">
<div class="section-head"><div><span class="section-index">05</span><h3>Runtime</h3></div></div>
<div class="runtime-list">
<div><span><i class="status-light ok"></i>Coordinator</span><strong id="runtime-coordinator">Active</strong></div>
<div><span><i class="status-light ok"></i>Outcome resolver</span><strong>Active</strong></div>
<div><span><i class="status-light ok"></i>Calibration</span><strong>Active</strong></div>
<div><span><i class="status-light ok"></i>Execution</span><strong id="runtime-execution">Shadow</strong></div>
<div><span><i class="status-light"></i>Historical queue</span><strong id="runtime-queue">—</strong></div>
</div>
<div class="account-card" id="account-card">
<span>Paper account</span>
<strong id="account-equity">Not sampled in shadow mode</strong>
<small id="account-meta">Alpaca Paper connected</small>
</div>
</aside>
</div>
<section class="panel" aria-label="Historical replay">
<div class="section-head"><div><span class="section-index">06</span><h3>Historical replay</h3></div><span class="mode-pill">Isolated from execution</span></div>
<div class="performance-facts">
<div><span>Run state</span><strong id="replay-status">Starting…</strong></div>
<div><span>Articles replayed</span><strong id="replay-articles">—</strong></div>
<div><span>Walk-forward evaluations</span><strong id="replay-evaluations">—</strong></div>
<div><span>Watermark</span><strong id="replay-watermark">—</strong></div>
</div>
</section>
<section class="panel" aria-label="Calibration cohorts">
<div class="section-head"><div><span class="section-index">07</span><h3>Calibration cohorts</h3></div><span class="mode-pill">Needs 30 samples · 5 tickers · max 50% in one</span></div>
<div class="table-wrap">
<table>
<thead><tr><th>Cohort</th><th>Source</th><th>Samples</th><th>Distinct tickers</th><th>Top ticker share</th><th>Status</th></tr></thead>
<tbody id="cohort-list"><tr><td colspan="6" class="empty-state">No calibration snapshots yet.</td></tr></tbody>
</table>
</div>
</section>
</main>
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/app.js?v=20260804-3"></script>
<script src="/admin/assets/js/autonomy.js?v=20260829-1"></script>
</body>
</html>
public/admin/pages/ingest/articles.html
+6 -6
View File
@@ -4,9 +4,9 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Duriin Admin — Articles</title>
<link rel="stylesheet" href="/admin/assets/css/base.css">
<link rel="stylesheet" href="/admin/assets/css/layout.css">
<link rel="stylesheet" href="/admin/assets/css/components.css">
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-3">
</head>
<body>
@@ -27,7 +27,7 @@
<div class="stats-bar" id="statsBar">
<div class="stat"><span class="label">Total articles</span><span class="value" id="s-total">—</span></div>
<div class="stat"><span class="label">With content</span><span class="value" id="s-content">—</span></div>
<div class="stat"><span class="label">Intelligence ready</span><span class="value" id="s-content">—</span></div>
<div class="stat"><span class="label">With embedding</span><span class="value" id="s-embed">—</span></div>
<div class="stat"><span class="label">Events</span><span class="value" id="s-events">—</span></div>
</div>
@@ -134,7 +134,7 @@
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/app.js"></script>
<script src="/admin/assets/js/articles.js"></script>
<script src="/admin/assets/js/app.js?v=20260804-3"></script>
<script src="/admin/assets/js/articles.js?v=20260804-3"></script>
</body>
</html>
public/admin/pages/ingest/events.html
+6 -6
View File
@@ -4,9 +4,9 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Duriin Admin — Events</title>
<link rel="stylesheet" href="/admin/assets/css/base.css">
<link rel="stylesheet" href="/admin/assets/css/layout.css">
<link rel="stylesheet" href="/admin/assets/css/components.css">
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-3">
</head>
<body>
@@ -27,7 +27,7 @@
<div class="stats-bar" id="statsBar">
<div class="stat"><span class="label">Total articles</span><span class="value" id="s-total">—</span></div>
<div class="stat"><span class="label">With content</span><span class="value" id="s-content">—</span></div>
<div class="stat"><span class="label">Intelligence ready</span><span class="value" id="s-content">—</span></div>
<div class="stat"><span class="label">With embedding</span><span class="value" id="s-embed">—</span></div>
<div class="stat"><span class="label">Events</span><span class="value" id="s-events">—</span></div>
</div>
@@ -94,7 +94,7 @@
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/app.js"></script>
<script src="/admin/assets/js/events.js"></script>
<script src="/admin/assets/js/app.js?v=20260804-3"></script>
<script src="/admin/assets/js/events.js?v=20260804-3"></script>
</body>
</html>
public/admin/pages/intelligence/graph.html
+8 -8
View File
@@ -4,10 +4,10 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Duriin Admin — Intelligence / Graph</title>
<link rel="stylesheet" href="/admin/assets/css/base.css">
<link rel="stylesheet" href="/admin/assets/css/layout.css">
<link rel="stylesheet" href="/admin/assets/css/components.css">
<link rel="stylesheet" href="/admin/assets/css/intel.css">
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/intel.css?v=20260804-3">
</head>
<body class="page-graph">
@@ -72,9 +72,9 @@
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/d3.min.js"></script>
<script src="/admin/assets/js/app.js"></script>
<script src="/admin/assets/js/intel-shared.js"></script>
<script src="/admin/assets/js/intel-graph.js"></script>
<script src="/admin/assets/js/d3.min.js?v=20260804-3"></script>
<script src="/admin/assets/js/app.js?v=20260804-3"></script>
<script src="/admin/assets/js/intel-shared.js?v=20260804-3"></script>
<script src="/admin/assets/js/intel-graph.js?v=20260804-3"></script>
</body>
</html>
public/admin/pages/intelligence/knowledge.html
+7 -7
View File
@@ -4,10 +4,10 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Duriin Admin — Intelligence / Knowledge</title>
<link rel="stylesheet" href="/admin/assets/css/base.css">
<link rel="stylesheet" href="/admin/assets/css/layout.css">
<link rel="stylesheet" href="/admin/assets/css/components.css">
<link rel="stylesheet" href="/admin/assets/css/intel.css">
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/intel.css?v=20260804-3">
</head>
<body>
@@ -90,8 +90,8 @@
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/app.js"></script>
<script src="/admin/assets/js/intel-shared.js"></script>
<script src="/admin/assets/js/intel-knowledge.js"></script>
<script src="/admin/assets/js/app.js?v=20260804-3"></script>
<script src="/admin/assets/js/intel-shared.js?v=20260804-3"></script>
<script src="/admin/assets/js/intel-knowledge.js?v=20260804-3"></script>
</body>
</html>
public/admin/pages/intelligence/predictions.html
+7 -7
View File
@@ -4,10 +4,10 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Duriin Admin — Intelligence / Predictions</title>
<link rel="stylesheet" href="/admin/assets/css/base.css">
<link rel="stylesheet" href="/admin/assets/css/layout.css">
<link rel="stylesheet" href="/admin/assets/css/components.css">
<link rel="stylesheet" href="/admin/assets/css/intel.css">
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/intel.css?v=20260804-3">
</head>
<body>
@@ -81,8 +81,8 @@
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/app.js"></script>
<script src="/admin/assets/js/intel-shared.js"></script>
<script src="/admin/assets/js/intel-predictions.js"></script>
<script src="/admin/assets/js/app.js?v=20260804-3"></script>
<script src="/admin/assets/js/intel-shared.js?v=20260804-3"></script>
<script src="/admin/assets/js/intel-predictions.js?v=20260804-3"></script>
</body>
</html>
public/admin/pages/intelligence/signals.html
+7 -7
View File
@@ -4,10 +4,10 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Duriin Admin — Intelligence / Signals</title>
<link rel="stylesheet" href="/admin/assets/css/base.css">
<link rel="stylesheet" href="/admin/assets/css/layout.css">
<link rel="stylesheet" href="/admin/assets/css/components.css">
<link rel="stylesheet" href="/admin/assets/css/intel.css">
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/intel.css?v=20260804-3">
</head>
<body>
@@ -137,8 +137,8 @@
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/app.js"></script>
<script src="/admin/assets/js/intel-shared.js"></script>
<script src="/admin/assets/js/intel-signals.js"></script>
<script src="/admin/assets/js/app.js?v=20260804-3"></script>
<script src="/admin/assets/js/intel-shared.js?v=20260804-3"></script>
<script src="/admin/assets/js/intel-signals.js?v=20260804-3"></script>
</body>
</html>
public/admin/pages/sql.html
+5 -5
View File
@@ -4,9 +4,9 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Duriin Admin — SQL</title>
<link rel="stylesheet" href="/admin/assets/css/base.css">
<link rel="stylesheet" href="/admin/assets/css/layout.css">
<link rel="stylesheet" href="/admin/assets/css/components.css">
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-3">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-3">
</head>
<body>
@@ -45,7 +45,7 @@
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/app.js"></script>
<script src="/admin/assets/js/sql.js"></script>
<script src="/admin/assets/js/app.js?v=20260804-3"></script>
<script src="/admin/assets/js/sql.js?v=20260804-3"></script>
</body>
</html>
public/admin/pages/stats.html
+10 -10
View File
@@ -4,10 +4,10 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Duriin Admin — Stats</title>
<link rel="stylesheet" href="/admin/assets/css/base.css">
<link rel="stylesheet" href="/admin/assets/css/layout.css">
<link rel="stylesheet" href="/admin/assets/css/components.css">
<link rel="stylesheet" href="/admin/assets/css/intel.css">
<link rel="stylesheet" href="/admin/assets/css/base.css?v=20260804-5">
<link rel="stylesheet" href="/admin/assets/css/layout.css?v=20260804-5">
<link rel="stylesheet" href="/admin/assets/css/components.css?v=20260804-5">
<link rel="stylesheet" href="/admin/assets/css/intel.css?v=20260804-5">
</head>
<body>
@@ -23,7 +23,7 @@
<div class="stats-bar" id="statsBar">
<div class="stat"><span class="label">Total articles</span><span class="value" id="s-total">—</span></div>
<div class="stat"><span class="label">With content</span><span class="value" id="s-content">—</span></div>
<div class="stat"><span class="label">Intelligence ready</span><span class="value" id="s-content">—</span></div>
<div class="stat"><span class="label">With embedding</span><span class="value" id="s-embed">—</span></div>
<div class="stat"><span class="label">Events</span><span class="value" id="s-events">—</span></div>
</div>
@@ -50,17 +50,17 @@
<div style="display:flex; gap:32px; flex-wrap:wrap; padding-top:4px">
<div>
<div class="section-heading">By source</div>
<div class="section-heading">Known sources</div>
<div class="table-wrap" style="width:auto; min-width:220px">
<table style="width:auto">
<thead><tr><th>Source</th><th style="text-align:right">Count</th></tr></thead>
<thead><tr><th>Source</th><th style="text-align:right">State</th></tr></thead>
<tbody id="sourceTable"></tbody>
</table>
</div>
</div>
<div>
<div class="section-heading">By content status</div>
<div class="section-heading">By archive readiness</div>
<div class="table-wrap" style="width:auto; min-width:180px">
<table style="width:auto">
<thead><tr><th>Status</th><th style="text-align:right">Count</th></tr></thead>
@@ -74,7 +74,7 @@
<div id="toast"><span class="toast-dot"></span><span id="toast-msg"></span></div>
<script src="/admin/assets/js/app.js"></script>
<script src="/admin/assets/js/stats.js"></script>
<script src="/admin/assets/js/app.js?v=20260804-5"></script>
<script src="/admin/assets/js/stats.js?v=20260804-5"></script>
</body>
</html>
scripts/analyze-reaction-conditioning.js
+184
View File
@@ -0,0 +1,184 @@
#!/usr/bin/env node
/*
* Lever 1: does conditioning on the initial market reaction reveal any
* discrimination in this pipeline?
*
* The coordinator is scored on excess return vs SPY starting at the information
* cutoff, so the announcement move itself sits OUTSIDE the scored window. That
* move is the best documented conditioner for post event drift, and we throw it
* away. This measures whether it is worth keeping.
*
* reaction = instrument excess vs SPY from the last close BEFORE the event's
* first article, up to the outcome's own entry price at the cutoff.
* forward = the already stored excess_return over the horizon.
* The two windows touch but never overlap, so there is no lookahead.
*
* Read only. Opens both databases read only and writes nothing but a price cache.
*
* PRE REGISTERED TESTS (declared before looking, so the buckets cannot be tuned):
* T1 does the reaction bucket predict the SIGN of forward excess return?
* (is there drift/reversal in this sample at all, model aside)
* T2 within a bucket, does the coordinator's direction discriminate?
* (does the model add anything on top of T1)
* Everything else printed is descriptive, not a test.
*/
const fs = require('fs');
const path = require('path');
const Database = require('better-sqlite3');
const ARCHIVE = process.env.DURIIN_DB || '/data/archive.sqlite';
const INTELLIGENCE = process.env.INTELLIGENCE_DB || '/data/intelligence.sqlite';
const CACHE = process.env.PRICE_CACHE || '/data/.price-cache';
const BENCHMARK = 'SPY';
// fixed bands on the reaction, declared up front. not quantiles, so the cut
// points cannot drift with the data.
const BANDS = [
{ key: 'strong_down', min: -Infinity, max: -0.05 },
{ key: 'down', min: -0.05, max: -0.015 },
{ key: 'flat', min: -0.015, max: 0.015 },
{ key: 'up', min: 0.015, max: 0.05 },
{ key: 'strong_up', min: 0.05, max: Infinity },
];
function band(reaction) {
return (BANDS.find((b) => reaction >= b.min && reaction < b.max) || { key: 'unknown' }).key;
}
function sleep(ms) { return new Promise((r) => setTimeout(r, ms)); }
async function fetchHistory(symbol) {
fs.mkdirSync(CACHE, { recursive: true });
const file = path.join(CACHE, `${symbol.replace(/[^A-Za-z0-9._-]/g, '_')}.json`);
if (fs.existsSync(file)) {
try { return JSON.parse(fs.readFileSync(file, 'utf8')); }
catch (error) { console.error(`[reaction] bad cache for ${symbol}, refetching:`, error.message); }
}
const url = `https://query1.finance.yahoo.com/v8/finance/chart/${encodeURIComponent(symbol)}`
+ `?period1=946684800&period2=${Math.floor(Date.now() / 1000)}&interval=1d`;
let rows = [];
try {
const response = await fetch(url, { headers: { 'User-Agent': 'Mozilla/5.0' } });
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const result = (await response.json())?.chart?.result?.[0];
rows = (result?.timestamp || []).map((ts, i) => ({
date: new Date(ts * 1000).toISOString().slice(0, 10),
close: result.indicators?.quote?.[0]?.close?.[i],
})).filter((r) => Number.isFinite(r.close));
} catch (error) {
console.error(`[reaction] history fetch failed for ${symbol}:`, error.message);
}
fs.writeFileSync(file, JSON.stringify(rows));
await sleep(250);
return rows;
}
const onOrAfter = (h, d) => h.find((r) => r.date >= d)?.close ?? null;
const strictlyBefore = (h, d) => { for (let i = h.length - 1; i >= 0; i--) if (h[i].date < d) return h[i].close; return null; };
// two proportion z test. returns z and a normal approx two sided p.
function ztest(x1, n1, x2, n2) {
if (!n1 || !n2) return null;
const p1 = x1 / n1, p2 = x2 / n2, p = (x1 + x2) / (n1 + n2);
const se = Math.sqrt(p * (1 - p) * (1 / n1 + 1 / n2));
if (!se) return null;
const z = (p1 - p2) / se;
// logistic approx to the normal cdf. has to be fed |z|, otherwise the two
// sided p comes back above 1 for negative z, which is nonsense.
const cdf = (v) => 1 / (1 + Math.exp(-0.07056 * v ** 3 - 1.5976 * v));
const pValue = Math.min(1, 2 * (1 - cdf(Math.abs(z))));
return { z, p: pValue, p1, p2 };
}
const pct = (x) => (x === null || x === undefined || Number.isNaN(x) ? ' n/a ' : `${(100 * x).toFixed(1)}%`);
async function main() {
const intel = new Database(INTELLIGENCE, { readonly: true });
const archive = new Database(ARCHIVE, { readonly: true });
const rows = intel.prepare(`
SELECT p.id, p.instrument, p.direction, p.event_id, p.information_cutoff, p.horizon_days, p.origin,
o.excess_return, o.direction_correct, o.price_0, o.benchmark_0
FROM autonomy_predictions p
JOIN autonomy_outcomes o ON o.prediction_id = p.id
WHERE o.excess_return IS NOT NULL
`).all();
console.log(`[reaction] matured outcomes: ${rows.length}`);
const eventStart = archive.prepare('SELECT MIN(pub_date_effective) AS first FROM articles WHERE event_id = ?');
const symbols = [...new Set(rows.map((r) => r.instrument))];
console.log(`[reaction] distinct instruments: ${symbols.length}, fetching history (cached)...`);
const hist = new Map();
for (const s of [BENCHMARK, ...symbols]) hist.set(s, await fetchHistory(s));
const bench = hist.get(BENCHMARK);
const scored = [];
let skipped = 0;
for (const r of rows) {
const first = r.event_id ? eventStart.get(r.event_id)?.first : null;
if (!first) { skipped++; continue; }
const startDay = String(first).slice(0, 10);
const cutoffDay = String(r.information_cutoff).slice(0, 10);
const h = hist.get(r.instrument) || [];
// baseline is the last close strictly before the first article, so the whole
// announcement move is inside the reaction window
const pPre = strictlyBefore(h, startDay);
const bPre = strictlyBefore(bench, startDay);
const pAt = Number.isFinite(r.price_0) ? r.price_0 : onOrAfter(h, cutoffDay);
const bAt = Number.isFinite(r.benchmark_0) ? r.benchmark_0 : onOrAfter(bench, cutoffDay);
if (![pPre, bPre, pAt, bAt].every(Number.isFinite) || !pPre || !bPre) { skipped++; continue; }
const reaction = (pAt - pPre) / pPre - (bAt - bPre) / bPre;
if (!Number.isFinite(reaction)) { skipped++; continue; }
scored.push({ ...r, reaction, bucket: band(reaction), fwdUp: r.excess_return > 0 ? 1 : 0 });
}
console.log(`[reaction] scored: ${scored.length}, skipped for missing prices/dates: ${skipped}\n`);
if (!scored.length) return;
const report = (label, set) => {
if (!set.length) return;
console.log(`\n================ ${label} (n=${set.length}) ================`);
console.log('bucket n fwd_up% model_acc% mean_fwd% P(up|pred+) P(up|pred-) z p');
for (const b of BANDS) {
const g = set.filter((r) => r.bucket === b.key);
if (!g.length) { console.log(`${b.key.padEnd(12)} 0 - - - - - - -`); continue; }
const pos = g.filter((r) => r.direction === 'positive');
const neg = g.filter((r) => r.direction === 'negative');
const t = ztest(pos.filter((r) => r.fwdUp).length, pos.length, neg.filter((r) => r.fwdUp).length, neg.length);
const acc = g.filter((r) => r.direction_correct).length / g.length;
const meanFwd = g.reduce((s, r) => s + r.excess_return, 0) / g.length;
console.log(
`${b.key.padEnd(12)} ${String(g.length).padStart(4)} ${pct(g.filter((r) => r.fwdUp).length / g.length)} ${pct(acc)} `
+ `${(100 * meanFwd).toFixed(2).padStart(6)}% ${pct(t ? t.p1 : null)} ${pct(t ? t.p2 : null)} `
+ `${t ? t.z.toFixed(2).padStart(6) : ' n/a'} ${t ? t.p.toFixed(3) : ' n/a'}`
);
}
// T1: does the reaction itself predict the forward sign?
const upSide = set.filter((r) => r.bucket === 'up' || r.bucket === 'strong_up');
const downSide = set.filter((r) => r.bucket === 'down' || r.bucket === 'strong_down');
const t1 = ztest(upSide.filter((r) => r.fwdUp).length, upSide.length, downSide.filter((r) => r.fwdUp).length, downSide.length);
console.log(`\n[T1] forward-up rate after a POSITIVE reaction vs after a NEGATIVE reaction`);
if (t1) {
console.log(` ${pct(t1.p1)} (n=${upSide.length}) vs ${pct(t1.p2)} (n=${downSide.length}) z=${t1.z.toFixed(2)} p=${t1.p.toFixed(3)}`);
console.log(` ${Math.abs(t1.z) >= 1.96 ? (t1.z > 0 ? '=> CONTINUATION (drift) is present' : '=> REVERSAL is present') : '=> no drift or reversal detectable'}`);
} else console.log(' insufficient data');
// T2: pooled model discrimination within buckets, vs unconditional
const allPos = set.filter((r) => r.direction === 'positive');
const allNeg = set.filter((r) => r.direction === 'negative');
const t2 = ztest(allPos.filter((r) => r.fwdUp).length, allPos.length, allNeg.filter((r) => r.fwdUp).length, allNeg.length);
console.log(`\n[T2] unconditional model discrimination P(up|pred+) - P(up|pred-)`);
if (t2) console.log(` ${pct(t2.p1)} vs ${pct(t2.p2)} z=${t2.z.toFixed(2)} p=${t2.p.toFixed(3)}`
+ ` ${Math.abs(t2.z) >= 1.96 ? '=> SIGNIFICANT' : '=> not distinguishable from zero'}`);
else console.log(' insufficient data');
};
report('ALL', scored);
const topTicker = [...scored.reduce((m, r) => m.set(r.instrument, (m.get(r.instrument) || 0) + 1), new Map())]
.sort((a, b) => b[1] - a[1])[0];
console.log(`\n\n(most common instrument: ${topTicker[0]} with ${topTicker[1]} of ${scored.length})`);
report(`EXCLUDING ${topTicker[0]}`, scored.filter((r) => r.instrument !== topTicker[0]));
console.log('\n[reaction] reminder: T1 and T2 were pre-registered. Per-bucket rows are');
console.log('[reaction] descriptive only, do not read a single bucket as a finding.');
}
main().catch((error) => { console.error('[reaction] fatal:', error.message, error.stack); process.exit(1); });
scripts/build-feedback-brief.js
+170
View File
@@ -0,0 +1,170 @@
#!/usr/bin/env node
/*
* Turn a replay run's own scored outcomes into a memo the next run is told
* before it predicts anything.
*
* This is the piece that was missing. The generator has never once seen its own
* results: nothing in coordinatorWorker, replayWorker, llm.js or graphContext
* reads autonomy_outcomes. Calibration reads them, but calibration only decides
* whether to ACT on a prediction, it never changes what gets predicted. So the
* only thing that has ever altered this system's output is a human editing the
* prompt. A memo generated from the data is not a human editing the prompt.
*
* Everything here is computed from a TRAIN slice bounded by --until. Nothing
* from the evaluation window may appear in the text or the comparison is just
* fitting to the answer sheet.
*
* node scripts/build-feedback-brief.js --run 1 --until "2026-09-04 19:30"
*/
const Database = require("better-sqlite3");
const INTELLIGENCE = process.env.INTELLIGENCE_DB || "/data/intelligence.sqlite";
// When duriin-api-replay-1 restarted onto the prompt it runs today. Not the
// commit timestamp, which is five minutes later and would have been wrong.
// Replay was between daily budgets across the restart, so there is an eight
// hour hole in predictions around it and every candidate split inside that
// hole partitions the data identically.
const SPLIT = "2026-09-04 19:17:43";
function pct(x, digits = 1) { return `${(x * 100).toFixed(digits)}%`; }
function erf(x) {
const sign = x < 0 ? -1 : 1;
const z = Math.abs(x);
const t = 1 / (1 + 0.3275911 * z);
const y = 1 - ((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-z * z);
return sign * y;
}
function twoSided(z) { return 2 * (1 - 0.5 * (1 + erf(Math.abs(z) / Math.SQRT2))); }
function loadTrain(db, { runId, createdBefore }) {
return db.prepare(`
SELECT p.direction, p.event_type, p.horizon_days, p.instrument,
o.direction_correct, o.excess_return
FROM autonomy_predictions p
JOIN autonomy_outcomes o ON o.prediction_id = p.id
WHERE p.origin = 'replay' AND p.replay_run_id = ? AND p.created_at < ?
`).all(runId, createdBefore);
}
// families and horizons that sit far enough below the constant to be worth
// naming. n floor keeps a handful of unlucky calls out of the memo.
function weakSlices(rows, key, { minimum = 40, factor }) {
const groups = new Map();
for (const row of rows) {
const k = String(row[key]);
if (!groups.has(k)) groups.set(k, []);
groups.get(k).push(row);
}
const kept = [...groups.entries()].filter(([, v]) => v.length >= minimum);
const adjust = factor || kept.length || 1;
return kept.map(([k, v]) => {
const hits = v.filter((r) => r.direction_correct).length;
const acc = hits / v.length;
const down = v.filter((r) => r.excess_return <= 0).length / v.length;
const bar = Math.max(down, 1 - down);
const se = Math.sqrt(bar * (1 - bar) / v.length);
const z = se > 0 ? (acc - bar) / se : 0;
return { key: k, n: v.length, acc, bar, p: Math.min(1, twoSided(z) * adjust) };
}).sort((a, b) => a.acc - b.acc);
}
function buildFeedbackBrief(db, { runId, createdBefore }) {
const rows = loadTrain(db, { runId, createdBefore });
if (rows.length < 200) {
throw new Error(`only ${rows.length} scored training predictions for run ${runId}, refusing to write a brief off that`);
}
const n = rows.length;
const positives = rows.filter((r) => r.direction === "positive");
const positiveShare = positives.length / n;
const actuallyUp = rows.filter((r) => r.excess_return > 0).length / n;
const acc = rows.filter((r) => r.direction_correct).length / n;
const alwaysNegative = 1 - actuallyUp;
const upGivenPositive = positives.filter((r) => r.excess_return > 0).length / (positives.length || 1);
const negatives = rows.filter((r) => r.direction === "negative");
const upGivenNegative = negatives.filter((r) => r.excess_return > 0).length / (negatives.length || 1);
const families = weakSlices(rows, "event_type", { minimum: 40 });
const horizons = weakSlices(rows, "horizon_days", { minimum: 40 });
const weakFamilies = families.filter((f) => f.acc < f.bar - 0.08).slice(0, 5);
const strongFamilies = [...families].reverse().filter((f) => f.acc > f.bar + 0.05).slice(0, 4);
const weakHorizons = horizons.filter((h) => h.acc < h.bar - 0.08).slice(0, 3);
const lines = [];
lines.push(`CALIBRATION FEEDBACK. The following comes from ${n} of your own earlier predictions on this`);
lines.push(`archive, every one of them scored on realised excess return against SPY. It describes how you`);
lines.push(`have actually performed, not how you think you perform. Use it.`);
lines.push("");
lines.push(`1. Your directional prior is wrong. You said "positive" on ${pct(positiveShare)} of those predictions.`);
lines.push(` Only ${pct(actuallyUp)} of the same bars actually beat SPY. Most individual names underperform a`);
lines.push(` cap weighted index over any horizon, so "positive" is the minority answer, not the default.`);
lines.push(` You were right ${pct(acc)} of the time. Answering "negative" to every single one of those bars`);
lines.push(` would have scored ${pct(alwaysNegative)}. You are currently below a constant.`);
lines.push("");
lines.push(`2. Beating SPY is the bar, not the company doing well. Good news that the index already had`);
lines.push(` priced, or that lifts the whole sector, is not a positive excess return. Ask whether this name`);
lines.push(` outperforms the market, never whether the story is upbeat.`);
lines.push("");
// a spread under two points is not worth telling it to keep, it would just be
// flattering noise back at itself
if (upGivenPositive - upGivenNegative >= 0.02) {
lines.push(`3. Your instinct on WHICH way is weakly right: bars you called positive beat SPY ${pct(upGivenPositive)}`);
lines.push(` of the time versus ${pct(upGivenNegative)} for the ones you called negative. That separation is small`);
lines.push(` enough that it could still be luck, and it is buried by how often you default to positive.`);
} else {
lines.push(`3. Your choice of direction carries no information yet: bars you called positive beat SPY`);
lines.push(` ${pct(upGivenPositive)} of the time versus ${pct(upGivenNegative)} for the ones you called negative. Only predict when`);
lines.push(` the evidence gives you a genuine mechanism, and return nothing otherwise.`);
}
lines.push("");
if (weakFamilies.length) {
lines.push(`4. Event families you read worst, accuracy against the constant on the same bars:`);
for (const f of weakFamilies) {
lines.push(` ${f.key}: you ${pct(f.acc)}, constant ${pct(f.bar)}, n=${f.n}`);
}
lines.push(` On these, prefer returning an empty predictions array over a weak call.`);
lines.push("");
}
if (strongFamilies.length) {
lines.push(`5. Event families you read best, where a confident call is warranted:`);
for (const f of strongFamilies) {
lines.push(` ${f.key}: you ${pct(f.acc)}, constant ${pct(f.bar)}, n=${f.n}`);
}
lines.push("");
}
if (weakHorizons.length) {
lines.push(`6. Horizons that went worst for you: ${weakHorizons.map((h) => `${h.key}d (${pct(h.acc)}, n=${h.n})`).join(", ")}.`);
lines.push(` The longer the horizon the more of the move is market and sector rather than the event.`);
lines.push("");
}
lines.push(`None of this tells you what to answer for the evidence below. It tells you which of your habits`);
lines.push(`have already cost you. An empty predictions array is always available and costs nothing.`);
return { text: lines.join("\n"), stats: { n, positiveShare, actuallyUp, acc, alwaysNegative,
upGivenPositive, upGivenNegative, weakFamilies, strongFamilies, weakHorizons } };
}
function main() {
const argv = process.argv.slice(2);
const opts = {};
for (let i = 0; i < argv.length; i += 1) {
if (!argv[i].startsWith("--")) continue;
const key = argv[i].slice(2);
const next = argv[i + 1];
opts[key] = (next && !next.startsWith("--")) ? (i += 1, next) : true;
}
const db = new Database(INTELLIGENCE, { readonly: true });
db.pragma("busy_timeout = 20000");
const { text, stats } = buildFeedbackBrief(db, {
runId: Number(opts.run || 1),
createdBefore: String(opts.until || SPLIT),
});
console.log(text);
console.log(`\n--- derived from ${stats.n} scored training predictions ---`);
db.close();
}
if (require.main === module) main();
module.exports = { buildFeedbackBrief };
scripts/import-legacy-intelligence.js
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env node
const path = require('path');
const Database = require('better-sqlite3');
const { initAutonomySchema } = require('../src/autonomy/schema');
const sourcePath = process.env.LEGACY_INTELLIGENCE_DB || path.resolve(process.cwd(), 'intelligence.sqlite');
const db = new Database(sourcePath);
db.pragma('journal_mode = WAL');
initAutonomySchema(db);
const insert = db.prepare(`
INSERT OR IGNORE INTO autonomy_legacy_records(source_table, source_id, payload)
VALUES (?, ?, ?)
`);
const tx = db.transaction(() => {
for (const table of ['event_predictions', 'trade_signals', 'company_facts', 'company_relationships']) {
const exists = db.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?").get(table);
if (!exists) continue;
const rows = db.prepare(`SELECT * FROM ${table}`).all();
for (const row of rows) insert.run(table, row.id, JSON.stringify(row));
console.log(`${table}: ${rows.length}`);
}
});
tx();
db.close();
scripts/initialize-autonomy.js
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env node
// Initializes the additive autonomy schema and creates one reconciliation job.
// It deliberately does not rewrite legacy intelligence or enqueue millions of
// historical records; reconciliation workers discover those in bounded batches.
const path = require('path');
const Database = require('better-sqlite3');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { enqueueJob } = require('../src/autonomy/jobs');
const intelligencePath = process.env.INTELLIGENCE_DB || path.resolve(process.cwd(), 'intelligence.sqlite');
const db = new Database(intelligencePath);
db.pragma('journal_mode = WAL');
initAutonomySchema(db);
const result = enqueueJob(db, {
jobType: 'reconcile_archive',
lane: 'maintenance',
priority: 100,
entityType: 'archive',
entityId: 'archive',
idempotencyKey: 'reconcile_archive:v1',
});
console.log(JSON.stringify({ intelligencePath, reconciliationJobInserted: result.inserted }));
db.close();
scripts/migrate-sqlite-to-postgres.js
+122
View File
@@ -0,0 +1,122 @@
#!/usr/bin/env node
/*
* Resumable logical migration for Duriin's two SQLite databases.
*
* It deliberately copies tables, not SQLite files: PostgreSQL receives usable
* relational data in `archive` and `intelligence` schemas. SQLite vec0
* implementation tables are indexes, not source-of-truth data; the canonical
* article_embedding_store is copied and can be used to rebuild pgvector later.
*/
const Database = require('better-sqlite3');
const { Pool } = require('pg');
const archivePath = process.env.SQLITE_ARCHIVE_PATH || '/data/archive.sqlite';
const intelligencePath = process.env.SQLITE_INTELLIGENCE_PATH || '/data/intelligence.sqlite';
const connectionString = process.env.DATABASE_URL;
if (!connectionString) throw new Error('DATABASE_URL is required');
const pool = new Pool({ connectionString, max: 2 });
const BATCH_SIZE = Math.max(1, Number(process.env.POSTGRES_MIGRATION_BATCH_SIZE) || 200);
const DERIVED_SQLITE_TABLES = new Set([
'article_embeddings', 'article_embeddings_chunks', 'article_embeddings_info',
'article_embeddings_rowids', 'article_embeddings_vector_chunks00',
]);
const NUL_BYTE = '\u0000';
const NUL_BYTES = /\u0000/g;
let sanitizedTextValues = 0;
function quote(name) { return `"${String(name).replaceAll('"', '""')}"`; }
function pgType(sqliteType) {
const type = String(sqliteType || '').toUpperCase();
if (type.includes('INT')) return 'BIGINT';
if (type.includes('REAL') || type.includes('FLOA') || type.includes('DOUB')) return 'DOUBLE PRECISION';
if (type.includes('BLOB')) return 'BYTEA';
return 'TEXT';
}
function normalizeValue(value) {
if (typeof value !== 'string' || !value.includes(NUL_BYTE)) return value ?? null;
sanitizedTextValues += 1;
return value.replace(NUL_BYTES, '');
}
async function ensureSchema(client, schema) {
await client.query(`CREATE SCHEMA IF NOT EXISTS ${quote(schema)}`);
await client.query(`CREATE TABLE IF NOT EXISTS ${quote(schema)}.${quote('_migration_progress')} (
table_name TEXT PRIMARY KEY, source_rows BIGINT NOT NULL, copied_rows BIGINT NOT NULL,
completed_at TIMESTAMPTZ, updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
)`);
}
function sourceTables(sqlite) {
return sqlite.prepare(`SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name`).all()
.map((row) => row.name).filter((name) => !DERIVED_SQLITE_TABLES.has(name));
}
async function createTable(client, schema, sqlite, table) {
const columns = sqlite.prepare(`PRAGMA table_info(${quote(table)})`).all();
if (!columns.length) return null;
const primary = columns.filter((column) => column.pk).sort((a, b) => a.pk - b.pk).map((column) => quote(column.name));
const definitions = columns.map((column) => `${quote(column.name)} ${pgType(column.type)}${column.notnull ? ' NOT NULL' : ''}`);
if (primary.length) definitions.push(`PRIMARY KEY (${primary.join(', ')})`);
await client.query(`CREATE TABLE IF NOT EXISTS ${quote(schema)}.${quote(table)} (${definitions.join(', ')})`);
return columns;
}
async function copyTable(client, schema, sqlite, table, columns) {
const sourceRows = Number(sqlite.prepare(`SELECT COUNT(*) AS count FROM ${quote(table)}`).get().count);
const progress = await client.query(`SELECT copied_rows, source_rows, completed_at FROM ${quote(schema)}.${quote('_migration_progress')} WHERE table_name=$1`, [table]);
const prior = progress.rows[0];
if (prior?.completed_at && Number(prior.source_rows) === sourceRows) {
console.log(`[migrate] ${schema}.${table}: already verified (${sourceRows})`);
return;
}
// Tables with an integer primary key are copied by key. Other tables use a
// deterministic row offset and are still restartable at batch boundaries.
const key = columns.find((column) => column.pk === 1 && /INT/i.test(column.type));
let cursor = key ? Number(prior?.copied_rows || 0) : 0;
const names = columns.map((column) => column.name);
const insert = `INSERT INTO ${quote(schema)}.${quote(table)} (${names.map(quote).join(', ')}) VALUES (${names.map((_, i) => `$${i + 1}`).join(', ')}) ON CONFLICT DO NOTHING`;
console.log(`[migrate] ${schema}.${table}: ${sourceRows} source rows`);
while (true) {
const rows = key
? sqlite.prepare(`SELECT ${names.map(quote).join(', ')} FROM ${quote(table)} WHERE ${quote(key.name)} > ? ORDER BY ${quote(key.name)} LIMIT ?`).all(cursor, BATCH_SIZE)
: sqlite.prepare(`SELECT ${names.map(quote).join(', ')} FROM ${quote(table)} LIMIT ? OFFSET ?`).all(BATCH_SIZE, cursor);
if (!rows.length) break;
await client.query('BEGIN');
try {
for (const row of rows) await client.query(insert, names.map((name) => normalizeValue(row[name])));
cursor = key ? Number(rows[rows.length - 1][key.name]) : cursor + rows.length;
await client.query(`INSERT INTO ${quote(schema)}.${quote('_migration_progress')} (table_name, source_rows, copied_rows, updated_at)
VALUES ($1,$2,$3,NOW()) ON CONFLICT (table_name) DO UPDATE SET source_rows=EXCLUDED.source_rows, copied_rows=EXCLUDED.copied_rows, updated_at=NOW()`, [table, sourceRows, cursor]);
await client.query('COMMIT');
} catch (error) { await client.query('ROLLBACK'); throw error; }
process.stdout.write(`\r[migrate] ${schema}.${table}: ${Math.min(cursor, sourceRows)}/${sourceRows}`);
}
const target = await client.query(`SELECT COUNT(*)::bigint AS count FROM ${quote(schema)}.${quote(table)}`);
if (Number(target.rows[0].count) < sourceRows) throw new Error(`${schema}.${table}: target count is short`);
await client.query(`UPDATE ${quote(schema)}.${quote('_migration_progress')} SET completed_at=NOW(), source_rows=$2, copied_rows=$2 WHERE table_name=$1`, [table, sourceRows]);
console.log(`\r[migrate] ${schema}.${table}: verified ${sourceRows}`);
}
async function migrateDatabase(client, schema, file) {
const sqlite = new Database(file, { readonly: true });
try {
await ensureSchema(client, schema);
for (const table of sourceTables(sqlite)) {
const columns = await createTable(client, schema, sqlite, table);
if (columns) await copyTable(client, schema, sqlite, table, columns);
}
} finally { sqlite.close(); }
}
(async () => {
const client = await pool.connect();
try {
await client.query('CREATE EXTENSION IF NOT EXISTS vector');
await migrateDatabase(client, 'archive', archivePath);
await migrateDatabase(client, 'intelligence', intelligencePath);
console.log('[migrate] SQLite logical data verified in PostgreSQL. SQLite remains the live source until application cutover.');
if (sanitizedTextValues) console.log(`[migrate] sanitized ${sanitizedTextValues} text values containing NUL bytes rejected by PostgreSQL text columns.`);
} finally { client.release(); await pool.end(); }
})().catch((error) => { console.error('[migrate] fatal:', error); process.exit(1); });
scripts/repair-autonomy-labels.js
+293
View File
@@ -0,0 +1,293 @@
#!/usr/bin/env node
/*
* Repairs autonomy prediction provenance labels.
*
* The coordinator worker never handed an origin down to acceptProposal(), and
* acceptProposal defaults metadata.origin to "live". Every prediction produced
* by a historical-lane coordinator job therefore landed in the table wearing an
* origin of "live" even though it was a backfill over an old information cutoff.
*
* Canonical origins after this runs:
* live - genuine real time work
* historical - coordinator historical lane backfill
* replay - walk forward replay
* and learning_eligible may only be 1 when origin is "live".
*
* The discriminator is how old the information cutoff is relative to the moment
* the row was written. A genuinely live prediction reasons about right now, so
* the gap is milliseconds. A backfill reasons about 2024 while being written in
* 2026, so the gap is months.
*
* Defaults to a dry run. Pass --apply to actually write. This touches
* production data so the flag is deliberately not optional.
*/
const path = require("path");
const { openRuntimeDb, isPostgresEnabled } = require("../src/db/runtime");
// A live prediction stamps its cutoff with new Date().toISOString() microseconds
// before the insert, so its gap is effectively zero. A backfill sits months
// behind. Anything in between does not exist in practice, which is why the exact
// threshold is not delicate - 24h just has to be far enough above clock skew,
// queue latency and a midnight rollover that a real live row can never trip it.
// A naive same-calendar-day comparison would misfile a row created at 00:00:00
// whose cutoff was stamped at 23:59:59 the night before, and that mistake is
// silent and unrecoverable once written.
const HISTORICAL_MIN_AGE_MS = 24 * 60 * 60 * 1000;
const LIVE_FILTER = "origin = 'live'";
const ELIGIBILITY_PREDICATE = "origin != 'live' AND learning_eligible != 0";
// postgres has a hard cap on bound parameters and huge IN lists are miserable to
// debug, so the id updates go out in bites.
const UPDATE_CHUNK = 100;
function parseArgs(argv) {
const flags = new Set(argv.slice(2));
if (flags.has("--help") || flags.has("-h")) {
console.log("usage: node scripts/repair-autonomy-labels.js [--dry-run|--apply]");
console.log(" --dry-run report what would change and write nothing (default)");
console.log(" --apply run the repair inside a transaction");
process.exit(0);
}
const apply = flags.has("--apply");
if (apply && flags.has("--dry-run")) {
console.error("[repair] --apply and --dry-run are mutually exclusive");
process.exit(2);
}
return { apply };
}
// Timestamps live in TEXT columns and arrive in two shapes: the ISO strings the
// coordinator writes, and sqlite's datetime('now') output which is UTC with a
// space and no zone marker. Handing the second one to new Date() unqualified
// makes node read it as local time, so we pin it to UTC ourselves.
function parseTimestamp(value) {
if (value === null || value === undefined) return null;
if (value instanceof Date) return Number.isNaN(value.getTime()) ? null : value;
let text = String(value).trim();
if (!text) return null;
text = text.replace(" ", "T");
if (/^\d{4}-\d{2}-\d{2}$/.test(text)) text += "T00:00:00";
if (!/(?:Z|[+-]\d{2}:?\d{2})$/i.test(text)) text += "Z";
const parsed = new Date(text);
return Number.isNaN(parsed.getTime()) ? null : parsed;
}
function count(db, where) {
const sql = `SELECT COUNT(*) AS count FROM autonomy_predictions${where ? ` WHERE ${where}` : ""}`;
return Number(db.prepare(sql).get().count || 0);
}
function allPredictionIds(db) {
// ids are BIGINT on the postgres side and come back as strings, so everything
// gets normalised to strings before it goes anywhere near a Set.
return new Set(db.prepare("SELECT id FROM autonomy_predictions").all().map((row) => String(row.id)));
}
// The date arithmetic happens in javascript rather than SQL. src/db/runtime.js
// rewrites date/datetime expressions on its way to postgres, and an interval
// comparison that survives both dialects untouched is not worth the risk on a
// script that edits production provenance.
function findHistoricalCandidates(db) {
const rows = db.prepare(`
SELECT id, information_cutoff, created_at
FROM autonomy_predictions
WHERE ${LIVE_FILTER}
`).all();
const ids = [];
const unparseable = [];
for (const row of rows) {
const cutoff = parseTimestamp(row.information_cutoff);
const created = parseTimestamp(row.created_at);
if (!cutoff || !created) {
unparseable.push({ id: String(row.id), informationCutoff: row.information_cutoff, createdAt: row.created_at });
continue;
}
if (created.getTime() - cutoff.getTime() > HISTORICAL_MIN_AGE_MS) ids.push(row.id);
}
return { ids, unparseable, scanned: rows.length };
}
function chunk(list, size) {
const out = [];
for (let index = 0; index < list.length; index += size) out.push(list.slice(index, index + size));
return out;
}
function snapshot(db) {
const byOrigin = db.prepare(`
SELECT origin, COUNT(*) AS count
FROM autonomy_predictions
GROUP BY origin
ORDER BY origin
`).all().map((row) => ({ origin: row.origin, count: Number(row.count || 0) }));
const byEligibility = db.prepare(`
SELECT learning_eligible, COUNT(*) AS count
FROM autonomy_predictions
GROUP BY learning_eligible
ORDER BY learning_eligible
`).all().map((row) => ({ learningEligible: Number(row.learning_eligible || 0), count: Number(row.count || 0) }));
return { total: count(db, null), byOrigin, byEligibility };
}
function printSnapshot(label, snap) {
console.log(`[repair] ${label} total rows: ${snap.total}`);
for (const row of snap.byOrigin) console.log(`[repair] ${label} origin=${row.origin}: ${row.count}`);
for (const row of snap.byEligibility) console.log(`[repair] ${label} learning_eligible=${row.learningEligible}: ${row.count}`);
}
// A repair script has no business creating schema, so instead of calling
// initAutonomySchema we just check the columns we are about to touch are there.
function assertColumns(db) {
const columns = db.prepare("PRAGMA table_info(autonomy_predictions)").all().map((row) => String(row.name));
const missing = ["id", "origin", "learning_eligible", "information_cutoff", "created_at"].filter((name) => !columns.includes(name));
if (missing.length) throw new Error(`autonomy_predictions is missing required columns: ${missing.join(", ")}`);
}
function reportUnparseable(unparseable) {
if (!unparseable.length) return;
console.error(`[repair] WARNING: ${unparseable.length} live rows have timestamps that could not be parsed and were left untouched`);
for (const row of unparseable.slice(0, 10)) {
console.error(`[repair] id=${row.id} information_cutoff=${JSON.stringify(row.informationCutoff)} created_at=${JSON.stringify(row.createdAt)}`);
}
if (unparseable.length > 10) console.error(`[repair] ... and ${unparseable.length - 10} more`);
}
let db = null;
function main() {
const { apply } = parseArgs(process.argv);
const intelligencePath = process.env.INTELLIGENCE_DB || path.resolve(process.cwd(), "intelligence.sqlite");
// readonly on a dry run means sqlite physically cannot be written to, and the
// postgres path ignores the flag entirely.
db = openRuntimeDb(intelligencePath, { schema: "intelligence", readonly: !apply });
console.log(`[repair] backend: ${isPostgresEnabled() ? "postgres" : `sqlite (${intelligencePath})`}`);
console.log(`[repair] mode: ${apply ? "APPLY (writes)" : "dry-run (no writes)"}`);
console.log(`[repair] historical threshold: created_at - information_cutoff > ${HISTORICAL_MIN_AGE_MS}ms (24h)`);
assertColumns(db);
const before = snapshot(db);
printSnapshot("before", before);
const candidates = findHistoricalCandidates(db);
const eligibilityCandidates = count(db, ELIGIBILITY_PREDICATE);
console.log(`[repair] live rows scanned: ${candidates.scanned}`);
console.log(`[repair] live rows older than the threshold (would become historical): ${candidates.ids.length}`);
console.log(`[repair] rows with a non-live origin but learning_eligible != 0: ${eligibilityCandidates}`);
reportUnparseable(candidates.unparseable);
if (!apply) {
console.log("[repair] dry run finished, nothing was written. re-run with --apply to commit.");
console.log(JSON.stringify({
mode: "dry-run",
total: before.total,
liveScanned: candidates.scanned,
wouldRelabel: candidates.ids.length,
wouldClearEligibility: eligibilityCandidates,
unparseableTimestamps: candidates.unparseable.length,
}));
return;
}
// The no-loss guarantee is an identity check, not a headcount. Workers are
// live and inserting while this runs, so a bigger table afterwards is normal;
// a row that was here before and is gone now is not, and neither is an equal
// sized DELETE+INSERT, which a plain total would happily wave through.
const beforeIds = allPredictionIds(db);
console.log(`[repair] tracking ${beforeIds.size} existing prediction ids through the transaction`);
const summary = { relabelled: 0, eligibilityCleared: 0, newRowsDuringRun: 0 };
const tx = db.transaction(() => {
// Recomputed inside the transaction so we act on a consistent read rather
// than on whatever the table looked like a few seconds ago.
const fresh = findHistoricalCandidates(db);
reportUnparseable(fresh.unparseable);
for (const ids of chunk(fresh.ids, UPDATE_CHUNK)) {
const placeholders = ids.map(() => "?").join(", ");
const result = db.prepare(`
UPDATE autonomy_predictions
SET origin = 'historical', learning_eligible = 0
WHERE id IN (${placeholders})
`).run(...ids);
summary.relabelled += Number(result.changes || 0);
}
// Second pass catches replay rows (and anything else non-live) that somehow
// carry an eligibility flag. We never set learning_eligible back to 1 here:
// the contract makes live a necessary condition, not a sufficent one, and
// the original write-time decision is not ours to reinvent.
const eligibility = db.prepare(`
UPDATE autonomy_predictions
SET learning_eligible = 0
WHERE ${ELIGIBILITY_PREDICATE}
`).run();
summary.eligibilityCleared = Number(eligibility.changes || 0);
const afterIds = allPredictionIds(db);
const missing = [...beforeIds].filter((id) => !afterIds.has(id));
if (missing.length) {
throw new Error(`${missing.length} prediction ids vanished during repair (first few: ${missing.slice(0, 5).join(", ")}), rolling back`);
}
summary.newRowsDuringRun = [...afterIds].filter((id) => !beforeIds.has(id)).length;
const stillBroken = findHistoricalCandidates(db).ids.length + count(db, ELIGIBILITY_PREDICATE);
if (stillBroken !== 0) {
throw new Error(`repair did not converge, ${stillBroken} rows still need repairing, rolling back`);
}
return snapshot(db);
});
let after;
try {
after = tx();
} catch (error) {
console.error("[repair] transaction rolled back:", error && error.stack ? error.stack : error);
throw error;
}
printSnapshot("after", after);
console.log(`[repair] all ${beforeIds.size} pre-existing prediction ids still present, no rows lost`);
if (summary.newRowsDuringRun) {
console.log(`[repair] note: ${summary.newRowsDuringRun} new rows were inserted by other workers while this ran (informational, not an error)`);
}
console.log(JSON.stringify({
mode: "apply",
totalBefore: before.total,
totalAfter: after.total,
idsPreserved: beforeIds.size,
relabelledToHistorical: summary.relabelled,
eligibilityCleared: summary.eligibilityCleared,
newRowsDuringRun: summary.newRowsDuringRun,
}));
}
// PgCompatDb has no close() and its pool keeps the event loop alive, hence the
// typeof guard plus the hard exit at the bottom.
function closeQuietly() {
if (db && typeof db.close === "function") {
try { db.close(); } catch (closeError) { console.error("[repair] close failed:", closeError && closeError.stack ? closeError.stack : closeError); }
}
}
try {
main();
} catch (error) {
console.error("[repair] fatal:", error && error.stack ? error.stack : error);
closeQuietly();
process.exit(1);
}
closeQuietly();
process.exit(0);
scripts/resync-postgres-schema.js
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env node
/*
* Re-sync one logical SQLite database into an existing PostgreSQL schema.
* Unlike the initial migration, this updates conflicting primary-key rows so
* mutable runtime state such as job status, replay cursors, and broker snapshots
* can be cut over safely.
*/
const Database = require('better-sqlite3');
const { Pool } = require('pg');
const sqlitePath = process.env.SQLITE_PATH || process.env.SQLITE_INTELLIGENCE_PATH || '/data/intelligence.sqlite';
const schema = process.env.POSTGRES_SCHEMA || 'intelligence';
const connectionString = process.env.DATABASE_URL;
if (!connectionString) throw new Error('DATABASE_URL is required');
const batchSize = Math.max(1, Number(process.env.POSTGRES_SYNC_BATCH_SIZE) || 500);
const sqlite = new Database(sqlitePath, { readonly: true });
const pool = new Pool({ connectionString, max: 2 });
function quote(name) { return `"${String(name).replaceAll('"', '""')}"`; }
function pgType(sqliteType) {
const type = String(sqliteType || '').toUpperCase();
if (type.includes('INT')) return 'BIGINT';
if (type.includes('REAL') || type.includes('FLOA') || type.includes('DOUB')) return 'DOUBLE PRECISION';
if (type.includes('BLOB')) return 'BYTEA';
return 'TEXT';
}
function normalizeValue(value) {
if (typeof value === 'string' && value.includes('\u0000')) return value.replace(/\u0000/g, '');
return value ?? null;
}
function sourceTables() {
return sqlite.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name").all().map((row) => row.name);
}
function columnsFor(table) {
return sqlite.prepare(`PRAGMA table_info(${quote(table)})`).all();
}
async function ensureTable(client, table, columns) {
const primary = columns.filter((column) => column.pk).sort((a, b) => a.pk - b.pk).map((column) => quote(column.name));
const definitions = columns.map((column) => `${quote(column.name)} ${pgType(column.type)}${column.notnull ? ' NOT NULL' : ''}`);
if (primary.length) definitions.push(`PRIMARY KEY (${primary.join(', ')})`);
await client.query(`CREATE TABLE IF NOT EXISTS ${quote(schema)}.${quote(table)} (${definitions.join(', ')})`);
}
async function repairIdentity(client, table, columns) {
const id = columns.find((column) => column.pk === 1 && column.name === 'id' && /INT/i.test(column.type));
if (!id) return;
const seq = `${schema}_${table}_id_seq`;
await client.query(`CREATE SEQUENCE IF NOT EXISTS ${quote(schema)}.${quote(seq)}`);
await client.query(`ALTER TABLE ${quote(schema)}.${quote(table)} ALTER COLUMN id SET DEFAULT nextval('${quote(schema)}.${quote(seq)}')`);
await client.query(`ALTER SEQUENCE ${quote(schema)}.${quote(seq)} OWNED BY ${quote(schema)}.${quote(table)}.id`);
await client.query(`SELECT setval('${quote(schema)}.${quote(seq)}', COALESCE((SELECT MAX(id) FROM ${quote(schema)}.${quote(table)}), 0) + 1, false)`);
}
async function syncTable(client, table) {
const columns = columnsFor(table);
if (!columns.length) return;
await ensureTable(client, table, columns);
const names = columns.map((column) => column.name);
const pk = columns.filter((column) => column.pk).sort((a, b) => a.pk - b.pk).map((column) => column.name);
const sourceRows = sqlite.prepare(`SELECT COUNT(*) AS count FROM ${quote(table)}`).get().count;
if (!sourceRows) {
await repairIdentity(client, table, columns);
console.log(`[sync] ${schema}.${table}: empty`);
return;
}
if (!pk.length) {
const targetRows = await client.query(`SELECT COUNT(*)::bigint AS count FROM ${quote(schema)}.${quote(table)}`);
if (Number(targetRows.rows[0].count) === Number(sourceRows)) {
console.log(`[sync] ${schema}.${table}: no primary key, count matches (${sourceRows})`);
return;
}
throw new Error(`${schema}.${table} has no primary key and count differs; refusing ambiguous sync`);
}
const placeholders = names.map((_, index) => `$${index + 1}`).join(', ');
const updates = names.filter((name) => !pk.includes(name)).map((name) => `${quote(name)}=EXCLUDED.${quote(name)}`).join(', ');
const conflict = pk.map(quote).join(', ');
const sql = `INSERT INTO ${quote(schema)}.${quote(table)} (${names.map(quote).join(', ')}) VALUES (${placeholders}) ON CONFLICT (${conflict}) ${updates ? `DO UPDATE SET ${updates}` : 'DO NOTHING'}`;
const key = columns.find((column) => column.pk === 1 && /INT/i.test(column.type));
let cursor = 0;
let copied = 0;
console.log(`[sync] ${schema}.${table}: ${sourceRows} source rows`);
while (true) {
const rows = key
? sqlite.prepare(`SELECT ${names.map(quote).join(', ')} FROM ${quote(table)} WHERE ${quote(key.name)} > ? ORDER BY ${quote(key.name)} LIMIT ?`).all(cursor, batchSize)
: sqlite.prepare(`SELECT ${names.map(quote).join(', ')} FROM ${quote(table)} LIMIT ? OFFSET ?`).all(batchSize, copied);
if (!rows.length) break;
await client.query('BEGIN');
try {
for (const row of rows) await client.query(sql, names.map((name) => normalizeValue(row[name])));
await client.query('COMMIT');
} catch (error) {
await client.query('ROLLBACK');
throw error;
}
copied += rows.length;
if (key) cursor = Number(rows[rows.length - 1][key.name]);
process.stdout.write(`\r[sync] ${schema}.${table}: ${copied}/${sourceRows}`);
}
await repairIdentity(client, table, columns);
console.log(`\r[sync] ${schema}.${table}: synced ${sourceRows}`);
}
(async () => {
const client = await pool.connect();
try {
await client.query(`CREATE SCHEMA IF NOT EXISTS ${quote(schema)}`);
for (const table of sourceTables()) await syncTable(client, table);
} finally {
client.release();
await pool.end();
sqlite.close();
}
})().catch((error) => {
console.error('[sync] fatal:', error);
process.exit(1);
});
scripts/score-replay-runs.js
+321
View File
@@ -0,0 +1,321 @@
#!/usr/bin/env node
/*
* Scoreboard and paired comparison for replay runs.
*
* Two jobs:
* 1. score one run against constant null models, so "50% accuracy" has to
* answer the question "compared to what". A model that always says
* negative scores whatever share of the sample actually went down, and if
* the system cannot beat that it has no directional skill at all.
* 2. compare two runs over the SAME articles. Replay walks the archive in
* cursor order, so different runs are the only way to hold article
* vintage fixed. Comparing two calendar periods of one run compares two
* market regimes, not two prompts.
*
* Read only. Opens intelligence read only and writes nothing.
*
* PRE REGISTERED TESTS (declared here so the buckets cannot be tuned later):
* T1 is run accuracy above the BEST constant baseline on the same sample?
* one sided binomial z. the best constant is used as the bar because it
* is the hardest of the two, which is conservative for us.
* T2 is the direction signed excess return above the BEST constant on the
* same bars? paired two sided t. testing it against zero was the first
* version and it flattered us: a unit short in everything also earns a
* positive number on this sample, so zero is not the bar.
* T3 DISCRIMINATION. is P(up | it said positive) above P(up | it said
* negative)? two proportion z. this is the only one of the four that a
* change of prior cannot fake: it asks whether the choice of direction
* carries information, separately from how often it picks each one.
* comparing a direction group's accuracy to "always that direction" on
* the same rows is an identity and tests nothing, which is what the
* first version of this file printed.
* T4 paired: on articles both runs answered, is the candidate's per article
* accuracy above the baseline's? two sided paired t on the differences.
* Everything under "descriptive" is NOT a test. Slice p values carry a
* bonferroni factor and are there to generate hypotheses, not confirm them.
*
* node scripts/score-replay-runs.js
* node scripts/score-replay-runs.js --run 1
* node scripts/score-replay-runs.js --baseline 1 --candidate 2
* node scripts/score-replay-runs.js --baseline 1 --candidate 2 --since 2026-02-01
*/
const Database = require("better-sqlite3");
const INTELLIGENCE = process.env.INTELLIGENCE_DB || "/data/intelligence.sqlite";
function args() {
const out = {};
const argv = process.argv.slice(2);
for (let i = 0; i < argv.length; i += 1) {
if (!argv[i].startsWith("--")) continue;
const key = argv[i].slice(2);
const next = argv[i + 1];
out[key] = (next && !next.startsWith("--")) ? (i += 1, next) : true;
}
return out;
}
// Abramowitz and Stegun 7.1.26. The last analysis used a logistic shortcut and
// it returned p > 1 for negative z, which is nonsense that survived because
// nobody looks at a p value and asks whether it is even in range.
function erf(x) {
const sign = x < 0 ? -1 : 1;
const z = Math.abs(x);
const t = 1 / (1 + 0.3275911 * z);
const y = 1 - ((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-z * z);
return sign * y;
}
function normalCdf(z) { return 0.5 * (1 + erf(z / Math.SQRT2)); }
function twoSided(z) { return 2 * (1 - normalCdf(Math.abs(z))); }
function oneSidedUpper(z) { return 1 - normalCdf(z); }
function pct(x, digits = 2) { return Number.isFinite(x) ? `${(x * 100).toFixed(digits)}%` : "n/a"; }
// accuracy of `hits` out of `n` against a fixed reference rate
function binomialZ(hits, n, p0) {
if (!n || p0 <= 0 || p0 >= 1) return { z: NaN, p: NaN };
const phat = hits / n;
const z = (phat - p0) / Math.sqrt(p0 * (1 - p0) / n);
return { z, p: oneSidedUpper(z) };
}
function tStat(values) {
const n = values.length;
if (n < 2) return { n, mean: NaN, z: NaN, p: NaN };
const mean = values.reduce((a, b) => a + b, 0) / n;
const variance = values.reduce((a, b) => a + (b - mean) ** 2, 0) / (n - 1);
const se = Math.sqrt(variance / n);
const z = se > 0 ? mean / se : 0;
return { n, mean, se, z, p: twoSided(z) };
}
// does the choice of direction carry information at all. invariant to how
// often it picks each side, unlike raw accuracy.
function discrimination(rows) {
const pos = rows.filter((r) => r.direction === "positive");
const neg = rows.filter((r) => r.direction === "negative");
const upPos = pos.filter((r) => r.excess_return > 0).length;
const upNeg = neg.filter((r) => r.excess_return > 0).length;
const p1 = pos.length ? upPos / pos.length : NaN;
const p2 = neg.length ? upNeg / neg.length : NaN;
const pooled = (upPos + upNeg) / (pos.length + neg.length);
const se = Math.sqrt(pooled * (1 - pooled) * (1 / pos.length + 1 / neg.length));
const z = se > 0 ? (p1 - p2) / se : 0;
return { pUpGivenPositive: p1, pUpGivenNegative: p2, nPositive: pos.length, nNegative: neg.length,
spread: p1 - p2, z, p: twoSided(z), positiveShare: pos.length / rows.length };
}
function signed(row) {
// what a unit position in the predicted direction actually earned
return row.direction === "negative" ? -row.excess_return : row.excess_return;
}
function articleOf(row) {
try {
const parsed = JSON.parse(row.evidence_article_ids || "[]");
return Array.isArray(parsed) && parsed.length ? String(parsed[0]) : null;
} catch (error) {
console.error(`[score] unparseable evidence on prediction ${row.id}:`, error.message);
return null;
}
}
function load(db, { runId, since, until, createdSince }) {
const where = ["p.origin = 'replay'", "o.prediction_id IS NOT NULL"];
const params = [];
if (runId) { where.push("p.replay_run_id = ?"); params.push(runId); }
if (since) { where.push("date(p.information_cutoff) >= date(?)"); params.push(since); }
if (until) { where.push("date(p.information_cutoff) <= date(?)"); params.push(until); }
// the train/test split is on when the prediction was MADE, because that is
// what fixes which prompt produced it. cutoff dates only correlate with it.
if (createdSince) { where.push("p.created_at >= ?"); params.push(createdSince); }
return db.prepare(`
SELECT p.id, p.instrument, p.direction, p.event_type, p.horizon_days, p.information_cutoff,
p.evidence_article_ids, p.replay_run_id, p.strategy_version,
pr.prompt_version, pr.coordinator_model,
o.direction_correct, o.excess_return
FROM autonomy_predictions p
JOIN autonomy_proposals pr ON pr.id = p.proposal_id
JOIN autonomy_outcomes o ON o.prediction_id = p.id
WHERE ${where.join(" AND ")}
ORDER BY p.information_cutoff ASC, p.id ASC
`).all(...params);
}
function baselines(rows) {
const n = rows.length;
const up = rows.filter((r) => r.excess_return > 0).length;
return {
n,
alwaysPositive: up / n,
alwaysNegative: (n - up) / n,
// mean of a unit long in every name, which is what always_positive earns
alwaysPositiveExcess: rows.reduce((a, r) => a + r.excess_return, 0) / n,
};
}
function scoreRun(rows, label) {
const n = rows.length;
if (!n) { console.log(`\n${label}: no scored predictions\n`); return null; }
const hits = rows.filter((r) => r.direction_correct).length;
const acc = hits / n;
const base = baselines(rows);
const best = Math.max(base.alwaysPositive, base.alwaysNegative);
const bestName = base.alwaysNegative >= base.alwaysPositive ? "always_negative" : "always_positive";
const t1 = binomialZ(hits, n, best);
// pair against the constant on the identical bars. where the system already
// agrees with the constant the difference is zero and contributes nothing,
// which is exactly right.
const constantSign = bestName === "always_negative" ? -1 : 1;
const t2 = tStat(rows.map((r) => signed(r) - constantSign * r.excess_return));
const rawSigned = tStat(rows.map(signed));
const constantExcess = rows.reduce((a, r) => a + constantSign * r.excess_return, 0) / n;
console.log(`\n=== ${label} ===`);
const models = [...new Set(rows.map((r) => r.coordinator_model))];
const prompts = [...new Set(rows.map((r) => r.prompt_version))];
console.log(` span ${rows[0].information_cutoff.slice(0, 10)} .. ${rows[n - 1].information_cutoff.slice(0, 10)}`);
console.log(` models ${models.join(", ")}`);
console.log(` prompts ${prompts.join(", ")}`);
console.log(` scored ${n} predictions over ${new Set(rows.map(articleOf)).size} articles`);
console.log("");
console.log(` system accuracy ${pct(acc)} (${hits}/${n})`);
console.log(` always_negative ${pct(base.alwaysNegative)} <- share of bars that underperformed SPY`);
console.log(` always_positive ${pct(base.alwaysPositive)}`);
console.log(` coin flip 50.00%`);
console.log("");
console.log(` T1 vs ${bestName}: z=${t1.z.toFixed(3)} p=${t1.p.toFixed(4)} (one sided, does the system beat the bar)`);
if (t1.z < 0) console.log(` the system is BELOW the constant. p=${twoSided(t1.z).toFixed(4)} two sided on being different from it.`);
console.log(` edge over the bar ${((acc - best) * 100).toFixed(2)} points`);
console.log(` T2 signed excess vs ${bestName}: ${pct(t2.mean, 3)} per prediction,`
+ ` t=${t2.z.toFixed(3)} p=${t2.p.toFixed(4)}`);
console.log(` system ${pct(rawSigned.mean, 3)} ${bestName} ${pct(constantExcess, 3)}`
+ ` always_positive ${pct(base.alwaysPositiveExcess, 3)}`);
const t3 = discrimination(rows);
console.log("");
console.log(` T3 discrimination: P(up | said positive) ${pct(t3.pUpGivenPositive)} (n=${t3.nPositive})`);
console.log(` P(up | said negative) ${pct(t3.pUpGivenNegative)} (n=${t3.nNegative})`);
console.log(` spread ${(t3.spread * 100).toFixed(2)} points, z=${t3.z.toFixed(3)} p=${t3.p.toFixed(4)}`);
console.log(` it says positive on ${pct(t3.positiveShare)} of calls while ${pct(base.alwaysPositive)}`
+ ` of the bars went up, so the prior is off by ${((t3.positiveShare - base.alwaysPositive) * 100).toFixed(1)} points`);
return { n, acc, hits, best, bestName, t1, t2, t3, base };
}
function slice(rows, key, label, minimum = 40) {
const groups = new Map();
for (const row of rows) {
const k = String(row[key]);
if (!groups.has(k)) groups.set(k, []);
groups.get(k).push(row);
}
const kept = [...groups.entries()].filter(([, v]) => v.length >= minimum);
if (!kept.length) return;
const factor = kept.length;
console.log(`\n descriptive by ${label} (n>=${minimum}, bonferroni x${factor}, NOT a test)`);
const scored = kept.map(([k, v]) => {
const hits = v.filter((r) => r.direction_correct).length;
const base = baselines(v);
const bar = Math.max(base.alwaysPositive, base.alwaysNegative);
const { z } = binomialZ(hits, v.length, bar);
return { k, n: v.length, acc: hits / v.length, bar, z, p: Math.min(1, twoSided(z) * factor),
excess: tStat(v.map(signed)).mean };
}).sort((a, b) => b.acc - a.acc);
for (const s of scored) {
// a small p here can mean significantly WORSE than the constant, which read
// like good news the first time this printed. say which side it fell on.
const side = s.acc >= s.bar ? "above" : "below";
const flag = s.p < 0.05 ? ` * ${side} bar` : "";
console.log(` ${s.k.padEnd(24)} n=${String(s.n).padEnd(5)} acc=${pct(s.acc).padEnd(8)}`
+ ` bar=${pct(s.bar).padEnd(8)} signed_excess=${pct(s.excess, 3).padEnd(9)} p_adj=${s.p.toFixed(3)}${flag}`);
}
}
// per article accuracy, so an article that produced 11 predictions does not
// count eleven times against one that produced a single call
function byArticle(rows) {
const map = new Map();
for (const row of rows) {
const id = articleOf(row);
if (!id) continue;
if (!map.has(id)) map.set(id, []);
map.get(id).push(row);
}
const out = new Map();
for (const [id, list] of map) {
out.set(id, {
accuracy: list.filter((r) => r.direction_correct).length / list.length,
signed: list.reduce((a, r) => a + signed(r), 0) / list.length,
count: list.length,
});
}
return out;
}
function paired(baseRows, candRows) {
const a = byArticle(baseRows);
const b = byArticle(candRows);
const shared = [...a.keys()].filter((id) => b.has(id));
console.log(`\n=== T4 paired comparison ===`);
console.log(` baseline articles ${a.size}, candidate articles ${b.size}, shared ${shared.length}`);
if (shared.length < 30) {
console.log(" not enough shared articles to say anything. run the candidate over the baseline's articles first.");
return;
}
const accDiff = shared.map((id) => b.get(id).accuracy - a.get(id).accuracy);
const excessDiff = shared.map((id) => b.get(id).signed - a.get(id).signed);
const baseAcc = shared.reduce((s, id) => s + a.get(id).accuracy, 0) / shared.length;
const candAcc = shared.reduce((s, id) => s + b.get(id).accuracy, 0) / shared.length;
const tAcc = tStat(accDiff);
const tExc = tStat(excessDiff);
const better = shared.filter((id) => b.get(id).accuracy > a.get(id).accuracy).length;
const worse = shared.filter((id) => b.get(id).accuracy < a.get(id).accuracy).length;
console.log(` baseline per article accuracy ${pct(baseAcc)}`);
console.log(` candidate per article accuracy ${pct(candAcc)}`);
console.log(` articles improved ${better}, degraded ${worse}, unchanged ${shared.length - better - worse}`);
console.log(` T4 accuracy delta ${pct(tAcc.mean)} t=${tAcc.z.toFixed(3)} p=${tAcc.p.toFixed(4)}`);
console.log(` signed excess delta ${pct(tExc.mean, 3)} t=${tExc.z.toFixed(3)} p=${tExc.p.toFixed(4)}`);
console.log(tAcc.p < 0.05
? (tAcc.mean > 0 ? " VERDICT: the candidate is better on the same articles." : " VERDICT: the candidate is WORSE on the same articles.")
: " VERDICT: no detectable difference on the same articles.");
}
function main() {
const opts = args();
const db = new Database(INTELLIGENCE, { readonly: true });
db.pragma("busy_timeout = 20000");
const runs = db.prepare("SELECT * FROM autonomy_replay_runs ORDER BY id").all();
console.log("replay runs on record:");
for (const run of runs) {
console.log(` #${run.id} ${run.status.padEnd(9)} watermark=${String(run.watermark_at).slice(0, 10)}`
+ ` ${run.strategy_version}/${run.prompt_version} ${run.coordinator_model}`
+ ` processed=${run.processed_articles}`);
}
if (opts.baseline && opts.candidate) {
const window = { since: opts.since, until: opts.until, createdSince: opts["created-since"] };
const baseRows = load(db, { runId: Number(opts.baseline), ...window });
const candRows = load(db, { runId: Number(opts.candidate), ...window });
scoreRun(baseRows, `run ${opts.baseline} (baseline)`);
scoreRun(candRows, `run ${opts.candidate} (candidate)`);
paired(baseRows, candRows);
db.close();
return;
}
const runId = opts.run && opts.run !== true ? Number(opts.run) : null;
const rows = load(db, { runId, since: opts.since, until: opts.until, createdSince: opts["created-since"] });
const summary = scoreRun(rows, runId ? `run ${runId}` : "all replay runs");
if (summary) {
slice(rows, "direction", "direction");
slice(rows, "horizon_days", "horizon");
slice(rows, "event_type", "event family");
slice(rows, "instrument", "instrument");
console.log("");
}
db.close();
}
main();
scripts/set-autonomy-instrument.js
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env node
const Database = require('better-sqlite3');
const path = require('path');
const { initAutonomySchema } = require('../src/autonomy/schema');
const symbol = String(process.argv[2] || '').trim().toUpperCase();
if (!/^[A-Z0-9._/-]+$/.test(symbol)) {
console.error('usage: node scripts/set-autonomy-instrument.js SYMBOL [broker]');
process.exit(2);
}
const broker = String(process.argv[3] || 'simulator');
const db = new Database(process.env.INTELLIGENCE_DB || path.resolve(process.cwd(), 'intelligence.sqlite'));
initAutonomySchema(db);
db.prepare(`
INSERT INTO autonomy_instruments(symbol, broker, active, tradable)
VALUES (?, ?, 1, 1)
ON CONFLICT(symbol) DO UPDATE SET broker=excluded.broker, active=1, tradable=1, updated_at=datetime('now')
`).run(symbol, broker);
console.log(JSON.stringify({ symbol, broker, active: true, tradable: true }));
db.close();
scripts/start-replay-run.js
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env node
/*
* Start the next replay run over the SAME articles as the previous one, with a
* feedback brief built from the previous run's own scored outcomes.
*
* The point is a single variable. The evaluation slice is the articles the
* parent answered under the current prompt and the current model, so run N+1
* differs from run N by the brief and nothing else. Articles the parent
* answered under an older prompt are the TRAINING half and are never replayed,
* because deriving the lesson and grading it on the same rows measures nothing.
*
* What it writes, all additive:
* - parent run status running -> paused. Its cursor is untouched, so it can
* be resumed later exactly where it stopped.
* - one new row in autonomy_replay_runs carrying the brief.
* - one row per evaluation article in autonomy_replay_run_articles.
* Nothing is deleted and no existing prediction, outcome or snapshot is touched.
*
* node scripts/start-replay-run.js --parent 1 --split "2026-09-04 19:30" --dry-run
* node scripts/start-replay-run.js --parent 1 --split "2026-09-04 19:30" --commit
*/
const Database = require("better-sqlite3");
const { buildFeedbackBrief } = require("./build-feedback-brief");
const { STRATEGY_VERSION, PROMPT_VERSION } = require("../workers/replayWorker");
const INTELLIGENCE = process.env.INTELLIGENCE_DB || "/data/intelligence.sqlite";
// When duriin-api-replay-1 restarted onto the prompt it runs today. Not the
// commit timestamp, which is five minutes later and would have been wrong.
// Replay was between daily budgets across the restart, so there is an eight
// hour hole in predictions around it and every candidate split inside that
// hole partitions the data identically.
const SPLIT = "2026-09-04 19:17:43";
function options() {
const argv = process.argv.slice(2);
const out = {};
for (let i = 0; i < argv.length; i += 1) {
if (!argv[i].startsWith("--")) continue;
const key = argv[i].slice(2);
const next = argv[i + 1];
out[key] = (next && !next.startsWith("--")) ? (i += 1, next) : true;
}
return out;
}
// the pinned table arrives with the schema migration, so a dry run from a
// container that has not been redeployed yet should still be able to report
function countOf(db, table) {
try {
return db.prepare(`SELECT COUNT(*) AS c FROM ${table}`).get().c;
} catch (error) {
console.error(`[replay-run] cannot count ${table}:`, error.message);
return 0;
}
}
function articleIdOf(raw, predictionId) {
try {
const parsed = JSON.parse(raw || "[]");
return Array.isArray(parsed) && parsed.length ? Number(parsed[0]) : null;
} catch (error) {
console.error(`[replay-run] unparseable evidence on prediction ${predictionId}:`, error.message);
return null;
}
}
function main() {
const opts = options();
const parentId = Number(opts.parent || 1);
const split = String(opts.split || SPLIT);
const commit = opts.commit === true;
const model = String(opts.model || process.env.OPEN_ROUTER_LLM_MODEL || "");
const db = new Database(INTELLIGENCE, { readonly: !commit });
db.pragma("busy_timeout = 20000");
const parent = db.prepare("SELECT * FROM autonomy_replay_runs WHERE id = ?").get(parentId);
if (!parent) throw new Error(`replay run ${parentId} does not exist`);
// row counts before, so the report can show nothing went missing
const before = {
runs: countOf(db, "autonomy_replay_runs"),
predictions: countOf(db, "autonomy_predictions"),
outcomes: countOf(db, "autonomy_outcomes"),
pinned: countOf(db, "autonomy_replay_run_articles"),
};
const evaluation = db.prepare(`
SELECT p.id, p.evidence_article_ids, p.information_cutoff
FROM autonomy_predictions p
WHERE p.origin = 'replay' AND p.replay_run_id = ? AND p.created_at >= ?
`).all(parentId, split);
const articles = new Map();
for (const row of evaluation) {
const id = articleIdOf(row.evidence_article_ids, row.id);
if (id && !articles.has(id)) articles.set(id, String(row.information_cutoff));
}
const { text: brief, stats } = buildFeedbackBrief(db, { runId: parentId, createdBefore: split });
console.log(`parent run #${parentId} ${parent.status}, watermark ${parent.watermark_at}`);
console.log(`split at ${split}`);
console.log(` training predictions (before split, feed the brief): ${stats.n}`);
console.log(` evaluation predictions (at or after split): ${evaluation.length}`);
console.log(` evaluation articles to replay: ${articles.size}`);
console.log(` brief: ${brief.split("\n").length} lines, ${brief.length} chars`);
console.log(` new run would be ${STRATEGY_VERSION}/${PROMPT_VERSION} on ${model || "(model from config at run time)"}`);
if (!articles.size) throw new Error("no evaluation articles, refusing to create an empty run");
// inheriting the parent's model here is how run 1 ended up labelled qwen for
// predictions deepseek made. a label nobody set is worse than a failure.
if (commit && !model) {
throw new Error("pass --model, or set OPEN_ROUTER_LLM_MODEL. refusing to guess what will run this");
}
if (!commit) {
console.log("\ndry run, nothing written. pass --commit to apply.");
db.close();
return;
}
const apply = db.transaction(() => {
db.prepare("UPDATE autonomy_replay_runs SET status = 'paused', updated_at = datetime('now') WHERE id = ? AND status = 'running'").run(parentId);
const created = db.prepare(`
INSERT INTO autonomy_replay_runs (watermark_at, strategy_version, prompt_version, coordinator_model, parent_run_id, feedback_brief)
VALUES (?, ?, ?, ?, ?, ?)
`).run(parent.watermark_at, STRATEGY_VERSION, PROMPT_VERSION, model, parentId, brief);
const runId = created.lastInsertRowid;
const insert = db.prepare("INSERT OR IGNORE INTO autonomy_replay_run_articles (run_id, article_id, effective_at) VALUES (?, ?, ?)");
for (const [articleId, effectiveAt] of articles) insert.run(runId, articleId, effectiveAt);
return runId;
});
const runId = apply();
const after = {
runs: countOf(db, "autonomy_replay_runs"),
predictions: countOf(db, "autonomy_predictions"),
outcomes: countOf(db, "autonomy_outcomes"),
pinned: countOf(db, "autonomy_replay_run_articles"),
};
console.log(`\ncreated replay run #${runId}, parent #${parentId} is now`
+ ` ${db.prepare("SELECT status FROM autonomy_replay_runs WHERE id = ?").get(parentId).status}`);
console.log("row counts before -> after");
for (const key of Object.keys(before)) {
const moved = after[key] - before[key];
console.log(` ${key.padEnd(12)} ${before[key]} -> ${after[key]} (${moved >= 0 ? "+" : ""}${moved})`);
}
if (after.predictions !== before.predictions || after.outcomes !== before.outcomes) {
console.error("predictions or outcomes changed, that should not happen here");
}
db.close();
}
main();
scripts/sync-paper-assets.js
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env node
const path = require('path');
const Database = require('better-sqlite3');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { createAlpacaPaperClient } = require('../src/brokers/alpacaPaper');
async function main() {
const client = createAlpacaPaperClient({
keyId: process.env.ALPACA_PAPER_KEY_ID,
secretKey: process.env.ALPACA_PAPER_SECRET_KEY,
});
const assets = await client.getAssets();
const db = new Database(process.env.INTELLIGENCE_DB || path.resolve(process.cwd(), 'intelligence.sqlite'));
initAutonomySchema(db);
const upsert = db.prepare(`
INSERT INTO autonomy_instruments(symbol, broker, asset_class, active, tradable, shortable, fractionable, updated_at)
VALUES (?, 'alpaca-paper', ?, ?, ?, ?, ?, datetime('now'))
ON CONFLICT(symbol) DO UPDATE SET
broker=excluded.broker, asset_class=excluded.asset_class, active=excluded.active,
tradable=excluded.tradable, shortable=excluded.shortable, fractionable=excluded.fractionable,
updated_at=datetime('now')
`);
const tx = db.transaction(() => assets.forEach((asset) => upsert.run(
asset.symbol, asset.class || 'us_equity', asset.status === 'active' ? 1 : 0,
asset.tradable ? 1 : 0, asset.shortable ? 1 : 0, asset.fractionable ? 1 : 0
)));
tx();
console.log(JSON.stringify({ synced: assets.length }));
db.close();
}
main().catch((error) => { console.error(error.message); process.exit(1); });
server.js
+7 -1
View File
@@ -6,6 +6,7 @@ const sourcesRoutes = require('./src/routes/sources');
const eventRoutes = require('./src/routes/events');
const adminRoutes = require('./src/routes/admin');
const devRoutes = require('./src/routes/dev');
const autonomyRoutes = require('./src/routes/autonomy');
const config = require('./src/config');
const { startScheduler } = require('./src/scheduler');
@@ -18,13 +19,18 @@ app.register(sourcesRoutes);
app.register(eventRoutes);
app.register(adminRoutes);
app.register(devRoutes);
app.register(autonomyRoutes);
app.get('/', async () => ({ ok: true }));
async function start() {
await app.listen({ port: config.server.port, host: config.server.host });
startScheduler();
if (process.env.DURIIN_RUN_SCHEDULER !== 'false') {
startScheduler();
} else {
app.log.warn('Background ingestion and enrichment scheduler is disabled');
}
}
start().catch((error) => {
src/adminStatsWorker.js
+39
View File
@@ -0,0 +1,39 @@
const { parentPort, workerData } = require('node:worker_threads');
const Database = require('better-sqlite3');
const fs = require('node:fs');
const path = require('node:path');
try {
const db = new Database(workerData.databasePath, { readonly: true, fileMustExist: true });
const counts = db.prepare(`
SELECT
COALESCE((SELECT seq FROM sqlite_sequence WHERE name='articles'), 0) AS total,
(SELECT COUNT(*) FROM article_embedding_meta) AS withContent,
(SELECT COUNT(*) FROM article_embedding_meta) AS withEmbedding,
COALESCE((SELECT seq FROM sqlite_sequence WHERE name='events'), 0) AS eventCount,
(SELECT COUNT(*) FROM articles WHERE ingested_at >= datetime('now', '-1 hour')) AS ingestedPerHour,
(SELECT COUNT(*) FROM article_embedding_meta WHERE embedded_at >= datetime('now', '-1 hour')) AS contentPerHour
`).get();
const sourceCatalog = JSON.parse(fs.readFileSync(path.resolve(__dirname, '..', 'sources.json'), 'utf8'));
const bySource = sourceCatalog
.map((source) => ({ source: source.label || source.id }))
.sort((a, b) => a.source.localeCompare(b.source));
// Avoid scanning the 3.7 GB article table for legacy content-status values.
// Readiness is the useful operational distinction and is available from the
// compact embedding metadata table.
const byStatus = [
{ status: 'ready', n: counts.withEmbedding },
{ status: 'unprocessed', n: Math.max(0, counts.total - counts.withEmbedding) },
];
let embeddingsPerHour = 0;
try {
embeddingsPerHour = db.prepare(`
SELECT COUNT(*) AS n FROM article_embedding_meta
WHERE embedded_at >= datetime('now', '-1 hour')
`).get().n;
} catch (_) {}
db.close();
parentPort.postMessage({ value: { ...counts, bySource, byStatus, embeddingsPerHour } });
} catch (error) {
parentPort.postMessage({ error: error.message });
}
src/autonomy/calibration.js
+130
View File
@@ -0,0 +1,130 @@
function clamp(value, min, max) { return Math.max(min, Math.min(max, value)); }
function betaMean(wins, total, priorWins = 1, priorLosses = 1) {
return (wins + priorWins) / (total + priorWins + priorLosses);
}
function quantile(values, q) {
if (!values.length) return null;
const sorted = [...values].sort((a, b) => a - b);
const position = (sorted.length - 1) * q;
const lower = Math.floor(position);
const upper = Math.ceil(position);
if (lower === upper) return sorted[lower];
return sorted[lower] + (sorted[upper] - sorted[lower]) * (position - lower);
}
// The coordinator emits event_type as free text, so production ended up with 200+
// distinct values across ~600 predictions. Keying calibration on the raw string
// gave cohorts of ~2.7 samples each, which can never clear any honest sample gate.
// These families are a closed set: order matters, first match wins, and anything
// we don't recognise lands in `other` rather than inventing its own cohort.
const EVENT_FAMILIES = [
['analyst_action', /\b(analysts?|upgrades?|downgrades?|price[_ ]?targets?|ratings?|initiations?|coverage|overweight|underweight|outperform)\b/],
['guidance', /\b(guidance|outlooks?|forecasts?|pre[_ ]?announce\w*|warns?|warning|raises?[_ ]guid\w*|cuts?[_ ]guid\w*|projections?)\b/],
['earnings', /\b(earnings?|results?|quarterly|eps|revenues?|margins?|beat|miss(ed|es)?|q[1-4]|fy\d{2,4}|financials?)\b/],
['m_and_a', /\b(m&a|merger|mergers|acquisitions?|acquires?|acquired|takeovers?|buyouts?|divestitures?|divests?|spin[_ ]?offs?|stake[_ ]sales?|tender[_ ]offers?)\b/],
['legal', /\b(lawsuits?|litigations?|courts?|patents?|settlements?|verdicts?|injunctions?|class[_ ]actions?|subpoenas?|infringements?|appeals?)\b/],
['regulatory', /\b(regulat\w*|antitrust|probes?|investigations?|sanctions?|export[_ ]controls?|tariffs?|bans?|banned|approvals?|approved|licens\w*|compliance|fda|ftc|doj|sec[_ ]filing|policy)\b/],
['leadership', /\b(ceo|cfo|coo|cto|chairman|executives?|resign\w*|appoint\w*|steps?[_ ]down|boards?|successions?|layoffs?|restructur\w*|hiring|departures?)\b/],
['supply_chain', /\b(supply|suppliers?|shortages?|capacity|production|fabs?|foundry|inventor\w+|logistics?|shipments?|recalls?|manufactur\w*|yields?|backlog)\b/],
['contract', /\b(contracts?|orders?|partnerships?|partners?|agreements?|collaborations?|deals?|customers?|wins?|awards?)\b/],
['product', /\b(products?|launch\w*|unveil\w*|releases?|announcements?|chips?|models?|features?|roadmaps?|platforms?)\b/],
['capital', /\b(buybacks?|repurchases?|dividends?|offerings?|debt|capital[_ ]raise|stock[_ ]splits?|ipos?|financing|bonds?)\b/],
['security_incident', /\b(hacks?|hacked|breach\w*|cyber\w*|ransomware|outages?|downtime|vulnerabilit\w+|exploits?)\b/],
['macro', /\b(macro\w*|fed|federal[_ ]reserve|interest[_ ]rates?|inflation|gdp|econom\w+|recession|currenc\w+|geopolit\w+|war|elections?|demand)\b/],
];
const EVENT_FAMILY_NAMES = EVENT_FAMILIES.map(([name]) => name).concat('other');
// snake_case, camelCase, "Supply Constraint" and "supply-constraint" all have to
// collapse onto the same token stream before we try to match anything.
function normalizeEventType(raw) {
if (raw === null || raw === undefined) return 'other';
const text = String(raw)
.replace(/([a-z0-9])([A-Z])/g, '$1 $2')
.toLowerCase()
.replace(/[^a-z0-9&]+/g, ' ')
.trim();
if (!text) return 'other';
for (const [family, pattern] of EVENT_FAMILIES) {
if (pattern.test(text)) return family;
}
return 'other';
}
// ALLOWED_HORIZONS is 1/5/10/20/30/60/90 in the coordinator. Seven horizons times
// two directions was another multiplier on the cohort explosion, and a 10 day and
// a 20 day call on the same event are not really different populations.
const HORIZON_BUCKETS = ['short', 'medium', 'long'];
function horizonBucket(horizonDays) {
const days = Number(horizonDays);
if (!Number.isFinite(days) || days <= 0) return 'unknown';
if (days <= 5) return 'short';
if (days <= 20) return 'medium';
return 'long';
}
const COHORT_KEY_VERSION = 'v2';
function cohortKey({ direction, eventType, horizonDays, sector = 'unknown' }) {
return [COHORT_KEY_VERSION, sector, normalizeEventType(eventType), horizonBucket(horizonDays), direction].join('|');
}
// Snapshots written before the taxonomy change still carry the raw key, this keeps
// them readable/joinable without a migration.
function legacyCohortKey({ direction, eventType, horizonDays, sector = 'unknown' }) {
return [sector, eventType || 'unknown', horizonDays, direction].join('|');
}
function instrumentOf(row) {
const symbol = row.instrument ?? row.symbol ?? null;
if (symbol === null || symbol === undefined) return null;
const trimmed = String(symbol).trim().toUpperCase();
return trimmed || null;
}
function calibrateOutcomes(rows, parent = null) {
const clean = rows.filter((row) => Number.isFinite(Number(row.excess_return)));
const wins = clean.filter((row) => Number(row.direction_correct) === 1).length;
const total = clean.length;
const priorProbability = parent ? parent.directionalProbability : 0.5;
const priorStrength = parent ? Math.max(2, Math.min(20, parent.effectiveSampleSize / 10)) : 2;
const probability = (wins + priorProbability * priorStrength) / (total + priorStrength);
const returns = clean.map((row) => Number(row.excess_return));
// Concentration matters as much as raw n here. A cohort of 300 outcomes that is
// 95% one ticker is one bet repeated, not 300 independant observations.
const counts = new Map();
for (const row of clean) {
const symbol = instrumentOf(row);
if (!symbol) continue;
counts.set(symbol, (counts.get(symbol) || 0) + 1);
}
const topCount = counts.size ? Math.max(...counts.values()) : 0;
return {
sampleSize: total,
effectiveSampleSize: total + priorStrength,
directionalProbability: clamp(probability, 0.01, 0.99),
expectedExcessReturn: returns.length ? returns.reduce((sum, value) => sum + value, 0) / returns.length : null,
lowerReturn: quantile(returns, 0.1),
upperReturn: quantile(returns, 0.9),
distinctInstruments: counts.size,
topInstrumentShare: total ? topCount / total : null,
};
}
module.exports = {
betaMean,
cohortKey,
legacyCohortKey,
calibrateOutcomes,
quantile,
normalizeEventType,
horizonBucket,
EVENT_FAMILY_NAMES,
HORIZON_BUCKETS,
COHORT_KEY_VERSION,
};
src/autonomy/coordinator.js
+166
View File
@@ -0,0 +1,166 @@
const { EVENT_FAMILY_NAMES, normalizeEventType } = require('./calibration');
const ALLOWED_DIRECTIONS = new Set(['positive', 'negative']);
const ALLOWED_HORIZONS = new Set([1, 5, 10, 20, 30, 60, 90]);
// The prompt used to say event_type was a 'stable_enum' and then never listed the
// enum, so the model invented one label per event and production ended up with 201
// distinct values. Same closed set the cohort key uses, so a label can never mean
// one thing in the prompt and another in calibration.
const ALLOWED_EVENT_TYPES = new Set(EVENT_FAMILY_NAMES);
// 96% of rejected proposals name an instrument we cannot trade: indices (SPX,
// DXY, ^TNX), fx (EURUSD, XAU/USD), futures (CL=F, BZ=F) and foreign listings
// (VOW3.DE, RHM.DE, 1211.HK). The analysis behind those is usually fine, it is
// the ticker that is unusable, so tell the model what the allowlist actually
// holds instead of paying for the call and discarding it at validation.
// Every symbol named below was checked against the live allowlist.
const INSTRUMENT_RULES = `Instrument rules. instrument must be a symbol tradable on a US exchange, and only these resolve:
- US listed common stock or ETF, by its US ticker.
- A foreign company only via its US listing or ADR, never its home listing. Volkswagen is VWAGY not VOW3.DE, Alibaba is BABA, Toyota is TM, Sony is SONY. If you do not know the company has a US listing, omit the prediction.
- An index, currency, rate or commodity only via a US listed ETF that tracks it: S&P 500 -> SPY, Nasdaq 100 -> QQQ, gold -> GLD, crude oil -> USO, US dollar -> UUP, treasuries -> TLT. Never emit SPX, DXY, ^TNX, EURUSD, XAU/USD, CL=F or any futures or fx symbol.
- Never SPY itself as the prediction, it is the benchmark and its excess return is always zero.
If the only instrument the evidence supports is untradable under these rules, leave it out rather than substituting something loosely related.`;
// An exact family is what we want. If the model ignores the list we try to salvage
// the label through the same mapper calibration uses, and only give up when it is
// unplaceable -- an explicit 'other' is a legitimate answer, unplaceable free text
// is not, and the difference is what stops 'other' quietly becoming the bin again.
function normalizeProposedEventType(raw, instrument) {
const value = String(raw || '').trim().toLowerCase().replace(/[\s-]+/g, '_');
if (ALLOWED_EVENT_TYPES.has(value)) return value;
const salvaged = normalizeEventType(raw);
if (salvaged !== 'other') {
console.warn(`[coordinator] ${instrument} event_type "${raw}" is not in the enum, mapped to "${salvaged}"`);
return salvaged;
}
throw new Error(`event_type must be one of ${EVENT_FAMILY_NAMES.join(', ')} (got "${raw}")`);
}
function normalizeProposal(raw, { informationCutoff, model = 'unknown', promptVersion = 'unknown' } = {}) {
if (!raw || typeof raw !== 'object') throw new Error('coordinator output must be an object');
const predictions = Array.isArray(raw.predictions) ? raw.predictions : [];
const normalized = predictions.map((item) => {
const instrument = String(item.instrument || item.ticker || '').trim().toUpperCase();
const direction = String(item.direction || '').trim().toLowerCase();
const horizonDays = Number(item.horizon_days || item.horizonDays);
if (!instrument) throw new Error('prediction instrument is required');
if (!ALLOWED_DIRECTIONS.has(direction)) throw new Error(`invalid direction: ${direction}`);
if (!ALLOWED_HORIZONS.has(horizonDays)) throw new Error(`invalid horizon_days: ${horizonDays}`);
const articleIds = Array.isArray(item.evidence_article_ids)
? item.evidence_article_ids.map(Number).filter(Number.isInteger)
: [];
if (articleIds.length === 0) throw new Error(`prediction for ${instrument} has no evidence`);
return {
instrument,
direction,
eventType: normalizeProposedEventType(item.event_type, instrument),
causalChannel: item.causal_channel ? String(item.causal_channel).trim() : null,
horizonDays,
evidenceArticleIds: [...new Set(articleIds)],
invalidationCondition: item.invalidation_condition ? String(item.invalidation_condition).trim() : null,
};
});
return {
schemaVersion: 1,
informationCutoff: informationCutoff || new Date().toISOString(),
coordinatorModel: model,
promptVersion,
predictions: normalized,
};
}
function verifyEvidence(archiveDb, articleIds, informationCutoff = null) {
const placeholders = articleIds.map(() => '?').join(',');
// No proposal, whatever lane produced it, may cite material which did not yet
// exist at its own information cutoff. This used to be a replay-only rule and
// that was a lookahead hole for every other origin.
const cutoffClause = informationCutoff ? ' AND datetime(COALESCE(pub_date_effective, pub_date, ingested_at)) <= datetime(?)' : '';
let rows;
try {
rows = archiveDb.prepare(`SELECT id FROM articles WHERE id IN (${placeholders})${cutoffClause}`)
.all(...articleIds, ...(informationCutoff ? [informationCutoff] : []));
} catch (error) {
// Minimal/test archives may not retain publication metadata at all, in which
// case the cutoff clause cannot even be prepared. We degrade to a plain
// existence check rather than blocking the pipeline, but the degredation is
// never silent - a production archive missing these columns is a real bug.
console.warn('[coordinator] evidence cutoff check unavailable, falling back to existence only.',
`cutoff=${informationCutoff} articles=${JSON.stringify(articleIds)} reason=${error && error.message}`);
if (error && error.stack) console.warn(error.stack);
rows = archiveDb.prepare(`SELECT id FROM articles WHERE id IN (${placeholders})`).all(...articleIds);
}
const found = new Set(rows.map((row) => row.id));
return articleIds.every((id) => found.has(id));
}
function acceptProposal(intelligenceDb, archiveDb, raw, metadata = {}) {
const proposal = normalizeProposal(raw, metadata);
// Tradability is a filter, not an integrity failure, so it is applied per
// prediction. One untradable ticker used to reject the entire proposal and take
// its valid siblings down with it: in a single day 93 proposals were rejected
// this way, discarding 171 predictions of which 76 named something we could
// trade perfectly well.
const allowlisted = intelligenceDb.prepare(
"SELECT tradable FROM autonomy_instruments WHERE symbol = ? AND active = 1 AND tradable = 1"
);
const dropped = [];
const tradable = proposal.predictions.filter((prediction) => {
if (allowlisted.get(prediction.instrument)) return true;
dropped.push(prediction.instrument);
return false;
});
if (dropped.length) {
console.warn(`[coordinator] dropped ${dropped.length} untradable instrument(s): ${dropped.join(', ')}`
+ ` (kept ${tradable.length})`);
}
// Lookahead stays all or nothing. A proposal citing evidence that did not exist
// at its own cutoff is corrupt rather than merely untradable, and quietly keeping
// the rest of it would hide exactly the thing we most need to see.
for (const prediction of tradable) {
if (!verifyEvidence(archiveDb, prediction.evidenceArticleIds, proposal.informationCutoff)) {
throw new Error(`proposal references missing evidence for ${prediction.instrument}`);
}
}
// what actually got stored, plus a record of what was filtered and why
const stored = { ...proposal, predictions: tradable, droppedInstruments: dropped };
const insert = intelligenceDb.prepare(`
INSERT INTO autonomy_proposals
(event_id, payload, information_cutoff, coordinator_model, prompt_version, status)
VALUES (?, ?, ?, ?, ?, 'accepted')
`);
const insertPrediction = intelligenceDb.prepare(`
INSERT INTO autonomy_predictions
(proposal_id, event_id, instrument, direction, event_type, causal_channel,
horizon_days, information_cutoff, evidence_article_ids, invalidation_condition, learning_eligible, strategy_version, origin, replay_run_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
const tx = intelligenceDb.transaction(() => {
const proposalResult = insert.run(metadata.eventId || null, JSON.stringify(stored), stored.informationCutoff,
stored.coordinatorModel, stored.promptVersion);
for (const prediction of tradable) {
insertPrediction.run(proposalResult.lastInsertRowid, metadata.eventId || null, prediction.instrument,
prediction.direction, prediction.eventType, prediction.causalChannel, prediction.horizonDays,
stored.informationCutoff, JSON.stringify(prediction.evidenceArticleIds), prediction.invalidationCondition,
metadata.learningEligible ? 1 : 0, metadata.strategyVersion || 'autonomy-1',
metadata.origin || 'live', metadata.replayRunId || null);
}
return Number(proposalResult.lastInsertRowid);
});
return { proposalId: tx(), predictionCount: tradable.length, droppedInstruments: dropped };
}
function recordRejectedProposal(intelligenceDb, raw, metadata = {}, reason = 'validation failed') {
const payload = raw && typeof raw === 'object' ? raw : { raw: String(raw) };
return intelligenceDb.prepare(`
INSERT INTO autonomy_proposals
(event_id, payload, information_cutoff, coordinator_model, prompt_version, status, rejection_reason, reviewed_at)
VALUES (?, ?, ?, ?, ?, 'rejected', ?, datetime('now'))
`).run(metadata.eventId || null, JSON.stringify(payload), metadata.informationCutoff || new Date().toISOString(),
metadata.model || 'unknown', metadata.promptVersion || 'unknown', String(reason).slice(0, 1000)).lastInsertRowid;
}
module.exports = { normalizeProposal, verifyEvidence, acceptProposal, recordRejectedProposal, INSTRUMENT_RULES };
src/autonomy/execution.js
+34
View File
@@ -0,0 +1,34 @@
function makeClientOrderId(decisionId) {
return `duriin-${String(decisionId)}`;
}
function validatePaperIntent(intent, constraints = {}) {
if (!intent || !intent.decisionId || !intent.instrument) throw new Error('decisionId and instrument are required');
if (!['BUY', 'SELL'].includes(intent.action)) throw new Error('only BUY and SELL create order intents');
if (!Number.isFinite(Number(intent.notional)) || Number(intent.notional) <= 0) throw new Error('notional must be positive');
const maxNotional = Number(constraints.maxNotional ?? 1000);
if (Number(intent.notional) > maxNotional) throw new Error('notional exceeds paper risk limit');
if (constraints.tradable !== true) throw new Error('instrument is not confirmed tradable');
return {
clientOrderId: makeClientOrderId(intent.decisionId),
instrument: String(intent.instrument).toUpperCase(),
side: intent.action === 'BUY' ? 'buy' : 'sell',
notional: Number(intent.notional),
mode: 'paper',
};
}
function createSimulator() {
const orders = new Map();
return {
submit(intent) {
if (orders.has(intent.clientOrderId)) return orders.get(intent.clientOrderId);
const order = { ...intent, brokerOrderId: `sim-${intent.clientOrderId}`, status: 'accepted' };
orders.set(intent.clientOrderId, order);
return order;
},
get(clientOrderId) { return orders.get(clientOrderId) || null; },
};
}
module.exports = { makeClientOrderId, validatePaperIntent, createSimulator };
src/autonomy/graphContext.js
+76
View File
@@ -0,0 +1,76 @@
// Relationship context for the coordinator.
//
// The graph has been built for months and fed nothing but a dashboard: no
// prediction, decision or order has ever seen an edge. That is the one piece of
// context a per-event coordinator genuinely cannot derive from its own article
// set, because "NVDA supplies X, so a capacity story at X matters for NVDA" is
// knowledge about companies rather than about this event.
//
// The hard rule here is the cutoff. company_relationships.first_seen_at is
// derived from article dates rather than processing time, so filtering on it
// keeps a historical proposal from seeing a relationship the world had not yet
// revealed. Without that filter this feature would quietly reintroduce exactly
// the lookahead the evidence check exists to prevent.
const MAX_COMPANIES = 3;
const MAX_PER_COMPANY = 8;
function buildGraphContext(intelligenceDb, eventId, informationCutoff, options = {}) {
if (!eventId || !informationCutoff) return '';
const maxCompanies = Number(options.maxCompanies) || MAX_COMPANIES;
const maxPerCompany = Number(options.maxPerCompany) || MAX_PER_COMPANY;
try {
const companies = intelligenceDb.prepare(`
SELECT DISTINCT tc.id, tc.name, tc.ticker
FROM event_knowledge ek
JOIN tracked_companies tc ON tc.id = ek.company_id
WHERE ek.event_id = ?
LIMIT ?
`).all(eventId, maxCompanies);
if (!companies.length) return '';
const relationships = intelligenceDb.prepare(`
SELECT relationship_type, to_entity, confidence, confirmation_count
FROM company_relationships
WHERE from_company_id = ?
AND first_seen_at IS NOT NULL
AND datetime(first_seen_at) <= datetime(?)
ORDER BY confirmation_count DESC, id ASC
LIMIT ?
`);
const blocks = [];
for (const company of companies) {
const edges = relationships.all(company.id, informationCutoff, maxPerCompany);
if (!edges.length) continue;
// de-duplicate on the entity name, the graph stores both casings for some
const seen = new Set();
const lines = [];
for (const edge of edges) {
const key = `${edge.relationship_type}:${String(edge.to_entity || '').toLowerCase()}`;
if (seen.has(key)) continue;
seen.add(key);
lines.push(` - ${edge.relationship_type}: ${edge.to_entity}`
+ ` (seen ${edge.confirmation_count}x, ${edge.confidence || 'unrated'})`);
}
const label = company.ticker ? `${company.name} (${company.ticker})` : company.name;
blocks.push(`${label}:\n${lines.join('\n')}`);
}
if (!blocks.length) return '';
return `Known company relationships, as they stood at the information cutoff:\n\n${blocks.join('\n\n')}\n\n`
+ `These are background knowledge, not evidence. They exist so you can reason about second order effects: `
+ `a story about one company may be the tradable event for a supplier, customer or competitor. `
+ `If you use one, say so in causal_channel, and still cite the article ids the story itself came from. `
+ `Do not predict an instrument the articles give you no reason to believe is affected, and do not treat a `
+ `relationship as evidence on its own.`;
} catch (error) {
// Context is an enhancement. Losing it should never cost us the prediction,
// but it must never be lost silently either.
console.error(`[graph-context] unavailable for event ${eventId}:`, error.message);
return '';
}
}
module.exports = { buildGraphContext, MAX_COMPANIES, MAX_PER_COMPANY };
src/autonomy/index.js
+17
View File
@@ -0,0 +1,17 @@
const { initAutonomySchema } = require('./schema');
const jobs = require('./jobs');
const coordinator = require('./coordinator');
const calibration = require('./calibration');
const policy = require('./policy');
const execution = require('./execution');
const orderIntents = require('./orderIntents');
module.exports = {
initAutonomySchema,
...jobs,
...coordinator,
...calibration,
...policy,
...execution,
...orderIntents,
};
src/autonomy/jobs.js
+64
View File
@@ -0,0 +1,64 @@
function enqueueJob(db, { jobType, lane = 'historical', priority = 0, entityType, entityId, idempotencyKey }) {
const result = db.prepare(`
INSERT OR IGNORE INTO autonomy_jobs
(job_type, lane, priority, entity_type, entity_id, idempotency_key)
VALUES (?, ?, ?, ?, ?, ?)
`).run(jobType, lane, priority, entityType, String(entityId), idempotencyKey);
return { inserted: result.changes > 0 };
}
function leaseNextJob(db, workerId, leaseSeconds = 60, jobTypes = null) {
const tx = db.transaction(() => {
const typeClause = Array.isArray(jobTypes) && jobTypes.length
? `AND job_type IN (${jobTypes.map(() => '?').join(',')})`
: '';
const typeParams = Array.isArray(jobTypes) && jobTypes.length ? jobTypes : [];
const job = db.prepare(`
SELECT * FROM autonomy_jobs
WHERE ((status = 'pending' AND datetime(available_at) <= datetime('now'))
OR (status = 'leased' AND datetime(lease_expires_at) <= datetime('now')))
${typeClause}
ORDER BY CASE lane WHEN 'live' THEN 3 WHEN 'maintenance' THEN 2 ELSE 1 END DESC,
priority DESC, id ASC
LIMIT 1
`).get(...typeParams);
if (!job) return null;
const updated = db.prepare(`
UPDATE autonomy_jobs
SET status = 'leased', leased_by = ?, lease_expires_at = datetime('now', ?),
attempts = attempts + 1
WHERE id = ?
`).run(workerId, `+${Math.max(1, Math.floor(leaseSeconds))} seconds`, job.id);
return updated.changes ? { ...job, status: 'leased', leased_by: workerId } : null;
});
try {
// Acquire the write reservation before selecting. A deferred transaction can
// otherwise read a snapshot, lose the writer race, and fail with
// SQLITE_BUSY_SNAPSHOT when it attempts the lease update.
return tx.immediate();
} catch (error) {
if (String(error.code || '').startsWith('SQLITE_BUSY')) return null;
throw error;
}
}
function completeJob(db, id, workerId) {
return db.prepare(`
UPDATE autonomy_jobs
SET status = 'complete', completed_at = datetime('now'),
leased_by = NULL, lease_expires_at = NULL
WHERE id = ? AND leased_by = ?
`).run(id, workerId).changes > 0;
}
function failJob(db, id, workerId, error, maxAttempts = 5) {
return db.prepare(`
UPDATE autonomy_jobs
SET status = CASE WHEN attempts >= ? THEN 'dead_letter' ELSE 'pending' END,
available_at = datetime('now', '+60 seconds'), last_error = ?,
leased_by = NULL, lease_expires_at = NULL
WHERE id = ? AND leased_by = ?
`).run(maxAttempts, String(error || 'unknown error').slice(0, 2000), id, workerId).changes > 0;
}
module.exports = { enqueueJob, leaseNextJob, completeJob, failJob };
src/autonomy/llm.js
+87
View File
@@ -0,0 +1,87 @@
function extractJson(text) {
const value = String(text || '').trim().replace(/^```(?:json)?\s*/i, '').replace(/\s*```$/, '');
try { return JSON.parse(value); } catch (_) {
const start = value.indexOf('{');
const end = value.lastIndexOf('}');
if (start >= 0 && end > start) return JSON.parse(value.slice(start, end + 1));
throw new Error('LLM response did not contain valid JSON');
}
}
// OpenRouter reserves max_tokens against the key's remaining budget up front, and
// that affordable ceiling shrinks as the balance depletes. An unbounded request is
// refused outright, so "no cap" is not an option on a limited key -- it produces no
// output at all rather than truncated output. This pulls the real ceiling out of the
// refusal so we can retry just under it instead of guessing a fixed number.
function affordableTokens(message) {
const match = /can only afford (\d+)/i.exec(String(message || ''));
if (!match) return null;
const affordable = Number(match[1]);
return Number.isFinite(affordable) && affordable > 256 ? affordable : null;
}
async function callCoordinator(config, prompt, options = {}) {
const apiKey = String(config?.openRouter?.apiKey || '').trim();
if (!apiKey) throw new Error('OpenRouter API key is not configured');
const timeoutMs = Math.max(1000, Number(config?.openRouter?.timeoutMs || process.env.OPEN_ROUTER_TIMEOUT_MS) || 60000);
// only set when a budget retry forced one, or an operator asked for one
const maxTokens = options.maxTokens
|| Number(config?.openRouter?.maxTokens || process.env.OPEN_ROUTER_MAX_TOKENS) || null;
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs);
let response;
try {
response = await fetch('https://openrouter.ai/api/v1/chat/completions', {
method: 'POST',
signal: controller.signal,
headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
body: JSON.stringify({
model: config.openRouter.llmModel,
temperature: 0,
response_format: { type: 'json_object' },
// No ceiling by default. Any number we pick is a number we invented, and
// 6000 was already tight enough to truncate a real replay article. A cap
// only exists to satisfy openrouter's affordability reservation, so it is
// supplied by the 402 handler below when the budget genuinely cannot cover
// an open ended request, and never otherwise. Set OPEN_ROUTER_MAX_TOKENS
// if you ever want one imposed deliberately.
...(maxTokens ? { max_tokens: maxTokens } : {}),
messages: [
{ role: 'system', content: 'You are a coordinator. Extract only evidence-backed categorical hypotheses. Never output probabilities, expected returns, confidence scores, position sizes, or trade actions.' },
{ role: 'user', content: prompt },
],
}),
});
} catch (error) {
const cause = error?.cause?.code || error?.code || error?.name || 'network_error';
throw new Error(`coordinator request failed before response (${cause})`);
} finally {
clearTimeout(timeout);
}
if (!response.ok) {
const body = await response.text().catch(() => '');
// A 402 names the ceiling the key can currently afford. Retry once just under
// it rather than failing the event, but only downwards, so a shrinking budget
// degrades output length instead of stopping the pipeline dead.
const affordable = response.status === 402 ? affordableTokens(body) : null;
if (affordable && !options.retriedForBudget) {
const retryTokens = Math.floor(affordable * 0.9);
console.warn(`[llm] budget only affords ${affordable} tokens, retrying with max_tokens=${retryTokens}`);
return callCoordinator(config, prompt, { maxTokens: retryTokens, retriedForBudget: true });
}
throw new Error(`coordinator request failed with ${response.status}: ${body.slice(0, 300)}`);
}
const body = await response.json();
const choice = body?.choices?.[0];
// Truncated json is worse than no json, because a partial object can occasionally
// still parse and quietly lose predictions. Fail loudly on the reason field rather
// than letting extractJson guess at a half-written response.
if (choice?.finish_reason === 'length') {
throw new Error('coordinator response was truncated by max_tokens, raise OPEN_ROUTER_MAX_TOKENS');
}
return extractJson(choice?.message?.content);
}
module.exports = { extractJson, callCoordinator };
src/autonomy/orderIntents.js
+25
View File
@@ -0,0 +1,25 @@
const { validatePaperIntent } = require('./execution');
function createOrderIntent(db, decisionId, notional, constraints = {}) {
const row = db.prepare(`
SELECT d.id AS decision_id, d.action, p.instrument,
COALESCE(ai.active, 0) AS active, COALESCE(ai.tradable, 0) AS tradable
FROM autonomy_decisions d
JOIN autonomy_predictions p ON p.id = d.prediction_id
LEFT JOIN autonomy_instruments ai ON ai.symbol = p.instrument
WHERE d.id = ?
`).get(decisionId);
if (!row) throw new Error(`decision ${decisionId} does not exist`);
const intent = validatePaperIntent({ decisionId, instrument: row.instrument, action: row.action, notional }, {
...constraints,
tradable: row.active === 1 && row.tradable === 1,
});
const result = db.prepare(`
INSERT OR IGNORE INTO autonomy_order_intents
(decision_id, client_order_id, instrument, side, notional, status)
VALUES (?, ?, ?, ?, ?, 'shadow')
`).run(decisionId, intent.clientOrderId, intent.instrument, intent.side, intent.notional);
return { ...intent, inserted: result.changes > 0 };
}
module.exports = { createOrderIntent };
src/autonomy/outcomes.js
+54
View File
@@ -0,0 +1,54 @@
// Yahoo writes class shares with a dash, BRK.B is BRK-B there. Our allowlist is
// full of dotted symbols and every one of them 404s forever otherwise.
function yahooSymbol(symbol) {
return String(symbol || '').trim().toUpperCase().replace(/\./g, '-');
}
function addTradingDays(date, days) {
const value = new Date(`${date}T00:00:00Z`);
let remaining = Math.max(0, Number(days) || 0);
while (remaining > 0) {
value.setUTCDate(value.getUTCDate() + 1);
const weekday = value.getUTCDay();
if (weekday !== 0 && weekday !== 6) remaining -= 1;
}
return value.toISOString().slice(0, 10);
}
function barOnOrAfter(history, date) {
return history.find((row) => row.date >= date) || null;
}
function nearestOnOrAfter(history, date) {
return barOnOrAfter(history, date)?.close ?? null;
}
function calculateOutcome(prediction, instrumentHistory, benchmarkHistory) {
const eventDate = String(prediction.information_cutoff).slice(0, 10);
const horizonDate = addTradingDays(eventDate, prediction.horizon_days);
const entryBar = barOnOrAfter(instrumentHistory, eventDate);
const exitBar = barOnOrAfter(instrumentHistory, horizonDate);
const benchEntryBar = barOnOrAfter(benchmarkHistory, eventDate);
const benchExitBar = barOnOrAfter(benchmarkHistory, horizonDate);
const price0 = entryBar?.close ?? null;
const priceHorizon = exitBar?.close ?? null;
const benchmark0 = benchEntryBar?.close ?? null;
const benchmarkHorizon = benchExitBar?.close ?? null;
if (![price0, priceHorizon, benchmark0, benchmarkHorizon].every(Number.isFinite)) return null;
// On a short horizon the entry and exit lookups can land on the same bar, which
// yields an excess return of exactly zero and gets scored as a directional miss.
// That is not a result, it means the horizon has not actually elapsed yet.
if (entryBar.date === exitBar.date || benchEntryBar.date === benchExitBar.date) return null;
const instrumentReturn = (priceHorizon - price0) / price0;
const benchmarkReturn = (benchmarkHorizon - benchmark0) / benchmark0;
const excessReturn = instrumentReturn - benchmarkReturn;
const directionCorrect = prediction.direction === 'positive' ? excessReturn > 0 : excessReturn < 0;
return {
price0, priceHorizon, benchmark0, benchmarkHorizon,
excessReturn, directionCorrect: directionCorrect ? 1 : 0,
eventDate, horizonDate,
};
}
module.exports = { addTradingDays, nearestOnOrAfter, barOnOrAfter, calculateOutcome, yahooSymbol };
src/autonomy/policy.js
+76
View File
@@ -0,0 +1,76 @@
// Thresholds live here so the worker, the replay evaluator and the tests all
// argue from the same numbers instead of sprinkling magic 30s around.
const DEFAULT_POLICY_RULES = {
minSampleSize: 30,
// A cohort has to be built from more than a handful of tickers. In production
// one name (NVDA) accounted for roughly half of every resolved outcome, so a
// pure sample-size gate was measuring one company, not an edge.
minDistinctInstruments: 5,
maxInstrumentConcentration: 0.5,
minProbability: 0.58,
minExpectedReturn: 0.005,
maxDownside: -0.08,
};
function decide({
direction = 'positive',
probability,
expectedExcessReturn,
lowerReturn,
upperReturn,
sampleSize,
distinctInstruments,
topInstrumentShare,
}, rules = {}) {
const minSampleSize = Number(rules.minSampleSize ?? DEFAULT_POLICY_RULES.minSampleSize);
const minDistinctInstruments = Number(rules.minDistinctInstruments ?? DEFAULT_POLICY_RULES.minDistinctInstruments);
const maxInstrumentConcentration = Number(rules.maxInstrumentConcentration ?? DEFAULT_POLICY_RULES.maxInstrumentConcentration);
const minProbability = Number(rules.minProbability ?? DEFAULT_POLICY_RULES.minProbability);
const minExpectedReturn = Number(rules.minExpectedReturn ?? DEFAULT_POLICY_RULES.minExpectedReturn);
const maxDownside = Number(rules.maxDownside ?? DEFAULT_POLICY_RULES.maxDownside);
if (![probability, expectedExcessReturn].every(Number.isFinite)) {
return { action: 'ABSTAIN', rationale: 'calibration unavailable' };
}
if (!Number.isFinite(Number(sampleSize)) || Number(sampleSize) < minSampleSize) {
return { action: 'ABSTAIN', rationale: `insufficient calibration sample (${sampleSize}/${minSampleSize})` };
}
// Snapshots written before diversification was tracked come back with the count
// missing. Unknown diversity is not the same as adequate diversity, abstain.
const instruments = distinctInstruments === null || distinctInstruments === undefined ? NaN : Number(distinctInstruments);
if (!Number.isFinite(instruments)) {
return { action: 'ABSTAIN', rationale: 'cohort instrument diversity unknown' };
}
if (instruments < minDistinctInstruments) {
return { action: 'ABSTAIN', rationale: `insufficient cohort diversity (${instruments}/${minDistinctInstruments} instruments)` };
}
// Same rule as the count above: a missing share is unknown, not safe. Number(null)
// is 0, which would sail straight through the cap, so check for absence first.
const concentration = topInstrumentShare === null || topInstrumentShare === undefined
? NaN
: Number(topInstrumentShare);
if (!Number.isFinite(concentration)) {
return { action: 'ABSTAIN', rationale: 'cohort instrument concentration unknown' };
}
if (concentration > maxInstrumentConcentration) {
return {
action: 'ABSTAIN',
rationale: `cohort dominated by a single instrument (${(concentration * 100).toFixed(0)}% > ${(maxInstrumentConcentration * 100).toFixed(0)}%)`,
};
}
const signedExpectedReturn = direction === 'negative' ? -expectedExcessReturn : expectedExcessReturn;
const signedLowerReturn = direction === 'negative'
? (Number.isFinite(upperReturn) ? -upperReturn : null)
: (Number.isFinite(lowerReturn) ? lowerReturn : null);
if (Number.isFinite(signedLowerReturn) && signedLowerReturn < maxDownside) {
return { action: 'HOLD', rationale: 'calibrated downside exceeds policy limit' };
}
if (probability >= minProbability && signedExpectedReturn >= minExpectedReturn) {
return { action: direction === 'negative' ? 'SELL' : 'BUY', rationale: 'calibrated edge clears policy thresholds' };
}
return { action: 'HOLD', rationale: 'calibrated edge does not clear policy thresholds' };
}
module.exports = { decide, DEFAULT_POLICY_RULES };
src/autonomy/schema.js
+283
View File
@@ -0,0 +1,283 @@
const AUTONOMY_SCHEMA_VERSION = 2;
// sqlite and postgres word this differently, and we re-run every ALTER on each
// boot, so a re-add is the expected case rather than a failure.
function isDuplicateColumn(error) {
const message = String(error && error.message || '').toLowerCase();
return message.includes('duplicate column') || message.includes('already exists');
}
function initAutonomySchema(db) {
if (db.dialect === 'postgres') return;
db.exec(`
CREATE TABLE IF NOT EXISTS autonomy_schema (
version INTEGER PRIMARY KEY,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS autonomy_jobs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_type TEXT NOT NULL,
lane TEXT NOT NULL CHECK (lane IN ('live', 'historical', 'maintenance')),
priority INTEGER NOT NULL DEFAULT 0,
entity_type TEXT NOT NULL,
entity_id TEXT NOT NULL,
idempotency_key TEXT NOT NULL UNIQUE,
status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'leased', 'complete', 'failed', 'dead_letter')),
attempts INTEGER NOT NULL DEFAULT 0,
available_at TEXT NOT NULL DEFAULT (datetime('now')),
leased_by TEXT,
lease_expires_at TEXT,
last_error TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
completed_at TEXT
);
CREATE INDEX IF NOT EXISTS idx_autonomy_jobs_claim
ON autonomy_jobs(status, lane, priority DESC, available_at);
CREATE TABLE IF NOT EXISTS autonomy_cursors (
key TEXT PRIMARY KEY,
value INTEGER NOT NULL DEFAULT 0,
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS autonomy_proposals (
id INTEGER PRIMARY KEY AUTOINCREMENT,
event_id INTEGER,
payload TEXT NOT NULL,
information_cutoff TEXT NOT NULL,
coordinator_model TEXT,
prompt_version TEXT,
status TEXT NOT NULL DEFAULT 'candidate'
CHECK (status IN ('candidate', 'accepted', 'rejected', 'superseded')),
rejection_reason TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
reviewed_at TEXT
);
CREATE TABLE IF NOT EXISTS autonomy_instruments (
symbol TEXT PRIMARY KEY,
broker TEXT NOT NULL,
asset_class TEXT NOT NULL DEFAULT 'us_equity',
active INTEGER NOT NULL DEFAULT 0,
tradable INTEGER NOT NULL DEFAULT 0,
shortable INTEGER NOT NULL DEFAULT 0,
fractionable INTEGER NOT NULL DEFAULT 0,
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS autonomy_legacy_records (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_table TEXT NOT NULL,
source_id INTEGER NOT NULL,
payload TEXT NOT NULL,
calibration_eligible INTEGER NOT NULL DEFAULT 0,
imported_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE(source_table, source_id)
);
CREATE TABLE IF NOT EXISTS autonomy_predictions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
proposal_id INTEGER NOT NULL REFERENCES autonomy_proposals(id),
event_id INTEGER,
instrument TEXT NOT NULL,
direction TEXT NOT NULL CHECK (direction IN ('positive', 'negative')),
event_type TEXT NOT NULL,
causal_channel TEXT,
horizon_days INTEGER NOT NULL,
information_cutoff TEXT NOT NULL,
evidence_article_ids TEXT NOT NULL,
invalidation_condition TEXT,
learning_eligible INTEGER NOT NULL DEFAULT 0,
strategy_version TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'open'
CHECK (status IN ('open', 'resolved', 'unresolvable', 'invalidated')),
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_autonomy_predictions_resolution
ON autonomy_predictions(status, information_cutoff, instrument);
CREATE TABLE IF NOT EXISTS autonomy_outcomes (
prediction_id INTEGER PRIMARY KEY REFERENCES autonomy_predictions(id),
price_0 REAL,
price_horizon REAL,
benchmark_0 REAL,
benchmark_horizon REAL,
excess_return REAL,
direction_correct INTEGER,
error_type TEXT,
evaluated_at TEXT NOT NULL DEFAULT (datetime('now')),
notes TEXT
);
CREATE TABLE IF NOT EXISTS autonomy_calibration_snapshots (
id INTEGER PRIMARY KEY AUTOINCREMENT,
cohort_key TEXT NOT NULL,
sample_size INTEGER NOT NULL,
effective_sample_size REAL NOT NULL,
directional_probability REAL NOT NULL,
expected_excess_return REAL,
lower_return REAL,
upper_return REAL,
parent_cohort_key TEXT,
version TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_autonomy_calibration_lookup
ON autonomy_calibration_snapshots(cohort_key, created_at DESC);
CREATE TABLE IF NOT EXISTS autonomy_decisions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
prediction_id INTEGER NOT NULL REFERENCES autonomy_predictions(id),
action TEXT NOT NULL CHECK (action IN ('BUY', 'SELL', 'HOLD', 'ABSTAIN')),
calibrated_probability REAL,
expected_excess_return REAL,
rationale TEXT NOT NULL,
strategy_version TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS autonomy_order_intents (
id INTEGER PRIMARY KEY AUTOINCREMENT,
decision_id INTEGER NOT NULL REFERENCES autonomy_decisions(id),
client_order_id TEXT NOT NULL UNIQUE,
instrument TEXT NOT NULL,
side TEXT NOT NULL CHECK (side IN ('buy', 'sell')),
notional REAL NOT NULL CHECK (notional > 0),
status TEXT NOT NULL DEFAULT 'shadow'
CHECK (status IN ('shadow', 'pending', 'submitted', 'filled', 'partially_filled', 'rejected', 'cancelled')),
broker_order_id TEXT,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS autonomy_broker_events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
broker TEXT NOT NULL,
event_type TEXT NOT NULL,
broker_id TEXT,
payload TEXT NOT NULL,
occurred_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE(broker, event_type, broker_id, occurred_at)
);
CREATE TABLE IF NOT EXISTS autonomy_account_snapshots (
id INTEGER PRIMARY KEY AUTOINCREMENT,
broker TEXT NOT NULL,
account_id TEXT,
equity REAL,
cash REAL,
buying_power REAL,
payload TEXT NOT NULL,
captured_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS autonomy_position_snapshots (
id INTEGER PRIMARY KEY AUTOINCREMENT,
broker TEXT NOT NULL,
instrument TEXT NOT NULL,
quantity REAL,
market_value REAL,
unrealized_pl REAL,
payload TEXT NOT NULL,
captured_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- Historical replay is a separate evidence path. It deliberately never
-- writes to autonomy_decisions or order intents.
CREATE TABLE IF NOT EXISTS autonomy_replay_runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
status TEXT NOT NULL DEFAULT 'running'
CHECK (status IN ('running', 'paused', 'complete', 'superseded', 'failed')),
start_at TEXT,
watermark_at TEXT,
cursor_article_id INTEGER NOT NULL DEFAULT 0,
cursor_effective_at TEXT,
processed_articles INTEGER NOT NULL DEFAULT 0,
strategy_version TEXT NOT NULL,
prompt_version TEXT NOT NULL,
coordinator_model TEXT,
price_provider TEXT NOT NULL DEFAULT 'yahoo',
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
completed_at TEXT,
last_error TEXT
);
CREATE INDEX IF NOT EXISTS idx_autonomy_replay_runs_active
ON autonomy_replay_runs(status, cursor_article_id);
-- An explicit article set for a run. When a run has rows here the scheduler
-- walks exactly these and nothing else, which is the only way to point two
-- runs at the same evidence. Runs without rows here keep walking the
-- archive by cursor exactly as before.
CREATE TABLE IF NOT EXISTS autonomy_replay_run_articles (
run_id INTEGER NOT NULL REFERENCES autonomy_replay_runs(id),
article_id INTEGER NOT NULL,
effective_at TEXT,
PRIMARY KEY (run_id, article_id)
);
CREATE INDEX IF NOT EXISTS idx_autonomy_replay_run_articles_walk
ON autonomy_replay_run_articles(run_id, effective_at, article_id);
CREATE TABLE IF NOT EXISTS autonomy_replay_evaluations (
prediction_id INTEGER PRIMARY KEY REFERENCES autonomy_predictions(id),
replay_run_id INTEGER NOT NULL REFERENCES autonomy_replay_runs(id),
snapshot_cutoff TEXT NOT NULL,
sample_size INTEGER NOT NULL,
action TEXT NOT NULL CHECK (action IN ('BUY', 'SELL', 'HOLD', 'ABSTAIN')),
calibrated_probability REAL,
expected_excess_return REAL,
rationale TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- Runtime knobs the operator can change without a redeploy. Execution mode used
-- to live only in AUTONOMY_EXECUTION_MODE, which means flipping it needed a
-- container restart, which is the last thing you want during an incident.
CREATE TABLE IF NOT EXISTS autonomy_settings (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_by TEXT
);
INSERT OR IGNORE INTO autonomy_schema(version) VALUES (${AUTONOMY_SCHEMA_VERSION});
`);
for (const statement of [
'ALTER TABLE autonomy_order_intents ADD COLUMN attempts INTEGER NOT NULL DEFAULT 0',
'ALTER TABLE autonomy_order_intents ADD COLUMN last_error TEXT',
'ALTER TABLE autonomy_predictions ADD COLUMN learning_eligible INTEGER NOT NULL DEFAULT 0',
"ALTER TABLE autonomy_predictions ADD COLUMN origin TEXT NOT NULL DEFAULT 'live'",
'ALTER TABLE autonomy_predictions ADD COLUMN replay_run_id INTEGER',
'ALTER TABLE autonomy_replay_runs ADD COLUMN cursor_effective_at TEXT',
// what this run was told about its predecessor's mistakes, kept on the run
// so a result can always be traced back to the text that produced it
'ALTER TABLE autonomy_replay_runs ADD COLUMN feedback_brief TEXT',
'ALTER TABLE autonomy_replay_runs ADD COLUMN parent_run_id INTEGER',
"ALTER TABLE autonomy_calibration_snapshots ADD COLUMN source TEXT NOT NULL DEFAULT 'live'",
'ALTER TABLE autonomy_calibration_snapshots ADD COLUMN replay_run_id INTEGER',
'ALTER TABLE autonomy_calibration_snapshots ADD COLUMN distinct_instruments INTEGER',
'ALTER TABLE autonomy_calibration_snapshots ADD COLUMN top_instrument_share REAL',
// The dashboard groups jobs by type/lane/status on every poll. autonomy_jobs is
// half a million rows and grows by roughly nine thousand a day from the archive
// reconcile loop, so without this that one query was 548ms and rising.
// created_at is included to keep the MAX() index-only.
'CREATE INDEX IF NOT EXISTS idx_autonomy_jobs_group ON autonomy_jobs(job_type, lane, status, created_at)',
'CREATE INDEX IF NOT EXISTS idx_calibration_snapshot_latest ON autonomy_calibration_snapshots(cohort_key, source, id)',
]) {
try {
db.exec(statement);
} catch (error) {
// Re-running these is normal, the column is already there. Anything else
// means a migration genuinely failed and we want to hear about it.
if (!isDuplicateColumn(error)) {
console.error(`[autonomy-schema] migration failed: ${statement}`, error.message, error.stack);
}
}
}
}
module.exports = { AUTONOMY_SCHEMA_VERSION, initAutonomySchema };
src/autonomy/settings.js
+67
View File
@@ -0,0 +1,67 @@
// Runtime settings for the autonomy stack. The env vars stay the default, the
// table is the override, so nothing changes behaviour until somebody deliberately
// writes a row. Kept deliberately tiny -- this is a control plane, not a config
// system, and every key here can move real money or stop the pipeline.
const EXECUTION_MODES = ['shadow', 'paper'];
const KEYS = {
executionMode: 'execution_mode',
killSwitch: 'execution_kill_switch',
};
function readSetting(db, key) {
try {
const row = db.prepare('SELECT value FROM autonomy_settings WHERE key = ?').get(key);
return row ? row.value : null;
} catch (error) {
// A missing table means an older schema, which should behave like "no override"
// rather than taking the worker down with it.
console.error(`[settings] could not read ${key}:`, error.message);
return null;
}
}
function writeSetting(db, key, value, updatedBy = 'admin') {
db.prepare(`
INSERT INTO autonomy_settings(key, value, updated_by) VALUES (?, ?, ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_by = excluded.updated_by,
updated_at = datetime('now')
`).run(key, String(value), updatedBy);
}
// Truthy strings people actually type, rather than only accepting 'true'
function isOn(value) {
return ['1', 'true', 'on', 'yes', 'engaged'].includes(String(value || '').trim().toLowerCase());
}
function getExecutionControls(db, env = process.env) {
const stored = readSetting(db, KEYS.executionMode);
const fallback = env.AUTONOMY_EXECUTION_MODE || 'shadow';
const mode = EXECUTION_MODES.includes(String(stored)) ? String(stored) : fallback;
return {
mode: EXECUTION_MODES.includes(mode) ? mode : 'shadow',
modeSource: EXECUTION_MODES.includes(String(stored)) ? 'settings' : 'env',
killSwitch: isOn(readSetting(db, KEYS.killSwitch)),
};
}
function setExecutionMode(db, mode, updatedBy) {
if (!EXECUTION_MODES.includes(mode)) {
throw new Error(`unsupported execution mode: ${mode} (expected ${EXECUTION_MODES.join(' or ')})`);
}
writeSetting(db, KEYS.executionMode, mode, updatedBy);
return getExecutionControls(db);
}
function setKillSwitch(db, engaged, updatedBy) {
writeSetting(db, KEYS.killSwitch, engaged ? 'true' : 'false', updatedBy);
return getExecutionControls(db);
}
module.exports = {
EXECUTION_MODES, KEYS, isOn,
readSetting, writeSetting,
getExecutionControls, setExecutionMode, setKillSwitch,
};
src/brokers/alpacaPaper.js
+36
View File
@@ -0,0 +1,36 @@
const PAPER_BASE_URL = 'https://paper-api.alpaca.markets';
function createAlpacaPaperClient({ keyId, secretKey } = {}) {
if (!keyId || !secretKey) throw new Error('Alpaca paper credentials are required');
async function request(path, options = {}) {
const response = await fetch(`${PAPER_BASE_URL}${path}`, {
...options,
headers: {
'APCA-API-KEY-ID': keyId,
'APCA-API-SECRET-KEY': secretKey,
'Content-Type': 'application/json',
...(options.headers || {}),
},
});
const text = await response.text();
let body = null;
try { body = text ? JSON.parse(text) : null; } catch (_) { body = { raw: text }; }
if (!response.ok) {
const error = new Error(body?.message || `Alpaca paper API returned ${response.status}`);
error.status = response.status;
error.body = body;
throw error;
}
return body;
}
return {
getAccount: () => request('/v2/account'),
getAssets: () => request('/v2/assets?status=active&asset_class=us_equity'),
getOrders: () => request('/v2/orders?status=all&limit=500&direction=desc'),
getPositions: () => request('/v2/positions'),
getOrderByClientId: (clientOrderId) => request(`/v2/orders:by_client_order_id?client_order_id=${encodeURIComponent(clientOrderId)}`),
submitOrder: (order) => request('/v2/orders', { method: 'POST', body: JSON.stringify(order) }),
};
}
module.exports = { PAPER_BASE_URL, createAlpacaPaperClient };
src/config.js
+4
View File
@@ -29,6 +29,10 @@ if (process.env.FINNHUB_API_KEY) config.finnhub.apiKey = process.env.FINNH
if (process.env.OPEN_ROUTER_API_KEY) config.openRouter.apiKey = process.env.OPEN_ROUTER_API_KEY;
if (process.env.OPEN_ROUTER_LLM_MODEL) config.openRouter.llmModel = process.env.OPEN_ROUTER_LLM_MODEL;
if (process.env.OPEN_ROUTER_EMBED_MODEL) config.openRouter.embeddingModel = process.env.OPEN_ROUTER_EMBED_MODEL;
// OPEN_ROUTER_CHEAP_MODEL was already set in the environment and nothing read it,
// so graph entity resolution ran on the reasoning model: 7,558 reasoning tokens to
// answer "reply with just the number", 113x the cost of a model that just answers.
if (process.env.OPEN_ROUTER_CHEAP_MODEL) config.openRouter.cheapModel = process.env.OPEN_ROUTER_CHEAP_MODEL;
if (process.env.GDELT_BQ_PROJECT) config.gdelt.bigQueryProject = process.env.GDELT_BQ_PROJECT;
if (process.env.GDELT_BQ_KEY_FILE) config.gdelt.bigQueryKeyFile = process.env.GDELT_BQ_KEY_FILE;
src/content.js
+34 -10
View File
@@ -68,8 +68,11 @@ const selectPartitionedArticlesMissingContent = db.prepare(`
SELECT id, url, title, description, source, pub_date_effective,
ROW_NUMBER() OVER (PARTITION BY source ORDER BY pub_date_effective DESC, id DESC) AS rn
FROM articles
WHERE (content IS NULL OR TRIM(content) = '')
AND (content_status IS NULL OR content_status = 'pending')
-- content_status is the authority on whether a row has been fetched, and it
-- agrees with the content column on all 2.2M rows. Testing TRIM(content) here
-- as well meant reading a 4GB blob column just to find out which rows to skip,
-- and it stopped the partial index below being usable at all.
WHERE (content_status IS NULL OR content_status = 'pending')
AND (content_retry_after IS NULL OR content_retry_after <= datetime('now'))
AND (id % ?) = ?
)
@@ -179,17 +182,39 @@ async function fetchPlainHtml(url) {
}
// Every individual step below has its own timeout, but acquiring the shared
// session does not, and it is awaited while holding a browser slot. A wedged
// chromium therefore parks all eight slots forever, and because nothing throws
// there is not a single line in the log to say so: content fetching simply stops.
// That is exactly what happened after the 4 Sept restart, ~328 articles in.
// This is the outer bound that guarantees the slot always comes back.
const BROWSER_HARD_TIMEOUT = 90000;
async function fetchBrowserHtml(url) {
await browserSemaphore.acquire();
try {
const maxConcurrentPages = Number(config.browser?.maxConcurrentPages) || 8;
const session = await getSharedBrowserSession({
requestTimeout: BROWSER_FETCH_TIMEOUT,
maxConcurrentPages,
let timer;
const expired = new Promise((_, reject) => {
timer = setTimeout(() => reject(new Error(
`browser fetch exceeded ${BROWSER_HARD_TIMEOUT}ms for ${url}, the session is probably wedged`)),
BROWSER_HARD_TIMEOUT);
});
const html = await session.fetchRenderedHtml(url, { timeout: BROWSER_FETCH_TIMEOUT });
return { html, finalUrl: url };
try {
return await Promise.race([
(async () => {
const session = await getSharedBrowserSession({
requestTimeout: BROWSER_FETCH_TIMEOUT,
maxConcurrentPages,
});
const html = await session.fetchRenderedHtml(url, { timeout: BROWSER_FETCH_TIMEOUT });
return { html, finalUrl: url };
})(),
expired,
]);
} finally {
clearTimeout(timer);
}
} finally {
browserSemaphore.release();
}
@@ -409,8 +434,7 @@ async function runBackfillWorker({ workerIndex, workerCount, perSource, batchSiz
function hasPendingContent() {
return Boolean(db.prepare(`
SELECT 1 FROM articles
WHERE (content IS NULL OR TRIM(content) = '')
AND (content_status IS NULL OR content_status = 'pending')
WHERE (content_status IS NULL OR content_status = 'pending')
AND (content_retry_after IS NULL OR content_retry_after <= datetime('now'))
LIMIT 1
`).get());
src/db/index.js
+25 -1
View File
@@ -48,8 +48,32 @@ db.exec(`
CREATE INDEX IF NOT EXISTS idx_articles_event_id ON articles(event_id);
CREATE INDEX IF NOT EXISTS idx_articles_has_embedding ON articles(has_embedding);
CREATE INDEX IF NOT EXISTS idx_articles_pub_date_effective ON articles(pub_date_effective DESC);
CREATE INDEX IF NOT EXISTS idx_articles_usable_pub_date
ON articles(pub_date_effective DESC, id DESC)
WHERE content IS NOT NULL
AND content != ''
AND is_index_page = 0
AND has_embedding = 1;
-- The content backfill picker partitions by source and orders by pub date, and
-- without this it built two temp b-trees over every unfetched row: 194 seconds
-- per call on a 2.2M row archive, synchronously, which froze the whole process.
-- Column order matches PARTITION BY source ORDER BY pub_date_effective DESC, id DESC
-- so the window function can just walk it.
CREATE INDEX IF NOT EXISTS idx_articles_pending_fetch
ON articles(source, pub_date_effective DESC, id DESC)
WHERE content_status IS NULL OR content_status = 'pending';
`);
// Without stats the planner ignores the partial index above and falls back to the
// content_status index plus a temp b-tree, which is roughly 70% slower. optimize
// only re-analyses what has actually drifted, so this is cheap after the first run.
try {
db.exec('PRAGMA optimize;');
} catch (error) {
console.error('[db] PRAGMA optimize failed, query plans may be stale:', error.message);
}
db.exec(`
CREATE TABLE IF NOT EXISTS article_embedding_store (
article_id INTEGER NOT NULL,
@@ -149,4 +173,4 @@ db.exec(`
);
`);
module.exports = db;
module.exports = db;
src/db/pgAsync.js
+24
View File
@@ -0,0 +1,24 @@
const { Pool } = require('pg');
const pools = new Map();
function postgresUrl() { return process.env.DURIIN_POSTGRES_URL || process.env.DATABASE_URL; }
function poolFor(schema = 'intelligence') {
const connectionString = postgresUrl();
if (!connectionString) throw new Error('DURIIN_POSTGRES_URL is required');
const key = `${connectionString}|${schema}`;
if (!pools.has(key)) {
pools.set(key, new Pool({
connectionString,
max: Math.max(1, Number(process.env.POSTGRES_HTTP_POOL_SIZE) || 4),
options: `-c search_path=${schema},public`,
}));
}
return pools.get(key);
}
async function all(schema, sql, params = []) { return (await poolFor(schema).query(sql, params)).rows; }
async function get(schema, sql, params = []) { return (await poolFor(schema).query(sql, params)).rows[0]; }
module.exports = { poolFor, all, get };
src/db/runtime.js
+179
View File
@@ -0,0 +1,179 @@
const { Pool } = require('pg');
const deasync = require('deasync');
const SqliteDatabase = require('better-sqlite3');
const pools = new Map();
function isPostgresEnabled() {
return String(process.env.DURIIN_DB_BACKEND || '').toLowerCase() === 'postgres' || Boolean((process.env.DURIIN_POSTGRES_URL || process.env.DATABASE_URL) && process.env.DURIIN_USE_POSTGRES === 'true');
}
function poolFor(schema) {
const connectionString = process.env.DURIIN_POSTGRES_URL || process.env.DATABASE_URL;
if (!connectionString) throw new Error('DURIIN_POSTGRES_URL is required for postgres runtime');
const key = `${connectionString}|${schema}`;
if (!pools.has(key)) {
pools.set(key, new Pool({
connectionString,
max: Math.max(1, Number(process.env.POSTGRES_RUNTIME_POOL_SIZE) || 4),
options: `-c search_path=${schema},public`,
}));
}
return pools.get(key);
}
function querySync(pool, sql, params = []) {
let done = false;
let result;
let error;
pool.query(sql, params).then((value) => { result = value; done = true; }).catch((err) => { error = err; done = true; });
deasync.loopWhile(() => !done);
if (error) throw error;
return result;
}
function connectSync(pool) {
let done = false;
let client;
let error;
pool.connect().then((value) => { client = value; done = true; }).catch((err) => { error = err; done = true; });
deasync.loopWhile(() => !done);
if (error) throw error;
return client;
}
function normalizeParams(params) {
if (params.length === 1 && params[0] && typeof params[0] === 'object' && !Array.isArray(params[0]) && !Buffer.isBuffer(params[0])) {
return params[0];
}
return params.flat();
}
function rewritePlaceholders(sql, params) {
if (params && !Array.isArray(params)) {
const values = [];
const text = sql.replace(/@([A-Za-z_][A-Za-z0-9_]*)/g, (_, name) => {
values.push(params[name]);
return `$${values.length}`;
});
return { sql: text, params: values };
}
let index = 0;
return { sql: sql.replace(/\?/g, () => `$${++index}`), params: params || [] };
}
function rewriteSql(sql, params) {
let text = String(sql).trim();
const pragmaTable = text.match(/^PRAGMA\s+table_info\((?:"([^"]+)"|'([^']+)'|([^)]+))\)$/i);
if (pragmaTable) {
const table = String(pragmaTable[1] || pragmaTable[2] || pragmaTable[3] || '').trim();
return {
sql: `
SELECT ordinal_position - 1 AS cid,
column_name AS name,
data_type AS type,
CASE WHEN is_nullable = 'NO' THEN 1 ELSE 0 END AS notnull,
column_default AS dflt_value,
0 AS pk
FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = $1
ORDER BY ordinal_position
`,
params: [table],
};
}
text = text.replace(/INSERT\s+OR\s+IGNORE\s+INTO/gi, 'INSERT INTO');
text = text.replace(/INSERT\s+OR\s+REPLACE\s+INTO\s+autonomy_outcomes\s*\(([^)]+)\)\s*VALUES\s*\(([^)]+)\)/i,
(match, columns, values) => {
const names = columns.split(',').map((item) => item.trim().replace(/"/g, ''));
const updates = names.filter((name) => name !== 'prediction_id').map((name) => `${name}=EXCLUDED.${name}`).join(', ');
return `INSERT INTO autonomy_outcomes (${columns}) VALUES (${values}) ON CONFLICT (prediction_id) DO UPDATE SET ${updates}`;
});
text = text.replace(/AUTOINCREMENT/gi, 'GENERATED BY DEFAULT AS IDENTITY');
text = text.replace(/INTEGER\s+PRIMARY\s+KEY\s+GENERATED BY DEFAULT AS IDENTITY/gi, 'BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY');
text = text.replace(/INTEGER\s+PRIMARY\s+KEY\s+AUTOINCREMENT/gi, 'BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY');
text = text.replace(/ingested_at\s*>=\s*datetime\('now',\s*'-48 hours'\)/gi, "ingested_at >= to_char(CURRENT_TIMESTAMP - interval '48 hours', 'YYYY-MM-DD HH24:MI:SS')");
text = text.replace(/datetime\('now',\s*\?\)/gi, 'CURRENT_TIMESTAMP + (?::interval)');
text = text.replace(/datetime\('now',\s*'\+60 seconds'\)/gi, "CURRENT_TIMESTAMP + interval '60 seconds'");
text = text.replace(/datetime\('now',\s*'([^']+)'\)/gi, "CURRENT_TIMESTAMP + interval '$1'");
text = text.replace(/datetime\('now'\)/gi, 'CURRENT_TIMESTAMP');
text = text.replace(/date\('now'\)/gi, 'CURRENT_DATE');
text = text.replace(/datetime\(COALESCE\(([^)]+)\)\)/gi, 'COALESCE($1)::timestamp');
text = text.replace(/datetime\((p\.information_cutoff),\s*'\+'\s*\|\|\s*(p\.horizon_days)\s*\|\|\s*' days'\)/gi, "($1::timestamp + ($2 || ' days')::interval)");
text = text.replace(/datetime\(([^)]+)\)/gi, '($1)::timestamp');
const rewritten = rewritePlaceholders(text, params);
let finalSql = rewritten.sql;
if (/^INSERT\s+INTO\s+autonomy_jobs\b/i.test(finalSql) && !/ON\s+CONFLICT/i.test(finalSql)) finalSql += ' ON CONFLICT DO NOTHING';
if (/^INSERT\s+INTO\s+autonomy_order_intents\b/i.test(finalSql) && !/ON\s+CONFLICT/i.test(finalSql)) finalSql += ' ON CONFLICT DO NOTHING';
if (/^INSERT\s+INTO\s+autonomy_schema\b/i.test(finalSql) && !/ON\s+CONFLICT/i.test(finalSql)) finalSql += ' ON CONFLICT DO NOTHING';
if (/^INSERT\s+INTO\s+autonomy_proposals\b/i.test(finalSql) && !/RETURNING\s+id/i.test(finalSql)) finalSql += ' RETURNING id';
return { sql: finalSql, params: rewritten.params };
}
class PgCompatDb {
constructor(schema) {
this.schema = schema;
this.dialect = 'postgres';
this.pool = poolFor(schema);
}
pragma() { return undefined; }
prepare(sql) {
const db = this;
const target = () => db.activeClient || db.pool;
return {
get(...rawParams) {
const { sql: text, params } = rewriteSql(sql, normalizeParams(rawParams));
return querySync(target(), text, params).rows[0];
},
all(...rawParams) {
const { sql: text, params } = rewriteSql(sql, normalizeParams(rawParams));
return querySync(target(), text, params).rows;
},
run(...rawParams) {
const { sql: text, params } = rewriteSql(sql, normalizeParams(rawParams));
const result = querySync(target(), text, params);
return { changes: result.rowCount || 0, lastInsertRowid: result.rows?.[0]?.id ?? null };
},
};
}
exec(sql) {
const statements = String(sql).split(';').map((statement) => statement.trim()).filter(Boolean);
for (const statement of statements) {
const { sql: text, params } = rewriteSql(statement, []);
querySync(this.activeClient || this.pool, text, params);
}
}
transaction(fn) {
const db = this;
const run = (...args) => {
const client = connectSync(db.pool);
try {
db.activeClient = client;
querySync(client, 'BEGIN', []);
const result = fn(...args);
querySync(client, 'COMMIT', []);
return result;
} catch (error) {
try { querySync(client, 'ROLLBACK', []); } catch (_) {}
throw error;
} finally {
db.activeClient = null;
client.release();
}
};
run.immediate = run;
return run;
}
}
function openRuntimeDb(path, { schema = 'intelligence', readonly = false } = {}) {
if (isPostgresEnabled()) return new PgCompatDb(schema);
return new SqliteDatabase(path, readonly ? { readonly: true } : undefined);
}
module.exports = { openRuntimeDb, isPostgresEnabled, PgCompatDb, rewriteSql };
src/ingest.js
+7 -1
View File
@@ -1,6 +1,7 @@
const db = require('./db');
const { normalizeTitle } = require('./dedup');
const { markSourceRun } = require('./state');
const { guardEffectivePubDate } = require('./pubDateGuard');
const sourcesById = Object.fromEntries(
require('../sources.json').map((s) => [s.id, s])
@@ -88,6 +89,11 @@ function ingestArticle(article) {
const ingestedAt = new Date().toISOString();
const language = (sourcesById[source] && sourcesById[source].language) || null;
// pub_date keeps whatever the source claimed (it is still useful for
// debugging a broken feed), but the effective date — the one the coordinator
// turns into an information cutoff — refuses anything from the future.
const effectivePubDate = guardEffectivePubDate(pubDate, ingestedAt, { source, url });
try {
const result = insertArticle.run(
title,
@@ -98,7 +104,7 @@ function ingestArticle(article) {
source,
pubDate,
ingestedAt,
pubDate || ingestedAt,
effectivePubDate,
language
);
src/pubDateGuard.js
+61
View File
@@ -0,0 +1,61 @@
// Guard against publication dates that sit in the future.
//
// pub_date_effective is what the autonomy coordinator uses to derive a
// prediction's information_cutoff (max pub_date_effective across an event's
// articles), so a single bogus feed date drags the cutoff forward and quietly
// breaks evidence-cutoff enforcement and outcome scoring. Production currently
// has exactly one such row, but one is enough to poison an event.
//
// Tolerance: 48 hours. It has to swallow the legitimate cases —
// * date only strings ("2026-08-29") are stored as midnight UTC, and a
// publisher in UTC+14 can legitimately stamp tomorrow's date,
// * feeds that emit local time without an offset, worst case ~14h ahead,
// * modest clock skew on the publisher's box.
// 48h covers all of that with room to spare while still catching anything
// genuinely wrong — the offending production row is about four months out.
const DEFAULT_TOLERANCE_MS = 48 * 60 * 60 * 1000;
function toleranceMs() {
const hours = Number(process.env.INGEST_FUTURE_PUB_DATE_HOURS);
if (Number.isFinite(hours) && hours > 0) return hours * 60 * 60 * 1000;
return DEFAULT_TOLERANCE_MS;
}
// Returns { ok, value, skewMs, toleranceMs }. `value` is null when the date is
// implausible so the caller can fall back to ingestion time. The article itself
// is never dropped for this — a bad date is not a bad article.
function checkPubDate(value, now = Date.now(), tolerance = toleranceMs()) {
if (!value) return { ok: true, value: null, skewMs: 0, toleranceMs: tolerance };
const parsed = new Date(value).getTime();
if (Number.isNaN(parsed)) return { ok: true, value: null, skewMs: 0, toleranceMs: tolerance };
const skewMs = parsed - now;
if (skewMs > tolerance) {
return { ok: false, value: null, skewMs, toleranceMs: tolerance };
}
return { ok: true, value, skewMs, toleranceMs: tolerance };
}
// Same check, but it also does the shouting. Keeps ingest.js readable and makes
// sure every clamp lands in the logs with the source and the offending value.
function guardEffectivePubDate(pubDate, fallback, context = {}) {
const verdict = checkPubDate(pubDate);
if (verdict.ok) return pubDate || fallback;
const days = (verdict.skewMs / 86400000).toFixed(1);
console.warn(
`[ingest] refusing future pub date from "${context.source || 'unknown source'}": ${pubDate} is ${days} days ahead ` +
`(tolerance ${Math.round(verdict.toleranceMs / 3600000)}h) — pub_date_effective falls back to ${fallback}. url=${context.url || 'n/a'}`
);
return fallback;
}
module.exports = {
checkPubDate,
guardEffectivePubDate,
DEFAULT_TOLERANCE_MS,
};
src/routes/admin.js
+502 -73
View File
@@ -1,27 +1,239 @@
const fs = require('fs');
const path = require('path');
const fastifyStatic = require('@fastify/static');
const { Worker } = require('node:worker_threads');
const db = require('../db');
const config = require('../config');
const Database = require('better-sqlite3');
const { openRuntimeDb, isPostgresEnabled } = require('../db/runtime');
const pg = require('../db/pgAsync');
const opsRoutes = require('./ops');
let idb = null;
let adb = null;
let statsSummaryCache = null;
let statsDetailCache = null;
const configDir = path.resolve(__dirname, '..', '..');
// The archive is resolved exactly like the workers do it (workers/index.js:31):
// DURIIN_DB wins, then config, and only then the repo relative default. The old
// code here went straight to config.database.path — a repo relative
// "./archive.sqlite" — which inside the container only ever pointed at the real
// data because of a build time symlink, and which quietly opens a brand new
// empty database when that symlink is not there.
function resolveArchivePath() {
const raw = process.env.DURIIN_DB
|| config.duriin_db
|| (config.database && config.database.path)
|| './archive.sqlite';
return path.isAbsolute(raw) ? raw : path.resolve(configDir, raw);
}
function resolveIntelligencePath() {
return process.env.INTELLIGENCE_DB
|| (config.intelligence_db
? (path.isAbsolute(config.intelligence_db) ? config.intelligence_db : path.resolve(configDir, config.intelligence_db))
: path.resolve(configDir, 'intelligence.sqlite'));
}
// Opens the archive and *proves* it is the archive before handing it back. Any
// failure throws with the resolved target in the message — serving the wrong
// database silently is far worse than an error on the sql console.
function getArchiveDb() {
if (adb) return adb;
const target = isPostgresEnabled() ? 'postgres schema "archive"' : resolveArchivePath();
try {
if (isPostgresEnabled()) {
const handle = openRuntimeDb(resolveArchivePath(), { schema: 'archive' });
const probe = handle.prepare("SELECT to_regclass('archive.articles') AS relation").get();
if (!probe || !probe.relation) throw new Error('the archive schema has no articles table');
adb = handle;
return adb;
}
const filePath = resolveArchivePath();
if (!fs.existsSync(filePath)) throw new Error('no such file');
const handle = new Database(filePath, { fileMustExist: true });
try {
const probe = handle.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name='articles'").get();
if (!probe) throw new Error('this file has no articles table, so it is not the archive');
} catch (probeError) {
handle.close();
throw probeError;
}
adb = handle;
return adb;
} catch (error) {
// never cached — if the volume shows up later the next request recovers
console.error(`[admin] archive database unavailable (${target}):`, error);
throw new Error(`archive database unavailable (${target}): ${error.message}`);
}
}
function calculateArchiveStats() {
const databasePath = resolveArchivePath();
const workerPath = path.resolve(__dirname, '..', 'adminStatsWorker.js');
return new Promise((resolve, reject) => {
const worker = new Worker(workerPath, { workerData: { databasePath } });
const timer = setTimeout(() => {
worker.terminate();
reject(new Error('archive statistics timed out'));
}, 60_000);
worker.once('message', (message) => {
clearTimeout(timer);
if (message.error) reject(new Error(message.error));
else resolve(message.value);
});
worker.once('error', (error) => {
clearTimeout(timer);
reject(error);
});
worker.once('exit', (code) => {
if (code !== 0) {
clearTimeout(timer);
reject(new Error(`archive statistics worker exited with code ${code}`));
}
});
});
}
function getIntelligenceDb() {
if (idb) return idb;
const configDir = path.resolve(__dirname, '..', '..');
const rawPath = process.env.INTELLIGENCE_DB
|| (config.intelligence_db
? (path.isAbsolute(config.intelligence_db) ? config.intelligence_db : path.resolve(configDir, config.intelligence_db))
: path.resolve(configDir, 'intelligence.sqlite'));
const rawPath = resolveIntelligencePath();
if (!fs.existsSync(rawPath)) return null;
if (!isPostgresEnabled() && !fs.existsSync(rawPath)) {
console.error(`[admin] intelligence database unavailable: no such file (${rawPath})`);
return null;
}
idb = new Database(rawPath);
idb = isPostgresEnabled() ? openRuntimeDb(rawPath, { schema: 'intelligence' }) : new Database(rawPath);
return idb;
}
// Prediction origins are not interchangeable. 'live' is genuine real time work,
// 'historical' is coordinator backfill over the archive and 'replay' is
// walk-forward replay. Averaging them into a single accuracy number reads like
// live edge when it is nothing of the sort, so the overview reports them side by
// side and lets the page say "nothing live yet" out loud.
const OUTCOME_ORIGINS = ['live', 'historical', 'replay'];
const OUTCOMES_BY_ORIGIN_SQL = `
SELECT p.origin AS origin,
COUNT(*) AS total,
SUM(o.direction_correct) AS correct,
AVG(o.excess_return) AS average_excess_return
FROM autonomy_outcomes o
JOIN autonomy_predictions p ON p.id = o.prediction_id
GROUP BY p.origin
`;
const PREDICTIONS_BY_ORIGIN_SQL = `
SELECT origin, status, COUNT(*) AS count
FROM autonomy_predictions
GROUP BY origin, status
`;
function summarizeOutcomeOrigins(rows) {
const buckets = new Map();
for (const name of OUTCOME_ORIGINS) {
buckets.set(name, { origin: name, total: 0, correct: 0, average_excess_return: null });
}
for (const row of rows || []) {
const origin = String(row.origin || 'unknown').toLowerCase();
if (!buckets.has(origin)) buckets.set(origin, { origin, total: 0, correct: 0, average_excess_return: null });
const bucket = buckets.get(origin);
bucket.total = Number(row.total || 0);
bucket.correct = Number(row.correct || 0);
bucket.average_excess_return = row.average_excess_return == null ? null : Number(row.average_excess_return);
}
const byOrigin = [...buckets.values()];
const live = byOrigin.find((bucket) => bucket.origin === 'live');
return { byOrigin, live };
}
// Diversification gate, mirrored from the policy layer. A cohort only earns the
// right to authorise a trade when it is big enough, spread over enough tickers
// and not dominated by a single one. Missing diversity data counts as a fail —
// the policy treats unknown as disqualifying and the admin view has to agree,
// otherwise the screen says "qualified" while the trader abstains.
const CALIBRATION_MIN_SAMPLES = 30;
const CALIBRATION_MIN_INSTRUMENTS = 5;
const CALIBRATION_MAX_CONCENTRATION = 0.5;
const CALIBRATION_BASE_COLUMNS = [
'cohort_key', 'sample_size', 'effective_sample_size', 'directional_probability',
'expected_excess_return', 'lower_return', 'upper_return', 'created_at',
];
// added by the diversification work; pre-existing rows/deployments may not have
// them yet so they are selected only when they really exist
const CALIBRATION_OPTIONAL_COLUMNS = ['distinct_instruments', 'top_instrument_share', 'source'];
function calibrationSnapshotSql(available) {
const columns = CALIBRATION_BASE_COLUMNS.slice();
for (const name of CALIBRATION_OPTIONAL_COLUMNS) {
columns.push(available.has(name) ? name : `NULL AS ${name}`);
}
return `SELECT ${columns.join(', ')} FROM autonomy_calibration_snapshots ORDER BY id DESC LIMIT 8`;
}
function gate(value, ok, threshold) {
return { value: value == null ? null : Number(value), threshold, ok, known: value != null };
}
function decorateCalibration(rows) {
return (rows || []).map((row) => {
const samples = row.sample_size == null ? null : Number(row.sample_size);
const instruments = row.distinct_instruments == null ? null : Number(row.distinct_instruments);
const share = row.top_instrument_share == null ? null : Number(row.top_instrument_share);
const checks = {
sample_size: gate(samples, samples != null && samples >= CALIBRATION_MIN_SAMPLES, CALIBRATION_MIN_SAMPLES),
distinct_instruments: gate(instruments, instruments != null && instruments >= CALIBRATION_MIN_INSTRUMENTS, CALIBRATION_MIN_INSTRUMENTS),
top_instrument_share: gate(share, share != null && share <= CALIBRATION_MAX_CONCENTRATION, CALIBRATION_MAX_CONCENTRATION),
};
const reasons = [];
if (!checks.sample_size.ok) {
reasons.push(samples == null ? 'sample size unknown' : `only ${samples} samples, needs ${CALIBRATION_MIN_SAMPLES}`);
}
if (!checks.distinct_instruments.ok) {
reasons.push(instruments == null ? 'instrument spread unknown' : `only ${instruments} distinct ticker${instruments === 1 ? '' : 's'}, needs ${CALIBRATION_MIN_INSTRUMENTS}`);
}
if (!checks.top_instrument_share.ok) {
reasons.push(share == null ? 'concentration unknown' : `${Math.round(share * 100)}% sits in one ticker, cap is ${Math.round(CALIBRATION_MAX_CONCENTRATION * 100)}%`);
}
// cohort keys are versioned now. legacy rows use the old key shape and will
// never match a current lookup, so they must not read as live calibration.
const legacy = !String(row.cohort_key || '').startsWith('v2|');
return {
...row,
source: row.source || null,
legacy_cohort_key: legacy,
qualification: { qualified: reasons.length === 0, checks, reasons },
};
});
}
function normalizeOriginCounts(rows) {
return (rows || []).map((row) => ({
origin: String(row.origin || 'unknown').toLowerCase(),
status: row.status,
count: Number(row.count || 0),
}));
}
const adminUser = (config.admin && config.admin.username) || 'admin';
const adminPass = (config.admin && config.admin.password) || 'changeme';
@@ -54,6 +266,8 @@ const pagesDir = path.join(publicDir, 'pages');
// map pretty url → page html file. keep these close to the routes so its
// obvious when a page gets added or renamed.
const pageMap = {
'/admin/console': path.join(publicDir, 'app.html'),
'/admin/autonomy': path.join(pagesDir, 'autonomy.html'),
'/admin/ingest/articles': path.join(pagesDir, 'ingest', 'articles.html'),
'/admin/ingest/events': path.join(pagesDir, 'ingest', 'events.html'),
'/admin/stats': path.join(pagesDir, 'stats.html'),
@@ -71,27 +285,35 @@ function sendPage(reply, filePath) {
}
async function adminRoutes(fastify) {
// Control plane for the ops dashboard. Lives in its own file, shares this one's
// auth and db handles so there is exactly one of each.
fastify.register(opsRoutes, { checkAuth, getIntelligenceDb, getArchiveDb });
// gate every request under /admin/* behind basic auth (covers pages, api, and assets)
fastify.addHook('onRequest', async (request, reply) => {
if (!checkAuth(request, reply)) return reply;
});
// static assets (css + js) under /admin/assets/*
// cache for an hour — avoids per-navigation revalidation round-trips
// for the admin panel. during dev use a hard-reload (cmd-shift-r)
// or bump the script src query string to bust it.
// Static assets are revalidated so an operator never runs stale UI code
// against a newly deployed autonomy API.
fastify.register(fastifyStatic, {
root: assetsDir,
prefix: '/admin/assets/',
decorateReply: false,
cacheControl: true,
maxAge: 3600 * 1000, // 1h, in ms (fastify-static forwards to send())
cacheControl: false,
etag: false,
lastModified: false,
setHeaders(res) {
res.setHeader('Cache-Control', 'no-store, no-cache, must-revalidate');
},
});
// top-level entry — redirect into ingest/articles
// top-level entry — the ops console is the primary surface now. The older
// per-page admin is still served underneath, the console frames the d3 graph
// from it rather than duplicating that visualisation.
fastify.get('/admin', async (request, reply) => {
reply.redirect('/admin/ingest/articles');
reply.redirect('/admin/console');
});
// ingest root — redirect to the articles subsection
@@ -113,6 +335,183 @@ async function adminRoutes(fastify) {
fastify.get(route, async (request, reply) => sendPage(reply, filePath));
}
// Autonomy control-room data. Keep this behind admin auth: it exposes model
// output, broker state and operational queue details that do not belong on the
// public status endpoint.
fastify.get('/admin/api/autonomy/overview', async (request, reply) => {
if (!checkAuth(request, reply)) return;
if (isPostgresEnabled()) {
const hasSchema = await pg.get('intelligence', "SELECT 1 FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2", ['intelligence', 'autonomy_jobs']);
if (!hasSchema) return { enabled: false, reason: 'autonomy schema is not initialized' };
const calibrationColumns = new Set((await pg.all('intelligence',
'SELECT column_name FROM information_schema.columns WHERE table_schema = $1 AND table_name = $2',
['intelligence', 'autonomy_calibration_snapshots'])).map((row) => row.column_name));
const [jobs, predictionCounts, predictionOriginRows, decisionCounts, proposalCounts, outcomeRows, instruments, latestRows, latestOrders, account, calibration, replay] = await Promise.all([
pg.all('intelligence', 'SELECT lane, status, COUNT(*) AS count FROM autonomy_jobs GROUP BY lane, status ORDER BY lane, status'),
pg.all('intelligence', 'SELECT status, COUNT(*) AS count FROM autonomy_predictions GROUP BY status'),
pg.all('intelligence', PREDICTIONS_BY_ORIGIN_SQL),
pg.all('intelligence', 'SELECT action, COUNT(*) AS count FROM autonomy_decisions GROUP BY action'),
pg.all('intelligence', 'SELECT status, COUNT(*) AS count FROM autonomy_proposals GROUP BY status'),
pg.all('intelligence', OUTCOMES_BY_ORIGIN_SQL),
pg.get('intelligence', 'SELECT COUNT(*) AS count FROM autonomy_instruments WHERE active=1 AND tradable=1'),
pg.all('intelligence', `
SELECT p.id, p.instrument, p.direction, p.event_type, p.causal_channel,
p.horizon_days, p.information_cutoff, p.evidence_article_ids,
p.invalidation_condition, p.learning_eligible, p.status, p.created_at,
d.action, d.calibrated_probability, d.expected_excess_return, d.rationale,
o.excess_return, o.direction_correct
FROM autonomy_predictions p
LEFT JOIN autonomy_decisions d ON d.id = (
SELECT MAX(d2.id) FROM autonomy_decisions d2 WHERE d2.prediction_id = p.id
)
LEFT JOIN autonomy_outcomes o ON o.prediction_id = p.id
ORDER BY p.id DESC LIMIT 12
`),
pg.all('intelligence', `
SELECT oi.id, oi.client_order_id, oi.instrument, oi.side, oi.notional,
oi.status, oi.broker_order_id, oi.attempts, oi.last_error,
oi.created_at, oi.updated_at, d.action
FROM autonomy_order_intents oi
JOIN autonomy_decisions d ON d.id = oi.decision_id
ORDER BY oi.id DESC LIMIT 12
`),
pg.get('intelligence', 'SELECT broker, equity, cash, buying_power, captured_at FROM autonomy_account_snapshots ORDER BY id DESC LIMIT 1'),
pg.all('intelligence', calibrationSnapshotSql(calibrationColumns)),
pg.get('intelligence', `
SELECT r.id, r.status, r.watermark_at, r.cursor_article_id, r.cursor_effective_at,
r.processed_articles, r.updated_at,
SUM(CASE WHEN p.status = 'resolved' THEN 1 ELSE 0 END) AS resolved_predictions,
COUNT(p.id) AS predictions,
SUM(o.direction_correct) AS correct_predictions,
(SELECT COUNT(*) FROM autonomy_replay_evaluations e WHERE e.replay_run_id = r.id) AS evaluations
FROM autonomy_replay_runs r
LEFT JOIN autonomy_predictions p ON p.replay_run_id = r.id
LEFT JOIN autonomy_outcomes o ON o.prediction_id = p.id
GROUP BY r.id ORDER BY r.id DESC LIMIT 1
`),
]);
const latestPredictions = latestRows.map((row) => {
let evidenceCount = 0;
try { evidenceCount = JSON.parse(row.evidence_article_ids || '[]').length; } catch (_) {}
const { evidence_article_ids: ignored, ...safeRow } = row;
return { ...safeRow, evidence_count: evidenceCount };
});
const origins = summarizeOutcomeOrigins(outcomeRows);
return {
enabled: true,
mode: process.env.AUTONOMY_EXECUTION_MODE || 'shadow',
broker: { name: 'Alpaca Paper', configured: Boolean(process.env.ALPACA_PAPER_KEY_ID && process.env.ALPACA_PAPER_SECRET_KEY) },
jobs, predictionCounts, decisionCounts, proposalCounts,
predictionsByOrigin: normalizeOriginCounts(predictionOriginRows),
outcomes: origins.live,
outcomesByOrigin: origins.byOrigin,
hasLiveOutcomes: origins.live.total > 0,
allowlistedInstruments: instruments.count, latestPredictions, latestOrders,
account: account || null, calibration: decorateCalibration(calibration), replay: replay || null,
generatedAt: new Date().toISOString(),
};
}
const intelligenceDb = getIntelligenceDb();
if (!intelligenceDb) return { enabled: false, reason: 'intelligence database unavailable' };
const hasSchema = isPostgresEnabled()
? intelligenceDb.prepare("SELECT 1 FROM information_schema.tables WHERE table_schema = ? AND table_name = ?").get('intelligence', 'autonomy_jobs')
: intelligenceDb.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='autonomy_jobs'").get();
if (!hasSchema) return { enabled: false, reason: 'autonomy schema is not initialized' };
const jobs = intelligenceDb.prepare(`
SELECT lane, status, COUNT(*) AS count
FROM autonomy_jobs GROUP BY lane, status ORDER BY lane, status
`).all();
const predictionCounts = intelligenceDb.prepare(`
SELECT status, COUNT(*) AS count FROM autonomy_predictions GROUP BY status
`).all();
const decisionCounts = intelligenceDb.prepare(`
SELECT action, COUNT(*) AS count FROM autonomy_decisions GROUP BY action
`).all();
const proposalCounts = intelligenceDb.prepare(`
SELECT status, COUNT(*) AS count FROM autonomy_proposals GROUP BY status
`).all();
const predictionOriginRows = intelligenceDb.prepare(PREDICTIONS_BY_ORIGIN_SQL).all();
const origins = summarizeOutcomeOrigins(intelligenceDb.prepare(OUTCOMES_BY_ORIGIN_SQL).all());
const instruments = intelligenceDb.prepare(`
SELECT COUNT(*) AS count FROM autonomy_instruments WHERE active=1 AND tradable=1
`).get();
const latestPredictions = intelligenceDb.prepare(`
SELECT p.id, p.instrument, p.direction, p.event_type, p.causal_channel,
p.horizon_days, p.information_cutoff, p.evidence_article_ids,
p.invalidation_condition, p.learning_eligible, p.status, p.created_at,
d.action, d.calibrated_probability, d.expected_excess_return, d.rationale,
o.excess_return, o.direction_correct
FROM autonomy_predictions p
LEFT JOIN autonomy_decisions d ON d.id = (
SELECT MAX(d2.id) FROM autonomy_decisions d2 WHERE d2.prediction_id = p.id
)
LEFT JOIN autonomy_outcomes o ON o.prediction_id = p.id
ORDER BY p.id DESC LIMIT 12
`).all().map((row) => {
let evidenceCount = 0;
try { evidenceCount = JSON.parse(row.evidence_article_ids || '[]').length; } catch (_) {}
const { evidence_article_ids: ignored, ...safeRow } = row;
return { ...safeRow, evidence_count: evidenceCount };
});
const latestOrders = intelligenceDb.prepare(`
SELECT oi.id, oi.client_order_id, oi.instrument, oi.side, oi.notional,
oi.status, oi.broker_order_id, oi.attempts, oi.last_error,
oi.created_at, oi.updated_at, d.action
FROM autonomy_order_intents oi
JOIN autonomy_decisions d ON d.id = oi.decision_id
ORDER BY oi.id DESC LIMIT 12
`).all();
const account = intelligenceDb.prepare(`
SELECT broker, equity, cash, buying_power, captured_at
FROM autonomy_account_snapshots ORDER BY id DESC LIMIT 1
`).get() || null;
const calibrationColumns = new Set(
intelligenceDb.prepare('PRAGMA table_info(autonomy_calibration_snapshots)').all().map((row) => row.name)
);
const calibration = decorateCalibration(
intelligenceDb.prepare(calibrationSnapshotSql(calibrationColumns)).all()
);
const replay = intelligenceDb.prepare(`
SELECT r.id, r.status, r.watermark_at, r.cursor_article_id, r.cursor_effective_at,
r.processed_articles, r.updated_at,
SUM(CASE WHEN p.status = 'resolved' THEN 1 ELSE 0 END) AS resolved_predictions,
COUNT(p.id) AS predictions,
SUM(o.direction_correct) AS correct_predictions,
(SELECT COUNT(*) FROM autonomy_replay_evaluations e WHERE e.replay_run_id = r.id) AS evaluations
FROM autonomy_replay_runs r
LEFT JOIN autonomy_predictions p ON p.replay_run_id = r.id
LEFT JOIN autonomy_outcomes o ON o.prediction_id = p.id
GROUP BY r.id ORDER BY r.id DESC LIMIT 1
`).get() || null;
return {
enabled: true,
mode: process.env.AUTONOMY_EXECUTION_MODE || 'shadow',
broker: {
name: 'Alpaca Paper',
configured: Boolean(process.env.ALPACA_PAPER_KEY_ID && process.env.ALPACA_PAPER_SECRET_KEY),
},
jobs,
predictionCounts,
predictionsByOrigin: normalizeOriginCounts(predictionOriginRows),
decisionCounts,
proposalCounts,
outcomes: origins.live,
outcomesByOrigin: origins.byOrigin,
hasLiveOutcomes: origins.live.total > 0,
allowlistedInstruments: instruments.count,
latestPredictions,
latestOrders,
account,
calibration,
replay,
generatedAt: new Date().toISOString(),
};
});
// list articles — all of them, not just the ones with embeddings
fastify.get('/admin/api/articles', async (request, reply) => {
if (!checkAuth(request, reply)) return;
@@ -156,18 +555,20 @@ async function adminRoutes(fastify) {
const where = conditions.length ? `WHERE ${conditions.join(' AND ')}` : '';
const total = db.prepare(`SELECT COUNT(*) as n FROM articles ${where}`).get(...params).n;
const total = conditions.length
? null
: (db.prepare("SELECT seq FROM sqlite_sequence WHERE name='articles'").get()?.seq || 0);
params.push(limit, offset);
const rows = db.prepare(`
params.push(limit + 1, offset);
const fetchedRows = db.prepare(`
SELECT id, title, url, source, pub_date, ingested_at, content_status, is_index_page, has_embedding, language
FROM articles
${where}
ORDER BY ingested_at DESC, id DESC
ORDER BY id DESC
LIMIT ? OFFSET ?
`).all(...params);
return { total, rows };
return { total, rows: fetchedRows.slice(0, limit), hasMore: fetchedRows.length > limit };
});
fastify.get('/admin/api/articles/:id', async (request, reply) => {
@@ -271,7 +672,7 @@ async function adminRoutes(fastify) {
// whitelist sort columns + direction so user input cant break the query
const sortMap = {
created_desc: 'e.created_at DESC',
created_desc: 'e.id DESC',
created_asc: 'e.created_at ASC',
articles_desc: 'article_count DESC',
articles_asc: 'article_count ASC',
@@ -279,32 +680,27 @@ async function adminRoutes(fastify) {
const orderBy = sortMap[q.sort] || sortMap.created_desc;
const whereClause = where.length ? `WHERE ${where.join(' AND ')}` : '';
const havingClause = having.length ? `HAVING ${having.join(' AND ')}` : '';
const countWhereClause = having.length ? `WHERE ${having.join(' AND ')}` : '';
// total count has to respect the HAVING clause too, so wrap the grouped query
const totalRow = db.prepare(`
SELECT COUNT(*) as n FROM (
SELECT e.id, COUNT(a.id) as article_count
FROM events e
LEFT JOIN articles a ON a.event_id = e.id
${whereClause}
GROUP BY e.id
${havingClause}
)
`).get(...whereParams, ...havingParams);
const rows = db.prepare(`
SELECT e.id, e.title, e.created_at, COUNT(a.id) as article_count
FROM events e
LEFT JOIN articles a ON a.event_id = e.id
${whereClause}
GROUP BY e.id
${havingClause}
ORDER BY ${orderBy}
// Avoid grouping the entire article archive for every page visit. The
// correlated count uses idx_articles_event_id and touches only the events
// that survive filtering/pagination.
const eventProjection = `
SELECT e.id, e.title, e.created_at,
(SELECT COUNT(*) FROM articles a WHERE a.event_id = e.id) AS article_count
FROM events e ${whereClause}
`;
const filteredEvents = `SELECT * FROM (${eventProjection}) ${countWhereClause}`;
const total = where.length || having.length
? null
: (db.prepare("SELECT seq FROM sqlite_sequence WHERE name='events'").get()?.seq || 0);
const fetchedRows = db.prepare(`
${filteredEvents}
ORDER BY ${orderBy.replaceAll('e.', '')}
LIMIT ? OFFSET ?
`).all(...whereParams, ...havingParams, limit, offset);
`).all(...whereParams, ...havingParams, limit + 1, offset);
return { total: totalRow.n, rows };
return { total, rows: fetchedRows.slice(0, limit), hasMore: fetchedRows.length > limit };
});
fastify.delete('/admin/api/events/:id', async (request, reply) => {
@@ -719,8 +1115,29 @@ async function adminRoutes(fastify) {
const { sql, database } = request.body || {};
if (!sql || !sql.trim()) { reply.code(400); return { error: 'no sql provided' }; }
const target = database === 'intelligence' ? getIntelligenceDb() : db;
if (!target) { reply.code(400); return { error: 'database not available' }; }
// empty/omitted means archive, the historic default. anything else has to be
// spelled correctly — a typo used to silently run against the archive.
const requested = String(database || 'archive').trim().toLowerCase() || 'archive';
if (requested !== 'archive' && requested !== 'intelligence') {
reply.code(400);
return { error: `unknown database "${requested}" — expected "archive" or "intelligence"` };
}
let target = null;
try {
target = requested === 'intelligence' ? getIntelligenceDb() : getArchiveDb();
} catch (error) {
console.error(`[admin] sql console cannot reach the ${requested} database:`, error);
reply.code(503);
return { error: error.message };
}
if (!target) {
const where = isPostgresEnabled() ? `postgres schema "${requested}"` : resolveIntelligencePath();
console.error(`[admin] sql console cannot reach the ${requested} database (${where})`);
reply.code(503);
return { error: `${requested} database unavailable (${where})` };
}
// split on semicolons, drop empty statements
const statements = sql.split(';').map(s => s.trim()).filter(s => s.length > 0);
@@ -731,13 +1148,18 @@ async function adminRoutes(fastify) {
for (const s of statements) {
try {
const stmt = target.prepare(s);
if (stmt.reader) {
// the postgres adapters dont expose better-sqlite3's `reader` flag, so
// without this fallback every SELECT went down the run() path and came
// back as a change count with no rows at all
const reads = typeof stmt.reader === 'boolean' ? stmt.reader : /^\s*(SELECT|WITH|PRAGMA|EXPLAIN|SHOW)\b/i.test(s);
if (reads) {
results.push({ sql: s, rows: stmt.all() });
} else {
const info = stmt.run();
results.push({ sql: s, changes: info.changes, lastInsertRowid: info.lastInsertRowid });
}
} catch (err) {
console.error(`[admin] sql console statement failed on ${requested}:`, s, err);
results.push({ sql: s, error: err.message });
}
}
@@ -745,38 +1167,45 @@ async function adminRoutes(fastify) {
return { results, elapsed: Date.now() - start };
});
// stats for dashboard header
fastify.get('/admin/api/stats', async (request, reply) => {
// Lightweight header summary, cached independently from the detailed stats
// page so navigation never waits for source/status aggregation.
fastify.get('/admin/api/stats/summary', async (request, reply) => {
if (!checkAuth(request, reply)) return;
if (statsSummaryCache && Date.now() - statsSummaryCache.at < 60_000) {
return statsSummaryCache.value;
}
const counts = db.prepare(`
SELECT
(SELECT COUNT(*) FROM articles) as total,
(SELECT COUNT(*) FROM articles WHERE content IS NOT NULL AND content != '') as withContent,
(SELECT COUNT(*) FROM articles WHERE has_embedding = 1) as withEmbedding,
(SELECT COUNT(*) FROM events) as eventCount,
(SELECT COUNT(*) FROM articles WHERE ingested_at >= datetime('now', '-1 hour')) as ingestedPerHour,
(SELECT COUNT(*) FROM articles WHERE content_attempted_at >= datetime('now', '-1 hour')) as contentPerHour
COALESCE((SELECT seq FROM sqlite_sequence WHERE name='articles'), 0) AS total,
(SELECT COUNT(*) FROM article_embedding_meta) AS withContent,
(SELECT COUNT(*) FROM article_embedding_meta) AS withEmbedding,
COALESCE((SELECT seq FROM sqlite_sequence WHERE name='events'), 0) AS eventCount
`).get();
statsSummaryCache = { at: Date.now(), value: counts };
return counts;
});
const bySource = db.prepare(`
SELECT source, COUNT(*) as n FROM articles GROUP BY source ORDER BY n DESC
`).all();
// Detailed statistics are cached because they summarize the full archive and
// do not need second-by-second precision.
fastify.get('/admin/api/stats', async (request, reply) => {
if (!checkAuth(request, reply)) return;
if (statsDetailCache && Date.now() - statsDetailCache.at < 5 * 60_000) {
return statsDetailCache.value;
}
const byStatus = db.prepare(`
SELECT COALESCE(content_status, 'null') as status, COUNT(*) as n
FROM articles GROUP BY content_status ORDER BY n DESC
`).all();
let embeddingsPerHour = 0;
try {
embeddingsPerHour = db.prepare(`
SELECT COUNT(*) as n FROM article_embedding_meta WHERE embedded_at >= datetime('now', '-1 hour')
`).get().n;
} catch (_) {}
return { ...counts, bySource, byStatus, embeddingsPerHour };
const value = await calculateArchiveStats();
statsDetailCache = { at: Date.now(), value };
statsSummaryCache = {
at: Date.now(),
value: {
total: value.total,
withContent: value.withContent,
withEmbedding: value.withEmbedding,
eventCount: value.eventCount,
},
};
return value;
});
}
module.exports = adminRoutes;
module.exports = adminRoutes;
src/routes/articles.js
+1 -1
View File
@@ -60,7 +60,7 @@ function buildArticlesQuery(query) {
return {
sql: `
SELECT id, title, description, content, ${includeEmbedding ? 'embedding,' : ''} url, normalized_title, source, pub_date, ingested_at
FROM articles
FROM articles INDEXED BY idx_articles_usable_pub_date
${whereClause}
ORDER BY ${orderBy}
LIMIT ? OFFSET ?
src/routes/autonomy.js
+59
View File
@@ -0,0 +1,59 @@
const path = require('path');
const { openRuntimeDb, isPostgresEnabled } = require('../db/runtime');
const pg = require('../db/pgAsync');
const intelligencePath = process.env.INTELLIGENCE_DB || path.resolve(process.cwd(), 'intelligence.sqlite');
const db = openRuntimeDb(intelligencePath, { schema: 'intelligence', readonly: true });
async function autonomyRoutes(fastify) {
fastify.get('/health', async () => ({ ok: true, service: 'duriin-api' }));
fastify.get('/autonomy/status', async () => {
if (isPostgresEnabled()) {
const exists = await pg.get('intelligence', "SELECT 1 FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2", ['intelligence', 'autonomy_jobs']);
if (!exists) return { enabled: false, reason: 'autonomy schema is not initialized' };
const [jobs, predictions, decisions, outcomes, legacy, instruments] = await Promise.all([
pg.all('intelligence', 'SELECT lane, status, COUNT(*) AS count FROM autonomy_jobs GROUP BY lane, status ORDER BY lane, status'),
pg.all('intelligence', 'SELECT status, COUNT(*) AS count FROM autonomy_predictions GROUP BY status ORDER BY status'),
pg.all('intelligence', 'SELECT action, COUNT(*) AS count FROM autonomy_decisions GROUP BY action ORDER BY action'),
pg.get('intelligence', 'SELECT COUNT(*) AS total, SUM(direction_correct) AS correct, AVG(excess_return) AS average_excess_return FROM autonomy_outcomes'),
pg.get('intelligence', 'SELECT COUNT(*) AS count FROM autonomy_legacy_records'),
pg.get('intelligence', 'SELECT COUNT(*) AS count FROM autonomy_instruments WHERE active=1 AND tradable=1'),
]);
return { enabled: true, jobs, predictions, decisions, outcomes, legacyRecords: legacy.count, allowlistedInstruments: instruments.count };
}
const exists = isPostgresEnabled()
? db.prepare("SELECT 1 FROM information_schema.tables WHERE table_schema = ? AND table_name = ?").get('intelligence', 'autonomy_jobs')
: db.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='autonomy_jobs'").get();
if (!exists) return { enabled: false, reason: 'autonomy schema is not initialized' };
const jobs = db.prepare(`
SELECT lane, status, COUNT(*) AS count
FROM autonomy_jobs
GROUP BY lane, status
ORDER BY lane, status
`).all();
const predictions = db.prepare(`
SELECT status, COUNT(*) AS count
FROM autonomy_predictions
GROUP BY status
ORDER BY status
`).all();
const decisions = db.prepare(`
SELECT action, COUNT(*) AS count
FROM autonomy_decisions
GROUP BY action
ORDER BY action
`).all();
const outcomes = db.prepare(`
SELECT COUNT(*) AS total,
SUM(direction_correct) AS correct,
AVG(excess_return) AS average_excess_return
FROM autonomy_outcomes
`).get();
const legacy = db.prepare('SELECT COUNT(*) AS count FROM autonomy_legacy_records').get();
const instruments = db.prepare('SELECT COUNT(*) AS count FROM autonomy_instruments WHERE active=1 AND tradable=1').get();
return { enabled: true, jobs, predictions, decisions, outcomes, legacyRecords: legacy.count, allowlistedInstruments: instruments.count };
});
}
module.exports = autonomyRoutes;
src/routes/ops.js
+254
View File
@@ -0,0 +1,254 @@
// Control plane for the operations dashboard.
//
// Kept apart from admin.js on purpose: everything in here either changes what the
// autonomy stack does or is read by an operator while something is on fire, so it
// wants to stay small enough to audit in one sitting.
const { execFile } = require('child_process');
const path = require('path');
const { getExecutionControls, setExecutionMode, setKillSwitch, EXECUTION_MODES } = require('../autonomy/settings');
// Which of the pipeline stages we consider "recent enough to be alive". These are
// generous, they exist to catch a stall not to police a few seconds of jitter.
const STALE_AFTER_MINUTES = { ingest: 90, prediction: 180, outcome: 24 * 60 };
const ANALYSES = {
reaction: {
label: 'Reaction conditioning',
script: 'scripts/analyze-reaction-conditioning.js',
detail: 'Does the initial market reaction predict anything. Read only, a few minutes.',
},
};
function minutesSince(value) {
if (!value) return null;
const stamp = String(value).includes('T') ? String(value) : `${String(value).replace(' ', 'T')}Z`;
const then = Date.parse(stamp);
if (!Number.isFinite(then)) return null;
return Math.max(0, (Date.now() - then) / 60000);
}
function one(db, sql, params = []) {
try {
return db.prepare(sql).get(...params) || {};
} catch (error) {
console.error('[ops] query failed:', sql.trim().slice(0, 80), error.message);
return {};
}
}
function many(db, sql, params = []) {
try {
return db.prepare(sql).all(...params) || [];
} catch (error) {
console.error('[ops] query failed:', sql.trim().slice(0, 80), error.message);
return [];
}
}
// One request for the whole dashboard. The old admin made the browser fire a
// handful of sequential calls and stitch them together, which is most of why it
// felt sluggish even though every individual endpoint was fast.
function buildOverview(intel) {
const predictions = one(intel, `
SELECT COUNT(*) AS total,
SUM(CASE WHEN origin='live' AND status='open' THEN 1 ELSE 0 END) AS live_open,
SUM(CASE WHEN origin='live' AND status='resolved' THEN 1 ELSE 0 END) AS live_resolved,
SUM(CASE WHEN origin IN ('historical','replay') AND status='resolved' THEN 1 ELSE 0 END) AS offline_resolved,
SUM(CASE WHEN status='unresolvable' THEN 1 ELSE 0 END) AS unresolvable,
MAX(created_at) AS latest
FROM autonomy_predictions
`);
const byOrigin = many(intel, `
SELECT p.origin, COUNT(*) AS total,
SUM(o.direction_correct) AS correct,
AVG(o.excess_return) AS mean_excess
FROM autonomy_predictions p JOIN autonomy_outcomes o ON o.prediction_id = p.id
GROUP BY p.origin
`);
// When live evidence actually arrives. This is the number that decides whether
// anything can ever qualify, and nothing in the old UI showed it.
const maturity = many(intel, `
SELECT horizon_days, COUNT(*) AS n,
MIN(date(information_cutoff, '+' || horizon_days || ' days')) AS first_matures
FROM autonomy_predictions
WHERE origin='live' AND status='open'
GROUP BY horizon_days ORDER BY horizon_days
`);
const decisions = many(intel, "SELECT action, COUNT(*) AS n FROM autonomy_decisions GROUP BY action");
const intents = one(intel, 'SELECT COUNT(*) AS n FROM autonomy_order_intents');
const jobs = many(intel, `
SELECT job_type, lane, status, COUNT(*) AS n, MAX(created_at) AS newest
FROM autonomy_jobs GROUP BY job_type, lane, status
`);
const deadLetters = many(intel, `
SELECT job_type, lane, COUNT(*) AS n, substr(MAX(last_error), 1, 160) AS sample_error
FROM autonomy_jobs WHERE status='dead_letter' GROUP BY job_type, lane
`);
let cohorts = [];
try {
cohorts = many(intel, `
SELECT cohort_key, source, sample_size, distinct_instruments, top_instrument_share,
directional_probability, expected_excess_return
FROM autonomy_calibration_snapshots
WHERE cohort_key LIKE 'v2|%'
GROUP BY cohort_key, source
HAVING MAX(created_at) = created_at
ORDER BY sample_size DESC LIMIT 40
`);
} catch (error) {
console.error('[ops] cohort snapshot read failed:', error.message);
}
const outcomes = one(intel, 'SELECT COUNT(*) AS n, MAX(evaluated_at) AS latest FROM autonomy_outcomes');
return {
generatedAt: new Date().toISOString(),
predictions,
byOrigin,
maturity,
decisions,
orderIntents: intents.n || 0,
outcomes,
jobs,
deadLetters,
cohorts,
freshness: {
predictionMinutes: minutesSince(predictions.latest),
outcomeMinutes: minutesSince(outcomes.latest),
thresholds: STALE_AFTER_MINUTES,
},
};
}
function opsRoutes(fastify, options) {
const { checkAuth, getIntelligenceDb, getArchiveDb } = options;
const withIntel = (reply) => {
const intel = getIntelligenceDb();
if (!intel) {
reply.code(503).send({ error: 'intelligence database unavailable' });
return null;
}
return intel;
};
fastify.get('/admin/api/ops/overview', async (request, reply) => {
if (!checkAuth(request, reply)) return;
const intel = withIntel(reply);
if (!intel) return;
const payload = buildOverview(intel);
try {
payload.controls = getExecutionControls(intel);
} catch (error) {
console.error('[ops] could not read execution controls:', error.message);
payload.controls = null;
}
// Archive counts are the one genuinely expensive thing here, so they are
// cheap approximations rather than COUNT(*) over 2.2M rows on every poll.
try {
const archive = getArchiveDb ? getArchiveDb() : null;
if (archive) {
payload.archive = one(archive, `
SELECT MAX(id) AS max_id, MAX(ingested_at) AS latest_ingest FROM articles
`);
payload.freshness.ingestMinutes = minutesSince(payload.archive.latest_ingest);
}
} catch (error) {
console.error('[ops] archive probe failed:', error.message);
payload.archive = null;
}
return payload;
});
fastify.get('/admin/api/ops/settings', async (request, reply) => {
if (!checkAuth(request, reply)) return;
const intel = withIntel(reply);
if (!intel) return;
return { controls: getExecutionControls(intel), modes: EXECUTION_MODES, analyses: ANALYSES };
});
fastify.post('/admin/api/ops/settings', async (request, reply) => {
if (!checkAuth(request, reply)) return;
const intel = withIntel(reply);
if (!intel) return;
const { mode, killSwitch } = request.body || {};
try {
if (mode !== undefined) setExecutionMode(intel, String(mode), 'admin-ui');
if (killSwitch !== undefined) setKillSwitch(intel, Boolean(killSwitch), 'admin-ui');
} catch (error) {
console.error('[ops] settings write rejected:', error.message);
reply.code(400).send({ error: error.message });
return;
}
const controls = getExecutionControls(intel);
console.log(`[ops] execution controls now mode=${controls.mode} kill=${controls.killSwitch}`);
return { controls };
});
// Requeue is deliberately narrow: it only ever moves dead_letter back to pending,
// it never deletes and never edits payloads, so the worst case is repeated work.
fastify.post('/admin/api/ops/dead-letters/requeue', async (request, reply) => {
if (!checkAuth(request, reply)) return;
const intel = withIntel(reply);
if (!intel) return;
const { jobType, lane } = request.body || {};
const filters = ["status = 'dead_letter'"];
const params = [];
if (jobType) { filters.push('job_type = ?'); params.push(String(jobType)); }
if (lane) { filters.push('lane = ?'); params.push(String(lane)); }
const where = filters.join(' AND ');
try {
const before = one(intel, `SELECT COUNT(*) AS n FROM autonomy_jobs WHERE ${where}`, params).n || 0;
if (!before) return { requeued: 0, remaining: 0 };
const result = intel.prepare(`
UPDATE autonomy_jobs
SET status='pending', attempts=0, available_at=datetime('now'),
leased_by=NULL, lease_expires_at=NULL,
last_error='requeued from the ops dashboard'
WHERE ${where}
`).run(...params);
const remaining = one(intel, `SELECT COUNT(*) AS n FROM autonomy_jobs WHERE ${where}`, params).n || 0;
console.log(`[ops] requeued ${result.changes} dead letters (jobType=${jobType || 'any'} lane=${lane || 'any'})`);
return { requeued: result.changes, before, remaining };
} catch (error) {
console.error('[ops] requeue failed:', error.message, error.stack);
reply.code(500).send({ error: error.message });
}
});
fastify.post('/admin/api/ops/analysis/:name', async (request, reply) => {
if (!checkAuth(request, reply)) return;
const spec = ANALYSES[request.params.name];
if (!spec) {
reply.code(404).send({ error: `unknown analysis: ${request.params.name}` });
return;
}
const script = path.resolve(__dirname, '..', '..', spec.script);
return new Promise((resolve) => {
execFile('node', [script], { timeout: 15 * 60 * 1000, maxBuffer: 8 * 1024 * 1024 },
(error, stdout, stderr) => {
if (error) console.error(`[ops] analysis ${request.params.name} failed:`, error.message);
resolve({
analysis: request.params.name,
label: spec.label,
ok: !error,
output: String(stdout || '').slice(-20000),
error: error ? String(stderr || error.message).slice(-4000) : null,
});
});
});
});
}
module.exports = opsRoutes;
module.exports.buildOverview = buildOverview;
module.exports.ANALYSES = ANALYSES;
src/routes/status.js
+14 -1
View File
@@ -6,7 +6,20 @@ let statusCacheAt = 0;
const STATUS_CACHE_TTL_MS = 30 * 1000;
async function statusRoutes(fastify) {
fastify.get('/status', async () => {
fastify.get('/status', async (request) => {
const deep = String(request.query?.deep || '').toLowerCase() === 'true';
if (!deep) {
const sequence = db.prepare("SELECT seq FROM sqlite_sequence WHERE name = 'articles'").get();
return {
total: sequence ? sequence.seq : 0,
usable: null,
lastIngestionBySource: getLastIngestionBySource(),
bySource: null,
embeddingModels: null,
mode: 'lightweight',
deep_status_url: '/status?deep=true',
};
}
const now = Date.now();
if (statusCache && now - statusCacheAt < STATUS_CACHE_TTL_MS) {
return statusCache;
src/scheduler.js
+18 -1
View File
@@ -65,7 +65,11 @@ async function runAllIngestions() {
return results;
}
const GDELT_BACKOFF_START_MS = 30 * 1000;
const GDELT_BACKOFF_MAX_MS = 30 * 60 * 1000;
function startScheduler() {
let gdeltBackoffMs = 0;
const runRss = async () => {
await runSource('rss', fetchRssArticles);
};
@@ -84,8 +88,14 @@ function startScheduler() {
await fetchGdeltArticles(async (articles) => {
await ingestBatch("gdelt", articles);
});
gdeltBackoffMs = 0;
} catch (error) {
console.error("gdelt ingestion failed:", error);
// No pause here at all previously, so once gdelt started refusing
// connections this span burned cpu and filled the log with the same
// stack indefinitely. It has been failing for days on end.
gdeltBackoffMs = Math.min(GDELT_BACKOFF_MAX_MS, gdeltBackoffMs ? gdeltBackoffMs * 2 : GDELT_BACKOFF_START_MS);
console.error(`gdelt ingestion failed, retrying in ${Math.round(gdeltBackoffMs / 1000)}s:`, error.message);
await sleep(gdeltBackoffMs);
}
}
};
@@ -119,7 +129,14 @@ function startScheduler() {
try {
const perSource = Number(config.contentBackfill?.perSource) || 50;
const batchSize = Number(config.contentBackfill?.batchSize) || 25;
// A round is long and silent. When content fetching wedged on 4 Sept there
// was not one line anywhere saying so, which is why it went unnoticed for
// three days while the live lane starved for want of enriched articles.
const startedAt = Date.now();
console.log(`[content] worker ${workerIndex} starting a round`);
const processed = await runBackfillWorker({ workerIndex, workerCount, perSource, batchSize });
console.log(`[content] worker ${workerIndex} finished ${processed} articles`
+ ` in ${Math.round((Date.now() - startedAt) / 1000)}s`);
// if a worker found nothing in its partition, brief sleep so we dont
// hammer the db with empty selects
src/sources/browserCrawler.js
+34 -6
View File
@@ -3,6 +3,11 @@ const { chromium } = require('playwright');
const BROWSER_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36';
const MAX_RENDERED_HTML_LENGTH = 1_500_000;
const DEFAULT_REQUEST_TIMEOUT = 20000;
// generous, this is the "something has gone wrong" bound rather than a normal wait
const PAGE_SLOT_WAIT_MS = 120000;
const PAGE_CLOSE_TIMEOUT_MS = 10000;
function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); }
const CONSENT_BUTTON_SELECTORS = [
'button[name="agree"]',
'input[name="agree"]',
@@ -104,15 +109,31 @@ async function buildBrowserSession(options = {}) {
let activePages = 0;
let closed = false;
async function acquirePageSlot() {
// A slot is never waited on forever. If every slot has leaked the callers used
// to park here silently with no log and no progress, which looks exactly like a
// dead worker, so time out and let the caller fail loudly instead.
async function acquirePageSlot(waitMs = PAGE_SLOT_WAIT_MS) {
if (activePages < maxConcurrentPages) {
activePages += 1;
return;
}
await new Promise((resolve) => {
waiters.push(resolve);
});
let waiter;
let timer;
try {
await new Promise((resolve, reject) => {
waiter = resolve;
waiters.push(waiter);
timer = setTimeout(() => {
const index = waiters.indexOf(waiter);
if (index !== -1) waiters.splice(index, 1);
reject(new Error(`timed out after ${waitMs}ms waiting for a browser page slot`
+ ` (${activePages}/${maxConcurrentPages} active)`));
}, waitMs);
});
} finally {
clearTimeout(timer);
}
activePages += 1;
}
@@ -142,10 +163,13 @@ async function buildBrowserSession(options = {}) {
}
await acquirePageSlot();
const page = await context.newPage();
// newPage() used to sit out here. When it threw or hung the slot was gone for
// good, and after maxConcurrentPages of those every caller blocked forever.
let page = null;
const timeout = normalizeTimeout(options.timeout || requestTimeout);
try {
page = await context.newPage();
await page.goto(url, {
waitUntil: 'domcontentloaded',
timeout,
@@ -167,7 +191,11 @@ async function buildBrowserSession(options = {}) {
return html;
} finally {
try {
await page.close();
// a wedged renderer can make close() hang too, and that would strand the
// slot just as badly as the original leak did
if (page) await Promise.race([page.close(), sleep(PAGE_CLOSE_TIMEOUT_MS)]);
} catch (error) {
console.error(`[browser] page close failed for ${url}:`, error.message);
} finally {
releasePageSlot();
}
test/autonomy.test.js
+538
View File
@@ -0,0 +1,538 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const Database = require('better-sqlite3');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { enqueueJob, leaseNextJob, completeJob } = require('../src/autonomy/jobs');
const { normalizeProposal, acceptProposal } = require('../src/autonomy/coordinator');
const { calibrateOutcomes, cohortKey } = require('../src/autonomy/calibration');
const { decide } = require('../src/autonomy/policy');
const { validatePaperIntent, createSimulator } = require('../src/autonomy/execution');
const { calculateOutcome, addTradingDays } = require('../src/autonomy/outcomes');
const { yahooSymbol } = require('../workers/outcomeAutonomyWorker');
const { createOrderIntent } = require('../src/autonomy/orderIntents');
const { enqueueCoordinatorEvent, reconcileArchiveBatch, reconcileLiveBatch, isTransientCoordinatorFailure } = require('../workers/autonomyWorker');
const { buildGraphContext } = require('../src/autonomy/graphContext');
const { buildPrompt } = require('../workers/coordinatorWorker');
const { scheduleNext, replayPrompt, runForJob } = require('../workers/replayWorker');
const { refreshHistoricalCalibration, createDecisions, ensureCalibrationColumns } = require('../workers/calibrationWorker');
test('autonomy schema and leased jobs are restart-safe', () => {
const db = new Database(':memory:');
initAutonomySchema(db);
assert.equal(enqueueJob(db, {
jobType: 'enrich', lane: 'live', priority: 10, entityType: 'article', entityId: 42,
idempotencyKey: 'enrich:42',
}).inserted, true);
assert.equal(enqueueJob(db, {
jobType: 'enrich', lane: 'live', priority: 10, entityType: 'article', entityId: 42,
idempotencyKey: 'enrich:42',
}).inserted, false);
const job = leaseNextJob(db, 'test-worker');
assert.equal(job.lane, 'live');
assert.equal(completeJob(db, job.id, 'test-worker'), true);
assert.equal(db.prepare("SELECT status FROM autonomy_jobs WHERE id = ?").get(job.id).status, 'complete');
});
test('coordinator proposals require evidence and contain no arbitrary numeric confidence', () => {
const normalized = normalizeProposal({ predictions: [{
instrument: 'nvda', direction: 'positive', event_type: 'supply_constraint',
horizon_days: 10, evidence_article_ids: [7],
}] }, { informationCutoff: '2026-01-01T00:00:00Z', model: 'test-model' });
assert.equal(normalized.predictions[0].instrument, 'NVDA');
assert.equal('probability' in normalized.predictions[0], false);
assert.throws(() => normalizeProposal({ predictions: [{ instrument: 'NVDA', direction: 'positive', horizon_days: 10 }] }));
});
test('accepted proposal preserves evidence and creates immutable prediction', () => {
const archive = new Database(':memory:');
archive.exec('CREATE TABLE articles (id INTEGER PRIMARY KEY)');
archive.prepare('INSERT INTO articles (id) VALUES (?)').run(7);
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
intelligence.prepare("INSERT INTO autonomy_instruments(symbol, broker, active, tradable) VALUES ('NVDA', 'test', 1, 1)").run();
const result = acceptProposal(intelligence, archive, {
predictions: [{ instrument: 'NVDA', direction: 'positive', event_type: 'earnings', horizon_days: 10, evidence_article_ids: [7] }],
}, { informationCutoff: '2026-01-01T00:00:00Z', strategyVersion: 'test' });
assert.equal(result.predictionCount, 1);
assert.deepEqual(JSON.parse(intelligence.prepare('SELECT evidence_article_ids FROM autonomy_predictions').get().evidence_article_ids), [7]);
});
test('replay evidence cannot look beyond its information cutoff and remains non-executable', () => {
const archive = new Database(':memory:');
archive.exec('CREATE TABLE articles (id INTEGER PRIMARY KEY, pub_date_effective TEXT, pub_date TEXT, ingested_at TEXT)');
archive.prepare("INSERT INTO articles VALUES (7, '2020-01-01T00:00:00Z', NULL, '2020-01-01T00:00:00Z')").run();
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
intelligence.prepare("INSERT INTO autonomy_instruments(symbol, broker, active, tradable) VALUES ('NVDA', 'test', 1, 1)").run();
assert.throws(() => acceptProposal(intelligence, archive, {
predictions: [{ instrument: 'NVDA', direction: 'positive', event_type: 'earnings', horizon_days: 10, evidence_article_ids: [7] }],
}, { informationCutoff: '2019-12-31T00:00:00Z', origin: 'replay', replayRunId: 1 }), /missing evidence/);
const proposal = intelligence.prepare("INSERT INTO autonomy_proposals(payload, information_cutoff, status) VALUES ('{}', datetime('now'), 'accepted')").run();
const prediction = intelligence.prepare(`INSERT INTO autonomy_predictions
(proposal_id, instrument, direction, event_type, horizon_days, information_cutoff, evidence_article_ids, strategy_version, origin)
VALUES (?, 'NVDA', 'positive', 'test', 10, datetime('now'), '[7]', 'test', 'replay')`).run(proposal.lastInsertRowid);
intelligence.prepare("INSERT INTO autonomy_decisions(prediction_id, action, rationale, strategy_version) VALUES (?, 'BUY', 'test', 'test')").run(prediction.lastInsertRowid);
const executable = intelligence.prepare(`SELECT d.id FROM autonomy_decisions d JOIN autonomy_predictions p ON p.id=d.prediction_id
WHERE d.action IN ('BUY','SELL') AND p.origin='live'`).all();
assert.equal(executable.length, 0);
});
test('replay scheduler skips terminal replay jobs instead of pinning the cursor', () => {
const archive = new Database(':memory:');
archive.exec(`
CREATE TABLE articles (
id INTEGER PRIMARY KEY,
title TEXT,
description TEXT,
content TEXT,
pub_date_effective TEXT,
is_index_page INTEGER
)
`);
archive.prepare("INSERT INTO articles VALUES (1, 'bad', '', 'content', '2020-01-01T00:00:00Z', 0)").run();
archive.prepare("INSERT INTO articles VALUES (2, 'next', '', 'content', '2020-01-02T00:00:00Z', 0)").run();
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
const runId = intelligence.prepare(`
INSERT INTO autonomy_replay_runs (watermark_at, strategy_version, prompt_version, coordinator_model)
VALUES ('2020-01-03T00:00:00Z', 'test', 'test', 'test')
`).run().lastInsertRowid;
enqueueJob(intelligence, {
jobType: 'replay_article', lane: 'historical', priority: 1, entityType: 'article', entityId: 1,
idempotencyKey: `replay:${runId}:article:1`,
});
intelligence.prepare("UPDATE autonomy_jobs SET status='dead_letter', attempts=5, last_error='fetch failed'").run();
const run = intelligence.prepare('SELECT * FROM autonomy_replay_runs WHERE id=?').get(runId);
const next = scheduleNext(intelligence, archive, run);
assert.equal(next.id, 2);
assert.equal(intelligence.prepare('SELECT cursor_article_id FROM autonomy_replay_runs WHERE id=?').get(runId).cursor_article_id, 1);
assert.equal(intelligence.prepare("SELECT COUNT(*) count FROM autonomy_jobs WHERE status='pending' AND entity_id='2'").get().count, 1);
});
test('a run with a pinned article set walks only that set, in date order', () => {
const archive = new Database(':memory:');
archive.exec(`
CREATE TABLE articles (
id INTEGER PRIMARY KEY, title TEXT, description TEXT, content TEXT,
pub_date_effective TEXT, is_index_page INTEGER
)
`);
for (const id of [1, 2, 3, 4]) {
archive.prepare('INSERT INTO articles VALUES (?, ?, ?, ?, ?, 0)')
.run(id, `a${id}`, '', 'content', `2020-01-0${id}T00:00:00Z`);
}
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
const runId = intelligence.prepare(`
INSERT INTO autonomy_replay_runs (watermark_at, strategy_version, prompt_version, coordinator_model)
VALUES ('2020-01-09T00:00:00Z', 'test', 'test', 'test')
`).run().lastInsertRowid;
// deliberately out of order and deliberately not article 1, the whole point
// is that the run ignores the archive walk and answers these
for (const [articleId, at] of [[4, '2020-01-04T00:00:00Z'], [2, '2020-01-02T00:00:00Z']]) {
intelligence.prepare('INSERT INTO autonomy_replay_run_articles (run_id, article_id, effective_at) VALUES (?, ?, ?)')
.run(runId, articleId, at);
}
const run = intelligence.prepare('SELECT * FROM autonomy_replay_runs WHERE id=?').get(runId);
const first = scheduleNext(intelligence, archive, run);
assert.equal(first.id, 2, 'earliest pinned article first, not article 1');
intelligence.prepare("UPDATE autonomy_jobs SET status='complete' WHERE entity_id='2'").run();
intelligence.prepare('UPDATE autonomy_replay_runs SET cursor_article_id=2, cursor_effective_at=? WHERE id=?')
.run('2020-01-02T00:00:00Z', runId);
const second = scheduleNext(intelligence, archive, intelligence.prepare('SELECT * FROM autonomy_replay_runs WHERE id=?').get(runId));
assert.equal(second.id, 4);
intelligence.prepare("UPDATE autonomy_jobs SET status='complete' WHERE entity_id='4'").run();
intelligence.prepare('UPDATE autonomy_replay_runs SET cursor_article_id=4, cursor_effective_at=? WHERE id=?')
.run('2020-01-04T00:00:00Z', runId);
const exhausted = scheduleNext(intelligence, archive, intelligence.prepare('SELECT * FROM autonomy_replay_runs WHERE id=?').get(runId));
assert.equal(exhausted, null, 'a pinned run stops when its set is done, it does not fall back to the archive');
});
test('a run with no pinned set still walks the archive exactly as before', () => {
const archive = new Database(':memory:');
archive.exec(`
CREATE TABLE articles (
id INTEGER PRIMARY KEY, title TEXT, description TEXT, content TEXT,
pub_date_effective TEXT, is_index_page INTEGER
)
`);
archive.prepare("INSERT INTO articles VALUES (7, 'only', '', 'content', '2020-01-01T00:00:00Z', 0)").run();
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
const runId = intelligence.prepare(`
INSERT INTO autonomy_replay_runs (watermark_at, strategy_version, prompt_version, coordinator_model)
VALUES ('2020-01-09T00:00:00Z', 'test', 'test', 'test')
`).run().lastInsertRowid;
const run = intelligence.prepare('SELECT * FROM autonomy_replay_runs WHERE id=?').get(runId);
assert.equal(scheduleNext(intelligence, archive, run).id, 7);
});
test('the feedback brief reaches the prompt and stays out of it when empty', () => {
const article = { id: 1, title: 't', content: 'body', effective_at: '2020-01-01T00:00:00Z' };
const withBrief = replayPrompt(article, 'CALIBRATION FEEDBACK. you over-call positive.');
assert.ok(withBrief.includes('you over-call positive'));
assert.ok(withBrief.indexOf('CALIBRATION FEEDBACK') < withBrief.indexOf('[Evidence 1]'),
'the brief has to land before the evidence, not after it');
assert.ok(!replayPrompt(article).includes('CALIBRATION FEEDBACK'));
});
test('a leftover job from an older run cannot hijack the active run', () => {
const db = new Database(':memory:');
initAutonomySchema(db);
const first = db.prepare(`
INSERT INTO autonomy_replay_runs (watermark_at, strategy_version, prompt_version, coordinator_model, status)
VALUES ('2020-01-09T00:00:00Z', 'autonomy-1', 'replay-coordinator-1', 'old-model', 'paused')
`).run().lastInsertRowid;
const second = db.prepare(`
INSERT INTO autonomy_replay_runs (watermark_at, strategy_version, prompt_version, coordinator_model, feedback_brief)
VALUES ('2020-01-09T00:00:00Z', 'autonomy-2', 'replay-coordinator-2', 'new-model', 'you over-call positive')
`).run().lastInsertRowid;
const active = db.prepare('SELECT * FROM autonomy_replay_runs WHERE id=?').get(second);
// a recovered dead letter from run 1, leased while run 2 is the active one
const owner = runForJob(db, { id: 9, idempotency_key: `replay:${first}:article:4242` }, active);
assert.equal(owner.id, first, 'the job belongs to the run that enqueued it');
assert.equal(owner.feedback_brief, null, 'and it must not be handed run 2 brief');
assert.equal(owner.prompt_version, 'replay-coordinator-1');
const own = runForJob(db, { id: 10, idempotency_key: `replay:${second}:article:1` }, active);
assert.equal(own.id, second);
// unattributable jobs fall back rather than being dropped, but loudly
assert.equal(runForJob(db, { id: 11, idempotency_key: null }, active).id, second);
assert.equal(runForJob(db, { id: 12, idempotency_key: 'replay:999:article:1' }, active).id, second);
});
test('calibration and policy abstain on insufficient evidence', () => {
const calibration = calibrateOutcomes([
{ excess_return: 0.02, direction_correct: 1 },
{ excess_return: -0.01, direction_correct: 0 },
]);
assert.equal(calibration.sampleSize, 2);
const result = decide({ ...calibration }, { minSampleSize: 30 });
assert.equal(result.action, 'ABSTAIN');
assert.equal(cohortKey({ sector: 'tech', eventType: 'earnings', horizonDays: 10, direction: 'positive' }), 'v2|tech|earnings|medium|positive');
assert.equal(decide({
direction: 'negative', probability: 0.8, expectedExcessReturn: -0.02, lowerReturn: -0.04,
sampleSize: 40, distinctInstruments: 9, topInstrumentShare: 0.25,
}).action, 'SELL');
});
test('historical replay outcomes create replay calibration snapshots', () => {
const db = new Database(':memory:');
initAutonomySchema(db);
const proposal = db.prepare("INSERT INTO autonomy_proposals(payload, information_cutoff, status) VALUES ('{}', '2020-01-01T00:00:00Z', 'accepted')").run();
const prediction = db.prepare(`
INSERT INTO autonomy_predictions
(proposal_id, instrument, direction, event_type, horizon_days, information_cutoff, evidence_article_ids,
learning_eligible, strategy_version, origin, replay_run_id, status)
VALUES (?, 'NVDA', 'positive', 'earnings', 10, '2020-01-01T00:00:00Z', '[1]', 0, 'test', 'replay', 7, 'resolved')
`).run(proposal.lastInsertRowid);
db.prepare(`
INSERT INTO autonomy_outcomes(prediction_id, excess_return, direction_correct)
VALUES (?, 0.04, 1)
`).run(prediction.lastInsertRowid);
// one per-run replay snapshot plus the pooled historical/replay snapshot
assert.equal(refreshHistoricalCalibration(db, 'test-cal'), 2);
const snapshot = db.prepare("SELECT source, replay_run_id, sample_size, directional_probability FROM autonomy_calibration_snapshots WHERE source='replay'").get();
assert.equal(snapshot.source, 'replay');
assert.equal(snapshot.replay_run_id, 7);
assert.equal(snapshot.sample_size, 1);
assert(snapshot.directional_probability > 0.5);
});
test('live decisions map calibration snapshot fields into policy inputs', () => {
const db = new Database(':memory:');
initAutonomySchema(db);
const proposal = db.prepare("INSERT INTO autonomy_proposals(payload, information_cutoff, status) VALUES ('{}', datetime('now'), 'accepted')").run();
const prediction = db.prepare(`
INSERT INTO autonomy_predictions
(proposal_id, instrument, direction, event_type, horizon_days, information_cutoff, evidence_article_ids,
learning_eligible, strategy_version, origin, status)
VALUES (?, 'NVDA', 'positive', 'earnings', 10, datetime('now'), '[1]', 1, 'test', 'live', 'open')
`).run(proposal.lastInsertRowid);
ensureCalibrationColumns(db);
db.prepare(`
INSERT INTO autonomy_calibration_snapshots
(cohort_key, sample_size, effective_sample_size, directional_probability, expected_excess_return,
lower_return, upper_return, parent_cohort_key, version, source)
VALUES ('v2|unknown|earnings|medium|positive', 40, 42, 0.7, 0.02, -0.01, 0.06, NULL, 'test-cal', 'live')
`).run();
db.prepare("UPDATE autonomy_calibration_snapshots SET distinct_instruments = 11, top_instrument_share = 0.2").run();
assert.equal(createDecisions(db), 1);
const decision = db.prepare('SELECT * FROM autonomy_decisions WHERE prediction_id=?').get(prediction.lastInsertRowid);
assert.equal(decision.action, 'BUY');
assert.equal(decision.calibrated_probability, 0.7);
assert.equal(decision.expected_excess_return, 0.02);
});
test('paper execution is allowlisted, bounded and idempotent', () => {
const intent = validatePaperIntent({ decisionId: 12, instrument: 'NVDA', action: 'BUY', notional: 100 }, { tradable: true, maxNotional: 500 });
const broker = createSimulator();
assert.deepEqual(broker.submit(intent), broker.submit(intent));
assert.throws(() => validatePaperIntent({ decisionId: 13, instrument: 'PRIVATE', action: 'BUY', notional: 100 }, { tradable: false }));
assert.throws(() => validatePaperIntent({ decisionId: 14, instrument: 'NVDA', action: 'BUY', notional: 501 }, { tradable: true, maxNotional: 500 }));
});
test('archive reconciliation prioritizes recent usable events and is bounded', () => {
const archive = new Database(':memory:');
archive.exec('CREATE TABLE articles (id INTEGER PRIMARY KEY, event_id INTEGER, ingested_at TEXT, content TEXT, has_embedding INTEGER)');
archive.prepare('INSERT INTO articles VALUES (1, 99, ?, ?, 1)').run(new Date().toISOString(), 'content');
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
const result = reconcileArchiveBatch(archive, intelligence, 1);
assert.equal(result.scanned, 1);
const job = intelligence.prepare("SELECT lane, priority FROM autonomy_jobs WHERE job_type='coordinator_event'").get();
assert.equal(job.lane, 'live');
assert.equal(job.priority, 100);
const live = reconcileLiveBatch(archive, intelligence, 1);
assert.equal(live.scanned, 1);
});
test('archive reconciliation recovers transient dead-lettered coordinator jobs', () => {
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
enqueueJob(intelligence, {
jobType: 'coordinator_event', lane: 'historical', priority: 10, entityType: 'event', entityId: 99,
idempotencyKey: 'coordinator_event:99',
});
intelligence.prepare(`
UPDATE autonomy_jobs
SET status='dead_letter', attempts=5, last_error='TypeError: fetch failed'
WHERE idempotency_key='coordinator_event:99'
`).run();
const result = enqueueCoordinatorEvent(intelligence, {
event_id: 99,
ingested_at: new Date().toISOString(),
content: 'content',
has_embedding: 1,
});
assert.equal(result.recovered, true);
const job = intelligence.prepare("SELECT status, lane, priority, attempts, last_error FROM autonomy_jobs WHERE idempotency_key='coordinator_event:99'").get();
assert.equal(job.status, 'pending');
assert.equal(job.lane, 'live');
assert.equal(job.priority, 100);
assert.equal(job.attempts, 0);
assert.match(job.last_error, /Recovered transient coordinator failure/);
});
test('archive reconciliation leaves non-transient coordinator dead letters alone', () => {
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
enqueueJob(intelligence, {
jobType: 'coordinator_event', lane: 'historical', priority: 10, entityType: 'event', entityId: 100,
idempotencyKey: 'coordinator_event:100',
});
intelligence.prepare(`
UPDATE autonomy_jobs
SET status='dead_letter', attempts=5, last_error='no tradable instruments are allowlisted'
WHERE idempotency_key='coordinator_event:100'
`).run();
const result = enqueueCoordinatorEvent(intelligence, {
event_id: 100,
ingested_at: new Date().toISOString(),
content: 'content',
has_embedding: 1,
});
assert.equal(result.recovered, false);
const job = intelligence.prepare("SELECT status, attempts, last_error FROM autonomy_jobs WHERE idempotency_key='coordinator_event:100'").get();
assert.equal(job.status, 'dead_letter');
assert.equal(job.attempts, 5);
assert.equal(job.last_error, 'no tradable instruments are allowlisted');
});
test('outcome calculation uses benchmark-relative return', () => {
const outcome = calculateOutcome(
{ information_cutoff: '2026-01-02T00:00:00Z', horizon_days: 5, direction: 'positive' },
[{ date: '2026-01-02', close: 100 }, { date: '2026-01-09', close: 110 }],
[{ date: '2026-01-02', close: 100 }, { date: '2026-01-09', close: 105 }]
);
assert.equal(outcome.directionCorrect, 1);
assert.equal(outcome.excessReturn, 0.05);
});
test('order intents require the explicit instrument allowlist', () => {
const db = new Database(':memory:');
initAutonomySchema(db);
db.prepare("INSERT INTO autonomy_instruments(symbol, broker, active, tradable) VALUES ('NVDA', 'sim', 1, 1)").run();
const proposal = db.prepare(`INSERT INTO autonomy_proposals(payload, information_cutoff, status) VALUES ('{}', datetime('now'), 'accepted')`).run();
const prediction = db.prepare(`INSERT INTO autonomy_predictions(proposal_id, instrument, direction, event_type, horizon_days, information_cutoff, evidence_article_ids, learning_eligible, strategy_version) VALUES (?, 'NVDA', 'positive', 'test', 10, datetime('now'), '[1]', 1, 'test')`).run(proposal.lastInsertRowid);
const decision = db.prepare(`INSERT INTO autonomy_decisions(prediction_id, action, rationale, strategy_version) VALUES (?, 'BUY', 'test', 'test')`).run(prediction.lastInsertRowid);
const intent = createOrderIntent(db, decision.lastInsertRowid, 100, { maxNotional: 100 });
assert.equal(intent.side, 'buy');
assert.equal(db.prepare('SELECT status FROM autonomy_order_intents').get().status, 'shadow');
});
test('event_type is held to the closed family enum', () => {
const base = { instrument: 'NVDA', direction: 'positive', horizon_days: 10, evidence_article_ids: [1] };
const typeOf = (eventType) => normalizeProposal({ predictions: [{ ...base, event_type: eventType }] }).predictions[0].eventType;
// the enum itself, in the shapes a model actually emits
assert.equal(typeOf('earnings'), 'earnings');
assert.equal(typeOf('Supply Chain'), 'supply_chain');
assert.equal(typeOf('M_AND_A'), 'm_and_a');
// "none of these fit" is a legitimate answer and has to survive
assert.equal(typeOf('other'), 'other');
// off-enum but placeable: salvaged onto the family calibration would have
// picked anyway, so we dont throw away a usable prediction over a label
assert.equal(typeOf('earnings_beat_q3'), 'earnings');
assert.equal(typeOf('ceo resignation'), 'leadership');
// unplaceable free text is the 201-distinct-values failure, and is rejected
assert.throws(() => typeOf('vibes_shifted'), /event_type must be one of/);
assert.throws(() => typeOf(''), /event_type must be one of/);
});
test('dotted tickers are translated to the format the price feed expects', () => {
// BRK.B 404d forever and the outcome worker retried it in a hot loop
assert.equal(yahooSymbol('BRK.B'), 'BRK-B');
assert.equal(yahooSymbol('ABR.PRD'), 'ABR-PRD');
assert.equal(yahooSymbol('aac.u'), 'AAC-U');
// ordinary symbols must pass through untouched
assert.equal(yahooSymbol('NVDA'), 'NVDA');
assert.equal(yahooSymbol(' spy '), 'SPY');
});
test('an outcome whose horizon has not actually elapsed is not a result', () => {
// ITW and WMT both scored exactly 0.00% excess in production because at
// horizon 1 the entry and exit lookups landed on the same bar. Zero is not a
// measurement, and it was being recorded as a directional miss.
const sameBar = calculateOutcome(
{ information_cutoff: '2026-01-02T00:00:00Z', horizon_days: 1, direction: 'positive' },
[{ date: '2026-01-05', close: 100 }],
[{ date: '2026-01-05', close: 100 }]
);
assert.equal(sameBar, null);
// the exit bar genuinely existing still scores normally
const real = calculateOutcome(
{ information_cutoff: '2026-01-02T00:00:00Z', horizon_days: 1, direction: 'positive' },
[{ date: '2026-01-02', close: 100 }, { date: '2026-01-05', close: 104 }],
[{ date: '2026-01-02', close: 100 }, { date: '2026-01-05', close: 102 }]
);
assert.equal(real.directionCorrect, 1);
assert.ok(Math.abs(real.excessReturn - 0.02) < 1e-9);
});
test('trading day arithmetic steps over weekends', () => {
// friday + 1 trading day is monday, not saturday. the old due-check counted
// calendar days and so called a friday horizon-1 prediction due on saturday,
// when monday's close cannot exist yet.
assert.equal(addTradingDays('2026-01-02', 1), '2026-01-05');
assert.equal(addTradingDays('2026-01-02', 5), '2026-01-09');
assert.equal(addTradingDays('2026-01-02', 0), '2026-01-02');
});
test('a budget failure is transient but a bad key is not', () => {
const quota = 'Error: coordinator request failed with 403: {"error":{"message":"Key limit exceeded (monthly limit)."}}';
const credits = 'LLM 402: {"error":{"message":"Insufficient credits. Add more using ..."}}';
const afford = 'coordinator request failed with 402: can only afford 3921 tokens';
assert.equal(isTransientCoordinatorFailure(quota), true);
assert.equal(isTransientCoordinatorFailure(credits), true);
assert.equal(isTransientCoordinatorFailure(afford), true);
// these must stay dead, retrying them forever helps nobody
assert.equal(isTransientCoordinatorFailure('request failed with 403: invalid api key'), false);
assert.equal(isTransientCoordinatorFailure('request failed with 401: unauthorized'), false);
assert.equal(isTransientCoordinatorFailure('proposal references missing evidence'), false);
// and the pre-existing transient cases still are
assert.equal(isTransientCoordinatorFailure('TypeError: fetch failed'), true);
assert.equal(isTransientCoordinatorFailure('request failed with 503'), true);
});
test('an untradable instrument drops itself, not its valid siblings', () => {
const archive = new Database(':memory:');
archive.exec('CREATE TABLE articles (id INTEGER PRIMARY KEY, pub_date_effective TEXT, pub_date TEXT, ingested_at TEXT)');
archive.prepare("INSERT INTO articles VALUES (7, '2020-01-01T00:00:00Z', NULL, '2020-01-01T00:00:00Z')").run();
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
intelligence.prepare("INSERT INTO autonomy_instruments(symbol, broker, active, tradable) VALUES ('NVDA','test',1,1)").run();
const pred = (instrument) => ({
instrument, direction: 'positive', event_type: 'earnings',
horizon_days: 10, evidence_article_ids: [7],
});
// EURUSD used to take NVDA down with it and lose the whole proposal
const result = acceptProposal(intelligence, archive,
{ predictions: [pred('NVDA'), pred('EURUSD')] },
{ informationCutoff: '2026-01-01T00:00:00Z', strategyVersion: 'test' });
assert.equal(result.predictionCount, 1);
assert.deepEqual(result.droppedInstruments, ['EURUSD']);
const stored = intelligence.prepare('SELECT instrument FROM autonomy_predictions').all();
assert.deepEqual(stored.map((r) => r.instrument), ['NVDA']);
});
test('lookahead still rejects the whole proposal, not just one prediction', () => {
const archive = new Database(':memory:');
archive.exec('CREATE TABLE articles (id INTEGER PRIMARY KEY, pub_date_effective TEXT, pub_date TEXT, ingested_at TEXT)');
archive.prepare("INSERT INTO articles VALUES (7, '2020-06-01T00:00:00Z', NULL, '2020-06-01T00:00:00Z')").run();
const intelligence = new Database(':memory:');
initAutonomySchema(intelligence);
intelligence.prepare("INSERT INTO autonomy_instruments(symbol, broker, active, tradable) VALUES ('NVDA','test',1,1)").run();
// evidence postdates the cutoff: corrupt, not merely untradable
assert.throws(() => acceptProposal(intelligence, archive, {
predictions: [{ instrument: 'NVDA', direction: 'positive', event_type: 'earnings', horizon_days: 10, evidence_article_ids: [7] }],
}, { informationCutoff: '2020-01-01T00:00:00Z' }), /missing evidence/);
assert.equal(intelligence.prepare('SELECT COUNT(*) AS n FROM autonomy_predictions').get().n, 0);
});
test('graph context never shows a relationship the world had not revealed yet', () => {
const db = new Database(':memory:');
initAutonomySchema(db);
db.exec(`
CREATE TABLE tracked_companies (id INTEGER PRIMARY KEY, name TEXT, ticker TEXT, aliases TEXT);
CREATE TABLE event_knowledge (id INTEGER PRIMARY KEY, event_id INTEGER, company_id INTEGER, type TEXT, data TEXT, event_date TEXT);
CREATE TABLE company_relationships (id INTEGER PRIMARY KEY, from_company_id INTEGER, relationship_type TEXT,
to_entity TEXT, to_company_id INTEGER, confidence TEXT, confirmation_count INTEGER, first_seen_at DATETIME,
last_seen_at DATETIME, supporting_event_ids TEXT);
`);
db.prepare("INSERT INTO tracked_companies VALUES (1,'Nvidia','NVDA','[]')").run();
db.prepare("INSERT INTO event_knowledge (event_id, company_id, type) VALUES (55, 1, 'x')").run();
const edge = db.prepare(`INSERT INTO company_relationships
(from_company_id, relationship_type, to_entity, confidence, confirmation_count, first_seen_at)
VALUES (1, ?, ?, 'high', ?, ?)`);
edge.run('supplier', 'Taiwan Semiconductor', 9, '2025-01-01T00:00:00Z');
edge.run('customer', 'Future Corp', 4, '2026-01-01T00:00:00Z');
// a proposal dated mid-2025 may only see what existed by then
const past = buildGraphContext(db, 55, '2025-06-01T00:00:00Z');
assert.match(past, /Taiwan Semiconductor/);
assert.equal(/Future Corp/.test(past), false);
// later, both are legitimately visible
const now = buildGraphContext(db, 55, '2026-06-01T00:00:00Z');
assert.match(now, /Taiwan Semiconductor/);
assert.match(now, /Future Corp/);
// an event we know nothing about contributes nothing rather than a stray header
assert.equal(buildGraphContext(db, 999, '2026-06-01T00:00:00Z'), '');
assert.equal(buildGraphContext(db, 55, null), '');
});
test('the coordinator prompt carries graph context only when there is some', () => {
const withCtx = buildPrompt({ id: 1, title: 't' }, [{ id: 9, title: 'a', content: 'c' }],
'Known company relationships, as they stood at the information cutoff:\n\nNvidia (NVDA):\n - supplier: TSMC');
assert.match(withCtx, /Known company relationships/);
assert.match(withCtx, /supplier: TSMC/);
const without = buildPrompt({ id: 1, title: 't' }, [{ id: 9, title: 'a', content: 'c' }]);
assert.equal(/Known company relationships/.test(without), false);
});
test/calibrationCohorts.test.js
+168
View File
@@ -0,0 +1,168 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const {
normalizeEventType,
horizonBucket,
cohortKey,
legacyCohortKey,
calibrateOutcomes,
EVENT_FAMILY_NAMES,
} = require('../src/autonomy/calibration');
const { decide, DEFAULT_POLICY_RULES } = require('../src/autonomy/policy');
test('event types collapse onto a small closed set of families', () => {
assert.equal(normalizeEventType('earnings_beat'), 'earnings');
assert.equal(normalizeEventType('Q3 Earnings Report'), 'earnings');
assert.equal(normalizeEventType('guidance_raise'), 'guidance');
assert.equal(normalizeEventType('supply_constraint'), 'supply_chain');
assert.equal(normalizeEventType('supplyConstraint'), 'supply_chain');
assert.equal(normalizeEventType('antitrust probe'), 'regulatory');
assert.equal(normalizeEventType('ceo_resignation'), 'leadership');
assert.equal(normalizeEventType('analyst-downgrade'), 'analyst_action');
assert.equal(normalizeEventType('share buyback'), 'capital');
assert.equal(normalizeEventType('data breach'), 'security_incident');
assert.equal(normalizeEventType('interest rate decision'), 'macro');
assert.equal(normalizeEventType('product_launch'), 'product');
assert.equal(normalizeEventType('acquisition_rumor'), 'm_and_a');
assert.equal(normalizeEventType('patent lawsuit'), 'legal');
});
test('unrecognised or empty event types fall back to other, never to their own cohort', () => {
assert.equal(normalizeEventType('zebra_convention'), 'other');
assert.equal(normalizeEventType(''), 'other');
assert.equal(normalizeEventType(' '), 'other');
assert.equal(normalizeEventType(null), 'other');
assert.equal(normalizeEventType(undefined), 'other');
assert.equal(normalizeEventType(42), 'other');
assert.ok(EVENT_FAMILY_NAMES.includes('other'));
assert.ok(EVENT_FAMILY_NAMES.length <= 15, `taxonomy grew to ${EVENT_FAMILY_NAMES.length} families`);
});
test('every allowed horizon lands in one of three buckets', () => {
assert.equal(horizonBucket(1), 'short');
assert.equal(horizonBucket(5), 'short');
assert.equal(horizonBucket(10), 'medium');
assert.equal(horizonBucket(20), 'medium');
assert.equal(horizonBucket(30), 'long');
assert.equal(horizonBucket(60), 'long');
assert.equal(horizonBucket(90), 'long');
assert.equal(horizonBucket(null), 'unknown');
assert.equal(horizonBucket('nope'), 'unknown');
});
test('cohort key is versioned, coarse and stable, and the legacy key is still available', () => {
assert.equal(
cohortKey({ direction: 'positive', eventType: 'earnings_beat', horizonDays: 10 }),
'v2|unknown|earnings|medium|positive'
);
// different raw event text, same family and horizon bucket -> same cohort
assert.equal(
cohortKey({ direction: 'positive', eventType: 'quarterly results miss', horizonDays: 20 }),
cohortKey({ direction: 'positive', eventType: 'earnings_beat', horizonDays: 10 })
);
assert.notEqual(
cohortKey({ direction: 'negative', eventType: 'earnings_beat', horizonDays: 10 }),
cohortKey({ direction: 'positive', eventType: 'earnings_beat', horizonDays: 10 })
);
assert.equal(
legacyCohortKey({ direction: 'positive', eventType: 'earnings_beat', horizonDays: 10 }),
'unknown|earnings_beat|10|positive'
);
});
test('the coarse taxonomy actually collapses a realistic spread of free text', () => {
const raw = [
'earnings_beat', 'earnings_miss', 'q2_earnings', 'revenue_growth', 'margin_expansion',
'guidance_raise', 'guidance_cut', 'outlook_downgrade', 'profit_warning',
'supply_constraint', 'chip_shortage', 'production_halt', 'capacity_expansion',
'analyst_upgrade', 'price_target_raise', 'ceo_departure', 'board_shakeup',
'antitrust_probe', 'export_controls', 'tariff_announcement',
];
const families = new Set(raw.map(normalizeEventType));
assert.ok(families.size <= 8, `expected heavy collapse, got ${families.size} families`);
});
test('calibrateOutcomes reports instrument diversity and concentration', () => {
const rows = [
{ excess_return: 0.02, direction_correct: 1, instrument: 'NVDA' },
{ excess_return: 0.01, direction_correct: 1, instrument: 'nvda' },
{ excess_return: -0.01, direction_correct: 0, instrument: 'NVDA' },
{ excess_return: 0.03, direction_correct: 1, instrument: 'AMD' },
];
const result = calibrateOutcomes(rows);
assert.equal(result.sampleSize, 4);
assert.equal(result.distinctInstruments, 2);
assert.equal(result.topInstrumentShare, 0.75);
const empty = calibrateOutcomes([]);
assert.equal(empty.distinctInstruments, 0);
assert.equal(empty.topInstrumentShare, null);
const unlabelled = calibrateOutcomes([{ excess_return: 0.01, direction_correct: 1 }]);
assert.equal(unlabelled.distinctInstruments, 0);
});
test('the diversification gate blocks single ticker cohorts however large they are', () => {
const base = { direction: 'positive', probability: 0.8, expectedExcessReturn: 0.02, lowerReturn: -0.01 };
// 300 samples, one name: this is the NVDA case, and it must not qualify
const concentrated = decide({ ...base, sampleSize: 300, distinctInstruments: 1, topInstrumentShare: 1 });
assert.equal(concentrated.action, 'ABSTAIN');
assert.match(concentrated.rationale, /diversity/);
// enough names but still dominated by one of them
const dominated = decide({ ...base, sampleSize: 300, distinctInstruments: 9, topInstrumentShare: 0.82 });
assert.equal(dominated.action, 'ABSTAIN');
assert.match(dominated.rationale, /dominated/);
// pre-diversification snapshots carry no count, unknown is not adequate
const unknown = decide({ ...base, sampleSize: 300, distinctInstruments: null, topInstrumentShare: null });
assert.equal(unknown.action, 'ABSTAIN');
assert.match(unknown.rationale, /unknown/);
const qualified = decide({ ...base, sampleSize: 40, distinctInstruments: 9, topInstrumentShare: 0.3 });
assert.equal(qualified.action, 'BUY');
});
test('both evidence thresholds are overridable and default conservatively', () => {
assert.equal(DEFAULT_POLICY_RULES.minSampleSize, 30);
assert.equal(DEFAULT_POLICY_RULES.minDistinctInstruments, 5);
assert.equal(DEFAULT_POLICY_RULES.maxInstrumentConcentration, 0.5);
const input = {
direction: 'positive', probability: 0.8, expectedExcessReturn: 0.02, lowerReturn: -0.01,
sampleSize: 12, distinctInstruments: 3, topInstrumentShare: 0.4,
};
assert.equal(decide(input).action, 'ABSTAIN');
assert.equal(decide(input, { minSampleSize: 10, minDistinctInstruments: 2 }).action, 'BUY');
assert.equal(decide(input, { minSampleSize: 10, minDistinctInstruments: 2, maxInstrumentConcentration: 0.3 }).action, 'ABSTAIN');
});
test('sample size gate still runs before the diversity gate', () => {
const result = decide({
direction: 'positive', probability: 0.9, expectedExcessReturn: 0.05,
sampleSize: 2, distinctInstruments: 40, topInstrumentShare: 0.1,
});
assert.equal(result.action, 'ABSTAIN');
assert.match(result.rationale, /insufficient calibration sample/);
});
test('a missing concentration share cannot sneak past the cap as a zero', () => {
const base = {
direction: 'positive', probability: 0.8, expectedExcessReturn: 0.02,
lowerReturn: -0.01, sampleSize: 300, distinctInstruments: 40,
};
// Number(null) is 0, which used to slide straight under the cap even though we
// had no idea what the real concentration was.
for (const share of [null, undefined]) {
const verdict = decide({ ...base, topInstrumentShare: share });
assert.equal(verdict.action, 'ABSTAIN');
assert.match(verdict.rationale, /concentration unknown/);
}
// a genuinely broad cohort still gets through, we havent just bolted it shut
const broad = decide({ ...base, topInstrumentShare: 0.12 });
assert.equal(broad.action, 'BUY');
});
test/calibrationLanes.test.js
+126
View File
@@ -0,0 +1,126 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const Database = require('better-sqlite3');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { cohortKey } = require('../src/autonomy/calibration');
const {
refreshCalibration,
refreshHistoricalCalibration,
createDecisions,
calibrationHealth,
} = require('../workers/calibrationWorker');
function seedDb() {
const db = new Database(':memory:');
initAutonomySchema(db);
db.prepare("INSERT INTO autonomy_proposals(payload, information_cutoff, status) VALUES ('{}', '2026-01-01T00:00:00Z', 'accepted')").run();
return db;
}
function addPrediction(db, { instrument, direction = 'positive', eventType = 'earnings_beat', horizonDays = 10,
origin = 'live', status = 'resolved', learningEligible = 0, replayRunId = null, excessReturn = null, correct = null }) {
const prediction = db.prepare(`
INSERT INTO autonomy_predictions
(proposal_id, instrument, direction, event_type, horizon_days, information_cutoff, evidence_article_ids,
learning_eligible, strategy_version, origin, replay_run_id, status)
VALUES (1, ?, ?, ?, ?, '2026-01-01T00:00:00Z', '[1]', ?, 'test', ?, ?, ?)
`).run(instrument, direction, eventType, horizonDays, learningEligible, origin, replayRunId, status);
if (excessReturn !== null) {
db.prepare('INSERT INTO autonomy_outcomes(prediction_id, excess_return, direction_correct) VALUES (?, ?, ?)')
.run(prediction.lastInsertRowid, excessReturn, correct);
}
return prediction.lastInsertRowid;
}
test('live calibration no longer starves on the never-set learning_eligible flag', () => {
const db = seedDb();
addPrediction(db, { instrument: 'NVDA', excessReturn: 0.03, correct: 1 });
addPrediction(db, { instrument: 'AMD', excessReturn: -0.01, correct: 0 });
assert.equal(refreshCalibration(db, 'live-cal'), 1);
const snapshot = db.prepare("SELECT * FROM autonomy_calibration_snapshots WHERE source='live'").get();
assert.equal(snapshot.sample_size, 2);
assert.equal(snapshot.distinct_instruments, 2);
// the old behaviour is still reachable on purpose, for once the flag is populated
assert.equal(refreshCalibration(db, 'strict-cal', { requireLearningEligible: true }), 0);
// counters report snapshots written, so a steady state poll is genuinely quiet
assert.equal(refreshCalibration(db, 'live-cal'), 0);
assert.equal(db.prepare("SELECT COUNT(*) c FROM autonomy_calibration_snapshots WHERE source='live'").get().c, 1);
});
test('historical calibration pools origin historical and replay together', () => {
const db = seedDb();
addPrediction(db, { instrument: 'NVDA', origin: 'historical', excessReturn: 0.02, correct: 1 });
addPrediction(db, { instrument: 'AMD', origin: 'historical', excessReturn: 0.01, correct: 1 });
addPrediction(db, { instrument: 'INTC', origin: 'replay', replayRunId: 3, excessReturn: -0.02, correct: 0 });
refreshHistoricalCalibration(db, 'hist-cal');
const pooled = db.prepare("SELECT * FROM autonomy_calibration_snapshots WHERE source='historical'").get();
assert.equal(pooled.sample_size, 3, 'the historical lane must not drop the relabelled rows');
assert.equal(pooled.distinct_instruments, 3);
const perRun = db.prepare("SELECT * FROM autonomy_calibration_snapshots WHERE source='replay'").get();
assert.equal(perRun.replay_run_id, 3);
assert.equal(perRun.sample_size, 1);
});
test('decisions are only written for open live predictions', () => {
const db = seedDb();
const open = addPrediction(db, { instrument: 'NVDA', status: 'open' });
addPrediction(db, { instrument: 'AMD', status: 'resolved', excessReturn: 0.01, correct: 1 });
addPrediction(db, { instrument: 'INTC', status: 'open', origin: 'historical' });
addPrediction(db, { instrument: 'MU', status: 'open', origin: 'replay', replayRunId: 3 });
assert.equal(createDecisions(db), 1);
const rows = db.prepare('SELECT prediction_id, action FROM autonomy_decisions').all();
assert.equal(rows.length, 1);
assert.equal(rows[0].prediction_id, open);
assert.equal(rows[0].action, 'ABSTAIN');
// second pass must not duplicate
assert.equal(createDecisions(db), 0);
});
test('a big single ticker historical cohort still cannot authorise a live buy', () => {
const db = seedDb();
for (let index = 0; index < 60; index++) {
addPrediction(db, { instrument: 'NVDA', origin: 'historical', excessReturn: 0.04, correct: 1 });
}
refreshHistoricalCalibration(db, 'hist-cal');
const prediction = addPrediction(db, { instrument: 'NVDA', status: 'open' });
assert.equal(createDecisions(db), 1);
const decision = db.prepare('SELECT * FROM autonomy_decisions WHERE prediction_id=?').get(prediction);
assert.equal(decision.action, 'ABSTAIN');
// offline evidence never authorises a live order, however much of it there is,
// and the abstain has to say the offline data existed so it isnt mistaken for
// "we know nothing about this cohort"
assert.match(decision.rationale, /no live calibration/);
assert.match(decision.rationale, /offline_only source=historical n=60/);
assert.match(decision.rationale, new RegExp(cohortKey({ direction: 'positive', eventType: 'earnings_beat', horizonDays: 10 }).replace(/\|/g, '\\|')));
});
test('a cohort with no evidence at all is distinguishable from an offline only one', () => {
const db = seedDb();
const prediction = addPrediction(db, { instrument: 'NVDA', status: 'open' });
assert.equal(createDecisions(db), 1);
const decision = db.prepare('SELECT * FROM autonomy_decisions WHERE prediction_id=?').get(prediction);
assert.equal(decision.action, 'ABSTAIN');
assert.match(decision.rationale, /no evidence/);
});
test('calibration health reports the stall instead of staying silent', () => {
const db = seedDb();
addPrediction(db, { instrument: 'NVDA', origin: 'historical', excessReturn: 0.02, correct: 1 });
addPrediction(db, { instrument: 'AMD', status: 'open' });
refreshHistoricalCalibration(db, 'hist-cal');
const health = calibrationHealth(db);
assert.equal(health.liveOpen, 1);
assert.equal(health.offlineResolved, 1);
assert.equal(health.learningEligible, 0);
assert.ok(health.cohorts >= 1);
assert.equal(health.qualifyingCohorts, 0);
});
test/pubDateGuard.test.js
+74
View File
@@ -0,0 +1,74 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const { checkPubDate, guardEffectivePubDate, DEFAULT_TOLERANCE_MS } = require('../src/pubDateGuard');
const NOW = Date.parse('2026-08-29T12:00:00.000Z');
const HOUR = 60 * 60 * 1000;
test('ordinary past publication dates pass straight through', () => {
const verdict = checkPubDate('2026-08-27T09:30:00.000Z', NOW);
assert.equal(verdict.ok, true);
assert.equal(verdict.value, '2026-08-27T09:30:00.000Z');
});
test('a date-only feed value from an eastern timezone is still accepted', () => {
// "2026-08-30" stored as midnight UTC is 12 hours ahead of now — legitimate
const verdict = checkPubDate('2026-08-30T00:00:00.000Z', NOW);
assert.equal(verdict.ok, true);
});
test('mild clock skew inside the tolerance is accepted', () => {
const verdict = checkPubDate(new Date(NOW + 47 * HOUR).toISOString(), NOW);
assert.equal(verdict.ok, true);
});
test('anything past the tolerance is rejected', () => {
const verdict = checkPubDate(new Date(NOW + 49 * HOUR).toISOString(), NOW);
assert.equal(verdict.ok, false);
assert.equal(verdict.value, null);
assert.ok(verdict.skewMs > DEFAULT_TOLERANCE_MS);
});
test('the real production offender is caught', () => {
const verdict = checkPubDate('2026-12-22T00:00:00.000Z', NOW);
assert.equal(verdict.ok, false);
});
test('missing and unparseable dates are not treated as future dates', () => {
assert.equal(checkPubDate(null, NOW).ok, true);
assert.equal(checkPubDate('', NOW).ok, true);
assert.equal(checkPubDate('not a date at all', NOW).ok, true);
assert.equal(checkPubDate('not a date at all', NOW).value, null);
});
test('the tolerance boundary itself is inclusive', () => {
assert.equal(checkPubDate(new Date(NOW + DEFAULT_TOLERANCE_MS).toISOString(), NOW).ok, true);
assert.equal(checkPubDate(new Date(NOW + DEFAULT_TOLERANCE_MS + 1).toISOString(), NOW).ok, false);
});
test('a rejected date falls back to ingestion time and never drops the article', () => {
const ingestedAt = new Date().toISOString();
const future = new Date(Date.now() + 120 * 24 * HOUR).toISOString();
const warnings = [];
const original = console.warn;
console.warn = (message) => warnings.push(message);
try {
const effective = guardEffectivePubDate(future, ingestedAt, { source: 'gdelt', url: 'https://example.com/a' });
assert.equal(effective, ingestedAt);
} finally {
console.warn = original;
}
assert.equal(warnings.length, 1);
assert.match(warnings[0], /gdelt/);
assert.match(warnings[0], /https:\/\/example\.com\/a/);
assert.ok(warnings[0].includes(future));
});
test('a good date is kept, and a missing one falls back quietly', () => {
const ingestedAt = '2026-08-29T12:00:00.000Z';
assert.equal(guardEffectivePubDate('2026-08-01T00:00:00.000Z', ingestedAt, {}), '2026-08-01T00:00:00.000Z');
assert.equal(guardEffectivePubDate(null, ingestedAt, {}), ingestedAt);
});
workers/augorWorker.js
+111 -12
View File
@@ -2,6 +2,11 @@ const https = require("https");
const http = require("http");
const { findMatchedCompaniesByEmbedding } = require("./embeddings");
const { getPriceContext, formatPriceContext } = require("./priceContext");
const REPROCESS_MIN_NEW_ARTICLES = 3;
const REPROCESS_COOLDOWN_HOURS = 6;
async function runAugorWorker(archiveDb, intelligenceDb, config) {
const loopDelay = config.workers?.augorLoopDelayMs ?? 1500;
@@ -11,6 +16,26 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
SELECT * FROM article_queue WHERE status = 'pending' LIMIT 1
`);
const getProcessingState = intelligenceDb.prepare(
"SELECT last_processed_at, articles_at_last_run FROM event_processing_state WHERE event_id = ?"
);
const upsertProcessingState = intelligenceDb.prepare(`
INSERT INTO event_processing_state (event_id, last_processed_at, articles_at_last_run)
VALUES (?, CURRENT_TIMESTAMP, ?)
ON CONFLICT(event_id) DO UPDATE SET
last_processed_at = CURRENT_TIMESTAMP,
articles_at_last_run = excluded.articles_at_last_run
`);
const getCompanyAccuracy = intelligenceDb.prepare(`
SELECT
COUNT(*) as total,
SUM(correct_10d) as correct
FROM prediction_outcomes
WHERE company_id = ? AND correct_10d IS NOT NULL
`);
const recordEvent = intelligenceDb.prepare(
`INSERT INTO worker_events (worker) VALUES ('augor')`
);
@@ -45,8 +70,8 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
VALUES (?, ?, ?, ?, ?)
`);
const insertPrediction = intelligenceDb.prepare(`
INSERT INTO event_predictions (event_id, company_id, type, direction, magnitude, timeframe, rationale, event_date)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
INSERT INTO event_predictions (event_id, company_id, type, direction, magnitude, timeframe, rationale, probability, event_date)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
const getEventDate = archiveDb.prepare(`
@@ -66,7 +91,6 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
const queueRow = getPending.get();
if (!queueRow) {
await sleep(loopDelay);
continue;
}
@@ -98,12 +122,29 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
const eventArticleIds = eventArticles.map(a => a.id);
// event-level batching guard — skip re-processing if we already ran on this event recently
// and not enough new articles have arrived to justify another LLM call
const state = getProcessingState.get(eventId);
if (state && state.last_processed_at) {
const newArticles = eventArticles.length - state.articles_at_last_run;
const lastRunMs = new Date(state.last_processed_at + "Z").getTime();
const hoursSince = (Date.now() - lastRunMs) / 3_600_000;
if (newArticles < REPROCESS_MIN_NEW_ARTICLES && hoursSince < REPROCESS_COOLDOWN_HOURS) {
for (const r of getEventArticleIds.all(eventId)) setStatusByArticleId.run(r.id);
console.log(`[augor] event ${eventId} — only ${newArticles} new articles in ${hoursSince.toFixed(1)}h, skipping re-process`);
continue;
}
}
const matchedCompanies = findMatchedCompaniesByEmbedding(
eventArticleIds, archiveDb, intelligenceDb, config
);
if (matchedCompanies.length === 0) {
for (const r of getEventArticleIds.all(eventId)) setStatusByArticleId.run(r.id);
upsertProcessingState.run(eventId, eventArticles.length);
console.log(`[augor] event ${eventId} — no company match, skipped`);
continue;
}
@@ -114,6 +155,7 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
const eventDateRow = getEventDate.get(eventId);
const eventDate = eventDateRow ? eventDateRow.pub_date_effective : null;
const eventDateOnly = eventDate ? eventDate.slice(0, 10) : null;
const articleText = eventArticles.map((a, i) => {
const body = (a.content || a.description || "").slice(0, 2000);
@@ -130,9 +172,30 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
factsBlock = `Known facts about ${company.name}:\n${lines}`;
}
const result = await callLlm(llmConfig, buildPrompt(company.name, event.title, articleText, factsBlock));
// pull live market context for the company at the time of the event
let priceBlock = null;
if (company.ticker && eventDateOnly) {
try {
const snapshot = await getPriceContext(intelligenceDb, company.ticker, eventDateOnly);
priceBlock = formatPriceContext(snapshot, company.ticker);
} catch (_) {}
}
// historical accuracy of past predictions for this company
let accuracyBlock = null;
const acc = getCompanyAccuracy.get(company.id);
if (acc && acc.total >= 5) {
const pct = (acc.correct / acc.total * 100).toFixed(0);
accuracyBlock = `Past prediction accuracy for ${company.name}: ${pct}% over ${acc.total} evaluated calls.`;
}
const result = await callLlm(llmConfig, buildPrompt(company.name, event.title, articleText, factsBlock, priceBlock, accuracyBlock));
if (result) {
const seenPreds = new Set();
const writeAll = intelligenceDb.transaction(() => {
for (const r of (result.knowledge?.relationships || [])) {
insertKnowledge.run(eventId, company.id, "relationship", JSON.stringify(r), eventDate);
@@ -145,7 +208,21 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
}
for (const p of (result.predictions || [])) {
insertPrediction.run(eventId, company.id, p.type, p.direction, p.magnitude, p.timeframe, p.rationale, eventDate);
// hard filter — neutral predictions are dead weight, skip them
if (p.direction === "neutral") continue;
if (p.direction !== "positive" && p.direction !== "negative") continue;
const key = `${p.type}|${p.direction}|${p.magnitude}|${p.timeframe}`;
if (seenPreds.has(key)) continue;
seenPreds.add(key);
const prob = typeof p.probability === "number" && p.probability >= 0 && p.probability <= 1
? p.probability
: null;
insertPrediction.run(
eventId, company.id, p.type, p.direction, p.magnitude, p.timeframe, p.rationale, prob, eventDate
);
}
});
@@ -158,6 +235,7 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
}
for (const r of getEventArticleIds.all(eventId)) setStatusByArticleId.run(r.id);
upsertProcessingState.run(eventId, eventArticles.length);
recordEvent.run();
pruneCounter++;
if (pruneCounter >= 100) { pruneEvents.run(); pruneCounter = 0; }
@@ -165,23 +243,44 @@ async function runAugorWorker(archiveDb, intelligenceDb, config) {
} catch (err) {
console.error("[augor] error:", err.message);
} finally {
// Enforce pacing on every path, including the many early `continue`
// branches for skipped or already-processed events.
await sleep(loopDelay);
}
}
}
function buildPrompt(companyName, eventTitle, articleText, factsBlock) {
function buildPrompt(companyName, eventTitle, articleText, factsBlock, priceBlock, accuracyBlock) {
const factsPart = factsBlock ? `${factsBlock}\n\n` : "";
const pricePart = priceBlock ? `Market context for ${companyName}:\n${priceBlock}\n\n` : "";
const accPart = accuracyBlock ? `${accuracyBlock}\n\n` : "";
return `You are a financial intelligence analyst focused on ${companyName}. Always respond in English regardless of the language of the input articles.
return `You are a financial intelligence analyst. Always respond in English.
${factsPart}Assess the impact of the following news event on ${companyName} given what you already know about the company.
Event: ${eventTitle}
${factsPart}${pricePart}${accPart}Event: ${eventTitle}
${articleText}
Return JSON only — no explanation. Shape:
Analyze the impact of this event on ${companyName}. Return JSON only — no explanation.
Strict rules — read carefully:
- predictions must be directly caused by THIS specific event — no speculation, no priced-in narrative
- only emit a prediction if the evidence is unambiguous AND the effect on ${companyName} is concrete and quantifiable
- the default answer is no prediction. an empty predictions array is the correct output for most news. only emit one when the event clearly moves the needle
- never emit two predictions that share type+direction+magnitude+timeframe — collapse them into one
- direction must be "positive" or "negative" only — no neutral predictions, ever. if the impact is unclear, emit nothing
- magnitude:
"high" = major revenue/market-share shift, expected >5% stock move
"medium" = measurable but limited, expected 1-5% stock move
omit any prediction that doesnt clear the medium bar
- timeframe:
"short" = days to 2 weeks (use sparingly — short-horizon predictions are unreliable)
"medium" = 2 weeks to 3 months
"long" = 3+ months — preferred when the thesis is structural
- probability: your honest calibrated probability that the directional call is correct over the stated timeframe, as a number between 0.5 and 0.95. if you cant honestly assign >= 0.6, dont emit the prediction
- if the market context shows the stock has already moved sharply (>10% in 30 days), be sceptical that this event adds new information — the move may already be priced in
{
"knowledge": {
"relationships": [
@@ -195,7 +294,7 @@ Return JSON only — no explanation. Shape:
]
},
"predictions": [
{ "type": "market_share|stock_price|competitive_position|other", "direction": "positive|negative|neutral", "magnitude": "high|medium|low", "timeframe": "short|medium|long", "rationale": "string" }
{ "type": "market_share|stock_price|competitive_position|other", "direction": "positive|negative", "magnitude": "high|medium", "timeframe": "short|medium|long", "probability": 0.0, "rationale": "string" }
]
}
workers/autonomy-entrypoint.js
+11
View File
@@ -0,0 +1,11 @@
const path = require('path');
const { runAutonomyWorker } = require('./autonomyWorker');
runAutonomyWorker({
archivePath: process.env.DURIIN_DB || path.resolve('/data/archive.sqlite'),
intelligencePath: process.env.INTELLIGENCE_DB || path.resolve('/data/intelligence.sqlite'),
pollMs: Number(process.env.AUTONOMY_POLL_MS) || 1000,
}).catch((error) => {
console.error('[autonomy] fatal:', error);
process.exit(1);
});
workers/autonomyWorker.js
+144
View File
@@ -0,0 +1,144 @@
const os = require('os');
const { openRuntimeDb } = require('../src/db/runtime');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { enqueueJob, leaseNextJob, completeJob, failJob } = require('../src/autonomy/jobs');
function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); }
function isTransientCoordinatorFailure(error) {
const value = String(error || '').toLowerCase();
return value.includes('fetch failed')
|| value.includes('network')
|| value.includes('timeout')
|| value.includes('before response')
|| /\b(408|429|5\d\d)\b/.test(value)
// A 402/403 for budget is temporary in a way an ordinary auth failure is not:
// monthly limits reset and credits get topped up. Without this, 380 jobs
// dead-lettered during one exhausted window and could never come back on
// their own, including 55 live events. A wrong key still fails permanently,
// because that says "invalid" or "unauthorized" rather than naming credits.
|| (/\b(402|403)\b/.test(value)
&& /credit|quota|key limit|afford|budget|exceeded/.test(value));
}
function enqueueCoordinatorEvent(intelligenceDb, row) {
const isLive = row.ingested_at && Date.now() - Date.parse(row.ingested_at) <= 48 * 60 * 60 * 1000;
const lane = isLive ? 'live' : 'historical';
const priority = isLive ? 100 : 10;
const idempotencyKey = `coordinator_event:${row.event_id}`;
const result = enqueueJob(intelligenceDb, {
jobType: 'coordinator_event',
lane,
priority,
entityType: 'event',
entityId: row.event_id,
idempotencyKey,
});
if (result.inserted) return { inserted: true, recovered: false };
const existing = intelligenceDb.prepare(`
SELECT id, status, last_error
FROM autonomy_jobs
WHERE idempotency_key = ? AND job_type = 'coordinator_event'
`).get(idempotencyKey);
if (!existing || existing.status !== 'dead_letter' || !isTransientCoordinatorFailure(existing.last_error)) {
return { inserted: false, recovered: false };
}
const recovered = intelligenceDb.prepare(`
UPDATE autonomy_jobs
SET status = 'pending',
lane = ?,
priority = ?,
attempts = 0,
available_at = datetime('now'),
leased_by = NULL,
lease_expires_at = NULL,
last_error = ?
WHERE id = ? AND status = 'dead_letter'
`).run(lane, priority, `Recovered transient coordinator failure: ${existing.last_error || 'unknown error'}`, existing.id);
return { inserted: false, recovered: recovered.changes > 0 };
}
function reconcileArchiveBatch(archiveDb, intelligenceDb, batchSize = 250) {
const cursor = intelligenceDb.prepare("SELECT value FROM autonomy_cursors WHERE key = 'archive_reconcile'").get();
const afterId = cursor ? cursor.value : 0;
const rows = archiveDb.prepare(`
SELECT id, event_id, ingested_at, content, has_embedding
FROM articles
WHERE id > ?
ORDER BY id ASC
LIMIT ?
`).all(afterId, batchSize);
if (!rows.length) {
intelligenceDb.prepare(`
INSERT INTO autonomy_cursors(key, value) VALUES ('archive_reconcile', 0)
ON CONFLICT(key) DO UPDATE SET value = 0, updated_at = datetime('now')
`).run();
return { scanned: 0, nextCursor: 0, reset: true };
}
const enqueue = intelligenceDb.transaction(() => {
for (const row of rows) {
const readyForIntelligence = row.event_id && row.content && row.has_embedding;
if (readyForIntelligence) {
enqueueCoordinatorEvent(intelligenceDb, row);
}
}
intelligenceDb.prepare(`
INSERT INTO autonomy_cursors(key, value) VALUES ('archive_reconcile', ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = datetime('now')
`).run(rows[rows.length - 1].id);
});
enqueue();
return { scanned: rows.length, nextCursor: rows[rows.length - 1].id, reset: false };
}
function reconcileLiveBatch(archiveDb, intelligenceDb, batchSize = 250) {
const rows = archiveDb.prepare(`
SELECT id, event_id, ingested_at, content, has_embedding
FROM articles
WHERE ingested_at >= datetime('now', '-48 hours')
ORDER BY ingested_at DESC, id DESC
LIMIT ?
`).all(batchSize);
let queued = 0;
for (const row of rows) {
if (!row.event_id || !row.content || !row.has_embedding) continue;
const result = enqueueCoordinatorEvent(intelligenceDb, row);
if (result.inserted || result.recovered) queued++;
}
return { scanned: rows.length, queued };
}
async function runAutonomyWorker({ archivePath, intelligencePath, workerId = `autonomy-${os.hostname()}-${process.pid}`, pollMs = 1000 } = {}) {
const archiveDb = openRuntimeDb(archivePath, { schema: 'archive', readonly: true });
const intelligenceDb = openRuntimeDb(intelligencePath, { schema: 'intelligence' });
intelligenceDb.pragma('journal_mode = WAL');
intelligenceDb.pragma('busy_timeout = 5000');
initAutonomySchema(intelligenceDb);
while (true) {
// This worker owns maintenance reconciliation only. Without the type filter it
// can lease coordinator_event jobs and complete them without analysis.
const job = leaseNextJob(intelligenceDb, workerId, 120, ['reconcile_archive']);
if (!job) { await sleep(pollMs); continue; }
try {
if (job.job_type === 'reconcile_archive') {
reconcileLiveBatch(archiveDb, intelligenceDb);
reconcileArchiveBatch(archiveDb, intelligenceDb);
// Keep the reconciler alive as a bounded maintenance loop.
enqueueJob(intelligenceDb, {
jobType: 'reconcile_archive', lane: 'maintenance', priority: 100,
entityType: 'archive', entityId: 'archive',
idempotencyKey: `reconcile_archive:${Date.now()}`,
});
}
completeJob(intelligenceDb, job.id, workerId);
} catch (error) {
failJob(intelligenceDb, job.id, workerId, error);
}
await sleep(pollMs);
}
}
module.exports = { enqueueCoordinatorEvent, reconcileArchiveBatch, reconcileLiveBatch, runAutonomyWorker, isTransientCoordinatorFailure };
workers/calibration-entrypoint.js
+10
View File
@@ -0,0 +1,10 @@
const path = require('path');
const { runCalibrationWorker } = require('./calibrationWorker');
runCalibrationWorker({
intelligencePath: process.env.INTELLIGENCE_DB || path.resolve('/data/intelligence.sqlite'),
pollMs: Number(process.env.AUTONOMY_CALIBRATION_POLL_MS) || 60000,
}).catch((error) => {
console.error('[calibration] fatal:', error);
process.exit(1);
});
workers/calibrationWorker.js
+337
View File
@@ -0,0 +1,337 @@
const os = require('os');
const { openRuntimeDb } = require('../src/db/runtime');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { calibrateOutcomes, cohortKey } = require('../src/autonomy/calibration');
const { decide, DEFAULT_POLICY_RULES } = require('../src/autonomy/policy');
function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); }
// Historical calibration has to pool the coordinator backfill lane and the
// walk-forward replay lane, they are the same kind of evidence and splitting them
// would drop the largest cohort on the floor.
const HISTORICAL_ORIGINS = ['historical', 'replay'];
const patchedDbs = new WeakSet();
// The snapshot table predates diversification tracking. Additive only, and the
// duplicate-column error is the expected path on every run after the first.
function ensureCalibrationColumns(db) {
if (patchedDbs.has(db)) return;
for (const statement of [
'ALTER TABLE autonomy_calibration_snapshots ADD COLUMN distinct_instruments INTEGER',
'ALTER TABLE autonomy_calibration_snapshots ADD COLUMN top_instrument_share REAL',
]) {
try {
db.exec(statement);
} catch (error) {
if (!/duplicate column|already exists/i.test(error.message)) {
console.error('[calibration] snapshot column patch failed:', error.message, error.stack);
}
}
}
patchedDbs.add(db);
}
function snapshotToDecisionInput(snapshot, direction) {
return {
direction,
probability: snapshot.directional_probability,
expectedExcessReturn: snapshot.expected_excess_return,
lowerReturn: snapshot.lower_return,
upperReturn: snapshot.upper_return,
sampleSize: snapshot.sample_size,
distinctInstruments: snapshot.distinct_instruments,
topInstrumentShare: snapshot.top_instrument_share,
};
}
function refreshCalibration(db, version = `cal-${Date.now()}`, {
origin = 'live',
origins = null,
source = (origins && origins.length ? origins[0] : origin),
replayRunId = null,
// learning_eligible has never been set to 1 by anything upstream, so requiring it
// starved the live lane permanently. origin='live' *is* the eligibility contract;
// flip this back on once the coordinator actually populates the flag.
requireLearningEligible = false,
} = {}) {
ensureCalibrationColumns(db);
const originList = origins && origins.length ? origins : [origin];
const params = {};
originList.forEach((value, index) => { params[`origin${index}`] = value; });
const originClause = originList.map((_, index) => `@origin${index}`).join(', ');
const learningClause = requireLearningEligible && originList.includes('live') ? 'AND p.learning_eligible = 1' : '';
let replayClause = '';
if (replayRunId !== null && replayRunId !== undefined) {
replayClause = 'AND p.replay_run_id = @replayRunId';
params.replayRunId = replayRunId;
}
const groups = db.prepare(`
SELECT p.direction, p.event_type, p.horizon_days, p.instrument, o.*
FROM autonomy_predictions p
JOIN autonomy_outcomes o ON o.prediction_id = p.id
WHERE p.status = 'resolved' AND p.origin IN (${originClause}) ${learningClause} ${replayClause}
`).all(params).reduce((map, row) => {
const key = cohortKey({ direction: row.direction, eventType: row.event_type, horizonDays: row.horizon_days });
if (!map.has(key)) map.set(key, []);
map.get(key).push(row);
return map;
}, new Map());
const insert = db.prepare(`
INSERT INTO autonomy_calibration_snapshots
(cohort_key, sample_size, effective_sample_size, directional_probability,
expected_excess_return, lower_return, upper_return, parent_cohort_key, version, source, replay_run_id,
distinct_instruments, top_instrument_share)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
// Count what we actually wrote, not how many cohorts exist. The old code returned
// groups.size, so a steady state system reported "work happened" on every poll and
// the log line lost all meaning.
let written = 0;
const tx = db.transaction(() => {
for (const [key, rows] of groups) {
if (db.prepare(`
SELECT 1 FROM autonomy_calibration_snapshots
WHERE cohort_key = ? AND version = ? AND source = ? AND COALESCE(replay_run_id, 0) = COALESCE(?, 0)
`).get(key, version, source, replayRunId)) continue;
const result = calibrateOutcomes(rows);
insert.run(key, result.sampleSize, result.effectiveSampleSize, result.directionalProbability,
result.expectedExcessReturn, result.lowerReturn, result.upperReturn, null, version, source, replayRunId,
result.distinctInstruments, result.topInstrumentShare);
written++;
}
});
tx();
return written;
}
function refreshHistoricalCalibration(db, version = `replay-cal-${Date.now()}`) {
const runs = db.prepare(`
SELECT DISTINCT replay_run_id AS replayRunId
FROM autonomy_predictions
WHERE origin = 'replay' AND replay_run_id IS NOT NULL
ORDER BY replay_run_id
`).all();
let written = 0;
for (const run of runs) {
written += refreshCalibration(db, `${version}-run-${run.replayRunId}`, {
origin: 'replay',
source: 'replay',
replayRunId: run.replayRunId,
});
}
// Pooled historical view across both offline origins. Reporting and research
// only, decisions never read it: live orders require live calibration.
written += refreshCalibration(db, version, {
origins: HISTORICAL_ORIGINS,
source: 'historical',
});
return written;
}
// Decisions stay scoped to open live predictions on purpose: a decision is a
// forward looking policy call, and writing one against a prediction whose outcome
// is already known would put lookahead straight into the executable ledger.
// The stall was never this predicate, it was that nothing upstream was producing
// open live predictions and nothing ever said so out loud.
function createDecisions(db, strategyVersion = 'autonomy-1', rules = {}) {
ensureCalibrationColumns(db);
const predictions = db.prepare(`
SELECT p.* FROM autonomy_predictions p
LEFT JOIN autonomy_decisions d ON d.prediction_id = p.id
WHERE d.prediction_id IS NULL AND p.status = 'open' AND p.origin = 'live'
`).all();
// Only calibration built from live outcomes may authorise a live order. Backfill
// and replay are legitimate evidence that the pipeline works, but they are not a
// live track record, and an order placed off them would be exactly the confusion
// this whole thing exists to avoid. No live snapshot means abstain, full stop.
const latest = db.prepare(`
SELECT * FROM autonomy_calibration_snapshots
WHERE cohort_key = ? AND source = 'live'
ORDER BY created_at DESC, id DESC LIMIT 1
`);
// Looked up purely so an abstain can say whether offline evidence exists for the
// cohort. It never feeds decide().
const offline = db.prepare(`
SELECT source, sample_size FROM autonomy_calibration_snapshots
WHERE cohort_key = ? AND source != 'live'
ORDER BY sample_size DESC, created_at DESC, id DESC LIMIT 1
`);
const insert = db.prepare(`
INSERT INTO autonomy_decisions
(prediction_id, action, calibrated_probability, expected_excess_return, rationale, strategy_version)
VALUES (?, ?, ?, ?, ?, ?)
`);
let created = 0;
const tx = db.transaction(() => {
for (const prediction of predictions) {
const key = cohortKey({ direction: prediction.direction, eventType: prediction.event_type, horizonDays: prediction.horizon_days });
const calibration = latest.get(key);
let decision;
let rationale;
if (calibration) {
decision = decide(snapshotToDecisionInput(calibration, prediction.direction), rules);
rationale = `${decision.rationale} [cohort=${key} source=live n=${calibration.sample_size}]`;
} else {
const fallback = offline.get(key);
decision = { action: 'ABSTAIN', rationale: 'no live calibration for this cohort' };
rationale = fallback
? `${decision.rationale} [cohort=${key} offline_only source=${fallback.source} n=${fallback.sample_size}]`
: `${decision.rationale} [cohort=${key} no evidence]`;
}
insert.run(prediction.id, decision.action, calibration?.directional_probability || null,
calibration?.expected_excess_return || null, rationale, strategyVersion);
created++;
}
});
tx();
return created;
}
// Replay evaluations are walk-forward: each historical prediction is scored
// against calibration data that had matured strictly before its cutoff. They
// are stored in their own ledger, never in autonomy_decisions.
function refreshReplayEvaluations(db, rules = {}) {
const predictions = db.prepare(`
SELECT p.*, o.excess_return, o.direction_correct
FROM autonomy_predictions p JOIN autonomy_outcomes o ON o.prediction_id = p.id
LEFT JOIN autonomy_replay_evaluations e ON e.prediction_id = p.id
WHERE p.origin = 'replay' AND p.status = 'resolved' AND e.prediction_id IS NULL
ORDER BY datetime(p.information_cutoff), p.id LIMIT 200
`).all();
const prior = db.prepare(`
SELECT p.direction, p.event_type, p.horizon_days, p.instrument, o.*
FROM autonomy_predictions p JOIN autonomy_outcomes o ON o.prediction_id = p.id
WHERE p.origin = 'replay' AND p.status = 'resolved'
AND datetime(p.information_cutoff, '+' || p.horizon_days || ' days') < datetime(?)
`);
const insert = db.prepare(`
INSERT INTO autonomy_replay_evaluations
(prediction_id, replay_run_id, snapshot_cutoff, sample_size, action, calibrated_probability, expected_excess_return, rationale)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`);
const tx = db.transaction(() => {
for (const prediction of predictions) {
const key = cohortKey({ direction: prediction.direction, eventType: prediction.event_type, horizonDays: prediction.horizon_days });
const rows = prior.all(prediction.information_cutoff).filter((row) =>
cohortKey({ direction: row.direction, eventType: row.event_type, horizonDays: row.horizon_days }) === key);
const calibration = rows.length ? calibrateOutcomes(rows) : null;
const decision = calibration ? decide({ ...calibration, direction: prediction.direction }, rules)
: { action: 'ABSTAIN', rationale: 'walk-forward calibration unavailable' };
insert.run(prediction.id, prediction.replay_run_id, prediction.information_cutoff, rows.length, decision.action,
calibration?.directionalProbability || null, calibration?.expectedExcessReturn || null, decision.rationale);
}
});
tx();
return predictions.length;
}
// A worker that only speaks when something happened looks identical to a worker
// that is dead. This is the "why is nothing moving" line.
function calibrationHealth(db, rules = {}) {
const minSampleSize = Number(rules.minSampleSize ?? DEFAULT_POLICY_RULES.minSampleSize);
const minDistinctInstruments = Number(rules.minDistinctInstruments ?? DEFAULT_POLICY_RULES.minDistinctInstruments);
try {
const predictions = db.prepare(`
SELECT
SUM(CASE WHEN origin = 'live' AND status = 'open' THEN 1 ELSE 0 END) AS live_open,
SUM(CASE WHEN origin = 'live' AND status = 'resolved' THEN 1 ELSE 0 END) AS live_resolved,
SUM(CASE WHEN origin IN ('historical', 'replay') AND status = 'resolved' THEN 1 ELSE 0 END) AS offline_resolved,
SUM(CASE WHEN learning_eligible = 1 THEN 1 ELSE 0 END) AS learning_eligible
FROM autonomy_predictions
`).get() || {};
// Only live snapshots can authorise anything, so counting offline cohorts as
// "qualifying" would overstate how close we are to being able to trade. They
// are still worth reporting, just in their own bucket.
const maxConcentration = Number(rules.maxInstrumentConcentration ?? DEFAULT_POLICY_RULES.maxInstrumentConcentration);
const gate = `sample_size >= ? AND COALESCE(distinct_instruments, 0) >= ?
AND COALESCE(top_instrument_share, 1) <= ?`;
const cohorts = db.prepare(`
SELECT
COUNT(*) AS total,
SUM(CASE WHEN source = 'live' AND ${gate} THEN 1 ELSE 0 END) AS qualifying,
SUM(CASE WHEN source != 'live' AND ${gate} THEN 1 ELSE 0 END) AS offline_qualifying
FROM autonomy_calibration_snapshots
`).get(minSampleSize, minDistinctInstruments, maxConcentration,
minSampleSize, minDistinctInstruments, maxConcentration) || {};
return {
liveOpen: Number(predictions.live_open || 0),
liveResolved: Number(predictions.live_resolved || 0),
offlineResolved: Number(predictions.offline_resolved || 0),
learningEligible: Number(predictions.learning_eligible || 0),
cohorts: Number(cohorts.total || 0),
qualifyingCohorts: Number(cohorts.qualifying || 0),
offlineQualifyingCohorts: Number(cohorts.offline_qualifying || 0),
};
} catch (error) {
console.error('[calibration] health probe failed:', error.message, error.stack);
return null;
}
}
function formatHealth(health) {
if (!health) return 'health=unavailable';
return `live_open=${health.liveOpen} live_resolved=${health.liveResolved} offline_resolved=${health.offlineResolved}`
+ ` learning_eligible=${health.learningEligible} cohorts=${health.cohorts}`
+ ` qualifying_live_cohorts=${health.qualifyingCohorts} qualifying_offline_cohorts=${health.offlineQualifyingCohorts}`;
}
async function runCalibrationWorker({
intelligencePath,
pollMs = 60000,
stallLogMs = 900000,
workerId = `calibration-${os.hostname()}-${process.pid}`,
} = {}) {
const db = openRuntimeDb(intelligencePath, { schema: 'intelligence' });
db.pragma('journal_mode = WAL');
db.pragma('busy_timeout = 5000');
initAutonomySchema(db);
ensureCalibrationColumns(db);
let lastStallLog = 0;
let lastStallSignature = null;
while (true) {
try {
const state = db.prepare('SELECT COUNT(*) AS count, COALESCE(MAX(prediction_id), 0) AS max_id FROM autonomy_outcomes').get();
const version = `cal-${state.count}-${state.max_id}`;
const snapshots = refreshCalibration(db, version);
const historicalSnapshots = refreshHistoricalCalibration(db, version);
const decisions = createDecisions(db);
const replayEvaluations = refreshReplayEvaluations(db);
if (snapshots || historicalSnapshots || decisions || replayEvaluations) {
console.log(`[${workerId}] calibration snapshots=${snapshots} historical_snapshots=${historicalSnapshots} decisions=${decisions} replay_evaluations=${replayEvaluations} ${formatHealth(calibrationHealth(db))}`);
lastStallSignature = null;
lastStallLog = 0;
} else {
// Nothing moved. Say so, but only when the picture changes or every
// stallLogMs, otherwise this is a zeroes-every-60-seconds firehose.
const health = calibrationHealth(db);
const signature = formatHealth(health);
const now = Date.now();
if (signature !== lastStallSignature || now - lastStallLog >= stallLogMs) {
console.log(`[${workerId}] calibration idle (no new cohorts, decisions or evaluations) ${signature}`);
lastStallSignature = signature;
lastStallLog = now;
}
}
} catch (error) {
console.error(`[${workerId}] calibration error:`, error.message, error.stack);
}
await sleep(pollMs);
}
}
module.exports = {
ensureCalibrationColumns,
refreshCalibration,
refreshHistoricalCalibration,
createDecisions,
refreshReplayEvaluations,
calibrationHealth,
runCalibrationWorker,
};
workers/coordinator-entrypoint.js
+11
View File
@@ -0,0 +1,11 @@
const path = require('path');
const { runCoordinatorWorker } = require('./coordinatorWorker');
runCoordinatorWorker({
archivePath: process.env.DURIIN_DB || path.resolve('/data/archive.sqlite'),
intelligencePath: process.env.INTELLIGENCE_DB || path.resolve('/data/intelligence.sqlite'),
pollMs: Number(process.env.AUTONOMY_POLL_MS) || 1000,
}).catch((error) => {
console.error('[coordinator] fatal:', error);
process.exit(1);
});
workers/coordinatorWorker.js
+104
View File
@@ -0,0 +1,104 @@
const os = require('os');
const fs = require('fs');
const path = require('path');
const { openRuntimeDb } = require('../src/db/runtime');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { leaseNextJob, completeJob, failJob } = require('../src/autonomy/jobs');
const { callCoordinator } = require('../src/autonomy/llm');
const { acceptProposal, recordRejectedProposal, INSTRUMENT_RULES } = require('../src/autonomy/coordinator');
const { EVENT_FAMILY_NAMES } = require('../src/autonomy/calibration');
const { buildGraphContext } = require('../src/autonomy/graphContext');
function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); }
// bumped when the prompt changes in a way that changes what a prediction means.
// autonomy-2 is the first version that can see the relationship graph.
const STRATEGY_VERSION = 'autonomy-2';
// This never moved across four prompt changes, so every proposal on record
// claims to come from the same prompt as the very first one. Attribution was
// impossible, which is why the only honest before/after we had was a timestamp.
const PROMPT_VERSION = 'coordinator-2';
function loadConfig() {
const configPath = path.resolve(process.env.DURIIN_CONFIG || path.join(__dirname, '..', 'config.json'));
const raw = JSON.parse(fs.readFileSync(configPath, 'utf8'));
require('dotenv').config({ path: path.resolve(path.dirname(configPath), '.env') });
raw.openRouter = { ...(raw.openRouter || {}) };
if (process.env.OPEN_ROUTER_API_KEY) raw.openRouter.apiKey = process.env.OPEN_ROUTER_API_KEY;
if (process.env.OPEN_ROUTER_LLM_MODEL) raw.openRouter.llmModel = process.env.OPEN_ROUTER_LLM_MODEL;
return raw;
}
function buildPrompt(event, articles, graphContext = '') {
const evidence = articles.map((article, index) =>
`[Evidence ${index + 1}] article_id=${article.id}\nTitle: ${article.title}\n${String(article.content || article.description || '').slice(0, 4000)}`
).join('\n\n---\n\n');
return `Event title: ${event.title}\n\n${evidence}\n\n${graphContext ? `${graphContext}\n\n` : ''}Return JSON only in this shape:\n${JSON.stringify({ predictions: [{
instrument: '<ticker supported by the articles>', direction: '<positive or negative>',
event_type: '<one value from the event_type list below>',
causal_channel: '<short description>', horizon_days: '<one of 1, 5, 10, 20, 30, 60, 90>',
evidence_article_ids: ['<article_id values from the evidence above>'],
invalidation_condition: '<what would falsify this>',
}] }, null, 2)}\n\nEvery value in that shape is a placeholder describing the field. Do not copy them. Choose instrument, direction and horizon_days from the evidence in front of you.\n\nevent_type must be exactly one of: ${EVENT_FAMILY_NAMES.join(', ')}. Pick the closest one. Use "other" only when none of them genuinely apply, and never invent a value outside this list.\n\n${INSTRUMENT_RULES}\n\nUse only instruments and evidence directly supported by the articles. Return an empty predictions array when there is no clear, tradable hypothesis. Never include probabilities, returns, confidence, position sizes, or actions.`;
}
async function runCoordinatorWorker({ archivePath, intelligencePath, workerId = `coordinator-${os.hostname()}-${process.pid}`, pollMs = 1000 } = {}) {
const archiveDb = openRuntimeDb(archivePath, { schema: 'archive', readonly: true });
const intelligenceDb = openRuntimeDb(intelligencePath, { schema: 'intelligence' });
intelligenceDb.pragma('journal_mode = WAL');
intelligenceDb.pragma('busy_timeout = 5000');
initAutonomySchema(intelligenceDb);
const config = loadConfig();
while (true) {
const job = leaseNextJob(intelligenceDb, workerId, 180, ['coordinator_event']);
if (!job) { await sleep(pollMs); continue; }
try {
const event = archiveDb.prepare('SELECT id, title FROM events WHERE id = ?').get(job.entity_id);
if (!event) throw new Error(`event ${job.entity_id} does not exist`);
const articles = archiveDb.prepare(`
SELECT id, title, description, content, pub_date_effective
FROM articles
WHERE event_id = ? AND content IS NOT NULL AND content != '' AND is_index_page = 0
ORDER BY pub_date_effective ASC, id ASC LIMIT 25
`).all(job.entity_id);
const allowlisted = intelligenceDb.prepare(
"SELECT 1 FROM autonomy_instruments WHERE active=1 AND tradable=1 LIMIT 1"
).get();
if (!allowlisted) throw new Error('no tradable instruments are allowlisted');
const historical = job.lane === 'historical';
const informationCutoff = historical
? (articles.map((article) => article.pub_date_effective).filter(Boolean).sort().pop() || new Date().toISOString())
: new Date().toISOString();
const graphContext = buildGraphContext(intelligenceDb, event.id, informationCutoff);
const raw = await callCoordinator(config, buildPrompt(event, articles, graphContext));
try {
acceptProposal(intelligenceDb, archiveDb, raw, {
eventId: event.id,
informationCutoff,
model: config.openRouter.llmModel || 'unknown',
promptVersion: PROMPT_VERSION,
strategyVersion: STRATEGY_VERSION,
// only a genuine live lane job may ever feed learning
origin: historical ? 'historical' : 'live',
learningEligible: !historical,
});
} catch (validationError) {
console.error(`[${workerId}] proposal rejected for event ${event.id}:`, validationError.message);
recordRejectedProposal(intelligenceDb, raw, {
eventId: event.id,
informationCutoff,
model: config.openRouter.llmModel || 'unknown',
promptVersion: PROMPT_VERSION,
origin: historical ? 'historical' : 'live',
learningEligible: !historical,
}, validationError.message);
}
completeJob(intelligenceDb, job.id, workerId);
} catch (error) {
failJob(intelligenceDb, job.id, workerId, error);
}
await sleep(pollMs);
}
}
module.exports = { buildPrompt, runCoordinatorWorker };
workers/db.js
+46
View File
@@ -1,5 +1,6 @@
const Database = require("better-sqlite3");
const sqliteVec = require("sqlite-vec");
const { initAutonomySchema } = require("../src/autonomy/schema");
let archiveDb = null;
let intelligenceDb = null;
@@ -17,6 +18,7 @@ function getIntelligenceDb(dbPath) {
if (!intelligenceDb) {
intelligenceDb = new Database(dbPath);
intelligenceDb.pragma("journal_mode = WAL");
initAutonomySchema(intelligenceDb);
}
return intelligenceDb;
}
@@ -109,6 +111,7 @@ function runMigrations(db) {
function runColumnMigrations(db) {
try { db.exec("ALTER TABLE event_predictions ADD COLUMN event_date TEXT"); } catch (_) {}
try { db.exec("ALTER TABLE event_knowledge ADD COLUMN event_date TEXT"); } catch (_) {}
try { db.exec("ALTER TABLE event_predictions ADD COLUMN probability REAL"); } catch (_) {}
db.exec(`
CREATE TABLE IF NOT EXISTS worker_events (
@@ -137,6 +140,49 @@ function runColumnMigrations(db) {
);
`);
// tracks last-processed state per event so augor doesnt redundantly re-run on every new article
db.exec(`
CREATE TABLE IF NOT EXISTS event_processing_state (
event_id INTEGER PRIMARY KEY,
last_processed_at DATETIME,
articles_at_last_run INTEGER NOT NULL DEFAULT 0
);
`);
// cached daily price snapshots so the augor prompt can include real market context
db.exec(`
CREATE TABLE IF NOT EXISTS price_snapshots (
ticker TEXT NOT NULL,
as_of TEXT NOT NULL,
price REAL,
price_30d_ago REAL,
price_90d_ago REAL,
vol_30d REAL,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (ticker, as_of)
);
`);
// outcomes — actual realized returns for each prediction, populated by the outcome worker
db.exec(`
CREATE TABLE IF NOT EXISTS prediction_outcomes (
prediction_id INTEGER PRIMARY KEY,
company_id INTEGER,
ticker TEXT,
event_date TEXT,
price_0 REAL,
price_5d REAL,
price_10d REAL,
r5 REAL,
r10 REAL,
correct_5d INTEGER,
correct_10d INTEGER,
evaluated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_prediction_outcomes_company ON prediction_outcomes (company_id);
`);
// prune rows older than 1 hour so the table doesnt grow unbounded
db.exec(`DELETE FROM worker_events WHERE completed_at < datetime('now', '-1 hour')`);
}
workers/execution-entrypoint.js
+12
View File
@@ -0,0 +1,12 @@
const path = require('path');
const { runExecutionWorker } = require('./executionWorker');
runExecutionWorker({
intelligencePath: process.env.INTELLIGENCE_DB || path.resolve('/data/intelligence.sqlite'),
pollMs: Number(process.env.AUTONOMY_EXECUTION_POLL_MS) || 10000,
mode: process.env.AUTONOMY_EXECUTION_MODE || 'shadow',
notional: Number(process.env.AUTONOMY_DEFAULT_NOTIONAL) || 100,
}).catch((error) => {
console.error('[execution] fatal:', error);
process.exit(1);
});
workers/executionWorker.js
+122
View File
@@ -0,0 +1,122 @@
const os = require('os');
const { openRuntimeDb } = require('../src/db/runtime');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { createOrderIntent } = require('../src/autonomy/orderIntents');
const { createAlpacaPaperClient } = require('../src/brokers/alpacaPaper');
const { getExecutionControls } = require('../src/autonomy/settings');
function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); }
async function runExecutionWorker({ intelligencePath, pollMs = 10000, mode = 'shadow', notional = 100, workerId = `execution-${os.hostname()}-${process.pid}` } = {}) {
if (!['shadow', 'paper'].includes(mode)) throw new Error(`unsupported execution mode: ${mode}`);
const db = openRuntimeDb(intelligencePath, { schema: 'intelligence' });
db.pragma('journal_mode = WAL');
db.pragma('busy_timeout = 5000');
initAutonomySchema(db);
// mode is re-read every poll now. the env var is only the default, so the kill
// switch actually works during an incident instead of needing a redeploy first.
let paperClient = null;
let lastMode = null;
let lastKill = null;
while (true) {
const controls = getExecutionControls(db, { AUTONOMY_EXECUTION_MODE: mode });
if (controls.mode !== lastMode) {
console.log(`[${workerId}] execution mode = ${controls.mode} (from ${controls.modeSource})`);
lastMode = controls.mode;
}
if (controls.killSwitch !== lastKill) {
console.log(`[${workerId}] kill switch ${controls.killSwitch ? 'ENGAGED, no orders will be placed' : 'released'}`);
lastKill = controls.killSwitch;
}
if (controls.mode === 'paper' && !paperClient) {
paperClient = createAlpacaPaperClient({ keyId: process.env.ALPACA_PAPER_KEY_ID, secretKey: process.env.ALPACA_PAPER_SECRET_KEY });
}
if (controls.mode !== 'paper') paperClient = null;
if (paperClient) {
try {
const [account, positions, orders] = await Promise.all([
paperClient.getAccount(), paperClient.getPositions(), paperClient.getOrders(),
]);
db.prepare(`
INSERT INTO autonomy_account_snapshots(broker, account_id, equity, cash, buying_power, payload)
VALUES ('alpaca-paper', ?, ?, ?, ?, ?)
`).run(account.id || null, Number(account.equity), Number(account.cash), Number(account.buying_power), JSON.stringify(account));
const insertPosition = db.prepare(`
INSERT INTO autonomy_position_snapshots(broker, instrument, quantity, market_value, unrealized_pl, payload)
VALUES ('alpaca-paper', ?, ?, ?, ?, ?)
`);
for (const position of positions || []) {
insertPosition.run(position.symbol, Number(position.qty), Number(position.market_value), Number(position.unrealized_pl), JSON.stringify(position));
}
for (const order of orders || []) {
if (!order.client_order_id) continue;
const mapped = { accepted: 'submitted', new: 'submitted', pending_new: 'submitted', partially_filled: 'partially_filled', filled: 'filled', canceled: 'cancelled', cancelled: 'cancelled', rejected: 'rejected' }[order.status];
if (!mapped) continue;
db.prepare(`
UPDATE autonomy_order_intents SET status=?, broker_order_id=?, updated_at=datetime('now')
WHERE client_order_id=?
`).run(mapped, order.id || null, order.client_order_id);
db.prepare(`
INSERT INTO autonomy_broker_events(broker, event_type, broker_id, payload)
VALUES ('alpaca-paper', ?, ?, ?)
`).run(order.status, order.id || null, JSON.stringify(order));
}
} catch (error) {
console.error(`[${workerId}] broker reconciliation:`, error.message);
}
}
// Broker reconciliation above still runs while the switch is engaged, we want
// to keep seeing the account. It is order creation specifically that stops.
const decisions = controls.killSwitch ? [] : db.prepare(`
SELECT d.id FROM autonomy_decisions d
JOIN autonomy_predictions p ON p.id = d.prediction_id
LEFT JOIN autonomy_order_intents oi ON oi.decision_id = d.id
WHERE oi.id IS NULL AND d.action IN ('BUY', 'SELL') AND p.origin = 'live'
ORDER BY d.created_at ASC LIMIT 25
`).all();
for (const decision of decisions) {
try {
const intent = createOrderIntent(db, decision.id, notional, { tradable: true, maxNotional: notional });
if (controls.mode === 'paper') db.prepare("UPDATE autonomy_order_intents SET status='pending', updated_at=datetime('now') WHERE client_order_id=?").run(intent.clientOrderId);
console.log(`[${workerId}] ${controls.mode} intent ${intent.clientOrderId}`);
} catch (error) {
console.error(`[${workerId}] decision ${decision.id}:`, error.message);
}
}
if (paperClient) {
const pending = db.prepare("SELECT * FROM autonomy_order_intents WHERE status='pending' ORDER BY created_at ASC LIMIT 25").all();
for (const intent of pending) {
try {
let order;
try { order = await paperClient.getOrderByClientId(intent.client_order_id); } catch (error) {
if (error.status !== 404) throw error;
}
if (!order) {
order = await paperClient.submitOrder({
symbol: intent.instrument,
notional: String(intent.notional),
side: intent.side,
type: 'market',
time_in_force: 'day',
client_order_id: intent.client_order_id,
});
}
db.prepare(`
UPDATE autonomy_order_intents
SET status = ?, broker_order_id = ?, updated_at = datetime('now')
WHERE id = ?
`).run(order.status === 'filled' ? 'filled' : 'submitted', order.id || null, intent.id);
} catch (error) {
console.error(`[${workerId}] paper order ${intent.client_order_id}:`, error.message);
db.prepare("UPDATE autonomy_order_intents SET attempts=attempts+1, last_error=?, updated_at=datetime('now') WHERE id=?")
.run(String(error.message).slice(0, 1000), intent.id);
}
}
}
await sleep(pollMs);
}
}
module.exports = { runExecutionWorker };
workers/graphWorker.js
+22 -1
View File
@@ -5,6 +5,19 @@ const http = require("http");
const VALID_TYPES = ["supplier", "customer", "competitor", "partner", "investor", "dependency"];
// A blown OpenRouter monthly limit comes back as an instant 403, so with no cooldown
// this resolver just hammers the endpoint: 2356 failures in 20 minutes, and a log so
// noisy nothing else in it is readable. Quota and auth problems dont fix themselves
// within seconds, so back off properly and stay quiet until the window is over.
const LLM_COOLDOWN_MS = 15 * 60 * 1000;
let llmCooldownUntil = 0;
function isQuotaOrAuthError(message) {
const text = String(message || "");
return /\b(401|402|403|429)\b/.test(text)
|| /key limit|quota|insufficient credit|rate limit/i.test(text);
}
const KEYWORD_MAP = [
["manufactur", "supplier"],
["suppli", "supplier"],
@@ -90,6 +103,8 @@ Reply with just the number of the match, or "none" if none apply. No explanation
temperature: 0,
});
if (Date.now() < llmCooldownUntil) return null;
const url = new URL("https://openrouter.ai/api/v1/chat/completions");
let responseText;
@@ -99,7 +114,13 @@ Reply with just the number of the match, or "none" if none apply. No explanation
"Authorization": `Bearer ${llmConfig.apiKey || ""}`,
});
} catch (err) {
console.warn("[graph] LLM resolve failed:", err.message);
if (isQuotaOrAuthError(err.message)) {
// one line per window rather than one per attempt, but never silent
console.error(`[graph] LLM quota/auth failure, pausing resolution for ${LLM_COOLDOWN_MS / 60000}m:`, err.message);
llmCooldownUntil = Date.now() + LLM_COOLDOWN_MS;
} else {
console.warn("[graph] LLM resolve failed:", err.message);
}
return null;
}
workers/index.js
+10
View File
@@ -8,6 +8,7 @@ const { ensureCompanyEmbeddings } = require("./embeddings");
const { runConsolidationWorker } = require("./consolidationWorker");
const { runGraphWorker } = require("./graphWorker");
const { runSignalWorker } = require("./signalWorker");
const { runOutcomeWorker } = require("./outcomeWorker");
require("dotenv").config({ path: path.resolve(__dirname, "../.env") });
@@ -25,6 +26,10 @@ const openRouter = { ...rawConfig.openRouter };
if (process.env.OPEN_ROUTER_API_KEY) openRouter.apiKey = process.env.OPEN_ROUTER_API_KEY;
if (process.env.OPEN_ROUTER_LLM_MODEL) openRouter.llmModel = process.env.OPEN_ROUTER_LLM_MODEL;
if (process.env.OPEN_ROUTER_EMBED_MODEL) openRouter.embeddingModel = process.env.OPEN_ROUTER_EMBED_MODEL;
// OPEN_ROUTER_CHEAP_MODEL was already set in the environment and nothing read it,
// so graph entity resolution ran on the reasoning model: 7,558 reasoning tokens to
// answer "reply with just the number", 113x the cost of a model that just answers.
if (process.env.OPEN_ROUTER_CHEAP_MODEL) openRouter.cheapModel = process.env.OPEN_ROUTER_CHEAP_MODEL;
const config = {
duriin_db: process.env.DURIIN_DB || resolvePath(rawConfig.duriin_db, path.resolve(configDir, "archive.sqlite")),
@@ -77,6 +82,11 @@ runSignalWorker(archiveDb, intelligenceDb, config).catch(err => {
process.exit(1);
});
runOutcomeWorker(archiveDb, intelligenceDb, config).catch(err => {
console.error("[outcome] fatal:", err);
process.exit(1);
});
process.on("SIGINT", () => {
console.log("[intelligence] shutting down");
process.exit(0);
workers/outcome-autonomy-entrypoint.js
+10
View File
@@ -0,0 +1,10 @@
const path = require('path');
const { resolveAutonomyOutcomes } = require('./outcomeAutonomyWorker');
resolveAutonomyOutcomes({
intelligencePath: process.env.INTELLIGENCE_DB || path.resolve('/data/intelligence.sqlite'),
pollMs: Number(process.env.AUTONOMY_OUTCOME_POLL_MS) || 60000,
}).catch((error) => {
console.error('[autonomy-outcome] fatal:', error);
process.exit(1);
});
workers/outcomeAutonomyWorker.js
+152
View File
@@ -0,0 +1,152 @@
const os = require('os');
const https = require('https');
const { openRuntimeDb } = require('../src/db/runtime');
const { initAutonomySchema } = require('../src/autonomy/schema');
const { calculateOutcome, addTradingDays, yahooSymbol } = require('../src/autonomy/outcomes');
function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); }
function httpGet(url) {
return new Promise((resolve, reject) => {
const request = https.get(url, { headers: { 'User-Agent': 'duriin-autonomy/1.0' } }, (response) => {
let body = '';
response.setEncoding('utf8');
response.on('data', (chunk) => { body += chunk; });
response.on('end', () => response.statusCode >= 200 && response.statusCode < 300
? resolve(body) : reject(new Error(`market data returned ${response.statusCode}`)));
});
request.setTimeout(15000, () => request.destroy(new Error('market data timeout')));
request.on('error', reject);
});
}
const MAX_OUTCOME_ATTEMPTS = 5;
// req.setTimeout only covers socket inactivity. A response that opens and then
// stalls, or a socket that never emits anything at all, leaves the promise
// pending forever and the whole loop with it. This worker sat "Up 3 days" and
// silent while predictions it could resolve in 400ms went unscored, which is the
// third time an unbounded await in a long lived loop has quietly stopped a
// worker. This is the bound that cannot be skipped.
const HISTORY_HARD_TIMEOUT = 30000;
function withTimeout(promise, ms, label) {
let timer;
const expired = new Promise((_, reject) => {
timer = setTimeout(() => reject(new Error(`${label} exceeded ${ms}ms`)), ms);
});
return Promise.race([promise, expired]).finally(() => clearTimeout(timer));
}
// how far past the horizon we keep trying before accepting there is no data
const UNRESOLVABLE_GRACE_DAYS = 3;
async function history(symbol) {
// GDELT backfills predate the normal rolling quote window. Use an explicit
// point-in-time range so replay outcomes do not silently become unresolvable.
const url = `https://query1.finance.yahoo.com/v8/finance/chart/${encodeURIComponent(yahooSymbol(symbol))}?period1=946684800&period2=${Math.floor(Date.now() / 1000)}&interval=1d`;
const body = JSON.parse(await httpGet(url));
const result = body?.chart?.result?.[0];
if (!result) return [];
return (result.timestamp || []).map((timestamp, index) => ({
date: new Date(timestamp * 1000).toISOString().slice(0, 10),
close: result.indicators?.quote?.[0]?.close?.[index],
})).filter((row) => Number.isFinite(row.close));
}
async function resolveAutonomyOutcomes({ intelligencePath, workerId = `outcome-${os.hostname()}-${process.pid}`, pollMs = 60000 } = {}) {
const db = openRuntimeDb(intelligencePath, { schema: 'intelligence' });
db.pragma('journal_mode = WAL');
db.pragma('busy_timeout = 5000');
initAutonomySchema(db);
const cache = new Map();
const failures = new Map();
while (true) {
// The sql filter is deliberately loose, it only counts calendar days and cannot
// know about weekends or when a close actually publishes. Trading day
// arithmetic, the same arithmetic calculateOutcome uses to find the exit bar,
// then decides what is genuinely ready.
const candidates = db.prepare(`
SELECT p.* FROM autonomy_predictions p
LEFT JOIN autonomy_outcomes o ON o.prediction_id = p.id
WHERE p.status = 'open' AND o.prediction_id IS NULL
AND datetime(p.information_cutoff, '+' || p.horizon_days || ' days') <= datetime('now')
ORDER BY p.information_cutoff ASC LIMIT 100
`).all();
const today = new Date().toISOString().slice(0, 10);
const predictions = candidates.filter((p) => {
const horizonDate = addTradingDays(String(p.information_cutoff).slice(0, 10), p.horizon_days);
// strictly before today, so the exit session has closed and published
return horizonDate < today;
}).slice(0, 25);
if (predictions.length) {
console.log(`[autonomy-outcome] ${workerId} scoring ${predictions.length} matured predictions`
+ ` (${candidates.length} candidates)`);
}
for (const prediction of predictions) {
try {
if (!cache.has(prediction.instrument)) {
cache.set(prediction.instrument, await withTimeout(history(prediction.instrument),
HISTORY_HARD_TIMEOUT, `market data for ${prediction.instrument}`));
}
if (!cache.has('SPY')) {
cache.set('SPY', await withTimeout(history('SPY'), HISTORY_HARD_TIMEOUT, 'market data for SPY'));
}
const result = calculateOutcome(prediction, cache.get(prediction.instrument), cache.get('SPY'));
if (!result) {
// A null here almost always means the exit bar has not published yet, not
// that the prediction can never be scored. The sql due-check counts
// calendar days while the price lookup counts trading days, so a friday
// horizon-1 call looks due on saturday when monday's close cannot exist.
// Retiring it there permanently destroyed exactly the short-horizon live
// predictions we are waiting on. Wait until the horizon is properly past
// before giving up on it.
const horizonDate = addTradingDays(String(prediction.information_cutoff).slice(0, 10), prediction.horizon_days);
const graceExpired = addTradingDays(horizonDate, UNRESOLVABLE_GRACE_DAYS) < new Date().toISOString().slice(0, 10);
if (graceExpired) {
db.prepare("UPDATE autonomy_predictions SET status = 'unresolvable' WHERE id = ?").run(prediction.id);
console.error(`[autonomy-outcome] ${workerId} prediction ${prediction.id} (${prediction.instrument})`
+ ` unresolvable: no market data ${UNRESOLVABLE_GRACE_DAYS} trading days past horizon ${horizonDate}`);
} else {
cache.delete(prediction.instrument);
}
continue;
}
db.prepare(`
INSERT INTO autonomy_outcomes
(prediction_id, price_0, price_horizon, benchmark_0, benchmark_horizon, excess_return, direction_correct, error_type)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(prediction_id) DO UPDATE SET
price_0=excluded.price_0,
price_horizon=excluded.price_horizon,
benchmark_0=excluded.benchmark_0,
benchmark_horizon=excluded.benchmark_horizon,
excess_return=excluded.excess_return,
direction_correct=excluded.direction_correct,
error_type=excluded.error_type,
evaluated_at=datetime('now')
`).run(prediction.id, result.price0, result.priceHorizon, result.benchmark0, result.benchmarkHorizon,
result.excessReturn, result.directionCorrect, result.directionCorrect ? null : 'direction_error');
db.prepare("UPDATE autonomy_predictions SET status = 'resolved' WHERE id = ?").run(prediction.id);
} catch (error) {
// A prediction that keeps failing stays 'open' and comes straight back on the
// next poll, so a symbol market data will never have just spins forever. Give
// it a few goes for genuinely transient failures, then retire it.
const attempts = (failures.get(prediction.id) || 0) + 1;
failures.set(prediction.id, attempts);
console.error(`[autonomy-outcome] ${workerId} prediction ${prediction.id} (${prediction.instrument})`
+ ` attempt ${attempts}/${MAX_OUTCOME_ATTEMPTS}:`, error.message);
if (attempts >= MAX_OUTCOME_ATTEMPTS) {
db.prepare("UPDATE autonomy_predictions SET status = 'unresolvable' WHERE id = ?").run(prediction.id);
failures.delete(prediction.id);
console.error(`[autonomy-outcome] ${workerId} prediction ${prediction.id} marked unresolvable`
+ ` after ${attempts} failed attempts on ${prediction.instrument}`);
}
cache.delete(prediction.instrument);
}
await sleep(800);
}
await sleep(pollMs);
}
}
module.exports = { calculateOutcome, resolveAutonomyOutcomes, yahooSymbol };
workers/outcomeWorker.js
+191
View File
@@ -0,0 +1,191 @@
// evaluates predictions older than 11 days against realized stock returns
// runs continuously, batching by ticker so we hit yahoo once per company per cycle
const { getPriceContext } = require("./priceContext");
const { yahooSymbol } = require("../src/autonomy/outcomes");
const https = require("https");
// PSTG and GROQ were re-requested every single poll, forever, because a fetch
// failure only logged and moved on. Predictions for a ticker with no market data
// never leave the pending set, so the loop retries them for as long as the process
// lives. Back off per ticker instead, doubling up to an hour, so a dead symbol
// costs one request an hour rather than one a minute.
const TICKER_BACKOFF_START_MS = 5 * 60 * 1000;
const TICKER_BACKOFF_MAX_MS = 60 * 60 * 1000;
async function runOutcomeWorker(archiveDb, intelligenceDb, config) {
const loopDelay = config.workers?.outcomeLoopDelayMs ?? 60000;
// pull predictions that are old enough to evaluate (>= 11 calendar days) and dont have an outcome yet
const getPending = intelligenceDb.prepare(`
SELECT ep.id, ep.company_id, ep.event_date, ep.direction, tc.ticker
FROM event_predictions ep
JOIN tracked_companies tc ON ep.company_id = tc.id
LEFT JOIN prediction_outcomes po ON po.prediction_id = ep.id
WHERE po.prediction_id IS NULL
AND ep.event_date IS NOT NULL
AND date(ep.event_date) <= date('now', '-11 days')
AND ep.direction IN ('positive', 'negative')
AND tc.ticker IS NOT NULL
AND tc.ticker NOT LIKE '%.%'
AND length(tc.ticker) <= 5
ORDER BY ep.event_date ASC
LIMIT 50
`);
const insertOutcome = intelligenceDb.prepare(`
INSERT OR REPLACE INTO prediction_outcomes
(prediction_id, company_id, ticker, event_date, price_0, price_5d, price_10d, r5, r10, correct_5d, correct_10d)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
const tickerBackoff = new Map();
while (true) {
try {
const pending = getPending.all();
if (pending.length === 0) {
await sleep(loopDelay);
continue;
}
// group by ticker so we only fetch each company's history once per cycle
const byTicker = new Map();
for (const p of pending) {
if (!byTicker.has(p.ticker)) byTicker.set(p.ticker, []);
byTicker.get(p.ticker).push(p);
}
let evaluated = 0;
for (const [ticker, preds] of byTicker.entries()) {
const cooling = tickerBackoff.get(ticker);
if (cooling && Date.now() < cooling.until) continue;
let history;
try {
history = await fetchYahooHistory(ticker, "1y");
tickerBackoff.delete(ticker);
} catch (err) {
const previous = cooling ? cooling.waitMs : 0;
const waitMs = Math.min(TICKER_BACKOFF_MAX_MS, previous ? previous * 2 : TICKER_BACKOFF_START_MS);
tickerBackoff.set(ticker, { until: Date.now() + waitMs, waitMs });
console.error(`[outcome] yahoo error for ${ticker}: ${err.message} — backing off ${Math.round(waitMs / 60000)}m`);
continue;
}
if (!history || history.length === 0) continue;
for (const pred of preds) {
const eventDate = pred.event_date.slice(0, 10);
const price0 = nearestOnOrAfter(history, eventDate);
if (price0 == null) continue;
const date5 = addTradingDays(eventDate, 5);
const date10 = addTradingDays(eventDate, 10);
const price5 = nearestOnOrAfter(history, date5);
const price10 = nearestOnOrAfter(history, date10);
const r5 = price5 != null ? (price5 - price0) / price0 * 100 : null;
const r10 = price10 != null ? (price10 - price0) / price0 * 100 : null;
const correct5 = r5 == null ? null : (pred.direction === "positive" ? (r5 > 0 ? 1 : 0) : (r5 < 0 ? 1 : 0));
const correct10 = r10 == null ? null : (pred.direction === "positive" ? (r10 > 0 ? 1 : 0) : (r10 < 0 ? 1 : 0));
insertOutcome.run(
pred.id, pred.company_id, ticker, eventDate,
price0, price5, price10, r5, r10, correct5, correct10
);
evaluated++;
}
// small delay between tickers so we dont hammer yahoo
await sleep(800);
}
if (evaluated > 0) {
console.log(`[outcome] evaluated ${evaluated} predictions across ${byTicker.size} tickers`);
}
await sleep(loopDelay);
} catch (err) {
console.error("[outcome] cycle error:", err.message);
await sleep(loopDelay);
}
}
}
async function fetchYahooHistory(ticker, range) {
const url = `https://query1.finance.yahoo.com/v8/finance/chart/${encodeURIComponent(yahooSymbol(ticker))}?range=${range}&interval=1d`;
const body = await httpGet(url, { "User-Agent": "Mozilla/5.0 (compatible; duriin-intelligence/1.0)" });
const parsed = JSON.parse(body);
const result = parsed?.chart?.result?.[0];
if (!result) return null;
const ts = result.timestamp || [];
const closes = result.indicators?.quote?.[0]?.close || [];
const out = [];
for (let i = 0; i < ts.length; i++) {
if (closes[i] == null) continue;
out.push({
date: new Date(ts[i] * 1000).toISOString().slice(0, 10),
close: closes[i],
});
}
return out;
}
function nearestOnOrAfter(history, dateStr) {
for (const row of history) {
if (row.date >= dateStr) return row.close;
}
return null;
}
function addTradingDays(dateStr, n) {
const dt = new Date(dateStr);
let count = 0;
while (count < n) {
dt.setDate(dt.getDate() + 1);
const dow = dt.getDay();
if (dow >= 1 && dow <= 5) count++;
}
return dt.toISOString().slice(0, 10);
}
function httpGet(url, headers) {
return new Promise((resolve, reject) => {
const u = new URL(url);
const req = https.request({
hostname: u.hostname,
path: u.pathname + u.search,
method: "GET",
headers,
}, (res) => {
let data = "";
res.on("data", chunk => data += chunk);
res.on("end", () => {
if (res.statusCode >= 200 && res.statusCode < 300) resolve(data);
else reject(new Error(`yahoo ${res.statusCode}: ${data.slice(0, 200)}`));
});
});
req.on("error", reject);
req.end();
});
}
function sleep(ms) {
return new Promise(r => setTimeout(r, ms));
}
module.exports = { runOutcomeWorker };
workers/priceContext.js
+171
View File
@@ -0,0 +1,171 @@
const https = require("https");
// fetches daily OHLC from yahoo finance v8 chart api
// no api key needed but rate limited so we cache aggressively
async function fetchYahooHistory(ticker, range = "6mo") {
const url = `https://query1.finance.yahoo.com/v8/finance/chart/${encodeURIComponent(ticker)}?range=${range}&interval=1d`;
const body = await httpGet(url, {
"User-Agent": "Mozilla/5.0 (compatible; duriin-intelligence/1.0)",
});
let parsed;
try {
parsed = JSON.parse(body);
} catch (_) {
throw new Error(`yahoo response not JSON: ${body.slice(0, 200)}`);
}
const result = parsed?.chart?.result?.[0];
if (!result) return null;
const ts = result.timestamp || [];
const closes = result.indicators?.quote?.[0]?.close || [];
const out = [];
for (let i = 0; i < ts.length; i++) {
if (closes[i] == null) continue;
out.push({
date: new Date(ts[i] * 1000).toISOString().slice(0, 10),
close: closes[i],
});
}
return out;
}
function nearestPriceOnOrBefore(history, dateStr) {
// history is sorted ascending by date
let last = null;
for (const row of history) {
if (row.date <= dateStr) last = row;
else break;
}
return last ? last.close : null;
}
function computeStdev(values) {
if (values.length < 2) return 0;
const mean = values.reduce((a, b) => a + b, 0) / values.length;
const sq = values.reduce((acc, v) => acc + (v - mean) ** 2, 0);
return Math.sqrt(sq / (values.length - 1));
}
function computeReturns(history) {
const rets = [];
for (let i = 1; i < history.length; i++) {
const prev = history[i - 1].close;
const cur = history[i].close;
if (prev > 0) rets.push((cur - prev) / prev);
}
return rets;
}
// returns { price, price_30d_ago, price_90d_ago, vol_30d } as_of a given date
async function getPriceContext(intelligenceDb, ticker, asOfDate) {
if (!ticker || !asOfDate) return null;
// skip private/synthetic tickers — yahoo wont know them
if (/^(OPENAI|ANTHROPIC|XAI|HUAWEI|BYTEDANCE|DEEPSEEK|MISTRAL|COHERE|GROQ|SCALEAI|MCKINSEY|DELOITTE|STABILITY|INFLECTION|SPACEX|BLUEORIGIN)$/i.test(ticker)) {
return null;
}
const cacheRow = intelligenceDb.prepare(
"SELECT price, price_30d_ago, price_90d_ago, vol_30d FROM price_snapshots WHERE ticker = ? AND as_of = ?"
).get(ticker, asOfDate);
if (cacheRow) return cacheRow;
let history;
try {
history = await fetchYahooHistory(ticker, "6mo");
} catch (err) {
// dont blow up the worker on a single bad ticker
return null;
}
if (!history || history.length === 0) return null;
const price = nearestPriceOnOrBefore(history, asOfDate);
if (price == null) return null;
const date30 = new Date(asOfDate);
date30.setDate(date30.getDate() - 30);
const price30 = nearestPriceOnOrBefore(history, date30.toISOString().slice(0, 10));
const date90 = new Date(asOfDate);
date90.setDate(date90.getDate() - 90);
const price90 = nearestPriceOnOrBefore(history, date90.toISOString().slice(0, 10));
// 30-day annualized vol from daily returns
const recent = history.filter(h => h.date <= asOfDate).slice(-30);
const vol30 = computeStdev(computeReturns(recent)) * Math.sqrt(252);
const snapshot = {
price,
price_30d_ago: price30,
price_90d_ago: price90,
vol_30d: vol30,
};
try {
intelligenceDb.prepare(
"INSERT OR REPLACE INTO price_snapshots (ticker, as_of, price, price_30d_ago, price_90d_ago, vol_30d) VALUES (?, ?, ?, ?, ?, ?)"
).run(ticker, asOfDate, snapshot.price, snapshot.price_30d_ago, snapshot.price_90d_ago, snapshot.vol_30d);
} catch (_) {}
return snapshot;
}
// formats the snapshot for inclusion in the LLM prompt
function formatPriceContext(snapshot, ticker) {
if (!snapshot || snapshot.price == null) return null;
const lines = [`${ticker} price as of event: $${snapshot.price.toFixed(2)}`];
if (snapshot.price_30d_ago) {
const ret30 = (snapshot.price - snapshot.price_30d_ago) / snapshot.price_30d_ago * 100;
lines.push(`30-day return: ${ret30 >= 0 ? "+" : ""}${ret30.toFixed(1)}%`);
}
if (snapshot.price_90d_ago) {
const ret90 = (snapshot.price - snapshot.price_90d_ago) / snapshot.price_90d_ago * 100;
lines.push(`90-day return: ${ret90 >= 0 ? "+" : ""}${ret90.toFixed(1)}%`);
}
if (snapshot.vol_30d) {
lines.push(`30-day annualized volatility: ${(snapshot.vol_30d * 100).toFixed(1)}%`);
}
return lines.join("\n");
}
function httpGet(url, headers) {
return new Promise((resolve, reject) => {
const u = new URL(url);
const req = https.request({
hostname: u.hostname,
path: u.pathname + u.search,
method: "GET",
headers,
}, (res) => {
let data = "";
res.on("data", chunk => data += chunk);
res.on("end", () => {
if (res.statusCode >= 200 && res.statusCode < 300) resolve(data);
else reject(new Error(`yahoo ${res.statusCode}: ${data.slice(0, 200)}`));
});
});
req.on("error", reject);
req.end();
});
}
module.exports = { getPriceContext, formatPriceContext };
workers/queueFeeder.js
+19 -1
View File
@@ -4,6 +4,10 @@
async function runQueueFeeder(archiveDb, intelligenceDb, config) {
const batchSize = config.workers?.queueFeederBatchSize ?? 100;
const loopDelay = config.workers?.queueFeederLoopDelayMs ?? 3000;
const maxPending = Math.max(
batchSize,
config.workers?.queueFeederMaxPending ?? 250
);
const getCursor = intelligenceDb.prepare(
"SELECT value FROM cursors WHERE key = 'queue_feeder'"
@@ -16,11 +20,21 @@ async function runQueueFeeder(archiveDb, intelligenceDb, config) {
INSERT OR IGNORE INTO article_queue (article_id, status, created_at)
VALUES (?, 'pending', CURRENT_TIMESTAMP)
`);
const getPendingCount = intelligenceDb.prepare(
"SELECT COUNT(*) AS count FROM article_queue WHERE status = 'pending'"
);
while (true) {
try {
const pending = getPendingCount.get().count;
if (pending >= maxPending) {
await sleep(loopDelay);
continue;
}
const cursorRow = getCursor.get();
const cursor = cursorRow ? cursorRow.value : 0;
const availableSlots = Math.min(batchSize, maxPending - pending);
const articles = archiveDb.prepare(`
SELECT id FROM articles
@@ -31,7 +45,7 @@ async function runQueueFeeder(archiveDb, intelligenceDb, config) {
AND event_id IS NOT NULL
ORDER BY id ASC
LIMIT ?
`).all(cursor, batchSize);
`).all(cursor, availableSlots);
if (articles.length === 0) {
await sleep(loopDelay);
@@ -53,6 +67,10 @@ async function runQueueFeeder(archiveDb, intelligenceDb, config) {
console.log(`[feeder] queued ${inserted} articles, cursor now ${newCursor}`);
}
// Always yield between archive scans. The query is synchronous and can
// otherwise monopolise the event loop while catching up a large archive.
await sleep(loopDelay);
} catch (err) {
console.error("[feeder] error:", err.message);
await sleep(loopDelay);

Some files were not shown because too many files have changed in this diff Show More