fix: stop three loops that retry forever, and let budget dead letters recover

The event outcome worker re-requested PSTG and GROQ on every poll for as long as
the process lived, because a fetch failure only logged and continued while the
prediction stayed pending. Ten requests every five minutes, indefinitely. Per
ticker backoff now doubles to an hour, so a symbol with no market data costs one
request an hour instead of one a minute. It also translates dotted tickers the
same way the autonomy worker does, which is why that helper moved into the
shared price module rather than being copied.

The gdelt loop had no pause on its error path at all, so once the api started
refusing connections it spun through failures continuously, burning cpu and
filling the log with the same stack. It has been doing that for days. Backs off
to half an hour now and resets on success.

isTransientCoordinatorFailure matched 408, 429 and 5xx but not a budget 402/403,
so the 380 jobs that dead-lettered during the exhausted quota window could never
come back on their own, including 55 live events. Budget failures are transient
in a way an ordinary auth failure is not, and a wrong key still dies permanently
because it says invalid or unauthorized rather than naming credits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb
This commit is contained in:
ImBenji
2026-09-03 16:42:28 +01:00
co-authored by Claude Opus 5
parent 859d0719b3
commit ca69ad0e73
6 changed files with 67 additions and 14 deletions
+20 -2
View File
@@ -2,8 +2,17 @@
// runs continuously, batching by ticker so we hit yahoo once per company per cycle
const { getPriceContext } = require("./priceContext");
const { yahooSymbol } = require("../src/autonomy/outcomes");
const https = require("https");
// PSTG and GROQ were re-requested every single poll, forever, because a fetch
// failure only logged and moved on. Predictions for a ticker with no market data
// never leave the pending set, so the loop retries them for as long as the process
// lives. Back off per ticker instead, doubling up to an hour, so a dead symbol
// costs one request an hour rather than one a minute.
const TICKER_BACKOFF_START_MS = 5 * 60 * 1000;
const TICKER_BACKOFF_MAX_MS = 60 * 60 * 1000;
async function runOutcomeWorker(archiveDb, intelligenceDb, config) {
const loopDelay = config.workers?.outcomeLoopDelayMs ?? 60000;
@@ -31,6 +40,8 @@ async function runOutcomeWorker(archiveDb, intelligenceDb, config) {
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
const tickerBackoff = new Map();
while (true) {
try {
const pending = getPending.all();
@@ -49,11 +60,18 @@ async function runOutcomeWorker(archiveDb, intelligenceDb, config) {
let evaluated = 0;
for (const [ticker, preds] of byTicker.entries()) {
const cooling = tickerBackoff.get(ticker);
if (cooling && Date.now() < cooling.until) continue;
let history;
try {
history = await fetchYahooHistory(ticker, "1y");
tickerBackoff.delete(ticker);
} catch (err) {
console.error(`[outcome] yahoo error for ${ticker}: ${err.message}`);
const previous = cooling ? cooling.waitMs : 0;
const waitMs = Math.min(TICKER_BACKOFF_MAX_MS, previous ? previous * 2 : TICKER_BACKOFF_START_MS);
tickerBackoff.set(ticker, { until: Date.now() + waitMs, waitMs });
console.error(`[outcome] yahoo error for ${ticker}: ${err.message} — backing off ${Math.round(waitMs / 60000)}m`);
continue;
}
@@ -101,7 +119,7 @@ async function runOutcomeWorker(archiveDb, intelligenceDb, config) {
async function fetchYahooHistory(ticker, range) {
const url = `https://query1.finance.yahoo.com/v8/finance/chart/${encodeURIComponent(ticker)}?range=${range}&interval=1d`;
const url = `https://query1.finance.yahoo.com/v8/finance/chart/${encodeURIComponent(yahooSymbol(ticker))}?range=${range}&interval=1d`;
const body = await httpGet(url, { "User-Agent": "Mozilla/5.0 (compatible; duriin-intelligence/1.0)" });
const parsed = JSON.parse(body);