fix: a wedged browser can no longer stop content fetching silently

Content fetching stopped dead on 4 Sept at 17:06 and nobody noticed for three
days. In that window it managed about 328 articles and then produced nothing:
no error, no timeout, not a single line in the log. Meanwhile the live lane
starved, because an article needs content before it can be embedded, clustered
and handed to the coordinator, and 6596 articles arrived in 48 hours with zero
of them ready.

Every individual browser step already had a timeout. Acquiring the shared
session did not, and it is awaited while holding one of eight browser slots, so
a wedged chromium parks every slot permanently and nothing ever throws. The page
slot timeout added earlier never fired because it sits downstream of the thing
that was actually stuck. There is now an outer bound around the whole browser
path so the slot always comes back.

The workers also log the start and end of each round. The reason this took days
to find is that a healthy content worker and a completely wedged one looked
identical from outside, and that is worth fixing on its own.

Verified on the box first: outbound fetches return 200, the picker returns rows
in 4.8s, and fetchAndStoreContent stores a real article in 354ms. Every part
worked in isolation, which is what made the silence so misleading.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnNxwxfXSbeNtjvtz5gayb

src/content.js

@@ -182,17 +182,39 @@ async function fetchPlainHtml(url) {
 }
 
 
+// Every individual step below has its own timeout, but acquiring the shared
+// session does not, and it is awaited while holding a browser slot. A wedged
+// chromium therefore parks all eight slots forever, and because nothing throws
+// there is not a single line in the log to say so: content fetching simply stops.
+// That is exactly what happened after the 4 Sept restart, ~328 articles in.
+// This is the outer bound that guarantees the slot always comes back.
+const BROWSER_HARD_TIMEOUT = 90000;
+
 async function fetchBrowserHtml(url) {
   await browserSemaphore.acquire();
   try {
     const maxConcurrentPages = Number(config.browser?.maxConcurrentPages) || 8;
+    let timer;
+    const expired = new Promise((_, reject) => {
+      timer = setTimeout(() => reject(new Error(
+        `browser fetch exceeded ${BROWSER_HARD_TIMEOUT}ms for ${url}, the session is probably wedged`)),
+        BROWSER_HARD_TIMEOUT);
+    });
+    try {
+      return await Promise.race([
+        (async () => {
           const session = await getSharedBrowserSession({
             requestTimeout: BROWSER_FETCH_TIMEOUT,
             maxConcurrentPages,
           });
-
           const html = await session.fetchRenderedHtml(url, { timeout: BROWSER_FETCH_TIMEOUT });
           return { html, finalUrl: url };
+        })(),
+        expired,
+      ]);
+    } finally {
+      clearTimeout(timer);
+    }
   } finally {
     browserSemaphore.release();
   }

src/scheduler.js

@@ -129,7 +129,14 @@ function startScheduler() {
       try {
         const perSource = Number(config.contentBackfill?.perSource) || 50;
         const batchSize = Number(config.contentBackfill?.batchSize) || 25;
+        // A round is long and silent. When content fetching wedged on 4 Sept there
+        // was not one line anywhere saying so, which is why it went unnoticed for
+        // three days while the live lane starved for want of enriched articles.
+        const startedAt = Date.now();
+        console.log(`[content] worker ${workerIndex} starting a round`);
         const processed = await runBackfillWorker({ workerIndex, workerCount, perSource, batchSize });
+        console.log(`[content] worker ${workerIndex} finished ${processed} articles`
+          + ` in ${Math.round((Date.now() - startedAt) / 1000)}s`);
 
         // if a worker found nothing in its partition, brief sleep so we dont
         // hammer the db with empty selects