add event_date column to event_knowledge and event_predictions tables; update related logic in admin panel and augorWorker

2026-04-23 13:13:28 +01:00
parent d972569e54
commit 29c6cdc6c3
8 changed files with 400 additions and 17 deletions
@@ -186,6 +186,56 @@ Use `domains[].policy` to diagnose why a source has high `skipped` or `failed` c
 - `normalized_title` is stored for deduplication and indexing.
 - `source` format is `<feed_type>:<label>` for GDELT and RSS (e.g. `gdelt:Bloomberg Markets`, `rss:TechCrunch`), or just the source name for other feeds (`alphavantage`, `edgar`, `finnhub`).

+## Intelligence layer
+
+A second process (`intelligence/index.js`) runs alongside the archive server and builds structured knowledge about tracked companies from ingested events.
+
+```bash
+npm run intelligence
+```
+
+In Docker it runs as a separate service (`intelligence`) sharing the same image and data volume.
+
+### How it works
+
+1. **Queue feeder** — continuously scans `archive.sqlite` for articles that have content, an embedding, and an event assignment. Inserts them into `article_queue` in `intelligence.sqlite`.
+2. **Augor worker** — pulls one pending article at a time from the queue. The article is a trigger — the unit of work is the event it belongs to. Fetches all articles in the event, matches them against tracked company embeddings via cosine similarity, calls the LLM once per matched company, writes structured knowledge and predictions, then marks all sibling articles in the event as processed.
+3. **Column migrations** — run on startup to safely add new columns to existing databases without data loss.
+
+### Output tables (`intelligence.sqlite`)
+
+| Table | Contents |
+|---|---|
+| `article_queue` | Per-article processing status (`pending` / `processed` / `skipped`) |
+| `tracked_companies` | Companies to watch, with names, tickers, and aliases |
+| `company_embeddings` | Pre-generated embeddings for each company (generated on startup via OpenRouter) |
+| `event_knowledge` | Extracted relationships, themes, and factors per event+company |
+| `event_predictions` | Forward-looking predictions (market share, stock price, competitive position) with `event_date` from the source articles |
+
+### Company matching
+
+Uses cosine similarity between company embeddings and article embeddings stored in `archive.sqlite`. A company is considered relevant to an event if any article in the event has similarity ≥ `config.intelligence.similarityThreshold` (default `0.35`). Both use the same OpenRouter embedding model (`openRouter.embeddingModel`).
+
+### LLM
+
+Uses `openRouter.llmModel` via the OpenRouter API. One call per matched company per event. Output is structured JSON — relationships, themes, factors, and predictions.
+
+### Config keys (in `config.json`)
+
+| Key | Purpose |
+|---|---|
+| `duriin_db` | Path to `archive.sqlite` (relative to config file, or absolute) |
+| `intelligence_db` | Path to `intelligence.sqlite` |
+| `openRouter.llmModel` | Chat model used for extraction |
+| `openRouter.embeddingModel` | Embedding model (shared with archive server) |
+| `intelligence.similarityThreshold` | Cosine similarity cutoff for company matching (default `0.35`) |
+| `workers.augorLoopDelayMs` | Delay between augor iterations when queue is empty (default `1500`) |
+| `workers.queueFeederBatchSize` | Articles pulled per feeder batch (default `100`) |
+
+### Admin panel
+
+The intelligence data is visible in the admin panel (`/admin`) under the **Intelligence** tab. Predictions can be sorted by event publication date. A **SQL** tab allows raw queries against either database.
+
 ## Notes

 - SQLite archive defaults to `./archive.sqlite`.