migrate database from SQLite to PostgreSQL; add PostgreSQL adapter and migration script

This commit is contained in:
ImBenji
2026-04-27 14:33:46 +01:00
parent 653a58b3d8
commit b4df02da0d
13 changed files with 1978 additions and 6 deletions
+9 -3
View File
@@ -1,6 +1,6 @@
# duriin_api
Node.js Fastify server that ingests news articles from RSS, SEC EDGAR 8-K filings, Alpha Vantage News Sentiment, Finnhub company news, and GDELT into a local SQLite archive.
Node.js Fastify server that ingests news articles from RSS, GDELT, SEC EDGAR 8-K filings, Alpha Vantage News Sentiment, Finnhub company news, and Google News into a local SQLite archive.
## Setup
@@ -202,7 +202,8 @@ In Docker it runs as a separate service (`intelligence`) sharing the same image
2. **Augor worker** — pulls one pending article at a time from the queue. The article is a trigger — the unit of work is the event it belongs to. Fetches all articles in the event, matches them against tracked company embeddings via cosine similarity, calls the LLM once per matched company, writes structured knowledge and predictions, then marks all sibling articles in the event as processed.
3. **Consolidation worker** — slow loop (default 60s). For each tracked company, reads all `event_knowledge` rows, builds a flat list of claims, and calls the LLM to normalize and deduplicate them into canonical grouped facts stored in `company_facts`. Preserves `first_seen_at` across cycles. Prunes facts that have only been confirmed once and haven't been seen in 90 days.
4. **Graph worker** — slow loop (default 90s). Reads all `company_facts` rows of type `relationship`, parses the claim format, resolves whether the target entity is a tracked company, and upserts edges into `company_relationships`. Inserts reciprocal edges automatically (supplier ↔ customer, etc.) when both endpoints are tracked.
5. **Column migrations** — run on startup to safely add new columns to existing databases without data loss.
5. **Signal worker** — slow loop (default 120s). Picks the tracked company with the oldest (or missing) signal that has at least 3 recent predictions within a 90-day window. Calls the LLM with the company's facts, predictions, and relationships to produce a structured trade signal (`buy` / `sell` / `hold` / `hold_monitor`) with confidence, timeframe, risk level, risk factors, and key drivers.
6. **Column migrations** — run on startup to safely add new columns to existing databases without data loss.
### Output tables (`intelligence.sqlite`)
@@ -215,6 +216,9 @@ In Docker it runs as a separate service (`intelligence`) sharing the same image
| `event_predictions` | Forward-looking predictions (market share, stock price, competitive position) with `event_date` from the source articles |
| `company_facts` | Deduplicated, canonical facts per company accumulated across all events. Each fact has a `confirmation_count` and a confidence tier (`low` / `medium` / `high` / `very_high`) |
| `company_relationships` | Cross-company relationship graph derived from `company_facts`. Includes reciprocal edges and `confirmation_count` |
| `trade_signals` | Generated investment signals per company — signal type, confidence, timeframe, risk level, risk factors, summary, and key drivers |
| `worker_events` | Lifecycle timestamps for each worker iteration. Pruned hourly to stay bounded. Used to compute per-worker processing rates in the admin panel |
| `cursors` | Key-value state store. Currently used by the queue feeder to persist its last-processed article ID across restarts |
### Company matching
@@ -237,6 +241,7 @@ Uses `openRouter.llmModel` via the OpenRouter API. One call per matched company
| `workers.queueFeederBatchSize` | Articles pulled per feeder batch (default `100`) |
| `workers.consolidationLoopDelayMs` | Delay between consolidation cycles (default `60000`) |
| `workers.graphWorkerLoopDelayMs` | Delay between graph worker cycles (default `90000`) |
| `workers.signalLoopDelayMs` | Delay between signal generation cycles (default `120000`) |
### Admin panel
@@ -244,7 +249,8 @@ The intelligence data is visible in the admin panel (`/admin`) under the **Intel
- **Knowledge** — raw extracted relationships, themes, and factors per event+company. Filterable by company and type, sortable by ingestion order or event date.
- **Predictions** — forward-looking LLM predictions per event+company, with direction, magnitude, timeframe, and rationale.
- **Graph** — interactive D3 force-directed network diagram of the cross-company relationship graph. Nodes are draggable and zoomable. Edge thickness reflects confirmation count. Hover an edge to see the relationship type and count. Click a tracked company node to see its top facts in a sidebar. Toggle untracked entities on/off with the checkbox in the legend. Use the Expand button to fill the full viewport.
- **Signals** — generated trade signals per company showing signal type (`buy` / `sell` / `hold` / `hold_monitor`), confidence, timeframe, risk level, risk factors, key drivers, and a generated-at timestamp.
- **Graph** — interactive D3 force-directed network diagram of the cross-company relationship graph. Nodes are draggable and zoomable. Multiple relationship types between the same pair of companies are merged into a single edge with a combined label and max confirmation count. Edge thickness reflects confirmation count. Hover an edge to see the relationship type and count. Click a tracked company node to see its top facts in a sidebar. Toggle untracked entities on/off with the checkbox in the legend. Use the Expand button to fill the full viewport.
A **SQL** tab allows raw queries against either database. Multiple statements separated by `;` are supported — each runs independently and results render as separate blocks.