IMBENJI.NET/Duriin-API
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add Docker configuration and news crawler implementation

Browse Source
This commit is contained in:
ImBenji
2026-04-16 22:54:27 +01:00
parent 7724fafbdc
commit c91e4ddb60
8 changed files with 915 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -1
View File
@@ -25,7 +25,8 @@ Node.js Fastify server that ingests news articles from RSS, SEC EDGAR 8-K filing
## Notes
- SQLite archive file defaults to `./archive.sqlite`.
- Deduplication is enforced on `url` and normalized title.
- Deduplication is enforced on `url`; normalized titles are stored and indexed for matching but are not unique.
- `newsCrawler.sites` can be configured with same-site seed pages for bounded HTML crawling and historical article discovery.
- Article body extraction runs asynchronously after insertion, with hourly retries for rows still missing content.
- Main article images are stored as ultra-compressed base64 WebP.
- Embeddings are generated asynchronously with OpenRouter `perplexity/pplx-embed-v1-0.6b` and indexed in `sqlite-vec` for similarity search.