refine article filtering to ensure only usable articles are returned

README.md

@@ -50,7 +50,7 @@ Use this to confirm the server is running, not to inspect ingestion state.
 
 ### `GET /articles`
 
-Returns articles from the `articles` table. Behavior changes based on the query params you send.
+Returns articles from the `articles` table. Only articles that are considered **usable** are exposed — meaning they have non-empty `content`, a stored embedding, and are not index/category pages. Behavior changes based on the query params you send.
 
 #### Query params
 
@@ -237,6 +237,7 @@ Returns one article by numeric ID.
 **Behavior**
 
 - Looks up the article directly in SQLite.
+- Same usability filter as the list endpoint — returns `404` if the article exists but is not usable.
 - Returns the same article fields as normal `/articles` list mode.
 - Does not return embedding data.
 - Returns `404` if the ID does not exist.
@@ -257,16 +258,10 @@ Returns ingestion and archive summary information.
 
 **Response fields**
 
-- `totalArticles`: total number of rows in `articles`
-- `countsBySource`: article counts grouped by source name
+- `total`: total number of rows in `articles` across all sources
+- `usable`: articles that have content, an embedding, and are not index pages
 - `lastIngestionBySource`: in-memory timestamps of the last successful batch run per source
-- `contentFetchCoverage.total`: total article count used for coverage math
-- `contentFetchCoverage.withContent`: rows whose `content` is present and non-empty
-- `contentFetchCoverage.withImage`: rows whose `image` is present and non-empty
-- `contentFetchCoverage.withEmbedding`: rows that have an embedding in `article_embeddings`
-- `contentFetchCoverage.contentRatio`: `withContent / total`
-- `contentFetchCoverage.imageRatio`: `withImage / total`
-- `contentFetchCoverage.embeddingRatio`: `withEmbedding / total`
+- `bySource`: per-source breakdown, each with `total` and `usable` counts
 
 **Important detail**
 
@@ -275,26 +270,18 @@ Returns ingestion and archive summary information.
 **Example response**
 ```json
 {
-  "totalArticles": 10234,
-  "countsBySource": {
-    "alphavantage": 120,
-    "edgar": 88,
-    "finnhub": 400,
-    "gdelt": 2100,
-    "rss": 7526
-  },
+  "total": 10234,
+  "usable": 8700,
   "lastIngestionBySource": {
     "rss": "2025-01-02T10:00:00.000Z",
     "gdelt": "2025-01-02T10:05:00.000Z"
   },
-  "contentFetchCoverage": {
-    "withContent": 9000,
-    "withImage": 6500,
-    "withEmbedding": 8700,
-    "total": 10234,
-    "contentRatio": 0.8794,
-    "imageRatio": 0.6351,
-    "embeddingRatio": 0.8501
+  "bySource": {
+    "alphavantage": { "total": 120, "usable": 98 },
+    "edgar": { "total": 88, "usable": 70 },
+    "finnhub": { "total": 400, "usable": 360 },
+    "gdelt": { "total": 2100, "usable": 1800 },
+    "rss": { "total": 7526, "usable": 6372 }
   }
 }
 ```