feat: let jev answer the crawler's classification, and keep its confidence
The crawler asked gpt-4.1-mini one three-way question per undecided page, and then threw away the confidence it asked for in the same prompt. Jev answers that question as a typed choice for $0.042 per million in and nothing out, with a confidence that falls out of the distribution rather than the model's opinion of itself. It cannot write learnedSignals though -- rule_value is free text -- so the expensive model is still what teaches a new site. Once a site has banked enough rules and jev is sure, we stop paying for it. A dead signal call now falls back to the jev answer instead of losing the page. confidence is stored on crawler_page_classifications, additively, in both dialects. Pattern, rule and signal-model rows keep a null rather than borrowing a number that belonged to a different answer. The openrouter slug is unverified -- jev is beta there and absent from /api/v1/models -- so both the slug and the endpoint are env overridable, and a bad slug degrades to the old path instead of breaking the crawl. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZ6bEoR6m6vrksSYRiXwVD
This commit is contained in:
@@ -182,6 +182,7 @@ exec(`
|
||||
url TEXT PRIMARY KEY,
|
||||
site_name TEXT NOT NULL,
|
||||
classification TEXT NOT NULL,
|
||||
confidence DOUBLE PRECISION,
|
||||
pattern TEXT,
|
||||
classified_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
)
|
||||
@@ -223,4 +224,12 @@ exec(`
|
||||
)
|
||||
`);
|
||||
|
||||
// same story as the sqlite side: the creates above skip a database that already
|
||||
// has the table, so additions land here too. postgres does have the IF NOT EXISTS
|
||||
// form so this one stays boring. Additive only.
|
||||
exec(`
|
||||
ALTER TABLE crawler_page_classifications
|
||||
ADD COLUMN IF NOT EXISTS confidence DOUBLE PRECISION
|
||||
`);
|
||||
|
||||
module.exports = pgDb;
|
||||
Reference in New Issue
Block a user