rerank_evaluate

io.github.JcJamet/ia-qa-toolbox · Verify this server

Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict from ground-truth relevance labels. Three modes, all keyless except the last. (1) BYO scores — give each passage the `score` your own reranker produced (Cohere, Jina, a self-hosted NIM, a cross-encoder): deterministic, offline, and it evaluates YOUR reranker rather than someone else's. This is the mode to gate CI on. (2) Default, no scores and no key — ranks with local BM25, a lexical keyword baseline: it answers "does a keyword floor already surface my relevant passages?", never "is my neural reranker good". (3) Live NVIDIA reranker — supply `api_key` for an NVIDIA account that still has reranking entitlement; NVIDIA retired its hosted reranking endpoints on 2026-05-18, so this one is for accounts that were grandfathered in.