build jev rag reranking

Retrieve First, Judge Second

Do not send an entire corpus to Jev. First-stage retrieval supplies recall and access control; Jev can only reconsider the candidates it receives. Missing documents remain missing. Preserve source identity so a later answer and evaluator can point back to the actual evidence.

Choose per-Candidate or Listwise Semantics Deliberately

The answer meaning determines the metric. Per-item judgments may support a relevance threshold; a Choice supports a winner among current candidates. Do not compare a Choice option probability with a separately asked Noul as if they were the same calibrated quantity.

Control State Length and Document Boundaries

Provide the query, one candidate or a small clearly labeled list, and only metadata needed to judge relevance. Remove navigation, repeated boilerplate and unrelated page sections. TypeSafe documents accuracy loss from irrelevant long state in Jev 1.13, which is especially relevant to retrieved documents.

Never merge passages so their citations become ambiguous. Keep document and passage IDs outside prose, enforce access control before inference, and reject stale records deterministically. See state projection for provenance and trust boundaries.

A Negative Public Result Is Useful Evidence, Not a Verdict

GoSailGlobal reported on X that one test involving 33,047 items and 164 queries did not improve its vector-retrieval baseline when Jev was added for reranking. Treat that as an attributable field report: it is a reason to benchmark, not proof that Jev never helps reranking. The post’s dataset, candidate construction, question wording, metric and released artifacts determine how far the result generalizes.

A useful reproduction publishes those details and compares the same top-k under BM25, vectors, a conventional reranker and Jev. The RAG use-case page covers groundedness as a separate task; relevance and evidence support should not be collapsed into one score.

Benchmark Ranking and the Final Answer

Freeze the corpus snapshot and relevance judgments, tune on a development set and report on held-out queries. Include the unchanged first-stage ordering as a baseline. A reranker is valuable only when the gain justifies extra inference, tail latency and operational complexity.

• Recall@k before reranking, because a reranker cannot recover absent relevant documents.
• nDCG@k, MRR or task-specific top-k judgment after reranking.
• Downstream answer correctness, citation support and refusal quality.
• Latency distribution and token cost by candidate count and passage length.
• Errors by query type, language, document length and ambiguity.

Production Controls

• Cap shortlist size and passage length; log truncation.
• Cache by query, candidate content hash, question and model version where privacy permits.
• Use deterministic stable sorting for ties and missing results.
• Fall back to the original ranking on timeout or malformed response.
• Monitor click/task outcomes without mistaking position-biased feedback for ground truth.