Retrieval Eval

BM25, dense and hybrid on a labelled query set · Evals and Retrieval

A small retrieval eval over 1,365 wine reviews. It scores BM25, a local embedding model and a reciprocal-rank-fusion hybrid on the same 55 labelled queries, with bootstrap intervals and paired differences.

View source on GitHub

limitations

Key features

BM25 wins when the query contains the label term. On paraphrases it mostly fails, and dense and hybrid do better
Hybrid improved MRR by 0.10 (interval 0.02 to 0.18) and tied BM25 on recall, so the README calls it the safe default here and not a clear winner
Query kind is computed from the labels instead of assigned by hand
The README lists how the labels favor BM25 and which queries are not independent

Impact

Shows that the right retriever depends on the query mix, and says how much of the result comes from this query set.

Tech stack

Python, sentence-transformers, pytest, GitHub Actions

interested in something like this?

Get in touch