Generate verified (query, passage) pairs from your own corpus in 25 languages, and fine-tune retrievers and rerankers on data that matches your domain.
Off-the-shelf embedding models are trained on English web text. Point one at your domain corpus (legal filings, medical guidance, product docs, Yorùbá news) and recall drops in ways no amount of prompt engineering fixes.
The fix is fine-tuning, and fine-tuning needs (query, positive passage) pairs. Nobody has those for their own corpus, and hand-labelling them is the reason most teams never do it.
lid-lite-608 tags every document on CPU. Map its label (yor_Latn) to the language code (yor) and keep only the 25 languages the generator covers, so generation happens in the right language and your eval splits stay honest.
lid-lite-608mist-qg-1.5b turns each passage into search-style questions in that language. This is the pipeline that built Olaverse's own qg-passages-multi dataset: roughly 50k passages and 150k questions.
mist-qg-1.5bEmbed each generated question and check that it retrieves its own source passage out of [source + distractors]. Discard the rest. What is left is a clean pair set you can train on.
Train your embedder or cross-encoder on the surviving pairs and evaluate on a held-out slice of your own corpus, not on a public benchmark that looks nothing like your data.
The Olaverse SDK wraps the models with sane defaults. If you would rather not add a dependency, the second tab is the same pipeline in plain transformers.
from olaverse import LIDLite608, MISTQuestionGenerator
from sentence_transformers import SentenceTransformer, util
lid = LIDLite608() # coverage mode, for corpus labelling
qg = MISTQuestionGenerator() # mist-qg-1.5b, 25 languages
qg.load()
embedder = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
QG_LANGS = {"afr", "amh", "deu", "eng", "fra", "hau", "hin", "ibo", "ind", "ita", "jpn", "kor", "nld",
"pol", "por", "rus", "sna", "som", "spa", "swh", "tur", "vie", "xho", "yor", "zul"}
def survives_round_trip(question, source, distractors):
pool = [source] + distractors
q = embedder.encode(question, convert_to_tensor=True)
p = embedder.encode(pool, convert_to_tensor=True)
return int(util.cos_sim(q, p).argmax()) == 0 # the source ranks first
pairs = []
for doc in corpus:
lang = lid.predict(doc["text"]).split("_")[0] # 'yor_Latn' → 'yor'
if lang not in QG_LANGS:
continue
for q in qg.generate(doc["text"], n=3, language=lang):
if survives_round_trip(q, doc["text"], sample_distractors(corpus, doc)):
pairs.append((q, doc["text"]))
# pairs → fine-tune your embedder or cross-encoder on your own domainEvery language the models in this pipeline were trained on, with the codes you pass and the codes you get back.
Generation covers these 25. Language ID covers 608, so documents in any other language are labelled and skipped rather than sent to the generator.
Every figure on this page comes from the published model cards. Download the weights and check us.
Generate natural, search-style questions from any passage across 25 languages, for quizzes, reading comprehension, help-centre FAQs, and assessment banks.
/ Data engineeringSort billions of web documents into 608 languages on CPUs, drop the junk, and pull out the African and low-resource text that general detectors misfile.
The weights are open, so you can build it yourself this afternoon. If you would rather we tuned it to your domain, evaluated it properly, and handed it over, tell us what you are working on.