Generate verified (query, passage) pairs from your own corpus, in 25 languages, and fine-tune retrievers and rerankers on data that matches your domain.
Off-the-shelf embedding models are trained on English web text. Point one at your domain corpus — legal filings, medical guidance, product docs, Yorùbá news — and recall drops in ways no amount of prompt engineering fixes.
The fix is fine-tuning, and fine-tuning needs (query, positive passage) pairs. Nobody has those for their own corpus, and hand-labelling them is the reason most teams never do it.
The way out is to generate the queries from the passages you already have, then verify each one by checking that it retrieves its own source back.
lid-neural-25.1 is the long-form checkpoint (99.9% accuracy on passages). Tag every document so generation happens in the right language and your eval splits stay honest.
lid-neural-25.1mist-qg-1.5b turns each passage into search-style questions in that language. This is the same pipeline that built Olaverse's own qg-passages-multi dataset — roughly 50k passages and 150k questions.
mist-qg-1.5bEmbed each generated question and check that it retrieves its own source passage out of [source + distractors]. Discard the rest. What is left is a clean pair set you can train on.
Train your embedder or cross-encoder on the surviving pairs and evaluate on a held-out slice of your own corpus — not on a public benchmark that looks nothing like your data.
The Olaverse SDK wraps the models with sane defaults. If you would rather not add a dependency, the second tab is the same pipeline in plain transformers.
from olaverse import MISTQuestionGenerator
from olaverse.nlp import LIDNeural25
from sentence_transformers import SentenceTransformer, util
lid = LIDNeural25(variant="passages") # long-form checkpoint
qg = MISTQuestionGenerator() # mist-qg-1.5b
embedder = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
lid.load(); qg.load()
def survives_round_trip(question, source, distractors):
pool = [source] + distractors
q = embedder.encode(question, convert_to_tensor=True)
p = embedder.encode(pool, convert_to_tensor=True)
return int(util.cos_sim(q, p).argmax()) == 0 # the source ranks first
pairs = []
for doc in corpus:
lang = lid.predict(doc["text"][:512])
for q in qg.generate(doc["text"], n=3, language=lang):
if survives_round_trip(q, doc["text"], sample_distractors(corpus, doc)):
pairs.append((q, doc["text"]))
# pairs → fine-tune your embedder or cross-encoder on your own domainEvery language the models in this pipeline were trained on, with the codes you pass and the codes you get back.
Both models in this pipeline cover the same 25 languages, because both were trained on olaverse/qg-passages-multi. Label with the ISO 639-3 code LID returns and pass it straight to the generator.
Every figure on this page comes from the published model cards, and every limitation is one the cards state themselves. Download the weights and check us.
Generate natural, search-style questions from any passage across 25 languages — for quizzes, reading comprehension, help-centre FAQs, and assessment banks.
/ Routing & moderationDetect the language of support tickets, chat messages, queries, and documents across 25 languages — including the African languages most detectors get wrong.
The weights are open, so you can build it yourself this afternoon. If you would rather we tuned it to your domain, evaluated it properly, and handed it over, tell us what you are working on.