Detect the language of support tickets, chat messages, queries, and documents across 25 languages — including the African languages most detectors get wrong.
Every multilingual pipeline starts with the same question: what language is this? Route on the wrong answer and the ticket lands with an agent who cannot read it, the moderation classifier scores gibberish, and the analytics dashboard reports a language mix that never existed.
Off-the-shelf detectors are trained on web-scale text where Yorùbá, Igbo, and Hausa barely register. They are also tuned for paragraphs, so they degrade exactly where real products live: a six-word chat message or a three-word search query.
lid-neural-25.2 is trained on short text — queries, chat messages, form fields. lid-neural-25.1 is trained on paragraph-length text — documents, articles, support emails. Same architecture, different training distribution, one variant= argument apart.
lid-neural-25.2Both checkpoints are XLM-RoBERTa sequence classifiers at 125M parameters. They load through one SDK class (or the plain transformers text-classification pipeline) and run on CPU in production.
lid-neural-25.1Send the ISO 639-3 label downstream to your queue, translation layer, or language-specific model — and hold low-confidence predictions for review rather than routing them wrong.
The Olaverse SDK wraps the models with sane defaults. If you would rather not add a dependency, the second tab is the same pipeline in plain transformers.
from olaverse.nlp import LIDNeural25
lid = LIDNeural25(variant="questions") # "passages" for long-form text
lid.load()
lid.predict("Kedu ka i mere?") # → 'ibo'
lid.predict_proba("Kedu ka i mere?") # → {'ibo': 0.998, 'yor': 0.001, ...}
# Route on the label, gate on the confidence
def route(message):
scores = lid.predict_proba(message)
lang, score = max(scores.items(), key=lambda kv: kv[1])
return LANGUAGE_QUEUES.get(lang, "human-review") if score > 0.9 else "human-review"Every language the models in this pipeline were trained on, with the codes you pass and the codes you get back.
*The models return ISO 639-3. Xhosa and Zulu are marked because they are confused with each other on short text (F1 ~0.79 each, against ≥0.98 everywhere else) — see the limitations below.
Every figure on this page comes from the published model cards, and every limitation is one the cards state themselves. Download the weights and check us.
Restore the marks your documents lost, then match on an accent-folded key — so a user typing "oko" finds "ọkọ", and still sees it written properly.
/ Text qualityRestore diacritics and tone marks on text that was typed, scraped, or keyed in without them — across 10 languages, from Yorùbá and Igbo to Vietnamese and Polish.
The weights are open, so you can build it yourself this afternoon. If you would rather we tuned it to your domain, evaluated it properly, and handed it over, tell us what you are working on.