Detect the language of support tickets, chat messages, queries, and documents across 608 languages, built African-first and accurate on the short text real products see.
Every multilingual pipeline starts with the same question: what language is this? Route on the wrong answer and the ticket lands with an agent who cannot read it, the moderation classifier scores gibberish, and the analytics dashboard reports a language mix that never existed.
Off-the-shelf detectors are trained on web-scale text where Yorùbá, Igbo, and Hausa barely register. They are also tuned for paragraphs, so they degrade exactly where real products live: a six-word chat message or a three-word search query.
Both models ship two modes. Traffic mode adjusts scores by each language’s real-world frequency, so short or ambiguous input leans toward the languages people actually write in. lid-neural-608 reaches 0.960 web-weighted accuracy, 0.906 on 1-5 word inputs and 0.943 on everyday phrases in this mode, ahead of GlotLID and Meta’s fastText LID.
lid-neural-608lid-lite-608 is a 37 MB fastText model with the same labels and modes, running about 6,800 texts per second on one CPU thread. Use it as a first pass and send only short or uncertain inputs to lid-neural-608.
lid-lite-608Labels are ISO 639-3 plus script, such as yor_Latn or srp_Cyrl. Send the label to your queue or language-specific model, hold low-confidence predictions for review, and drop zxx_Zxxx, the noise class for numbers, URLs and code.
The Olaverse SDK wraps the models with sane defaults. If you would rather not add a dependency, the second tab is the same pipeline in plain transformers.
from olaverse import LIDLite608, LIDNeural608
fast = LIDLite608(mode="traffic") # 37 MB, CPU
accurate = LIDNeural608(mode="traffic") # 140M, GPU or CPU
def detect(message):
probs = fast.predict_proba(message, top_k=1)
lang, score = next(iter(probs.items()))
# Short or uncertain input goes to the neural model
if score < 0.9 or len(message.split()) <= 5:
lang, score = max(accurate.predict_proba_batch([message])[0].items(), key=lambda kv: kv[1])
return lang, score
def route(message):
lang, score = detect(message)
if lang == "zxx_Zxxx": # numbers, URLs, code
return "no-language"
return LANGUAGE_QUEUES.get(lang, "human-review") if score > 0.9 else "human-review"
route("Ẹ kú àárọ̀, ṣé dáadáa ni?") # yor_Latn → the Yorùbá queueEvery figure on this page comes from the published model cards. Download the weights and check us.
Sort billions of web documents into 608 languages on CPUs, drop the junk, and pull out the African and low-resource text that general detectors misfile.
/ ML engineeringGenerate verified (query, passage) pairs from your own corpus in 25 languages, and fine-tune retrievers and rerankers on data that matches your domain.
The weights are open, so you can build it yourself this afternoon. If you would rather we tuned it to your domain, evaluated it properly, and handed it over, tell us what you are working on.