lid-neural-608
Know the language of any text, even when it is three words long.
lid-neural-608 tells you which of 608 languages a piece of text is written in. We built it African-first and trained it on the messy input real products receive: chat messages, typos, missing accents, emoji and stray URLs.
The problem it solves
Most language detectors were built for paragraphs of web text in a few hundred well-resourced languages. Hand them a short Yorùbá greeting or a Kinyarwanda search query and they guess, often confidently and often wrong.
That guess decides everything downstream: which support queue a ticket lands in, which moderation model scores a message, which language your analytics think your users speak. We wanted a detector that holds up on exactly that kind of input.
The numbers
Head to head on the same test rows
Against lid-lite-608, GlotLID v3 and Meta’s fastText LID across six test sets. It leads the held-out test (0.867), conversational Tatoeba (0.930) and, in traffic mode, real-world 1-5 word input (0.906).
Accuracy by input length
Ahead of GlotLID at every length, with the biggest lead on the shortest text: 0.666 against 0.486 on 1-5 words.
Highlights
Two modes, one model
Coverage mode treats every language as equally likely, for finding rare languages. Traffic mode leans on how often languages actually appear, for user input. You choose per call.
Strong on short text
In traffic mode it reads 1-5 word inputs at 0.906 web-weighted accuracy and everyday phrases at 0.943, the best of every model we compared.
Tells language from noise
A dedicated zxx_Zxxx class catches numbers, URLs and code, so junk never gets forced into a real language.
Free for commercial use
Fine-tuned from MIT-licensed mmBERT-small on commercially usable data, and released under Apache-2.0.
Run it in a few lines
from olaverse import LIDNeural608
lid = LIDNeural608(device="cuda") # "cpu", "mps" or "auto"
lid.predict("Habari za asubuhi") # → 'swh_Latn'
lid.predict_batch(["Mo fẹ́ lọ sí ọjà lónìí", "Bonjour mon ami"])
# → ['yor_Latn', 'fra_Latn']
# For user input, switch to traffic mode
chat = LIDNeural608(mode="traffic")
chat.predict_batch(["Good morning", "Thank you very much"])
# → ['eng_Latn', 'eng_Latn']Which LID-608 model?
Need raw speed on a CPU instead? lid-lite-608 has the same languages, labels and modes in 37 MB.
| lid-neural-608THIS PAGE | lid-lite-608 → | |
|---|---|---|
| Size | 140M parameters | 37 MB |
| Speed | ~1,100 texts/s on an A100 | ~6,800 texts/s on one CPU thread |
| Held-out accuracy | 0.867 | 0.858 |
| 1-5 words, traffic mode | 0.906 | 0.829 |
| Everyday phrases, traffic mode | 0.943 | 0.838 |
| Best for | User input and short text | Bulk filtering, CPU and edge |
Best for
- Routing chat messages, search queries and support tickets, where text is short and typed by real users.
- Products for African users that need to tell Yorùbá, Hausa, Igbo, Swahili and Pidgin apart reliably.
- Accuracy-first pipelines with a GPU, with lid-lite-608 in front as a fast first pass at scale.
- Separating real language from numbers, URLs and code before anything downstream runs.
Use cases
Language routing
Detect the language of support tickets, chat messages, queries, and documents across 608 languages, built African-first and accurate on the short text real products see.
/ Data engineeringLow-resource corpus mining
Sort billions of web documents into 608 languages on CPUs, drop the junk, and pull out the African and low-resource text that general detectors misfile.
More production models
All modelsPurpleMIST-Flash-1.0
typed-decisions 0.720 · African TD 0.668
PurpleMISTPurpleMIST-Mini-1.0
typed-decisions 0.601 · 1.9B
MIST-ExtractMIST-Extract-1.0
91.8% docs fully right · 4B
MIST-ExtractMIST-Extract-Flash-1.0
9.7 docs/s · 2B
DiacNetdiacnet-2.0
DER 0.0117 · 582M
DiacNetdiacnet-mini-2.0
DER 0.0141 · 300M
DiacNetdiactag-2.0
DER 0.0130 · 37.9M
MIST QGmist-qg-1.5b
0.955 keep-rate · 25 langs