lid-lite-608
608 languages in 37 MB, on a single CPU thread.
lid-lite-608 is the fast member of the LID-608 line. It knows the same 608 languages and returns the same labels as lid-neural-608, but it is a small fastText model that runs anywhere, with no GPU and no deep-learning stack.
The problem it solves
Language identification often has to run on everything: every page of a web crawl, every message through a gateway, every document in an archive. At that volume a GPU model is too slow and too expensive.
The existing fast detectors are either far larger or know far fewer languages, and they misfile low-resource African languages into bigger neighbours. lid-lite-608 is small enough to run everywhere and wide enough to find the languages others lose.
The numbers
20 African languages
Higher F1 than GlotLID on 18 of 20 major African languages, averaging 0.839 against 0.757. The biggest jumps are Kinyarwanda, Xhosa, Wolof and Lingala.
Against detectors many times its size
At 37 MB it beats GlotLID and Meta’s fastText LID on the held-out test (0.858), 1-5 word input (0.672), Tatoeba (0.922) and UDHR-LID (0.910).
Highlights
Tiny and fast
About 3× the languages of Meta’s fastText LID at 1/32 of its size, and 1/45 the size of GlotLID.
Better on African languages
Average F1 0.839 across 20 major African languages against 0.757 for GlotLID, with big gains on Kinyarwanda, Xhosa, Wolof and Lingala.
Same modes as the neural model
Coverage mode for corpus building, traffic mode for user input. Swap to lid-neural-608 without changing any labels.
Filters junk
The noise class for numbers, URLs and code scores F1 0.961, so it doubles as a cleaning pass on raw crawls.
Run it in a few lines
from olaverse import LIDLite608
lid = LIDLite608() # coverage mode (default)
lid.predict("Ẹ kú àárọ̀, ṣé dáadáa ni?") # → 'yor_Latn'
lid.predict_proba("Habari za asubuhi", top_k=3)
# → {'swh_Latn': 0.916, 'swc_Latn': 0.084, 'hau_Latn': 0.0}
# Traffic mode for user-facing input
LIDLite608(mode="traffic").predict("Bonjour mon ami") # → 'fra_Latn'Which LID-608 model?
Need the best accuracy on short text? lid-neural-608 shares the labels and modes, so you can send it only the hard cases.
| lid-neural-608 → | lid-lite-608THIS PAGE | |
|---|---|---|
| Size | 140M parameters | 37 MB |
| Speed | ~1,100 texts/s on an A100 | ~6,800 texts/s on one CPU thread |
| Held-out accuracy | 0.867 | 0.858 |
| 1-5 words, traffic mode | 0.906 | 0.829 |
| Everyday phrases, traffic mode | 0.943 | 0.838 |
| Best for | User input and short text | Bulk filtering, CPU and edge |
Best for
- Filtering web crawls and building language corpora at scale on ordinary CPUs.
- Edge and serverless deployments, where a 37 MB model with no deep-learning stack matters.
- Finding low-resource African languages that other detectors misfile.
- A fast first pass in front of lid-neural-608, sending it only the hard cases.
Use cases
Language routing
Detect the language of support tickets, chat messages, queries, and documents across 608 languages, built African-first and accurate on the short text real products see.
/ Search & retrievalSearch that matches
Restore the marks your documents lost, then match on an accent-folded key, so a user typing "oko" finds "ọkọ" and still sees it written properly.
/ Data engineeringLow-resource corpus mining
Sort billions of web documents into 608 languages on CPUs, drop the junk, and pull out the African and low-resource text that general detectors misfile.
/ ML engineeringRetrieval training data
Generate verified (query, passage) pairs from your own corpus in 25 languages, and fine-tune retrievers and rerankers on data that matches your domain.
More production models
All modelsPurpleMIST-Flash-1.0
typed-decisions 0.720 · African TD 0.668
PurpleMISTPurpleMIST-Mini-1.0
typed-decisions 0.601 · 1.9B
MIST-ExtractMIST-Extract-1.0
91.8% docs fully right · 4B
MIST-ExtractMIST-Extract-Flash-1.0
9.7 docs/s · 2B
DiacNetdiacnet-2.0
DER 0.0117 · 582M
DiacNetdiacnet-mini-2.0
DER 0.0141 · 300M
DiacNetdiactag-2.0
DER 0.0130 · 37.9M
MIST QGmist-qg-1.5b
0.955 keep-rate · 25 langs