Restore the marks your documents lost, then match on an accent-folded key, so a user typing "oko" finds "ọkọ" and still sees it written properly.
A user searches "oko" and gets nothing, because the article says "ọkọ". A marketplace listing typed with tone marks never surfaces for the buyer who typed without them. Neither side did anything wrong: the index and the query are in different orthographies.
Accent-folding both sides fixes the matching in one line of Unicode, and plenty of teams stop there. But a corpus that was scraped, OCR’d, or user-typed without marks is still broken everywhere it is displayed, and folding just hides that.
So do both, and keep them separate: restore the marks so the text reads correctly, and fold a copy so it matches however anyone types.
lid-lite-608 labels documents at about 6,800 texts per second on one CPU thread, so a whole archive can be bucketed before anything else runs. Where you already know a source’s language, set it per source instead.
lid-lite-608Run diactag-2.0 over text missing its marks: scraped pages, OCR output, user submissions. It copies the base letters and only predicts marks, so a stray pass can at worst re-decide an accent; it cannot rewrite the sentence. Marks an editor already typed are kept by default.
diactag-2.0Fold the restored text down to unmarked characters and store that as the search field, keeping the restored text for display. This is pure Unicode (NFD, drop the combining marks), so no model sits on your indexing path beyond the restore.
One function, both sides. A query typed with marks and one typed without collapse to the same key, and no model runs on the query path at all, so search latency is untouched.
The Olaverse SDK wraps the models with sane defaults. If you would rather not add a dependency, the second tab is the same pipeline in plain transformers.
import unicodedata
from olaverse.nlp import Diacritizer
# Your corpus language is something you usually know; omit lang= to let the model detect it
d = Diacritizer(model="diactag-2.0", lang="yor", onnx=True)
def fold(s: str) -> str:
"""Accent-fold to a match key. Pure Unicode, no model."""
return "".join(c for c in unicodedata.normalize("NFD", s)
if unicodedata.category(c) != "Mn").lower()
def index_document(doc):
# Restore only what's missing its marks
text = doc["text"] if doc["has_diacritics"] else d.restore(doc["text"])
return {**doc, "display_text": text, "match_text": fold(text)}
corpus = [
{"id": 1, "text": "Ṣé ẹranko náà sì gbọ́ ọ?", "has_diacritics": True}, # clean CMS copy
{"id": 2, "text": "se eranko naa si gbo o?", "has_diacritics": False}, # scraped / OCR
]
indexed = [index_document(c) for c in corpus]
# both share one match_text, and doc 2 now displays as 'ṣé ẹranko náà sì gbọ́ ọ?'
# The query path is folding only: no model, no added latency
key = fold("SE ERANKO NAA")
[c["id"] for c in indexed if key in c["match_text"]] # → [1, 2]Every language the models in this pipeline were trained on, with the codes you pass and the codes you get back.
Restoration covers these eleven. Anything else still benefits from the folding half of the pipeline: it is indexed as typed, with no restoration pass.
Every figure on this page comes from the published model cards. Download the weights and check us.
Restore diacritics and tone marks on text that was typed, scraped, or keyed in without them, across 11 languages from Yorùbá and Igbo to Vietnamese, Polish and Arabic.
/ Text quality · ArabicRestore Arabic tashkīl, with or without case endings, for text-to-speech, teaching material and readable copy, from models that also cover ten other languages.
The weights are open, so you can build it yourself this afternoon. If you would rather we tuned it to your domain, evaluated it properly, and handed it over, tell us what you are working on.