Put the tone marks back into text that lost them

    Restore diacritics and tone marks on text that was typed, scraped, or keyed in without them, across 11 languages from Yorùbá and Igbo to Vietnamese, Polish and Arabic.

    Media & publishingArchives & librariesEdtechGovernment records
    0.0117
    MEAN DER, DIACNET-2.0 (10 LANGUAGES)
    1.0000
    COMPLIANCE, BY CONSTRUCTION
    11
    LANGUAGES, ONE MODEL
    / The problem

    Why this breaks today

    Almost nobody types diacritics. Phone keyboards make them slow, legacy databases strip them, and OCR of older print drops them entirely. The result is an archive where "ọkọ" (husband), "oko" (farm), and "òkò" (stone) all collapse into the same string.

    For a publisher that means unreadable copy. For an archive it means text that no longer says what the author wrote. For any search index built on top, it means queries that silently fail to match.

    Restoring the marks by hand is a proofreading job measured in person-months, and it needs speakers of every language involved. Handing it to a general-purpose generative model trades one problem for another: anything that writes the sentence out again can also reword it.

    / How it works

    The pipeline, step by step

    01

    Pick the model for your constraint

    diacnet-2.0 (582M) is the most accurate Olaverse diacritizer on 8 of 10 languages, mean DER 0.0117. diacnet-mini-2.0 (300M) keeps the same features at half the size, mean DER 0.0141. diactag-2.0 (37.9M) tags characters instead of generating text, is strongest on Igbo, Vietnamese and Modern Standard Arabic, and runs on a CPU.

    diacnet-2.0
    02

    Name the language, or let the model find it

    Pass one of the eleven language tags, or leave it out. diacnet-2.0 with <auto> stays within 0.0005 DER of the explicit tag; diactag-2.0 has a built-in language-detection head. One model covers every language, so there is nothing to route between.

    diacnet-2.0
    03

    Keep the letters, take only the marks

    diactag-2.0 copies the base characters, so its output is always your input with marks added: compliance 1.0000 by construction. diacnet-2.0 generates text, so run its output through the alignment helper on the model card (the SDK does it by default). Every input letter is kept and only the marks are taken, which makes compliance 100% too.

    diactag-2.0
    04

    Steer the ambiguous words

    Some words take different marks for different meanings. diacnet-2.0 accepts meaning hints like [g: ranh=free (time)], which lift Vietnamese ambiguous-word accuracy from 0.63 to 0.90. Marks already in the text are kept (99.9–100%) and the rest are filled in, so half-marked input is completed rather than undone.

    diacnet-2.0
    / The code

    Running in about ten lines

    The Olaverse SDK wraps the models with sane defaults. If you would rather not add a dependency, the second tab is the same pipeline in plain transformers.

    $pip install "olaverse[deeplearning]"
    restore.py
    from olaverse.nlp import Diacritizer
    
    # Most accurate: diacnet-2.0 (output is aligned by default, so only marks change)
    d = Diacritizer(model="diacnet-2.0", lang="yor", device="cuda")   # "cpu", "mps" or "auto"
    d.restore("O so fun ara re pe oun ko ni isoro kankan.")
    # → 'Ó sọ fún ara rẹ̀ pé òun kò ní ìṣòro kankan.'
    
    # Meaning hints for words whose marks depend on meaning
    d.restore("Chi ay chi that su ranh vao nhung buoi toi sau khi da cho con ngu say.",
              lang="vie", hints={"ranh": "free (time)"})
    
    # Mixed languages in one batch; "auto" when you don't know
    d.restore_batch(texts, lang=["yor", "ara", "auto"])
    
    # CPU serving: diactag-2.0 never alters letters, int8 ONNX (pip install "olaverse[onnx]")
    fast = Diacritizer(model="diactag-2.0", lang="yor", onnx=True)
    fast.restore("se eranko naa si gbo o?")    # → 'ṣé ẹranko náà sì gbọ́ ọ?'

    Language coverage

    / 11 LANGUAGES

    Every language the models in this pipeline were trained on, with the codes you pass and the codes you get back.

    Yorùbá
    yo · yor
    Igbo
    ig · ibo
    Hausa
    ha · hau
    Vietnamese
    vi · vie
    Polish
    pl · pol
    Turkish
    tr · tur
    Portuguese
    pt · por
    Spanish
    es · spa
    French
    fr · fra
    Italian
    it · ita
    Arabic
    ar · ara

    All three models cover the same eleven. Scoped deliberately to languages where diacritics are lexically meaningful. Arabic has its own page, since vowel marks work differently there.

    / This is you if
    • You hold a corpus of undiacritized text: CMS content, scanned print, user submissions, or an old database export.
    • Your users type without tone marks but expect to find correctly written text.
    • You publish in Yorùbá, Igbo, Hausa, Vietnamese, Polish, Turkish, Portuguese, Spanish, French, Italian, or Arabic.
    • You cannot ship anything that might quietly reword the text it was asked to accent: records, legal copy, or an archive of record.

    Every figure on this page comes from the published model cards. Download the weights and check us.

    Want this on your data?

    The weights are open, so you can build it yourself this afternoon. If you would rather we tuned it to your domain, evaluated it properly, and handed it over, tell us what you are working on.