A diacritic model that cannot corrupt your text
Diacritic restoration is easy to state. Given se eranko naa si gbo o, produce ṣé ẹranko náà sì gbọ́ ọ. Every character keeps its identity; some acquire marks.
The standard approach treats it as translation. Bare text in, marked text out, generated token by token. That is what diacnet did, ByT5 fine-tuned on 7 million sentences.
It works, mostly. But “mostly” hides something. On Hausa, only 94.7% of diacnet-1.1 outputs still stripped back to their input. The other 5.3% weren’t mis-accented, they were different text. Words dropped. Clauses rewritten. Punctuation invented. A model asked to add accents had quietly decided to edit.
That is not a tuning problem. A generative decoder can emit any token at any position, so nothing in the architecture prevents it. You can penalise it in training and it gets rarer, but it never becomes impossible, and “rare” is not a property you can put in a contract.
We wanted one where it is impossible.
Reframing: classify, don’t generate
The task has a hard structural constraint that generation throws away. The output has exactly as many characters as the input, in the same order, with the same base letters. Only the marks change.
So don’t generate. Classify. For each character, predict which diacritics it carries, and copy the base character through.
input s e r a n k o
│ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼
╔═══════════════════════════════════════════════════╗
║ character transformer encoder ║
╚═══════════════════════════════════════════════════╝
│ │ │ │ │ │ │
SHAPE DOT · · · · · ·
TONE · · · ACUTE · · ·
│ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼
output ṣ e r á n k o
└──────┴──────┴──────┴──────┴──────┴──────┘
base letters COPIED, never predicted
The base character is not an output of the network. There is no mechanism by which the model can change it, so strip(output) == strip(input) holds by construction, for a trained model, an untrained one, or the int8 export.
We test exactly that: the invariant suite runs against a randomly initialised model. If the guarantee only held for a good model, it would not be a guarantee.
The model
Twelve layers, 512 hidden, 37.6M parameters, against 580M for the seq2seq model it replaces.
graphemes [B,T] language id [B]
│ │
▼ ▼
┌─────────────┐ ┌─────────────┐
│ char emb │ │ lang emb │ 11th slot =
│ 403×512 │ │ 11×512 │ "unknown language"
└──────┬──────┘ └──────┬──────┘
│ │
└──────────────► + ◄───────────┘ additive: no <yor> token,
│ no position spent
▼
╔═══════════════════════════════════════╗
║ × 12 encoder block ║
║ ┌─────────────────────────────────┐ ║ bidirectional, ║ │ RMSNorm → MHA(8 heads, RoPE) │ ║ a diacritic depends on
║ │ + residual │ ║ what follows as much
║ │ RMSNorm → SwiGLU 512→1344→512 │ ║ as what precedes
║ │ + residual │ ║
║ └─────────────────────────────────┘ ║
╚═══════════════════╤═══════════════════╝
│ H [B, T, 512]
┌──────────┬────────┴────────┬───────────┐
▼ ▼ ▼ ▼
┌─────────┐┌─────────┐ ┌──────────┐┌──────────┐
│ SHAPE ││ TONE │ │ MLM ││ LID │
│ 512→15 ││ 512→8 │ │ 512→403 ││ 512→10 │
│per char ││per char │ │ tied emb ││mean-pool │
└────┬────┘└────┬────┘ └────┬─────┘└────┬─────┘
│ │ │ │
└────┬─────┘ training only language
▼
legality mask → argmax → compose(base, shape, tone)
Where the parameters go:
| component | shape | params |
|---|---|---|
| character embedding | 403 × 512 | 206k |
| language embedding | 11 × 512 | 6k |
| 12 × encoder block | attn 4·512² + FFN 3·512·1344 | 37.4M |
| SHAPE head | 512 × 15 | 7.7k |
| TONE head | 512 × 8 | 4.1k |
| LID head | 512 × 10 | 5.1k |
| MLM head | tied to embedding | 0 |
The task heads together are 12k parameters, 0.03% of the model. Nearly everything is shared encoder, which is why ten languages cost roughly the same as one.
loss = 1.0·CE(shape) + 1.0·CE(tone) + 0.3·CE(masked_char) + 0.05·CE(language)
SHAPE and TONE are the task. MLM is auxiliary and dropped at inference, it forces lexical knowledge into the encoder, which is what tone disambiguation needs. Its target is always the fully marked character; masking ú and accepting u would teach the encoder that bare vowels are valid Yorùbá, the exact failure being fixed. LID is nearly free and makes the API restore(text) instead of restore(text, lang).
Three choices that carry weight:
RoPE, not learned positions. Documents are processed with a sliding window, so the model sees offsets at inference it never saw in training. Learned absolute positions generalise badly to that; rotary ones don’t.
Language as an additive embedding. No <yor> prefix token, nothing to confuse with content, no position spent. During training the language embedding is dropped 12% of the time, giving a genuine “unknown language” mode, so being wrong about the language degrades gracefully instead of catastrophically.
Trained from scratch. We considered starting from an existing encoder, but its tokenizer was trained on diacritized text while our inputs are stripped, every subword boundary would be wrong in the direction that matters most.
Splitting shape from tone
The naive version assigns each character one label from a flat set: acute, grave, underdot, circumflex-plus-acute. That works for Spanish and falls apart on Vietnamese, where stacked marks produce ~60 classes, most of them rare.
Instead, factorize:
ẹ́ → e + DOT_BELOW + acute (Yorùbá)
ộ → o + CIRCUMFLEX + dot-below (Vietnamese)
ữ → u + HORN + tilde (Vietnamese)
ń → n +, + acute (Polish)
SHAPE is marks that change letter identity. TONE is the mark that changes pitch or stress. Two heads, two label sets, two loss terms.
The subtlety, and the reason it works, is that which marks belong to which slot is language-dependent:
U+0323 dot below → SHAPE in Yorùbá (ẹ and e are different letters)
→ TONE in Vietnamese (nặng, one of six tones)
U+0303 tilde → SHAPE in Spanish (ñ is a letter)
→ TONE in Vietnamese (ngã)
A flat label set cannot express that. It sees ọ and assigns one class, blind to the fact that the same codepoint plays different grammatical roles in the two languages.
Factorized, Vietnamese collapses to 5 shapes × 6 tones, every component seen thousands of times. Across all ten languages the final space is 15 shapes and 8 tones.
The bigger payoff is diagnostic. Yorùbá’s dominant failure is tone direction, the model knows a mark belongs there and picks the wrong one, writing pátápátá for pàtàpàtà. In a flat label space that is indistinguishable from any other error. Split, it gets its own head, its own confidence and its own metric:
| checkpoint | Yorùbá DER | tone portion |
|---|---|---|
| 10k steps | 0.1306 | 87% |
| 30k steps | 0.1138 | 81% |
| 120k steps | 0.0836 | 83% |
Four fifths of the remaining error, at every stage of training, is one failure mode. Aggregate DER would have shown a number going down and told us nothing about what to do next.
Graphemes, not codepoints
Yorùbá stacks marks: ọ̀ is o with an underdot and a grave. Unicode has no precomposed codepoint for that combination, so even after NFC normalisation it stays two codepoints. len("ọ̀rọ̀") is 5, not 3.
Iterating over the string misaligns every label after the first stacked character. Everything here operates on graphemes, a base codepoint plus any combining marks that follow it. A small detail that silently determines whether the hardest language in the set trains at all.
What the corpus was teaching us
This is the finding that changed the result more than any architectural decision.
Before training, we plotted diacritic density, the fraction of eligible characters actually carrying a mark, per language. Nine of ten gave a single hump. Yorùbá gave two:
count
│ ▁▄██▇▅▃▂▁
│ ▂████████▃ ▁▂▃▃▂▁
│ ▃██████████▅▂ ▁▃██████▃▁
└──────┴───────────┴──────────────────┴──────────┴────► density
0.10 0.43
89% of sentences 11% of sentences
UNDER-MARKED correct
Correctly marked Yorùbá lands around 0.30–0.45, nearly half its characters are vowels, most carry a high or low tone, plus underdots on ẹ, ọ and ṣ.
Which means the large mode is the broken one. Eighty-nine per cent of our Yorùbá training data was under-marked: real words with their tone marks simply omitted. And we were using it as ground truth. Every one of those sentences was teaching the model that bare vowels are correct.
That is diacnet-1.1‘s documented failure, sitting in a histogram. It also explains something we had noticed and shrugged at: diacnet-1.1 is worse than 1.0 on all three African languages. More data, trained longer, learned to omit tone more confidently.
The fix is not to delete the data
The obvious move is to drop the low-density mode. We didn’t, because under-marked Yorùbá is still Yorùbá, the words are real, the morphology is real. It is excellent data for learning the language. It is poisonous data for learning tone marks.
Since we already had two objectives on one encoder, we could separate them. Sentences below the detected threshold keep their masked-character targets and contribute no diacritic supervision at all:
density 0.458 ABOVE floor → 23 diacritic labels, 8 MLM labels
density 0.000 BELOW floor → 0 diacritic labels, 8 MLM labels
└── still teaches vocabulary
The MLM head trains on all 1.9M Yorùbá sentences. The tagger heads train on the clean ~205k. Same corpus, two different notions of what counts as a label.
Detecting it without eyeballing a threshold
We wanted this automatic. The first attempt used Otsu’s method, the standard image-binarisation algorithm, and it reported every language as bimodal.
Otsu returns a split for any distribution. On a single right-skewed hump, which is what density distributions look like, that split lands near the left edge, where “the peak below the cut” is two nearly-empty bins and every statistic derived from it is noise. Our test fixtures used symmetric Gaussians, which hid the failure completely.
The fix was to find peaks directly and measure how deep the trough between them falls, requiring both sides to hold at least 5% of the mass so a tail cannot masquerade as a mode. Measured that way the answer is not close:
yor ████████████████████████ 0.55 BIMODAL
pol ████ 0.10
tur ███ 0.07
ita ▊ 0.02
ibo 0.00
Igbo and Hausa are correctly left alone, and that matters: standard Igbo orthography omits tone by convention, and Hausa rarely writes it at all. Their low density is a property of the writing system, not a defect. No threshold should touch them.
What falls out of the architecture
Per-character classification gives you several things that are awkward or impossible in a generative setup.
Automatic language detection. A fourth head classifies the language from the mean-pooled encoder state. It costs almost nothing, DER 0.0132 with the language supplied, 0.0133 with it detected, and most integrations don’t reliably know the input language.
Per-character confidence. The joint shape × tone distribution gives a calibrated probability per character (temperature-fitted on validation, T = 1.14). Threshold it, and characters below are left exactly as the user typed them:
| threshold | coverage | DER on committed |
|---|---|---|
| 0.00 | 100.0% | 0.0132 |
| 0.90 | 97.1% | 0.0039 |
| 0.99 | 91.9% | 0.0008 |
At 0.90: 97% of characters restored at 99.6% accuracy, the rest flagged. This is the number to put in front of a customer, because it answers the question they have. A wrong tone mark changes meaning; a missing one is merely incomplete. The threshold is per-request, so a CMS pre-fill and a legal pipeline sit at different points on the same curve.
Legality masking. For each (language, base character) we record which (shape, tone) pairs were ever observed and mask the rest to −∞ at decode time. No acute on Turkish vowels, no Vietnamese horn on Polish. An interesting result: turning it off changes DER by zero to four decimal places. The model learned the orthographies well enough that the constraint never binds. We keep it as a guarantee for out-of-distribution input, it costs nothing.
Protected spans. URLs, emails, @handles, inline code and CONSTANT_NAMES pass through untouched, and were excluded from the training loss. An accent inside a URL is never correct, and it is the error a reader notices first.
Documents, not sentences. No sentence splitting. Overlapping windows are planned across the input and only the centre of each is trusted, so every character is predicted with context on both sides. The window plan is proved to partition the input exactly, 500 randomised cases in CI.
Results
diacbench, 1000 sentences per language.
| lang | DER | shape | tone | exact | compliance |
|---|---|---|---|---|---|
| ita | 0.0002 | 0.0000 | 0.0002 | 0.991 | 1.0000 |
| fra | 0.0012 | 0.0003 | 0.0009 | 0.952 | 1.0000 |
| tur | 0.0016 | 0.0016 | 0.0000 | 0.961 | 1.0000 |
| por | 0.0019 | 0.0006 | 0.0014 | 0.925 | 1.0000 |
| pol | 0.0022 | 0.0018 | 0.0003 | 0.935 | 1.0000 |
| spa | 0.0022 | 0.0001 | 0.0021 | 0.917 | 1.0000 |
| hau | 0.0041 | 0.0040 | 0.0001 | 0.741 | 1.0000 |
| ibo | 0.0122 | 0.0110 | 0.0013 | 0.483 | 1.0000 |
| vie | 0.0164 | 0.0073 | 0.0124 | 0.650 | 1.0000 |
| yor | 0.0836 | 0.0203 | 0.0695 | 0.084 | 1.0000 |
| all | 0.0132 | 0.0055 | 0.0086 | 0.764 | 1.0000 |
Yorùbá 0.2006 → 0.0836, a 58% reduction. Hausa 0.0593 → 0.0041, 93%, though much of the Hausa gain is the compliance guarantee rather than better modelling, since a large share of the old error was corruption rather than wrong accents.
37.6M parameters against 580M.
Against frontier LLMs
300 sentences per language, matching how the baselines were run.
| lang | diactag-1.0 | Claude Sonnet 4.5 | GPT-4o-mini |
|---|---|---|---|
| yor | 0.0933 | 0.1913 | 0.2811 |
| ibo | 0.0110 | 0.0427 | 0.1277 |
| hau | 0.0041 | 0.0178 | 0.1432 |
| vie | 0.0166 | 0.0107 | 0.0399 |
| fra | 0.0014 | 0.0052 | 0.0023 |
Best on 7 of 10. Vietnamese and Portuguese genuinely lose.
Those LLM numbers deserve a footnote, because they are the charitable ones. They come from a harness that checks whether the output still strips back to the input and throws it away when it doesn’t. Raw, against the floor of copying the input unchanged:
| lang | copy-input floor | Claude, raw |
|---|---|---|
| hau | 0.0236 | 0.3509 |
| ibo | 0.1171 | 0.2261 |
| fra | 0.0471 | 0.1171 |
On three languages, a frontier model makes the text worse than doing nothing at all. Hausa is already 97.6% correct if you leave it alone; Claude returns it 35% wrong.
That fallback harness is our architecture, reimplemented externally and imperfectly. The difference is that here there is nothing to discard.
Deployment
| backend | chars/s | p50 | size | DER |
|---|---|---|---|---|
| PyTorch CPU | 105 | 591 ms | 150 MB | 0.0105 |
| ONNX fp32 | 202 | 286 ms | 150.9 MB | 0.0105 |
| ONNX int8 | 244 | 200 ms | 38.3 MB | 0.0108 |
Three times faster and four times smaller for +0.03pp DER. Compliance stays 1.0000 under quantisation, the guarantee is architectural, not a property of a particular numeric precision.
A 38MB artifact at 244 characters per second on one CPU core takes the GPU off the serving bill entirely.
One honest caveat: p95 is 788–1099 ms across all backends while p50 varies 3×. The tail is length-driven, long sentences need more windows, not backend-driven. Quantisation doesn’t fix it.
Reproducing it
pip install olaverse[deeplearning]
from olaverse.nlp import Diacritizer
d = Diacritizer(model="diactag-1.0", lang="yo")
d.restore("se eranko naa si gbo o?")
# → 'ṣé ẹranko náà sì gbọ́ ọ?'
Data. 7.0M sentences from a web + Wikipedia corpus, plus 131k from a Q&A passage set, the only non-encyclopedic register in the mix, gated on diacritic density and deduplicated against the eval splits by stripped-form hash.
Two filters worth copying if you build something similar. Script filtering: all ten languages are Latin, but Hausa Wikipedia carries Ajami and Yorùbá articles carry Korean and Devanagari names; those contribute nothing but identity labels. Cross-language filtering: a Vietnamese name inside a Turkish sentence factorizes into composite classes that exist in no language, because Turkish declares no tone slot for those marks to occupy. Without both, the label space inflates from 15 shape classes to 99, most of them junk competing in the same softmax.
Training. AdamW, lr 3e-4 with 2000 warmup steps and cosine decay to 5%, weight_decay 0.05, batch 128 × 2 accumulation = 256, seq len 256, bf16, 120k steps. About 13 hours on a single A100.
Language balancing. Temperature sampling at α=0.5. At α=1 Yorùbá and Igbo would be 54% of the mix; at α=0 Hausa’s 400k sentences get seen 5× per Yorùbá epoch and overfit.
Augmentation. partial_mark_prob=0.30 with keep range [0.05, 0.9], case perturbation (2% upper, 3% lower, 1% title), 3% protected-span injection, mlm_prob=0.15, lang_dropout=0.12.
Evaluation. Report shape_DER and tone_DER separately, and compliance as a hard assertion rather than a metric. If compliance is not exactly 1.0, the architecture is broken and nothing else in the table means anything.
What it still gets wrong
Yorùbá. DER 0.0836, 83% of it tone direction. Sentence-level exact match is 0.084, 92 of every 100 Yorùbá sentences contain at least one wrong mark. A harsher framing than DER, and a truer one.
Igbo and Hausa tone numbers are not achievements. tone_DER of 0.0013 and 0.0001 looks superb and means little, because those orthographies barely write tone. Nothing to learn, nothing to get wrong.
No typo correction. The architecture cannot insert or delete characters, so it cannot fix Ile → Ilé and teh → the in one pass. That is the price of the guarantee, and we think it is the right trade, a model that reliably does one thing beats one that sometimes does two. The per-character confidence is a natural trigger for a separate corrector.
Dense input degrades. The Polish pangram Zażółć gęślą jaźń has density 0.565 against a Polish median of 0.068, nine times denser than real Polish, and the model misses six characters, despite a Polish DER of 0.0022.
Some errors are irreducible. Viaggio and Viaggiò are both valid Italian and the stripped form contains no information distinguishing them. Every language has a nonzero floor.
Three things worth taking away
Plot your data before you train. The bimodal Yorùbá distribution took ten minutes to find and changed the outcome more than any modelling decision. We nearly skipped it.
Make your metric decompose along your failure modes. Aggregate DER would have gone down and taught us nothing. Splitting shape from tone told us, at every checkpoint, that four fifths of the problem was one thing, and that the other nine languages had already converged.
Put your guarantees in the architecture, not the loss function. A constraint the model can violate is one you have to monitor, apologise for, and wrap in a fallback. A constraint it cannot violate is one you can sell.
Weights, ONNX exports and label space: olaverse/diactag-1.0. Documentation: olaverse-labs.github.io/olaverse/models/diactag. Apache 2.0.
Enjoyed this? Every model we write about is open weight and free to download.
