Pretrained From Scratch for Naija: Introducing mist-encoder-base-ng

A 30.9M-parameter encoder built from the ground up for Hausa, Yoruba, Igbo, and Nigerian Pidgin, with a tokenizer that beats GPT-4o’s on all four.

---

Most “African language support” in NLP is an afterthought: a massive multilingual model where Yoruba got a sliver of the training budget and the tokenizer shreds every word into fragments. Today we’re releasing something different, mist-encoder-base-ng, a compact encoder pretrained entirely from scratch on Nigerian text, open-sourced on Hugging Face under Apache 2.0.

It covers the four major languages of Nigeria, Hausa, Yoruba, Igbo, and Nigerian Pidgin, and it’s built to be a workhorse base: load the encoder body, attach a head, and fine-tune for classification, sentiment, language ID, or sentence embeddings.

from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("olaverse/mist-encoder-base-ng")
enc = AutoModel.from_pretrained("olaverse/mist-encoder-base-ng")

Why from scratch?

Fine-tuning an existing multilingual model inherits that model’s tokenizer, and for Nigerian languages, general-purpose tokenizers are quietly terrible. Every extra subword a tokenizer needs per word means less real signal fits in a fixed context window, slower inference, and weaker per-token representations.

So before training the model, we built the tokenizer: otk-bpe-50k, a unified byte-level BPE for Naija text with a ~50k vocabulary, NFC diacritic preservation, code-mixed English support, and 0% unknown tokens. The fertility numbers (tokens per word, lower is better) speak for themselves:

Tokenizer	Hausa	Yoruba	Igbo	Pidgin
otk-bpe-50k (ours)	1.231	1.296	1.416	1.249
GPT-4o (o200k)	1.589	1.687	1.807	1.304
AfroXLMR	1.604	2.277	2.570	1.401

Our tokenizer is more efficient on Nigerian text than the tokenizer behind GPT-4o, on all four languages. On Igbo, AfroXLMR needs over 2.5 tokens per word; ours needs 1.4.

The model

On top of that tokenizer sits a ModernBERT architecture, 6 layers, hidden size 384, 1024 max positions, pretrained with masked language modeling on ~480k carefully prepared chunks. The corpus combines FineWeb-2, the audited WURA corpus, and a fresh Nigerian Pidgin dataset, all cross-deduplicated (FineWeb-2 and WURA both descend from Common Crawl, so overlap was real), language-balanced so abundant Hausa didn’t drown out scarce Igbo and Pidgin, and chunked into windows so no text was wasted on truncation. Every source is attribution-only licensed, which is what makes the clean Apache 2.0 release possible.

The result is 30.9M parameters, and since the embedding table is about two-thirds of that, the transformer itself is only ~11M. For comparison: AfriBERTa is 126M, mBERT is 178M, XLM-R is 270M.

What a 30.9M specialist can do

On MasakhaNEWS topic classification (macro-F1), the model is genuinely competitive at a quarter of the baselines’ size:

Model	Params	Hausa	Yoruba	Igbo	Pidgin
mist-encoder-base-ng	30.9M	0.878	0.859	0.803	0.898
AfriBERTa	126M	0.924	0.921	0.914	0.991
mBERT	178M	0.806	0.886	0.805	0.967

It beats mBERT on Hausa and ties it on Igbo, with under a fifth of the parameters. That efficiency is the point: this is a model you can fine-tune on a laptop and deploy on-device or in low-resource environments where 126M+ models are impractical.

What it can’t do (yet), and exactly why

We publish our weaknesses with the same rigor as our wins. Named-entity recognition is this model’s honest weak spot: it trails AfriBERTa and mBERT by roughly 10–20 entity-F1 on MasakhaNER 2.0, with Hausa NER lowest at 0.656.

And we didn’t just report the gap, we diagnosed it. The deficit is structural, not a tuning artifact: it persists across random seeds and labeling schemes. Two causes: the unified 50k tokenizer fragments entity words (person and place names) more than language-specific tokenizers, on Hausa NER text, ~61% of entity words split into multiple subwords versus ~21% for AfriBERTa, and at 30.9M parameters the model has less capacity to reassemble meaning from those fragments. If NER accuracy is critical to your application, use a larger model. If sentence-level understanding at minimal cost is what you need, this is your base.

The model card also lays out an evidence-backed v2 roadmap: more capacity (~50–80M), a less-fragmenting tokenizer for token-level tasks, and more Pidgin and Hausa data. Notably, longer pretraining isn’t on the list, eval loss plateaued by epoch 11, so we know that lever is spent.

Part of a bigger picture

mist-encoder-base-ng joins the Mist Embedder collection and sits alongside our LID models (which are fine-tuned classifiers) as complementary infrastructure: a from-scratch, Nigerian-first foundation others can build on. Two fine-tunes of it already exist on the Hub, and there’s a demo Space to try it.

• Model: https://huggingface.co/olaverse/mist-encoder-base-ng
• Tokenizer: https://huggingface.co/olaverse/otk-bpe-50k
• Collection: https://huggingface.co/collections/olaverse/mist-embedder

Apache 2.0, free for commercial use. If you fine-tune it for a Nigerian-language task, sentiment, topic tagging, embeddings, anything, open a discussion on the model page and show us. That’s the whole point of releasing the base.

---

Olaverse Lab builds open-source AI with first-class support for underserved languages. Explore all our models at huggingface.co/olaverse.