AI Models
    OTK BPE 50k

    OlaverseOTK BPE 50k

    OTK Tokenizers family · open weights on Hugging Face.

    Byte-level BPE tokenizer built for Nigerian languages, Yorùbá, Igbo, Hausa, and Pidgin, efficient subword segmentation where general vocabularies fall apart.

    Tokenizers
    BPE
    Tokenizer
    Nigerian Languages
    Yoruba
    Igbo
    Hausa
    Nigerian Pidgin
    Naija
    Licence: apache-2.0
    0
    Likes
    0
    Downloads
    May 2026
    Created

    Model Card

    Built for production use.

    Open-weights repository on Hugging Face

    Integrated ecosystem protocol tier.

    Compatible with Transformers library

    Integrated ecosystem protocol tier.

    Optimised for low-latency inference

    Integrated ecosystem protocol tier.

    Community engagement: 0 likes

    Integrated ecosystem protocol tier.

    Quick Start

    Use OTK BPE 50k, straight from its model card.

    example.py
    from the model card
    from transformers import AutoTokenizer
    # Load the custom Yoruba tokenizer
    tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe-50k", subfolder="yo")
    # Encode text
    text = "Ẹ kú àbọ̀ ooo, ṣé dáadáa ni? 😂"
    inputs = tokenizer(text)
    print("Tokens:", tokenizer.tokenize(text))
    print("IDs:", inputs["input_ids"])
    print("Decoded:", tokenizer.decode(inputs["input_ids"]))
    Model Card

    Official Repository README

    Dynamically loaded from Hugging Face

    Model card metadata is available on Hugging Face.

    Built with

    Transformers PyTorch Python
    Built by Olaverse Labs

    Ready to try OTK BPE 50k?

    Byte-level BPE tokenizer built for Nigerian languages, Yorùbá, Igbo, Hausa, and Pidgin, efficient subword segmentation where general vocabularies fall apart.

    Model Ecosystem

    Explore More Models