AI Models
    OTK BPE 50k

    OTK BPE 50k

    OTK Tokenizers family · open weights on Hugging Face.

    Byte-level BPE tokenizer built for Nigerian languages, Yorùbá, Igbo, Hausa, and Pidgin, efficient subword segmentation where general vocabularies fall apart.

    TokenizersBPETokenizerNigerian LanguagesYorubaIgboHausaNigerian PidginNaijaLicence: apache-2.0
    0
    Likes
    0
    Downloads
    May 2026
    Created
    / Model card

    Built for production use

    Open-weights repository on Hugging Face

    Integrated ecosystem protocol tier.

    Compatible with Transformers library

    Integrated ecosystem protocol tier.

    Optimised for low-latency inference

    Integrated ecosystem protocol tier.

    Community engagement: 0 likes

    Integrated ecosystem protocol tier.

    / Quick start

    Use OTK BPE 50k, straight from its model card.

    example.pyfrom the model card
    from transformers import AutoTokenizer
    # Load the custom Yoruba tokenizer
    tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe-50k", subfolder="yo")
    # Encode text
    text = "Ẹ kú àbọ̀ ooo, ṣé dáadáa ni? 😂"
    inputs = tokenizer(text)
    print("Tokens:", tokenizer.tokenize(text))
    print("IDs:", inputs["input_ids"])
    print("Decoded:", tokenizer.decode(inputs["input_ids"]))
    / Model card

    Official repository README

    Dynamically loaded from Hugging Face

    Model card metadata is available on Hugging Face.

    / Built with
    Transformers PyTorch Python

    Ready to try OTK BPE 50k?

    Byte-level BPE tokenizer built for Nigerian languages, Yorùbá, Igbo, Hausa, and Pidgin, efficient subword segmentation where general vocabularies fall apart.

    / Model ecosystem

    Explore more models