AI Models
    OTK BPE

    OTK BPE

    OTK Tokenizers family Β· open weights on Hugging Face.

    Byte-level BPE tokenizer built for multilingual text, efficient subword segmentation where general vocabularies fall apart.

    TokenizerByte Level BpeMultilingualSwRwFrEnglishLicence: apache-2.0
    0
    Likes
    0
    Downloads
    Jul 2026
    Created
    / Model card

    Built for production use

    Open-weights repository on Hugging Face

    Integrated ecosystem protocol tier.

    Compatible with Transformers library

    Integrated ecosystem protocol tier.

    Optimised for low-latency inference

    Integrated ecosystem protocol tier.

    Community engagement: 0 likes

    Integrated ecosystem protocol tier.

    / Quick start

    Use OTK BPE, straight from its model card.

    example.pyfrom the model card
    from transformers import AutoTokenizer
    # Load the Swahili tokenizer at your chosen size
    tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="sw-150k")
    text = "Habari yako? Leo ni siku nzuri sana 😊"
    inputs = tokenizer(text)
    print("Tokens:", tokenizer.tokenize(text))
    print("IDs:", inputs["input_ids"])
    print("Decoded:", tokenizer.decode(inputs["input_ids"]))
    / Model card

    Official repository README

    Dynamically loaded from Hugging Face

    Model card metadata is available on Hugging Face.

    / Built with
    Transformers PyTorch Python

    Ready to try OTK BPE?

    Byte-level BPE tokenizer built for multilingual text, efficient subword segmentation where general vocabularies fall apart.

    / Model ecosystem

    Explore more models