AI Models
    OTK BPE

    OlaverseOTK BPE

    OTK Tokenizers family · open weights on Hugging Face.

    Byte-level BPE tokenizer built for multilingual text, efficient subword segmentation where general vocabularies fall apart.

    Tokenizer
    Byte Level Bpe
    Multilingual
    Sw
    Rw
    Fr
    English
    Licence: apache-2.0
    0
    Likes
    0
    Downloads
    Jul 2026
    Created

    Model Card

    Built for production use.

    Open-weights repository on Hugging Face

    Integrated ecosystem protocol tier.

    Compatible with Transformers library

    Integrated ecosystem protocol tier.

    Optimised for low-latency inference

    Integrated ecosystem protocol tier.

    Community engagement: 0 likes

    Integrated ecosystem protocol tier.

    Quick Start

    Use OTK BPE, straight from its model card.

    example.py
    from the model card
    from transformers import AutoTokenizer
    # Load the Swahili tokenizer at your chosen size
    tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="sw-150k")
    text = "Habari yako? Leo ni siku nzuri sana 😊"
    inputs = tokenizer(text)
    print("Tokens:", tokenizer.tokenize(text))
    print("IDs:", inputs["input_ids"])
    print("Decoded:", tokenizer.decode(inputs["input_ids"]))
    Model Card

    Official Repository README

    Dynamically loaded from Hugging Face

    Model card metadata is available on Hugging Face.

    Built with

    Transformers PyTorch Python
    Built by Olaverse Labs

    Ready to try OTK BPE?

    Byte-level BPE tokenizer built for multilingual text, efficient subword segmentation where general vocabularies fall apart.

    Model Ecosystem

    Explore More Models