OTK Tokenizers

    OTK Tokenizers

    The ground floor of the stack: otk-bpe and otk-bpe-50k are BPE vocabularies trained for Yorùbá, Igbo, Hausa, and Nigerian Pidgin, so African-language text stops being tokenised into confetti.

    OTK Tokenizers

    otk-bpe-50k

    Byte-level BPE tokenizer built for Nigerian languages, Yorùbá, Igbo, Hausa, and Pidgin, efficient subword segmentation where general vocabularies fall apart.

    0 0
    View
    OTK Tokenizers

    otk-bpe

    Byte-level BPE tokenizer built for multilingual text, efficient subword segmentation where general vocabularies fall apart.

    0 0
    View

    Explore Other Families

    More of the stack, organised the same way.

    5 models

    MIST

    Frontier-merged LLMs: from MIST-Mini-8B to MIST-1-140B, including reasoning-tuned Thinking variants.

    Explore
    6 models

    LID

    Language identification: from 5 Nigerian languages to 25 languages worldwide.

    Explore
    6 models

    DiacNet

    Diacritic & tone restoration for Yorùbá, Igbo, Hausa and beyond.

    Explore