AMHTokenizer

Amharic, spoken by over 58 million people in Ethiopia and the diaspora, has been largely overlooked in modern AI. While English and other Latin-based languages benefit from advanced NLP tools, Amharic presents a unique challenge: its Fidel script contains over 300 distinct characters, each representing a consonant–vowel syllable. Standard tokenizers treat each symbol separately, leading to large vocabularies, sparse data, and slow, inefficient model training.
To address this, I developed the Amharic Tokenizer, an open-source Python library designed specifically for Amharic. By decomposing each Fidel into its base consonant and vowel components, the tokenizer reduces the effective vocabulary from over 300 tokens to just 40, while preserving the full meaning of the language.
This approach improves AI training efficiency by up to 70%, enables rare syllables to be composed dynamically, and significantly reduces model size. It also sets the foundation for scaling to other African languages, which share similar scripts, potentially impacting millions of speakers.
Funding will support continued development, dataset creation, and open-source model training, helping African languages achieve parity with English in AI systems. With your support, we can ensure these languages are not left behind, but become central to the next generation of AI innovation.