Dr. Sherzod Hakimov

Dr. Sherzod Hakimov

TurkicNLP: An Open Toolkit and a Free Book for All 24 Turkic Languages

3 minute read

Published:

Turkic languages are spoken from the Balkans and Anatolia to Central Asia, western China and north-eastern Siberia. They share much of their grammar, yet the tools available for them range from production-ready for Turkish to almost nothing for Khakas or Tuvan. They’re written in Latin, Cyrillic and Perso-Arabic scripts, sometimes all three for a single language.

Over the past months I’ve been working on two things to close that gap: an open-source toolkit and a free online book. Both are out now.

TurkicNLP: toolkit and book

🔧 The toolkit: TurkicNLP

TurkicNLP is a pip-installable Python library with one Stanza-style pipeline for the whole family. It currently has:

  • 24 languages across the Oghuz, Kipchak, Karluk, Siberian, Oghur and Arghu branches, plus Ottoman Turkish and Old Turkic
  • 15 pipeline processors: tokenization, morphology, POS tagging, lemmatization, dependency parsing, NER, translation, embeddings, sentiment, spelling correction, subword tokenization and speech recognition
  • 7 backend families combined behind one interface: rule-based, Apertium finite-state transducers (20 languages), Stanza/UD models (9 languages, 5 of them custom-trained), multilingual Glot500 models (POS/parsing for 15 languages, morphology for 22), NLLB-200 translation and embeddings (11 languages), Omnilingual ASR and LLMs
  • Speech → text → analysis for 20 Turkic languages, with transcripts returned in each language’s own script
  • An LLM backend for all 24 languages. Morphology, NER, translation, sentiment and spelling correction can run on a local vLLM/Ollama model, OpenAI, Claude or a Hugging Face model. Answers are validated against the toolkit’s tokens, so a failed answer stays empty rather than being guessed.
  • A hybrid morpheme tokenizer that combines FSTs with neural models and segments 89% of words exactly across 20 languages (Apertium alone: 79%, neural alone: 68%)
  • Script handling built in: transliteration between Cyrillic, Latin and Perso-Arabic, the Common Turkic Alphabet for 23 languages, and Old Turkic runes. Text in a script without models (e.g. Uzbek Cyrillic) is transliterated automatically and the results are mapped back.
pip install "turkicnlp[all]"

import turkicnlp

nlp = turkicnlp.Pipeline("kaz", processors=["tokenize", "pos", "lemma", "depparse"])
doc = nlp("Мен мектепке бардым.")
print(doc.to_conllu())

The library is Apache 2.0. Apertium data is GPL and downloaded separately, so the package stays permissively licensed.

📖 The book: Natural Language Processing Across Turkic Languages

The toolkit answers how to run things. The book explains why they work, where they fail, and what to build next. It’s free to read online, licensed CC BY-SA 4.0, with a DOI on Zenodo.

17 chapters + 4 appendices, built around three recurring questions: How does morphology affect a task? What changes when the script changes? What transfers between related languages?

Every chapter has exercises, and most have a hands-on section that uses the toolkit, so you can reproduce what you read.

🤝 How you can help

This is a community effort, and the most valuable contributions don’t require writing code:

  • Speakers: check analyses in your language. Native speakers catch wrong lemmas and segmentations that no metric does.
  • Readers: found a mistake in the book? Please open a pull request on github.com/turkic-nlp/book.
  • Researchers & developers: the roadmap lists open tasks, and items marked S are good first issues.
  • Everyone: star the repo, share it with someone working on a Turkic language, or join us on Discord.

Website: turkic-nlp.github.io

Toolkit: github.com/turkic-nlp/turkicnlp · PyPI · Notebooks · 🤗 Models & data

Book: Read online · Table of contents · Source · DOI 10.5281/zenodo.23324599

Paper: TurkicNLP: An NLP Toolkit for Turkic Languages (arXiv:2602.19174)