Unsupervised text tokenizer for Neural Network-based text generation.
-
Updated
Feb 11, 2026 - C++
#
word-segmentation
Here are 152 public repositories matching this topic...
百度NLP：分词，词性标注，命名实体识别，词重要性
python java named-entity-recognition lexical-analysis chinese-nlp word-segmentation part-of-speech-tagger chinese-word-segmentation
-
Updated
May 25, 2021 - C++
SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm
spellcheck fuzzy-search fuzzy-matching edit-distance levenshtein levenshtein-distance spelling spell-check chinese-text-segmentation word-segmentation approximate-string-matching spelling-correction damerau-levenshtein text-segmentation chinese-word-segmentation symspell
-
Updated
Jan 20, 2026 - C#
Thai natural language processing in Python
python natural-language-processing thai-language thai computational-linguistics text-processing soundex nlp-library word-segmentation thai-nlp hacktoberfest thai-nlp-library thai-soundex hacktoberfest-accepted
-
Updated
Feb 13, 2026 - Python
Unsupervised text tokenizer focused on computational efficiency
-
Updated
Mar 29, 2024 - C++
Python port of SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm
python spellcheck fuzzy-search fuzzy-matching edit-distance levenshtein levenshtein-distance spelling spell-check chinese-text-segmentation word-segmentation approximate-string-matching spelling-correction damerau-levenshtein text-segmentation chinese-word-segmentation symspell
-
Updated
Feb 13, 2026 - Python
CKIP Transformers
-
Updated
Apr 21, 2023 - Python
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization
-
Updated
Jun 2, 2025 - Python
Kiwi(지능형 한국어 형태소 분석기)
nlp cpp morphology korean word-segmentation morphological-analysis korean-text-processing korean-tokenizer korean-nlp
-
Updated
Feb 2, 2026 - C++
A Vietnamese natural language processing toolkit (NAACL 2018)
java nlp natural-language-processing parsing vietnamese python3 named-entity-recognition ner word-segmentation pos-tagging dependency-parsing pos-tagger vietnamese-nlp sentence-segmentation vietnamese-tokenizer vncorenlp word-segmenter rdrsegmenter vnmarmot
-
Updated
Feb 12, 2023 - Java
BERT for Multitask Learning
nlp text-classification transformer named-entity-recognition pretrained-models part-of-speech ner word-segmentation bert cws encoder-decoder multi-task-learning multitask-learning
-
Updated
Apr 12, 2023 - Jupyter Notebook
AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models
nlp natural-language-processing crf pytorch information-extraction named-entity-recognition chinese-nlp ner word-segmentation bert sequence-labeling relation-extraction natural-language-understanding entity-typing token-classification multi-modal-ner
-
Updated
Nov 15, 2023 - Python
A Japanese tokenizer based on recurrent neural networks
nlp natural-language-processing japanese tokenizer nlp-library word-segmentation dynet pos-tagging sequence-labeling
-
Updated
Feb 12, 2026 - Python
Juman++ (a Morphological Analyzer Toolkit)
nlp japanese tokenizer cjk word-segmentation pos-tagging part-of-speech-tagger morphological-analysis pos-tagger morphological-analyser juman
-
Updated
Oct 3, 2023 - C++
Cantonese Linguistics and NLP
python nlp natural-language-processing linguistics cantonese computational-linguistics word-segmentation jyutping pycantonese stop-words part-of-speech-tagging
-
Updated
May 23, 2024 - Python
Python API for Kiwi
-
Updated
Dec 25, 2025 - Python
中文文本分类、序列标注工具包（pytorch），支持中文长文本、短文本的多类、多标签分类任务，支持中文命名实体识别、词性标注、分词、抽取式文本摘要等序列标注任务。 Chinese text classification and sequence labeling toolkit, supports multi class and multi label classification, text similsrity, text summary and NER.
text-classification transformers python3 pytorch named-entity-recognition chinese-text-segmentation pretrained-models word-segmentation bert pos-tagging sequence-labeling chinese-text-classification
-
Updated
Jul 18, 2024 - Python
This repository is archived! The maintained MeCab can be found https://github.com/shogo82148/mecab
-
Updated
Oct 15, 2024 - C++
A PyTorch implementation of the BI-LSTM-CRF model.
nlp crf pytorch ner word-segmentation pos-tagging sequence-labeling bi-lstm-crf bilstm crf-model lstm-crf bilstm-crf sequence-tagging
-
Updated
Oct 30, 2024 - Python
MONPA 罔拍是一個提供正體中文斷詞、詞性標註以及命名實體辨識的多任務模型
nlp named-entity-recognition pos ner word-segmentation albert bert pos-tagging chinese-word-segmentation
-
Updated
Feb 20, 2025 - Python
Improve this page
Add a description, image, and links to the word-segmentation topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the word-segmentation topic, visit your repo's landing page and select "manage topics."