Home > Research > Publications & Outputs > UNLT

Electronic data

Links

Text available via DOI:

View graph of relations

UNLT: Urdu Natural Language Toolkit

Research output: Contribution to Journal/MagazineJournal articlepeer-review

Published

Standard

UNLT: Urdu Natural Language Toolkit. / Shafi, Jawad; Iqbal, Hafiz Rizwan; Nawab, Rao Muhammad Adeel et al.
In: Natural Language Engineering, Vol. 29, No. 4, 31.07.2023, p. 942-977.

Research output: Contribution to Journal/MagazineJournal articlepeer-review

Harvard

Shafi, J, Iqbal, HR, Nawab, RMA & Rayson, P 2023, 'UNLT: Urdu Natural Language Toolkit', Natural Language Engineering, vol. 29, no. 4, pp. 942-977. https://doi.org/10.1017/S1351324921000425

APA

Shafi, J., Iqbal, H. R., Nawab, R. M. A., & Rayson, P. (2023). UNLT: Urdu Natural Language Toolkit. Natural Language Engineering, 29(4), 942-977. https://doi.org/10.1017/S1351324921000425

Vancouver

Shafi J, Iqbal HR, Nawab RMA, Rayson P. UNLT: Urdu Natural Language Toolkit. Natural Language Engineering. 2023 Jul 31;29(4):942-977. Epub 2022 Jan 19. doi: 10.1017/S1351324921000425

Author

Shafi, Jawad ; Iqbal, Hafiz Rizwan ; Nawab, Rao Muhammad Adeel et al. / UNLT : Urdu Natural Language Toolkit. In: Natural Language Engineering. 2023 ; Vol. 29, No. 4. pp. 942-977.

Bibtex

@article{5dceeea1cafe4978bba7d31319eb79e7,
title = "UNLT: Urdu Natural Language Toolkit",
abstract = "This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other languages. There is also a dire need for standard text processing tools and methods for Urdu, despite it being widely spoken in different parts of the world with a large amount of digital text being readily available. This study presents the first version of the UNLT (Urdu Natural Language Toolkit) which contains three key text processing tools required for an Urdu NLP pipeline; word tokenizer, sentence tokenizer, and part-of-speech (POS) tagger. The UNLT word tokenizer employs a morpheme matching algorithm coupled with a state-of-the-art stochastic n-gram language model with back-off and smoothing characteristics for the space omission problem. The space insertion problem for compound words is tackled using a dictionary look-up technique. The UNLT sentence tokenizer is a combination of various machine learning, rule-based, regular-expressions, and dictionary look-up techniques. Finally, the UNLT POS taggers are based on Hidden Markov Model and Maximum Entropy-based stochastic techniques. In addition, we have developed large gold standard training and testing data sets to improve and evaluate the performance of new techniques for Urdu word tokenization, sentence tokenization, and POS tagging. For comparison purposes, we have compared the proposed approaches with several methods. Our proposed UNLT, the training and testing data sets, and supporting resources are all free and publicly available for academic use.",
keywords = "Part-of-speech tagging, Text segmentation, Urdu NLP toolkit, Urdu corpora, Word segmentation",
author = "Jawad Shafi and Iqbal, {Hafiz Rizwan} and Nawab, {Rao Muhammad Adeel} and Paul Rayson",
year = "2023",
month = jul,
day = "31",
doi = "10.1017/S1351324921000425",
language = "English",
volume = "29",
pages = "942--977",
journal = "Natural Language Engineering",
publisher = "Cambridge University Press",
number = "4",

}

RIS

TY - JOUR

T1 - UNLT

T2 - Urdu Natural Language Toolkit

AU - Shafi, Jawad

AU - Iqbal, Hafiz Rizwan

AU - Nawab, Rao Muhammad Adeel

AU - Rayson, Paul

PY - 2023/7/31

Y1 - 2023/7/31

N2 - This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other languages. There is also a dire need for standard text processing tools and methods for Urdu, despite it being widely spoken in different parts of the world with a large amount of digital text being readily available. This study presents the first version of the UNLT (Urdu Natural Language Toolkit) which contains three key text processing tools required for an Urdu NLP pipeline; word tokenizer, sentence tokenizer, and part-of-speech (POS) tagger. The UNLT word tokenizer employs a morpheme matching algorithm coupled with a state-of-the-art stochastic n-gram language model with back-off and smoothing characteristics for the space omission problem. The space insertion problem for compound words is tackled using a dictionary look-up technique. The UNLT sentence tokenizer is a combination of various machine learning, rule-based, regular-expressions, and dictionary look-up techniques. Finally, the UNLT POS taggers are based on Hidden Markov Model and Maximum Entropy-based stochastic techniques. In addition, we have developed large gold standard training and testing data sets to improve and evaluate the performance of new techniques for Urdu word tokenization, sentence tokenization, and POS tagging. For comparison purposes, we have compared the proposed approaches with several methods. Our proposed UNLT, the training and testing data sets, and supporting resources are all free and publicly available for academic use.

AB - This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other languages. There is also a dire need for standard text processing tools and methods for Urdu, despite it being widely spoken in different parts of the world with a large amount of digital text being readily available. This study presents the first version of the UNLT (Urdu Natural Language Toolkit) which contains three key text processing tools required for an Urdu NLP pipeline; word tokenizer, sentence tokenizer, and part-of-speech (POS) tagger. The UNLT word tokenizer employs a morpheme matching algorithm coupled with a state-of-the-art stochastic n-gram language model with back-off and smoothing characteristics for the space omission problem. The space insertion problem for compound words is tackled using a dictionary look-up technique. The UNLT sentence tokenizer is a combination of various machine learning, rule-based, regular-expressions, and dictionary look-up techniques. Finally, the UNLT POS taggers are based on Hidden Markov Model and Maximum Entropy-based stochastic techniques. In addition, we have developed large gold standard training and testing data sets to improve and evaluate the performance of new techniques for Urdu word tokenization, sentence tokenization, and POS tagging. For comparison purposes, we have compared the proposed approaches with several methods. Our proposed UNLT, the training and testing data sets, and supporting resources are all free and publicly available for academic use.

KW - Part-of-speech tagging

KW - Text segmentation

KW - Urdu NLP toolkit

KW - Urdu corpora

KW - Word segmentation

U2 - 10.1017/S1351324921000425

DO - 10.1017/S1351324921000425

M3 - Journal article

VL - 29

SP - 942

EP - 977

JO - Natural Language Engineering

JF - Natural Language Engineering

IS - 4

ER -