NUWT: Jawi-specific buckwalter corpus for Malay word tokenization

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

This paper describes the design and creation of a monolingual parallel corpus for the Malay language written in Jawi. This paper proposes a new corpus called the National University of Malaysia Word Tokenization (NUWT) corpora To the best of our knowledge, currently, there is no sufficiently comprehensive, well-designed standard corpus that is annotated and made available for the public for the Jawi script corpora. This corpus contains the Jawi-specific Buckwalter character code and can be used to evaluate the performance of word tokenization tasks, as well as further language processing. The objective of this work is to conform and standardize the corpora between similar characters in Jawi. It consists of three subcorporas with documents from different genres. The gathering and processing steps, as well as the definition of several evaluation tasks regarding the use of these corpora, are included in this paper. One of the important roles and fundamental tasks of the corpus, which is the tokenization, is also presented in this paper. The development of the Malay language tokenizer is based on the syntactic data compatibility of Malay words written in Jawi. A series of experiments were performed to validate the corpus and to fulfill the requirement of the Jawi script tokenizer with an average error rate of 0.020255. Based on this promising result, the token will be used for the disambiguation and unknown word resolution, such as out-ofvocabulary (OOV) problem in the tagging process.

Original languageEnglish
Pages (from-to)107-131
Number of pages25
JournalJournal of Information and Communication Technology
Volume15
Issue number1
Publication statusPublished - 2016

Fingerprint

Malaysia
Syntactics
Processing
Experiments
Corpus
Tagging
Compatibility
Error Rate
Unknown

Keywords

  • Buckwalter character code
  • Malay corpora
  • Regular expression
  • Word tokenization

ASJC Scopus subject areas

  • Computer Science(all)
  • Mathematics(all)

Cite this

@article{e564428f607c48ec8a1153eb5cf7f263,
title = "NUWT: Jawi-specific buckwalter corpus for Malay word tokenization",
abstract = "This paper describes the design and creation of a monolingual parallel corpus for the Malay language written in Jawi. This paper proposes a new corpus called the National University of Malaysia Word Tokenization (NUWT) corpora To the best of our knowledge, currently, there is no sufficiently comprehensive, well-designed standard corpus that is annotated and made available for the public for the Jawi script corpora. This corpus contains the Jawi-specific Buckwalter character code and can be used to evaluate the performance of word tokenization tasks, as well as further language processing. The objective of this work is to conform and standardize the corpora between similar characters in Jawi. It consists of three subcorporas with documents from different genres. The gathering and processing steps, as well as the definition of several evaluation tasks regarding the use of these corpora, are included in this paper. One of the important roles and fundamental tasks of the corpus, which is the tokenization, is also presented in this paper. The development of the Malay language tokenizer is based on the syntactic data compatibility of Malay words written in Jawi. A series of experiments were performed to validate the corpus and to fulfill the requirement of the Jawi script tokenizer with an average error rate of 0.020255. Based on this promising result, the token will be used for the disambiguation and unknown word resolution, such as out-ofvocabulary (OOV) problem in the tagging process.",
keywords = "Buckwalter character code, Malay corpora, Regular expression, Word tokenization",
author = "Bakar, {Juhaida Abu} and Khairuddin Omar and Nasrudin, {Mohammad Faidzul} and Murah, {Mohd. Zamri}",
year = "2016",
language = "English",
volume = "15",
pages = "107--131",
journal = "Journal of Information and Communication Technology",
issn = "1675-414X",
publisher = "Universiti Utara Malaysia Press",
number = "1",

}

TY - JOUR

T1 - NUWT

T2 - Jawi-specific buckwalter corpus for Malay word tokenization

AU - Bakar, Juhaida Abu

AU - Omar, Khairuddin

AU - Nasrudin, Mohammad Faidzul

AU - Murah, Mohd. Zamri

PY - 2016

Y1 - 2016

N2 - This paper describes the design and creation of a monolingual parallel corpus for the Malay language written in Jawi. This paper proposes a new corpus called the National University of Malaysia Word Tokenization (NUWT) corpora To the best of our knowledge, currently, there is no sufficiently comprehensive, well-designed standard corpus that is annotated and made available for the public for the Jawi script corpora. This corpus contains the Jawi-specific Buckwalter character code and can be used to evaluate the performance of word tokenization tasks, as well as further language processing. The objective of this work is to conform and standardize the corpora between similar characters in Jawi. It consists of three subcorporas with documents from different genres. The gathering and processing steps, as well as the definition of several evaluation tasks regarding the use of these corpora, are included in this paper. One of the important roles and fundamental tasks of the corpus, which is the tokenization, is also presented in this paper. The development of the Malay language tokenizer is based on the syntactic data compatibility of Malay words written in Jawi. A series of experiments were performed to validate the corpus and to fulfill the requirement of the Jawi script tokenizer with an average error rate of 0.020255. Based on this promising result, the token will be used for the disambiguation and unknown word resolution, such as out-ofvocabulary (OOV) problem in the tagging process.

AB - This paper describes the design and creation of a monolingual parallel corpus for the Malay language written in Jawi. This paper proposes a new corpus called the National University of Malaysia Word Tokenization (NUWT) corpora To the best of our knowledge, currently, there is no sufficiently comprehensive, well-designed standard corpus that is annotated and made available for the public for the Jawi script corpora. This corpus contains the Jawi-specific Buckwalter character code and can be used to evaluate the performance of word tokenization tasks, as well as further language processing. The objective of this work is to conform and standardize the corpora between similar characters in Jawi. It consists of three subcorporas with documents from different genres. The gathering and processing steps, as well as the definition of several evaluation tasks regarding the use of these corpora, are included in this paper. One of the important roles and fundamental tasks of the corpus, which is the tokenization, is also presented in this paper. The development of the Malay language tokenizer is based on the syntactic data compatibility of Malay words written in Jawi. A series of experiments were performed to validate the corpus and to fulfill the requirement of the Jawi script tokenizer with an average error rate of 0.020255. Based on this promising result, the token will be used for the disambiguation and unknown word resolution, such as out-ofvocabulary (OOV) problem in the tagging process.

KW - Buckwalter character code

KW - Malay corpora

KW - Regular expression

KW - Word tokenization

UR - http://www.scopus.com/inward/record.url?scp=85010207165&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85010207165&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:85010207165

VL - 15

SP - 107

EP - 131

JO - Journal of Information and Communication Technology

JF - Journal of Information and Communication Technology

SN - 1675-414X

IS - 1

ER -