Domain-specific Chinese word segmentation using suffix tree and mutual information

Daniel Zeng, Donghua Wei, Michael Chau, Feiyue Wang

Research output: Contribution to journalArticle

23 Scopus citations

Abstract

As the amount of online Chinese contents grows, there is a critical need for effective Chinese word segmentation approaches to facilitate Web computing applications in a range of domains including terrorism informatics. Most existing Chinese word segmentation approaches are either statistics-based or dictionary-based. The pure statistical method has lower precision, while the pure dictionary-based method cannot deal with new words beyond the dictionary. In this paper, we propose a hybrid method that is able to avoid the limitations of both types of approaches. Through the use of suffix tree and mutual information (MI) with the dictionary, our segmenter, called IASeg, achieves high accuracy in word segmentation when domain training is available. It can also identify new words through MI-based token merging and dictionary updating. In addition, with the proposed Improved Bigram method IASeg can process N-grams. To evaluate the performance of our segmenter, we compare it with two well-known systems, the Hylanda segmenter and the ICTCLAS segmenter, using a terrorism-centric corpus and a general corpus. The experiment results show that IASeg performs better than the benchmarks in both precision and recall for the domain-specific corpus and achieves comparable performance for the general corpus.

Original languageEnglish (US)
Pages (from-to)115-125
Number of pages11
JournalInformation Systems Frontiers
Volume13
Issue number1
DOIs
StatePublished - Mar 2011

Keywords

  • Chinese segmentation
  • Heuristic rules
  • Mutual information
  • N-gram
  • Suffix tree
  • Ukkonen algorithm

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Information Systems
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Domain-specific Chinese word segmentation using suffix tree and mutual information'. Together they form a unique fingerprint.

  • Cite this