parsivar:...

Abstract

With the growth of Internet usage, a massive amount of textual data is generated on social media and the Web. As the text on the Web are generated by different authors with various types of writing styles and different encodings , a preprocessing step is required before applying any NLP task .  The goal of preprocessing is to convert text into a standard format that make s it easy to extract  information from documents and sentences. Moreover, the problem is more acute when we deal with  Arabic script -based languages, in which there are some different kinds of encoding schemes ,    different kinds of writing styles and the spaces between or within the words. This paper introduces a   preprocessing toolkit named as Parsivar ,  which is a comprehensive set of tools for Persian text preprocessing tasks . This toolkit performs various kinds of activities comprised of normalization, space correction, tokenization, stemming, parts of speech tagging and shallow parsing. To evaluate the performance of the proposed toolkit, both intrinsic and extrinsic approaches for evaluation have been applied. A Persian plagiarism detection system has been exploited as a downstream task for extrinsic evaluation of the proposed toolkit. The results have revealed that our toolkit outperforms the available Persian preprocessing toolkits by about 8 percent in terms of F1.