LDC Arabic Treebanks and Associated Corpora: Data Divisions Manual

Mona T. Diab; Nizar Y. Habash; Owen C. Rambow; Ryan M. Roth

LDC Arabic Treebanks and Associated Corpora: Data Divisions Manual
Diab, Mona T.
Habash, Nizar Y.
Rambow, Owen C.
Roth, Ryan M.
Center for Computational Learning Systems
Persistent URL:
CCLS Technical Report
Part Number:
Center for Computational Learning Systems, Columbia University
Publisher Location:
New York
The Linguistic Data Consortium (LDC) has developed hundreds of data corpora for natural language processing (NLP) research. Among these are a number of annotated treebank corpora for Arabic. Typically, these corpora consist of a single collection of annotated documents. NLP research, however, usually requires multiple data sets for the purposes of training models, developing techniques, and final evaluation. Therefore it becomes necessary to divide the corpora used into the required data sets (divisions). Unfortunately, there is no universally accepted convention or standard for dividing bulk corpora. This caused different research groups to either define their own divisions (which makes comparison to similar research results difficult) or adopt existing published divisions (which do not adapt as new corpora versions are released). When a new treebank is released, a new division needs to be developed, which may or may not be consistent with the other treebank divisions. This document details a set of rules that have been defined to enable consistent divisions for old and new Arabic treebanks (ATB) and related corpora. These rules have been applied to the currently available LDC Modern Standard Arabic Treebanks (ATB1 - ATB12), the Egyptian Arabic Treebanks (ARZ1 - ARZ8) and the spoken Levantine ATB, and the exact divisions are listed in tables.
Computer science
Information science
Item views
text | xml
Suggested Citation:
Mona T. Diab, Nizar Y. Habash, Owen C. Rambow, Ryan M. Roth, , LDC Arabic Treebanks and Associated Corpora: Data Divisions Manual, Columbia University Academic Commons, .

Columbia University Libraries | Policies | FAQ