Two Telugu corpora (Blog Texts and Newspapers) are merged for braoder content in the corpus. First the source, type and topic arguments provided in the corpus are removed. All the alpha-numerics, special symbols and foreign language words (present in the corpus due to imperfection of the language detector, as mentioned) are removed. Finally, the occurences of all the unique Telugu words in the corpus is sent as input to the Undivide++ which then gave the morphological analysis of the words. The entire code is written in bash script using following commands:
A bash script is written to filter out the frequent(>4) words with atleast 10 characters. 10 characters restriction was chosen for the sake of presence of considerable number of morphemes in each test word. It was found by observation that in Telugu, each letter was taking on average 4 non-ASCII characters. So, words with more than 40 (10 lettters with 4 non-ASCII each) non-ASCII characters are extracted. From this set of words, randomly chosen 300 are used as the test set. They are hand annotated to get "groundtruth".
Five more arguments passed through command line: To Run: LOW_FREQUENCY_DROPOUTS - YES. Helped in removing most of the proper nouns, borrowed words (English words written in Telugu) and drastically boosted the performance. PROMOTE_LONG_SEGMENTATION - NO. Since all the test words are long, changing this parameter improved the performance. SUFFIX_CUTOFF_THRESHOLD and PREFIX_CUTOFF_THRESHOLD. Lowered to prevent over-segmentation. ALLOMORPH_THRESHOLD. Decreasing these parameters didn't help. _NO_ orthographic rules were learnt by the system.
Final Performance:
REMARKS: I have used really long words in my test set. Even manually, they were really hard to divide into morphemes. For smaller words, Undivide++ was performing exceptionally well (based on my observation).
Undivided++ Parameters Used -
#define SMALL_ROOT_LENGTH 5
#define LOW_FREQUENCY_DROPOUTS 2
#define LOW_FREQUENCY_DROPOUTS_LEARNING 3
#define SUFFIX_CUTOFF_THRESHOLD 60
#define PREFIX_CUTOFF_THRESHOLD 60
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 5
#define WRFR_PREFIX_THRESHOLD 1.0
#define SLS_NORMALIZATION_CONSTANT 5
#define ALLOMORPH_REPLACEMENT_THRESHOLD 2
#define ALLOMORPH_DELETION_THRESHOLD 2
#define ALLOMORPH_ADDITION_THRESHOLD 2
#define PROMOTE_LONG_SEGMENTATION 0
#define PROMOTE_LONG_SEGMENTATION_LENGTH 15
#define INDUCE_OUTOFVOCABULARY_ROOTS 0
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5
1. Induce Prefixes/Suffixes/Roots - YES
2. Detecting Incorrect Attachments Using Relative Frequency - YES
3. Applying Suffix Level Similarity - NO (No betterment in performance)
4. Inducing Orthographic Rules and Allomorphs - YES
5. Handle Small Roots. - NO (To prevent over-segmentation)
./a.out wordlist.tel 1 1 0 1 0
Results -
Precision: 64.752%
Recall: 67.554%
F-Score: 66.123%
PART B: Resources -
Hand-Annotated test set
Pruned Dataset(Partial)
Undivide++ output on test set
Telugu Spelling Errors List
References -
@article{goldsmith-01-unsupervised-morphology-learning,
title={Unsupervised learning of the morphology of a natural language},
author={Goldsmith, J.},
journal={Computational linguistics},
volume={27},
number={2},
pages={153--198},
year={2001},
}
@inproceedings{dasgupta-ngV-07_language-independent-morphological-segmentation,
title={High-Performance, Language-Independent Morphological Segmentation.},
author={Dasgupta, Sajib and Ng, Vincent},
booktitle={Proceedings HLT-NAACL},
pages={155--163},
year={2007}
}