Heliohost corpora for Hindi (both Hi_Blogs and Hi_Newspapers) was used to train the UnDivide++ tool. A python script was written to split the corpus at every non-devanagari char to obtain only relevant words (hence removing foreign/Roman-script words). The same script also kept track of the frequency of each Hindi word with length >= 4 (the number 4 gave the best F-Score on running UnDivide++, in the range of 1-5). I retained the low frequency words too since these could be either used or discarded by UnDivide++ later, based on parameter tuning.
The test word list used was "list 32.txt". It contained many transliterated English words in devanagari script (ex: co-ordinator, architect, attendant, ...) and proper nouns such as Argentina, which I didn't count in the evaluation. Also, there were quite a few words which weren't present in the Heliohost corpora which I removed before annotating the morphological structure by hand (to the best of my ability) in UnDivide++ format. The hand-annotated test corpus can be found on this
link
Initially, on running UnDivide++ on the training corpus with the default parameter settings, I obtained a low accuracy of ~30% on comparing with the hand-annotated test set. Most of the errors were caused by over-segmentation since the test words list were quite long in size, thereby resulting in a large number of segments. Setting the 3rd command line arguments (Applying Suffix Level Similarity) improved the recall by a very small fraction, and hence the defaults were retained.
However, on tweaking the UnDivide++ parameters to the values shown below, huge improvement in accuracy was achieved. In particular, setting PROMOTE_LONG_SEGMENTATION (forcibly addign extra segments for longer words) to 0 greatly helped reduce the segmentation to about the right number. Also, gradually increasing the suffix/prefix threshold to downsize the respective lists and increasing the minimum size of root to 5 and minimum frequency of words to 2 had a noticable improvement in reducing small size (1-2 character) erroneous segments.
Parameters which achieved one of the best F-Score:
Command line arguments: 1 1 0 1 0
- SMALL_ROOT_LENGTH 5
- LOW_FREQUENCY_DROPOUTS 2
- LOW_FREQUENCY_DROPOUTS_LEARNING 3
- SUFFIX_CUTOFF_THRESHOLD 70
- PREFIX_CUTOFF_THRESHOLD 70
- COMPOSITE_SUFFIX_THRESHOLD 0.65
- WRFR_SUFFIX_THRESHOLD 5
- WRFR_PREFIX_THRESHOLD 1.5
- SLS_NORMALIZATION_CONSTANT 5
- ALLOMORPH_REPLACEMENT_THRESHOLD 3
- ALLOMORPH_DELETION_THRESHOLD 3
- ALLOMORPH_ADDITION_THRESHOLD 3
- PROMOTE_LONG_SEGMENTATION 0
- PROMOTE_LONG_SEGMENTATION_LENGTH 15
- INDUCE_OUTOFVOCABULARY_ROOTS 0
- INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5
|
Results
- Accuracy: 74.73% (139/186)
- Precision: 0.755
- Recall: 0.815
- F-Score: 0.784
|
Error cases:
- Over-Segmentation: These can be handled by increasing suffix/prefix threshold, but it would inversely affect the precision. (ex: निर्यातकों : निर्यात+ क+ों and सामान्यतया : सामान+्य+त+या)
- Under-Segmentation: Conversely, these can be handled by decreasing suffix/prefix threshold, or easing on minimum size/frequency restrictions, but it could cause a drop in recall due to occurance of 1-2 character segments. Most of the cases in this category weren't segmented at all (i.e, the segment was the word itself). (ex: पिण्डलियों : पिण्डलियों)
- Others: Presence of transliterated English words with hindi suffixes (ex: रेस्तरांओं), and abbreviations (ex: आरडीसीआईएस )
Part-C: Links