corpus pruning and trials
The corpus from blog and newspaper was used in this .the corpus was cleaned using python and bash. Non hindi unicode characters and hindi numerals were removed.apart from this,numbers ,english alphabets,punctuation and words with frequency less than threshold(decided in trials) were removed
Undivided++ was run using the following parametes : ./a.out hindilist.txt 1 1 0 1 0
algorithm tries to learn prefixes and suffixes based on their relative frequency in the corpus while training.
so these parameters were critical and their variation resulted in performance of undivide++ for hindi.
WRFR_SUFFIX_THRESHOLD
WRFR_PREFIX_THRESHOLD
prefix_cutoff_threshold
suffix_cutoff_threshold
promote_long_segmentation
| parameter | value |
| small_root_length | 3 |
| low_frequency_dropouts | 1 |
| low_frequency_dropouts_learning | 1 |
| suffix_cutoff_threshold | 70 |
| prefix_cutoff_threshold | 60 |
| composite_suffix_threshold | 0.65 |
| wrfr_suffix_threshold | 10 |
| wrfr_prefix_threshold | 1.5 |
| sls_normalization_constant | 5 |
| allomorph_replacement_threshold | 3 |
| allomorph_deletion_threshold | 3 |
| allomorph_addition_threshold | 3 |
| promote_long_segmentation | 1 |
| promote_long_segmentation_length | 15 |
| induce_outofvocabulary_roots | 0 |
| induce_outofvocabulary_roots_threshold | 5 |
| hits | 225 |
| insertion | 50 |
| Deletion | 481 |
| precision | 225/275=0.819 |
| recall | 225/706=0.312 |
| f-score | 450/981=0.4587 |
Error cases
The following observations were made while analyzing the errors in final output.1.Non-Hindi words
The non-hindi words in corpus contributes significantly to error cases.The program is not able to learn any morphology of such words , thus segmenting them on the basis of of hindi prefixes and suffixes and leading to wrong structure discovery
example:
कॉम+्बी+ने+शन(undivide++)
कॉम्बीनेशन(hand)
क्+र+िस्ट+ो+फ़र(undivide++)
क्रिस्टोफ़र(and)
2.Over-segmentation if values of suffix and prefix threshold is keft low then words are over segmented,almost at each character boundary.
example:
भवानीपुरा:भव+ान+ी+पुर+ा
चिड़चिड़ेपन:चि+ड़+चि+ड़+े+पन
3.Undersegmentation Sometimes words are not segmented correctly because of the parameter PROMOTE_LONG_SEGMENTATION set to 0.it basically helps in eliminating over segmentation but sometimes it even leads to under segmentation .
example:
प्रति : it is very common prefix in hindi but often this is not segmentation.
undivide ++ doesnot segment प्रति in प्रतिनिघिमण्डल
conclusion
Hindi is much more morphologically complex then many of the languages.While word formation aka संधि non linear changes takes place which is impossible to capture in unsupervised learning program like undivide++ ,results can be improved if somehow program can capture this