Morphological Structure Discovery
Observations
Part A
Corpus Pruning Trials and Effects
The initial Telugu corpus (from blogs and newspapers) had both the alphanumeric and other special characters, which needed to be removed. For the pruning process a python script was used which replaced both the corrupted text and many other alphanumeric characters awith the newline character.Then the resultant words are sorted by the number of occurrences, which was then used as an input to the program's Undivide++ "UnsupervisedWordSegmentation.cpp".
For cleaning up the corpus,the parameters like Droping the words with frequency less than or equal to 1 (second and third parameter) and The smallest-root length of words to be taken as 5 (first parameter). These parameters were changed, and it was observed that increasing the minimum frequency pruned the dataset but did not significantly improve the segmentation accuracy. And also for the comparison purpose another code was implemented which calculated the HIT,Insertion and deletion frequency to give the precision , recall and F-score .
The dataset of 300 for comparison had to be chosen by us,so care was taken not to include translated words from english to telugu and also the words which are significantly longer in length from others ,and words which are proper names,and this showed a significant improvement than randomly picking the words from the dataset.Initially, the trials with randomly picking the words to compare were giving around 23 % perfect accuracy, but when used with the default parameters given in Undivided++ and all the above said cases were taken care of the accuracy went significantly, Accuracy-40 % , Precision - 82.3 % , Recall -96.5 % and F-score - 88.8 %Undivided++ Parameters Used
The parameters used for Undivided++ which gave the best accuracy were. #define SMALL_ROOT_LENGTH 3 #define LOW_FREQUENCY_DROPOUTS 2 #define LOW_FREQUENCY_DROPOUTS_LEARNING 3 #define SUFFIX_CUTOFF_THRESHOLD 70 #define PREFIX_CUTOFF_THRESHOLD 50 #define COMPOSITE_SUFFIX_THRESHOLD 0.65 #define WRFR_SUFFIX_THRESHOLD 6 #define WRFR_PREFIX_THRESHOLD 1.5 #define SLS_NORMALIZATION_CONSTANT 5 #define ALLOMORPH_REPLACEMENT_THRESHOLD 3 #define ALLOMORPH_DELETION_THRESHOLD 3 #define ALLOMORPH_ADDITION_THRESHOLD 3 #define PROMOTE_LONG_SEGMENTATION 0 #define PROMOTE_LONG_SEGMENTATION_LENGTH 15 #define INDUCE_OUTOFVOCABULARY_ROOTS 0 #define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5 The parameters used while compiling were ./a.out file_name.csv 1 1 0 1 0 As mentioned before,trying to ignore longer strings improved the performance,especially the parameter which had the maximum positive effect was setting PROMOTE_LONG_SEGMENTATION to 0, which reduced a lot of cases of segmentation of long words. And also the parameter LOW_FREQUENCY_DROPOUTS and LOW_FREQUENCY_DROPOUTS_LEARNING largely reduced the number of words considered and improved the performance.