There are three Corpora that I worked with, all provided on the course website. I used the HC Corpora(both blogs corpus and the newspaper corpus) and the Corpus available from CFILT, IIT Bombay. I used these scripts to extract the words stepwise, ensuring that the loss of words is minimized. Once this was done, I used the perl One Liner perl -0777 -lape's/\s+/\n/g' input.txt | sort | uniq -c | sort -nr > output.txt to sort the words, count unique words and then sort them in decreasing order of magnitude. So, by now I had the set of all words and their count and the words stored in a file, along with their count, according to their count. As the next step this file was used as input to the program "UnsupervisedWordSegmentation.cpp". But the program didn't run properly on such a large input size data, and started to consume around 3.5 GB RAM and eventually got "Killed" after running for about 2 hours. So, we needed to reduce the number of words in the input file. We then tried out with the three Corpora individually but still it failed to deliver any result.
So, we planned to remove infrequent words, words with frequency less than 4 as the "program" didn't learn from these words and were useless as far as their importance for improving performance of the Morphological Analisis of other Words was concerned.
Initially the F Score was around 32%.
The changes introduced in the parameters, apart from the default ones, are as follows:
LOW_FREQUENCY_DROPOUTS changed from 1 to 3. So, I ignored the less frequent words.
LOW_FREQUENCY_DROPOUTS_LEARNING changed from 5 to 3. This was done in order to learn more from the comparitively less frequent words. It actually improved performance to a great extent.
SUFFIX_CUTOFF_THRESHOLD and PREFIX_CUTOFF_THRESHOLD were interchanged due to the nature of the language. It also improved performance to a great extent.
WRFR_SUFFIX_THRESHOLD was decreased gradally to control over-segmentation of long words and its value of 7 seemed to yield the optimal results.
ALLOMORPH_REPLACEMENT_THRESHOLD, ALLOMORPH_DELETION_THRESHOLD and ALLOMORPH_ADDITION_THRESHOLD were all changed from 4 to 3 to enable more character change rules learning.
PROMOTE_LONG_SEGMENTATION was changed to 0 ensure that long words with no suffixes/prefixes were not forced to be segmented.
A lot of tests were conducted by changing the values of INDUCE_OUTOFVOCABULARY_ROOTS, but finally changing it to 0 resulted in the best output. Although it resulted in a lot of words not being segmented as they should have been, it ensured that large words are not over-segmented.
These changes resulted in change of F Score to around 75%.
Values of Final Parameters Used
SMALL_ROOT_LENGTH 3
LOW_FREQUENCY_DROPOUTS 3
LOW_FREQUENCY_DROPOUTS_LEARNING 3
SUFFIX_CUTOFF_THRESHOLD 70
PREFIX_CUTOFF_THRESHOLD 50
COMPOSITE_SUFFIX_THRESHOLD 0.65
WRFR_SUFFIX_THRESHOLD 7
WRFR_PREFIX_THRESHOLD 1.5
SLS_NORMALIZATION_CONSTANT 5
ALLOMORPH_REPLACEMENT_THRESHOLD 3
ALLOMORPH_DELETION_THRESHOLD 3
ALLOMORPH_ADDITION_THRESHOLD 3
PROMOTE_LONG_SEGMENTATION 0
PROMOTE_LONG_SEGMENTATION_LENGTH 9
INDUCE_OUTOFVOCABULARY_ROOTS 0
INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5
Section 2: Output Data from Undivide++ and Manually Processed Data
The Output data from Undivide++ can be found here and the manually processed data here.
Section 3: Results
The results observed in the whole process are listed below:
Words from list found in the Corpus: 244
Total Words Correctly Segmented: 135
Precision: 0.85269121813
Recall: 0.670378619154
F Score: 0.750623441397
Section 4: Errors
The errors that occured in the process can be classified broadly as:
Over Segmentation: Many words are segmented excessively when compared to the correct result. For example: कुरूक्षेत्र: क+ुर+ू+क्षेत्र should have been कुरूक्षेत्र: कुरू+क्षेत्र. This can be corrected by adjusting Prefix/Suffix threshold. Setting the SUFFIX_CUTOFF_THRESHOLD greater than PREFIX_CUTOFF_THRESHOLD yields better esults as already mentioned above. It can be improved by setting PROMOTE_LONG_SEGMENTATION to 0.
Under Segmentation: Many words were not segmnted sufficiently. For example: उच्चाघिकारियों: उच्चाघिकारियों should have been उच्चाघिकारियों: उच्च+अघिकारी+इयों. This can be corrected by increasing PREFIX_CUTOFF_THRESHOLD. However, increasing it beyond a limit can have adverse effects. But changing PROMOTE_LONG_SEGMENTATION to 0, has adverse effects on Under Segmentation.
Words from different languages: Due to the presence of words from english language, the segmentation of words is irratic. For example: एनालिस्ट्स: ए+ना+लिस्ट+्स should have been एनालिस्ट्स: एनालिस्ट्स.