Part 1
Corpus was cleaned by using the grep,sed,tr,uniq and sort commands taught in the class. Commands used can be found here.
After test running the undivide++ and analysing the results following parameters were adjusted.
Observations made about the relevant parameters:
SMALL_ROOT_LENGTH 3
It was kept at 3 so that the words are not oversegmented because of the very small root length in our program.
LOW_FREQUENCY_DROPOUTS 1
LOW_FREQUENCY_DROPOUTS_LEARNING 2
LOW_FREQUENCY_DROPOUTS parameter is used to discard the words with the specified frequency from the vocabulary and never used. This was kept at one as single words do not consider any valueable information.
LOW_FREQUENCY_DROPOUTS_LEARNING parameter is used to discard the words with specified frequency at the time of learning phase. Since we have many words with low frequency we keep the parameter to 2.
WRFR_SUFFIX_THRESHOLD 8
WRFR_PREFIX_THRESHOLD 1.5
After doing analysis of the freqFile.txt file from the output of undivide++ it was seen that the the near about correct threshold should be 8 for suffix as many correct suffixes had a WRFR below 8. Prefix threshold was kept to 1.5.
Undivide++ was executed on the cleaned corpus. The ouput obtained is here.
Part 2:
The list provided had lot of english words which were removed as they had no significance in the accuracy of our results. The hand annotated word list is here.
Part 3:
The results were calculated by counting the number of Hits(H), Insertion(I), Deletion(D) for every word in the test list, as explained in the evaluation report provided to us.
Results obtained are as follows:
Precision = H/(I+H)= 46.55%
Recall = H/(D+H)= 81.40%
F-score = 2H/(2H+I+D)= 59.23%
Code:
Code used can be downloaded here.
Spell Check:
File having spelling mistakes can be found here.