Corpus Pruning Text and Trials.
The initial corpus from Heliohost had a lot of roman numerals and albhabets. I first combined the two corpora(blogs and news) together in one large single corpus. Then I used a python script to prune characters which were different from Hindi characters. I removed the title and other numbers in the begining of line and replaced them with a whitespaces. I then used UNIX tools like sed and grep to prepare a frequency and word list. I sorted it in decreasing order of frequency and stored top 100,000 distinct words in a separate file. Then I removed all the words that had frequency less than 4 and final list was used as input for undivide++.
You can download relevant scripts and commands here
Important Considerations while cleaning the corpus:
Roman Alphabets and Numbers are removed
Words with frequency <=3 are removed.
English Language punctuation marks are removed except '-' which is used to form joint words (sanyukt shabd) in hindi.
There was a significant effect of changing PROMOTE_LONG_SEGMENTATION to 0 after which significantly reduce the unnecessary segmentation of the word. Suffix and prefix cutoff threshold was also increased, so that unwanted splitting of words do not take place. There was about 50% increase in accuracy after these values were tweaked. For more information, see the statistics below.
The parameters used while compiling were ./a.out wordlist.hin 1 1 0 1 0
Discussion of Error Cases
Before studying distribution of error cases, following statistics need to be considered imperatively,
Total words in given test list
300
Total words with frequency >= 4
175
Only these 175 words have been used to draw any inferences.
It is also noteworthy to mention that in these given list of 175 words, a large number of words were of english origin. (e.g. कॉस्मेटिक्स, इंडेक्सेशन etc.)
Some of these words could further be segmented into other meanigful words in Hindi. For example, ब्लॉगस्पॉट can be segmented into ब्लॉग and स्पॉट; both of them being valid words individually in Hindi.
However most of them could not be segmented further. (e.g. कॉस्मेटिक्स). There were 35 such words (out of remaining 175).
We now discuss error cases as follows:
Over segmentation of Words: This refers to splitting of words more than expected. I observed that this was the largest class of error cases in the outcome. One such example of Over-segmenting is splitting of प्रोग्राम्स into प्रो+ग्+राम+्स.
Almost all the English derived words(as described above) were over segmented.(e.g. कॉस्मेटिक्स = कॉ+स्+मे+टिक+्स)
Under segmentation of Words: The number of such error cases were far smaller than over segmentation case, but still very significant. For eg.
प्रतिक्रियाएं is segmented as प्रतिक्रिया+एं instead of प्र+ति+क्रिया+एं(expected).
Mis-placement of segmentation boundary: For eg. परिस्थितिजन्य is segmented as परिस्थि+ति+जन+्य instead of परि+स्थिति+जन+्य
Precision, Recall and F-Score
Total Relevant segmentation boundaries(TP+FN)
222
Total Retrieved segmentation boundaries(TP+FP)
391
Relevant & Retrieved segmentation boundaries(TP)
184
*Excluding words of English Origin
Recall of given sample = 184/222 = 82.88%
As we notice that given test sample is higly skewed with high number of unsegmentable English origin words (which are incorrectly and highly segmented by undivide++), I report precision without taking them into consideration*.
Precision = 184/391 = 47%(approx.)
F-Score = 2*Precision*Recall / Precision+Recall = 2*83*47/47+83 = 60.01%