for part 2b part b spell checker
In this assignment i tried to implement a word segmentor for hindi on the ideas of Peter Norwig.The segmentor uses the unigram frequencys of the segmented words to check whether they are the word in vocabulary.The corpus used for creating frequency list is from IIT Bombay.The total number of words were 5726784. The length of longest word in corpus was found to be 132
Example segmentation
input :आमआदमीसेलेकरखासआदमीऔरनेतासेलेकरअभिनेतातकउनकेप्रशंशकऔरअनुयायीथे
output: आम आदमी से लेकर खास आदमी और नेता से लेकर अभिनेता तक उनके प्रशंशकऔरअनुयायीथे
correct:आम आदमी से लेकर खास आदमी और नेता से लेकर अभिनेता तक उनके प्रशंशक और अनुयायी थे
note : remove the non hindi unicodes to improve the performance
For given test corpus
Hindi sentence file
Hindi ground truth file
Approach 1(based on actual word comparision)
The list of words segmented on the basis of segmentor was compared(basic string comparision) with the set of correct words for that sentence .
true_in_mytrue :768
mytrue :894
true :1156
precision :85.91%
recall :66.43%
Approach 2(based on the Sajib & Dasgupta's work)
In this ,i compare the word boundaries as suggested by das and sajib .
For given Unknown corpus contributed by me (taken from bbc hindi blog)
it has around 525 words and 27 sentences
Hindi sentence file
Hindi ground truth file
Approach 1(based on actual word comparision)
The list of words segmented on the basis of segmentor was compared(basic string comparision) with the set of correct words for that sentence .
true_in_mytrue : 276
mytrue : 333
true : 525
precision : 82.8828828829
recall : 52.5714285714
Approach 2(based on the Sajib & Dasgupta's work)
In this ,i compare the word boundaries as suggested by das and sajib