Home Work 2

Word Segmenter

Mohit Garg (10409)

Part A

Processing Corpus

Corpus Used: Hindi HC Corpora at www.corpora.heliohost.org/

The Hindi corpus is processed to obtain all possible sets of unigrams and bigrams with their respective frequency.
However it should be noted that I have used only unigrams to carry out this experiment as it provided a very good F-Score.

Part B

Approach Used

My approach is based on Norvig's paper, where in probability of occurrence of each word is proportional to it's frequency. All the non-existing words in corpus are discouraged by imposing heavy penalty which is proportional to word length. The probability of occurrence of a string of words is proportional to product of probabilities of individual word. The algorithm is self explained by code which can be retrieved by clicking on code section.

Part C

Results

The following results were obtained on given test data.

Total Relevant segmentation boundaries(TP+FN)	1089
Total Retrieved segmentation boundaries(TP+FP)	930
Relevant & Retrieved segmentation boundaries(TP)	908

Precision, Recall and F-Score

Recall of given sample = 908/1089 = 83.37%
Precision = 908/930 = 97.63%(approx.)
F-Score = 2*Precision*Recall / Precision+Recall = 89.93%

Part D

Code and Other Files

The Code and other relevant files are available here.