HA 2:CS671 Natural Language Processing
Vishal Kumar Gupta 10817
Test Data Set Language: Hindi
A uni-gram model is prepared to for word segmentation from text. The model is also evaluated for two different data sets picked from internet to show that the model is not undergoing over-fittingand the results so found are good enough. The only parameter to tweak is "L". In my case it is set to 20.The results are as follow:
1). Dataset: hi-wordseg & hi-wseg-gt
Precision = 0.98
Recall = 0.94
F-score = 0.96
2). Dataset: hi-wordseg2 & hi-wseg-gt3
Precision = 0.99
Recall = 0.91
F-score = 0.95
2). Dataset: hi-wordseg3 & hi-wseg-gt3
Precision = 0.98
Recall = 0.94
F-score = 0.96