Training Corpus
Heliohost corpora for Hindi (containing both Hi_Blogs and Hi_Newspapers) was used to train
unigram and
bigram models.
Validation Sets
4 random online hindi newspaper articles were taken as validation set and the parameters were tuned to improve the F-Score result. Mainly, setting the max word length (L parameter used in split function in ngrams.py) to 40 resulted in noticeable improvement in F-Score.
The validation input files used are -
input1,
input2,
input3,
input4.
And, the corresponding ground truth files are -
groundTruth1,
groundTruth2,
groundTruth3,
groundTruth4.
Test Set
The code was finally run on the
test set with corresponding
groundTruth file. There were few alignment mis-match between the 2 files, which were corrected.
Results
| Input |
Model |
Precision |
Recall |
F-Score |
| Validation set | UniGram |
0.891 |
0.929 |
0.91 |
| Validation set | BiGram |
0.897 |
0.944 |
0.92 |
| Test set | UniGram |
0.927 |
0.94 |
0.934 |
| Test set | BiGram |
0.931 |
0.953 |
0.942 |
The complete code used to generate the language models and evaluate the word segmentation can be found on this
link.