Word segmentation is the problem of dividing a string of written language into its component words. In this work I have primarily used the idea described in Peter Norvig's Chapter 14 in the book 'Beautiful Data'. The idea is to use the knowledge of n-grams and probabilistic inference to guess the word boundaries!
I combined all the three Hindi corpus available to us. Then using the python program getSentence.py and getSentence2.py written by me, I extracted the sentences from the corpus. Then words were simply obtained from the corpus by using getWords.py and bigrams and trigrams were extracted from the Sentence data collected from the corpus using getGram2.py and getGram3.py. In case of the starting of sentence the tag "S within <>" was used.
Trial and effects -
After obtaining the unigrams and bigrams for Hindi, I ran the Peter Norvig's code to segment the Hindi sentences. The result obtained are as follows: Unigram (F-score) - 95% and Bigram (F-score) - 96%
Obsevation -
There were some problems in the ground truth given to us. I modified them manually. However, this is not a necessary step but just an independent effort on my side.
Along with what was already given to us for testing, I pulled out some more Hindi articles from Wikipedia, from different fields (Science, News, Descriptive Essay, etc). They all are present in my code folder (attached below) in the name of (test1.txt, .., test5.txt).
Different kind of modifications were made by me at different point of time while testing the different validation data.
Rules for punctuation: There were some segmentations such as: हे नारी रत्न! मेरी बात मान लो : हे नारी रत्न ! मेरी बात मान लो ..... : . . . . . हस्ताक्षरित की गई.[8][9] : हस्ताक्षरित की गई. [8] [9] Now these were not really errors in the context of word segmentation. Therefore, I created rules for punctuation symbols classifying them as those having space after them, those having space before them and those not having space before them if there occurs a punctuation symbol before them. The punctuations were considered as word boundaries and while printing it was printed according to the standard punctuation rules.
Dealing with foreign words: The characters which didn't belong to Hindi were considered as word boundaries and printed as they occured in the input. For example, in वहीं आरोह में 7 और अवरोग में 6 स्वर होने पर ‘सम्पूर्ण षाडव जाति’ बनती है, 7 and 6 are foreign and in इस सिद्धांत ने क्रन्तिकारी द्रव्यमान-ऊर्जा सम्बन्ध E=mc2 दिया, E=mc2 is foreign and therefore printed as it occured in the input.
Low probability for not possible words: Some words were segmented as स्टोर ्टिंग. Clearly ्टिंग is not a word. Now for the Hindi language we can assign some rules to not include such type of word segmentations. Therefore, I extended the code to give the lowest possible probability to the words which are not possible to occur in Hindi (words starting with bindu or halant). The probability assigned is 10./(N * 10**LONGEST_WORD).
Trigram analysis: I extended Dr. Norvig's code to include a trigram model for word segmentation. However, the recursive implementation didn't give a nice result and it was taking a lot of time which made optimizing and debugging this model impossible.
Longest word size: After testing for the various validation set, changing the longest word size from 10 to 15 to 20 to 25 to 30, 20 was chosen to be the best one. However, 30 gave better results if only an unigram analysis was done.
Roughly all the test files had a precision, recall or f-score above 95%
Score obtained for the test set provided to us:
Total no. of correct calculations - 49 Total no. of incorrect calculations - 19 Exact accuracy = 72.06 Precision = 99.07 Recall = 98.80 F-Score = 98.93
There were some errors like: