Methodology

A language model can be used to accomplish a lot of task related to the language. Here we look at how to make a spell-checker based on a N-Gram based language model for Hindi.

Unigram Based Model

For a unigram based model, we consider words which do not exist in the corpus. For such word we construct a list of possible words at an edit distance of 1 & 2 and find which words exist in the corpus and with what probablility. We give 3 top suggestions based on the probablity of them occuring in the corpus.

Bigram Based Model

For the list of words in the edit distance list we got in the previous part, we construct their bigram with adjacent word the in the test case. These bigrams are then searched in the bigram corpus and arranged on the basis of decreasing order of probability.

Adding Phonetic Similarity

Spelling errors also occur because of phonetic similarity in words.

For example: खांचेदार can be written as खाँचेदार due their phonetic similarity.

Also letters like फ़, ण, ब are phonetically similar to फ, न, भ and many more.

The phonetic similarity is accounted for by considering all possible cases where the letter in a word can be replaced by a phonetically simillar word. The list of these words are again subjected to the unigram and bigram models suggested above.

Results

To manually check the results on a big file as given hi-spell2.txt is impossible. Qualitative analysis shows the suggestions given by our code are promising.

Example: पर्लोभन was suggested to be प्रलोभन

चाहिऐ was suggested to be चाहिए

Example with phonetic similarity: आदरनीय was correctly suggested to be आदरणीय.

To match results with "aspell" is a cumbersome task because it suggests spelling correction even for words which are perfectly correct. Quantitative comparision of corrections with aspell is thus not possible.

Link to Code

Codes : Please supply a hindi corpus named 'hindicorpus.txt' with the format "freq word". Run the 'Make' file to obtain the bigram corpus named hindibigramfreq.txt.
Unigram frequency corpus as used in the earliar assignment is used as hindiunigramfreq.txt.
Run the code using:
python spellchecker.py hi-spell2.txt > results.txt

The results.txt file gives suggestions for words for each line. Here is the file for results run on hi-spell2.txt.

Results