Spell Checker for hindi

Spell checker in Hindi

Sugam Anand 10730

python code for unigram based spell checker

Unigram Model
In this part of the assignment i modeled the language using unigrams.In case of non words errors, i created a possible candidates set for the correct words on the basis of ideas of norwig.The strings were those which are 1 and 2 edit distance away from it .The ranking of suggestions were then done according to their occurence probability

Results
As compared on the test file provided by the tutor:
Number of errors :680
Number of correct suggestion provided by spell checker:278
Accuracy :41 %
Reasons for bad performance

Due to less standardization in hindi.There are lot of errors , incorrect forms , out of language words etc are widely used in the writtrn form of hindi. हजारों (Incorrect according to Tutor file) occurs with high frequency then the correct हज़ारों (with nukta) .Such things can be handled by using quality corpus.
The file provided for testing is highly corrupted , a large percentage of corrected words are not in Hindi wordnet while some of the errors are there in wordnet.So the distinction between correct and incorrect is doubtful.

Words which consist of two different root words fused together also is in high percentage and obviously they will occur in less frequency.These errors can be attributed to bad design of test file.

example
आचारव्यवहार
गायकगायिकाओं

Bigram model

In this part i modeled the language using bigrams.The approach was to find the nonword in the sentence and for that word create the possible corrected words using earlier edit distance metric.Then search all the possible bigrams in the corpus and replace with that word whose bigram frequency is high. python code for bigram based spell checker
file used for testing

file used for evaluation

Aspell Results for the test file is aspell output
usage :cat file.txt | aspell --lang=hi pipe --encoding utf-8

Phonetic distance based Spell checking

The spell checker could be made more smart by adding strings which are phoneticaly nearer to error word. Similiar sounds

स श ष
ब भ
र ऋ ऱ

Norwig's code for english was modifed for hindi and also comparing with ground truth is included