Here we used an N-gram based approach language model for Hindi to perform spell checker.
Our spell checker draws inspiration from Peter Norvig's Spell Checker with the following modifications.
Use of Phonetic Data in calculating edit distance:Most of the times hindi errors which are caused by replacement, occur because of replacement with phonetic similar alphabets.
Restricting to Edit Distance 1 as it is observed that edit distance 1 is good enough for Hindi Language.
For a bigram based model, we consider words which do not exist in the corpus or which occur in error file. First we look at bigram data for all the possible candidates and return the one with maximum frequency. If none of the candidates (with preceding word) are there in bigram data, we report one with highest frequency same as in unigram case.
Since corpus is not big enough(compared to other languages) and generalized (biased for news articles), 3 gram model does not improve performance.
Secondly, since the bigram data is not sufficiently large, it is difficult to find word errors. So we restrict it to non word errors.
Results
File output.txt contains output of our checker when run on hi-spell2.txt
| Total Non-word Errors detected | 206 |
| Total Replaced | 88 |
The aspell directory contains relevant files.
When aspell and our checker were run on hi-spell.txt, following results were obtained.
We have taken into account top suggestion only for both our checker and aspell.
| Correct by Aspell | 148/340 |
| Correct by Our Checker | 13/340 |
The spelling checker when considered with top 3 suggestions, gives approximately 80% accuracy including word errors.
For this sake, I ran my code on a small paragraph with errors already known to me and inspected top 3 suggestion for each word manually.