Home Work 2B

Spell Checker

Devanshu Arya (10229)

Unigram Model

Words which are at edit distance of 1 and 2 from the given word(not in the dictionary) are arranged on the basis of their probability of occurance in the corpus. Corpus used was previous one which has also been used for the assignment one. Spelling correction has been only done for non word errors. Top word with the probability is being suggested.

Bigram Model

For bigram based model we look at bigram data for all the possible pair of words occuring together and return the one with maximum frequency. If none of the pairs are there in bigram data, then one with highest frequency is reported same as in unigram case.

Adding Phonetic Similarity

Spelling errors also occur because of phonetic similarity in words. There is high amount of irregularity on the use of 'nukta' by different users for same spellings.
For Example: आदरनीय - आदरणीय
सोर - शोर

Results

Since corpus is not large enough and also due to irregularity in hindi language results are not very promising. Unigram model is being tested on the file obtained by combining all the spelling mistakes file by the class and out of 680 words, 290 words are retrieved correcty. This is due to the fact that corpus data is corrupted and the same word can be written differently by different person and sometimes frequency of incorrectly written words is more than the correct one.
File prvided to test "hi-spell2.txt" is a kind of very diverse test data. This method doen't work efficiently on this test set because the corpus is too small to correct the spell errors.But this code has been checked for some simple sentences which is used in our daily life quite frequently and gives good result.
Aspell Result:

Aspell Result

Code

The code is here.