Home Work 2B
While checking for spelling errors major factors are:
Language Model ie unigram , bigram
Edit distance and confusion matrix
Other errors like phonetics and space
Proper smoothing for words not present in corpus
Preparation
First I pruned the given corpus to get the unigram frequency of each words and then also removed the words with wrong spellings.
I created a script for constructing the confusion matrix from the wrong words.
Unigram
Without phonetics error(only confusion matrix)
This gave an less accurate results.
With phonetics error taken into consideration
The accuracy improved many folds.
Three Top Suggestions for each word are also given
Bigram
With phonetics error taken into consideration
Three Top Suggestions for each word are also given
The python files used are
code unigram
The result files can be seen at:
data unigram files
The python files used are
code bigram
The result files can be seen at:
data bigram files
(can't be inserted as disk quota exceeded)
The python file to run
aspell file