Homework 2 B 

Anant Raj   Roll No - 10086

Spell-Checker

Introduction

Spelling correction is a very important task in NLP. A probabilistic language model is build to accomplish many purpose in NLP tasks. In this assign we develop languge models (Unigrams, Bigraams) for checking spelling errors.

Unigram Model
For words which do not exist in the corpus we construct the possible list of such words which are at edit distance of 1 and 2 from the given word. Then suggestions are made on the basis of their probability of occurance. The corpus used for finding the frequency of words is Newspaper+Blog+IITB corpus.
Results
As compared on the test file obtained obtained by combining all the SPELL-CHECKER files from each of the group meetings
Number of errors :680
Number of correct suggestion provided by spell checker:291
Accuracy :43 %
Reason for bad results
1. Differnt form of same word exists and each form is used quite frequently by different users and sometimes incorrect forms occur more frequently then correct one.
2. There are errors in file provided.
3.Some words consist of fusing two different words and obviously their frequency in the corpus will be quite less.
Bigram Model

Also analysis has been carried out for checking spelling error by using Bigram Language model. For getting the bigram count the code given by "Prof Katrin Erk" is used. Pair of words which have frequency less than or equal to 3 has been removed from the file. This biagram file has already been uploaded in the first part of the Assignment. For a bigram based model, those words are considered which do not exist in the corpus or which occur in error file. For the list of words in the edit distance list we got in the previous part, their bigram probability is caalculted with adjacent word the in the test case. That biagram is being suggested which has more frequency in the biagram data.

python code for bigram based spell checker
File prvided to test "hi-spell2.txt" is a kind of very diverse test data. This method doen't work efficiently on this test set. But this code has been checked for some simple sentences which is used in our daily life quite frequently and gives good result.
Aspell Results for the test file obtained obtained by combining all the SPELL-CHECKER files from each of the group meetings :
aspell output
usage :cat file.txt | aspell --lang=hi pipe --encoding utf-8

Phonetic distance based Spell checking

Spelling error also occurs due to phonetic similarity of sounds. Some examples of phonetic similar words where replacement can occur in between :
All possible similar phonetic replacement is considered for edit distance calculation and the same unigram or bigram model is applied. Some results :
ऱखणा --- रखना
मुऱणा --- मुड़ना

Links to all files