CS671 Natural Language Processing
HW2B: Spell Checker
Vishal Kumar Gupta 10817
Test Data Set Language: Hindi
Introduction: Spelling errors often happen in languages due to various reasons like proximity of different keys in the keyboard
or due to phonetic reasons. This makes spell checking one of the important task in NLP. Various approaches have been adopted,
probabilistic model being one of them in which errors are dected by modelling the language based on n-grams model(unigram, bigram) etc.
Unigram Model: In this model we consider the occcurence of a given word independent of words in the vicinity.
Spell Checking: For the list of words not in the corpus a possible list fo words at a edit distance of 1 and is constructed.
Based on probability of occurence, suggestions are made.
Results: The test file is obtained by combiniing all the Spell-Checker files from previous group meetings.
Number of errors :680
Number of correct suggestion provided by spell checker:291
Accuracy :43 %
Conclusion:
=>The same word exists in the corpus in different forms resulting in less overall probability of the given word. Since
we are using unigram model, detecting different forms of same word is quite difficult.
Bigram Model: This model is developed based on the fact that we calculate the probability of a word given the previous
(later) word. The code provided by "Prof Katrin Erk" is used for calculating the bigram frequency and probability. Instances
having frequency less than three words have been removed.The bigram data is available here.
In this model words are considered which do not occur in the test file or which occur in the error file. Bigram probability
for words in the edit-distance set is calculated same way using adjacent word in the test case. Bigrams with high frequency
are suggested.
Code for bigram based spell checker. For the file hi-spell.txt(provided) the results are not
good because that data is quite diverse. But for simple sentences, from daily use or picked from internet, the results are
good enough.
Aspell Output for word list obtained from group meetings!
Phonetic distance based Spell checking
Spelling errors also occur due to phonetic similarity. Some examples are as follow:
- स श ष
- ब भ
- र ऋ ऱ ड़
- ँं
- फ़ फ
All possible similar phonetic replacement is considered for edit distance calculation and the same unigram or bigram model is applied.
Some results :
ऱखणा --- रखना
मुऱणा --- मुड़ना