Homework 2

N-gram based Language Models

Language: Hindi

Method

• Unigram model is constructed with smoothing.
• For every word(Both word and non-word) we calculate words in unigram distribution which are at levenshtein distance less than 2 from it, and choose the most probable word. Levenshtein distance of 2 is allowed if both the edits are insertions(this was done considering that half letters in hindi are made of 2 characters)
• Phonetic similarity was added. Words which are phonetically similar are more probable to be changed than other words.

Results

on hi-spell2.txt

Suggestions for each word of the file: suggestions_2.txt

corrected_2.txt: This file replaces the incorrected word with the top prediction of the spell checker

We see that most of the words corrected by aspell in it are also corrected by the spell checker

on hi-spell1.txt

Using phonetic similarity

suggestions_ph.txt: This file gives the suggestions for each incorrect word in the file

corrected_ph.txt: This file replaces the incorrected word with the top prediction of the spell checker

Total Correct corrections: 94/210

Total Correct suggestions: 124/210 (where the correct word appeared in 1 of the 3 top suggetions)

Without using phonetic similarity

suggestions.txt: This file gives the suggestions for each incorrect word in the file

corrected.txt: This file replaces the incorrected word with the top prediction of the spell checker

Total Correct corrections: 85/210

Total Correct suggestions: 121/210 (where the correct word appeared in 1 of the 3 top suggetions)

Aspell got around 160 suggestions correct(compared to our 124)

Difficulties

In hindi words which are very close to each other also have a levenshtein distance of greater than 2 for eg:
अनिश्तिकालीन & अनिश्चितकालीन have a levenshtein distance of 3, although it seems that there is only 1 letter missing!

Similarly there are loads of other words. This problem mainly occurs because half words require 2 charaters and problem persists even after including 2 insertions. Implementing levenshtein distance of 2 or greater is not feasible as it takes lots of time

code