We score each unigram by its frequency of occurrence in our corpus. The candidate corrections are sorted in the decreasing order of this score and the top 5 are output.
for e.g. for "भाग", unigram score is its frequency which is 4392
We score any given word as the product of its unigram frequency, its left-bigram frequency and its right bigram frequency. The candidate corrections are sorted in the decreasing order of this score and the top 3 are output.
for e.g. for "भाग" in "दो भाग होते", left-bigram frequency is frequency of "दो भाग" which is 39 and right-bigram frequency corresponding to "भाग होते" is 19.
For each error word, the words at unit distance are computed. A list of hindi alphabets is used and a set of replacements, transpositions, deletions and insertions are computed.
for e.g. for भाग - भागा, भार, भोग ...etc. are unit distance words
From ASPELL, we use the hi_phone.dat file for phinding fonetically similar characters. We generate unit distance phonetically similar words as candidate corrections.
for e.g. for भाग - बाग, भग ...etc. are unit distance phonetically similar words
While computing unigram and bigram frequencies, +1 smoothing is applied.
Using the unigram checker on the concatenated spell-checker files of all groups hi-spell1.txt, most of the unit distance spelling errors were corrected (234/252 ~ 96%).
The major chunk of the ones not corrected (90/108 ~ 83%) are those with distance more than one from the correct word like:
कोषिष which is 2 distances away from कोशिश (correct).
Some unit distance errors could not be corrected due to rare words, i.e. words not in the corpus like:
आलोचनाकर्म - आलोचनाकर्मी
गैरजिम्मेदार - गैरज़िम्मेदार
निशानदार - निशानेदार
बांकेबाजार - बांकेबाज़ार
Some were not corrected because they were equally close to words which are more common which made the actual word fall out of the top 5 limit
घाटक - नाटक, घाटा, घाट... push back the actual correct word घातक
नास - नाम, पास, ना, खास...etc push back नाष
Results (blank lines to denoted non-corrected unit distance errors): hi-spell1_corr.txt
The bigram checker gave good enough results on the hi-spell2.txt file. Using ASPELL to compare the results seemed to be a cumbersome task since it gave 6 suggesstions for each word, whether it is erroneous or not.
On inspection, we find that errors like
दंतमूल to दन्तमूल
रहनेवाला to रहने वाला
धारवाले to धार वाले
सँकरी to संकरी
क़रीब to करीब
आदरनीय to आदरणीय
are corrected successfully.
Some words in the test file did not exist in our corpus and hence, could not be corrected. Several of such words were however, morphologically segmented
रुधिरवाहिकाएँ to रुधिर वाहिकाएँ
दंतशिखर to दंत शिखर
हाथीदांत to हाथी दांत
संधिस्थल to संधि स्थल
Results: hi-spell2_corr.txt
H. Codes
Link to codes used for the assignment (readme included):
codes.zip