Assignment 2

• Trials and Effects

  First of all we generated smoothed unigram and bigram data using all the wrong spellings from the groups.
  Then generated confusion matrix using edit distance between word and non word.
  WITHOUT PHONETICS
  I executed the code in two ways:
  1. Took only unigrams as imput data. The accuracy came to be 39.9%
  2. Took only bigrams . The accuracy came to be 36.09%
  
  
  WITH PHONETICS
  Again in three ways
  1. Took only unigrams as imput data. The accuracy came to be 46.9%
  2. Took only bigrams . The accuracy came to be 42.68%
  3. Took gm of bigram and unigram prob. the accuracy came to be 48.02%
  NOTE:
  I Generated bigram and unigram from the corpus of hindi words (newspapers, blogs and HC corpora).
  I used the same smoothing using norvig.
  To generate confusion matrix, the substitution cost was assumed to be 1 as well as insertion and deletion costs.
  I installed and ran aspell for the words not found in our data.
  
  Result on "hi-spell1.txt" hindi test set:

  Trial	Unigram	Bigram	Uni+Bi gram
  Without phonetics	39.9%	-	36.09%
  With phonetics	46.09%	48.92%	42.04%

  
  
  

  I used uni+bigram and phonetics on second data set. If the word was not in our dictionary then it was evaluated by both our and aspell. The results were stored in two seperate files. Now the most difference was due to multiple (more than 2) edit distance also covered by aspell.
  But most of our results were present in aspell recommendation but not necessarily in top 3.
  But in calculation of accuracy only considered top 2 of both
  The accuracy came to be 28.82%
  
  

• CODE

  For the test data for smoothing(also the confusion matrix and new datas), the link is Download
  For the code for spell1 and spell2, the link is Download