Assignment 2

• Trials and Effects

  I executed the code in two ways:
  1. Took only unigrams as imput data. The Precision came to be 97.3% recall to 93% and fscore 95.3%
  2. Took both bigrams and unigrams. The Precision came out to be 93.2% recall 95.3%.
  
  Then I changed the parameters of L longest word allowed to 25, this increased the Precision to 95.5%.
  Then changed to 28, The Precision came to be 96.4%
  Then changed to 32, the precision came to be 97%.
  On further increasing the precision increased, but this could result in overfitting. As in case of new word our code will try to fit more known words when the newer word would be correct segmentation.
  I also tried changing non known word frequency but the effect was not much.
  I Generated bigram and unigram from the corpus of hindi words (newspapers, blogs and HC corpora).
  
  Result on "given" hindi test set:

  Trial	Precision	Recall	F-Score
  Final	97.09%	95.01%	96.04%

  
  
  
  Result on "new" hindi set:

  Trial	Precision	Recall	F-Score
  Final	98.6%	98.05%	98.33%

  
  
  

  Main errors occured due to:
  
  1. Unseen Words - When unseen words occur, our code tries to fit in known words with larger probability first creating some errors. This can also be understood from overfit.
  
  2. Out of Context segmentation - Sometimes the same words are broken into two words depending on the context of the sentence. Sometimes the while segmentation changes due to the meaning of the sentence
  
  
  
  

• Extra Test Data

  For the test data, the link is Download
  
  
  
  

• Code

  For the code for handling the corpus , the link is Download all code