Assignment 1

• Corpus Pruning Trials and Effects

  I took the corpus from Heliost and blogs and newspapers. The corpus had non hindi words and numbers. To remove them I used "sed" commands to prune it. Then catenated the whole corpus into a single file using "cat" command. Then I used python script to generate frequency wordlist. Then further tweaking used "sed" to remove spaces and other non-hindi words and other errors.

  I executed the code in two ways:
  1. Took top 90k words in a file and ran the code on this file. The Accuracy came to be 22/244
  2. Took all the words -5L (The code could not give segmentation for all the words in corpus).The segmentation was generated for 1L words. The accuracy came out to be 61/283.
  
  Then I changed the parameters of LOW_FREQUENCY_DROPOUTS_LEARNING to 3 and WRFR_PREFIX_THRESHOLD to 2, this increased the accuracy further, but not significantly. The change on changing WRFR_SUFFIX_THRESHOLD to 5 was more in comparision to prefix.Accuracy came out to be 107/283
  

  Final Undivided++ parameters used :
  
  #define SMALL_ROOT_LENGTH 3
  #define LOW_FREQUENCY_DROPOUTS 2
  #define LOW_FREQUENCY_DROPOUTS_LEARNING 3
  #define SUFFIX_CUTOFF_THRESHOLD 75 //CRITICAL
  #define PREFIX_CUTOFF_THRESHOLD 55 //CRITICAL
  #define COMPOSITE_SUFFIX_THRESHOLD 0.65
  #define WRFR_SUFFIX_THRESHOLD 6
  #define WRFR_PREFIX_THRESHOLD 1.5
  #define SLS_NORMALIZATION_CONSTANT 5
  #define ALLOMORPH_REPLACEMENT_THRESHOLD 3
  #define ALLOMORPH_DELETION_THRESHOLD 3
  #define ALLOMORPH_ADDITION_THRESHOLD 3
  #define PROMOTE_LONG_SEGMENTATION 0
  #define PROMOTE_LONG_SEGMENTATION_LENGTH 12
  #define INDUCE_OUTOFVOCABULARY_ROOTS 0
  #define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5
  
  
  

  Trial	Precision	Recall	F-Score
  Final	0.54	0.81	0.65

  The parameters used while compiling were
  ./a.out wordlist.hin 1 1 1 1 0
  
  

  Discussion of Error Cases

  Mainly four types of errors occured:
  
  1. Over-segmentation - Long words are prone to over-segmentation. Mostly these are words are english converted to Hindi language. This also happen when prefixes and suffixes very common are extracted even when they are not acting as one. But over-segmentation is reduced when PROMOTE_LONG_SEGMENTATION is set to 0 and by increasing SUFFIX_CUTOFF_THRESHOLD. An example
  
  बॉन्डहोल्डर्स:बॉन+्ड+हो+ल+्डर+्स
  
  परेस्थेटिका:पर+े+स्थ+े+टिक+ा
  
  कालाबाजारियों:काला+बा+जा+र+ियों
  
  पेंसिलवेनिया:पें+सिल+वे+निया
  
  
  2. Under-segmentation - Setting PROMOTE_LONG_SEGMENTATION to 0 and by increasing SUFFIX_CUTOFF_THRESHOLD some words become prone to under segmentation.This may be rectified using a larger segmentation set. eg.
  
  प्रज्जवलित:प्रज्जवलित
  
  आग्नेयास्त्र:आग्नेयास्त्र
  
  आत्मावलोकन:आत्मावलोकन
  
  
  3. Shift in Segmentation - Word is segmented at the wrong place, and not segmented where it was supposed to be. This is due to identification of prefixes and suffixes and it usage at every place. Increasing/Decreasing the threshold of SUFFIX_CUTOFF_THRESHOLD adversely affects the whole test data. Eg
  
  महाप्रबन्धक:महाप्र+बन्+धक
  
  महत्वपूर्ण:मह+त्व+पूर्ण
  
  आसक्तिरहित:आस+क+्ति+रहित
  
  
  4. Shift of characters - Word is segmented but segmented words dont follow rules of grammer of hindi.A single character (generally vowel) shifts to other word . This is because word joining is much more complicated in Hindi in terms of changing the characters joined. Eg
  
  उच्चायुक्त:उच्चा+युक्त
  
  
  
  
  

• Hand Annotated Test Data

  For the hand-annotated test data, the link is Download Hand annotated test data.
  
  
  
  

• Code for handling the corpus

  For the code for handling the corpus , the link is Download all code
  It contains three python scripts and some bash commands.
  The scripts are used to convert regular data into frequency data, take out test case results and match the handwritten results to them.