The following was removed from the corpus

• English Alphabet & Numerals
• Hindi Numerals
  
• Punctuation (both Hindi and English)
  
• Special Characters
  

Spelling errors were not corrected.

A count of all unique words in the corpus was stored in a file with "frequency word" in each line. Further, the words with frequency less than 3 and with less than 2.5 letters(in Hindi) are discarded. The resulting 83344 frequency-word corpus is what is fed to UnDivide++. All codes are written in python.

The removal of less frequency words improved the results slightly as some spelling errors/misprints and some non-Hindi words were ridden of.
The removal of shorter words had an immense performance improvement by reducing a lot of over segmentation

The WR/FR ratios were tweaked to work best at the values:

• WRFR_SUFFIX_THRESHOLD 12
  
• WRFR_PREFIX_THRESHOLD 5

PROMOTE_LONG_SEGMENTATION was changed to 0 from its default 1. This resulted in a huge performance improvement by segmenting several words which were not being segmented earlier.

Parameters used to call Undivide++:
./a.out 1 1 1 0 0

From the list of 300 test words, 258 existed in our final output (possibly due to removal low frequency and shorter words).

• Correct segmentations: 61.8%
  
• Precision: 74.5%
  
• Recall: 84.3%
  
• F-Score: 79.1%

In Hindi, there are several rules when joining two words for e.g. ा+ा is still an ा or ि becomes ै, ि+अ becomes य ‍‌‌and so on. These rules are not identified by the program.

It is observed that the program has learned prefixes िय, ियां and ियों in cases where it should further segment them into ि+य, ि+यां and ि+यों respectively. Some other undersegmentation errors may occur due to less occurence of root words in the corpus.
तीर्थयात्रियों : तीर्थ+यात्रि+यों (actual): तीर्थ+यात्र+ियों (code)
प्रवृत्तियों : प्र+वृत्त+ि+यों (actual): प्र+वृत्त+ियों (code)
रूपान्तरित : रूप+ान्तर+ित (actual): रूपान्तर+ित (code)

The program does not know that a word is a name of a person or a place and tries to segment it.
for e.g.
राजस्थान is broken to राज+स्थान
रमेशचन्द्र is broken to रमेश+चन्द्र

Some errors also occur because of hindi-ized english words like वेजिटेबल्स, रिसेप्टर्स, टेक्नीशियन and so on. Though surprisingly, the program learns the suffix ्स and breaks
वेजिटेबल्स to वेजिटेबल+्स and
रिसेप्टर्स to रिसेप्टर+्स

A few errors occur because of spelling errors in the corpus.
for e.g.
आपत्तिाजनक
र्दुव्‍यवहार

Link to hand-annotated TEST corpus: manualseg.txt

Link to wrong segmentations by the program: mistakes.txt

Link to mis-spellings in the test corpus: spell-error-Hindi.txt

Link to codes used for the assignment (readme included): codes.zip

Link to groundtruth-morph.txt: groundtruth-morph.txt