Part a

Corpse pruning trials and effects

The corpse was taken from magazines and news paper articles do contained a lot of alpha-numeric characters other than hindi which were trimmed by using a python script.The script took a long time to clean the corpse.The same task using bash script is a lot faster and time saving.The corpse after cleaning is here.

Observations

The average values for the following terms were :
Precision=0.876
Recall=0.452
F-score=0.566
The value of precision is higher becuase of large value of SUFFIX_CUTOFF_THRESHOLD and PREFIX_CUTOFF_THRESHOLD. The parameters used for Undivided++ which gave the best accuracy were.
#define SMALL_ROOT_LENGTH 3
#define LOW_FREQUENCY_DROPOUTS 2
#define LOW_FREQUENCY_DROPOUTS_LEARNING 3
#define SUFFIX_CUTOFF_THRESHOLD 70
#define PREFIX_CUTOFF_THRESHOLD 60
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 6
#define WRFR_PREFIX_THRESHOLD 1.5
#define SLS_NORMALIZATION_CONSTANT 5
#define ALLOMORPH_REPLACEMENT_THRESHOLD 3
#define ALLOMORPH_DELETION_THRESHOLD 3
#define ALLOMORPH_ADDITION_THRESHOLD 3
#define PROMOTE_LONG_SEGMENTATION 0
#define PROMOTE_LONG_SEGMENTATION_LENGTH 15
#define INDUCE_OUTOFVOCABULARY_ROOTS 0
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5

The parameters used while compiling were ./a.out file_name.csv 1 1 0 1 0

Error Cases

Over-segmentation

There were many words which were oversegmented as shown in these example.It happened a lot with words which were not from originally from hindi language or had very unusual root words.
Done by tool--- > पं+थन+िर+पे+क्ष

Done by tool--- > पं+थ+निरपेक्ष when SUFFIX_CUTOFF_THRESHOLD 60 PREFIX_CUTOFF_THRESHOLD 50
(Hand annotated)पंथ+निरपेक्ष
Done by tool--- > डि+जि+ट+ल+ीकरण
Done by tool--- >डि+जि+ट+ली+करण when SUFFIX_CUTOFF_THRESHOLD 60 PREFIX_CUTOFF_THRESHOLD 50
(Hand annotated)डिजिटली+करण

Under-segmentation
These were the words which occurred rarely and the prefixes and suffixes of these were also used very rare.
Done by tool--- > राष्‍ट्रपति
(Hand annotated)राष्‍ट्र+पति

Done by tool--- >निरुद्देश्य
(Hand annotated)निर+उद्देश्य

Part b

Link to hand-annotated TESTcorpse.

Part c

Code Snippets
Click here to download the code
Spell check words