Assignment 1 : NLP

Morphological Structure Discovery

I am using pyhton script for comparing the hand segmented test set to the Undivided ++ segmented test set. I removed the unnecessary word by using UNIX command (tr , sed) .This cleard test set was given as input to Undivided++ Script (UnsupervisedWordSegmentation.cpp) which segmented each word of test set. Then I passed this Undivided ++ output and manually segmented word file to a python sccript fot camparision and error analysis.



Undivided++ Parameters

#define SMALL_ROOT_LENGTH 3
#define LOW_FREQUENCY_DROPOUTS 1
#define LOW_FREQUENCY_DROPOUTS_LEARNING 5
#define SUFFIX_CUTOFF_THRESHOLD 50
#define PREFIX_CUTOFF_THRESHOLD 70
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 10
#define WRFR_PREFIX_THRESHOLD 1.5
#define SLS_NORMALIZATION_CONSTANT 5
#define ALLOMORPH_REPLACEMENT_THRESHOLD 3
#define ALLOMORPH_DELETION_THRESHOLD 3
#define ALLOMORPH_ADDITION_THRESHOLD 3
#define PROMOTE_LONG_SEGMENTATION 1
#define PROMOTE_LONG_SEGMENTATION_LENGTH 20
#define INDUCE_OUTOFVOCABULARY_ROOTS 1
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5

The parameters passed to UnsupervisedWordSegmentation.cpp were:
./a.out train_set 1 1 0 1 0

 Error Analysis:

 Unidentified Rules
many words couldn't be correctly segmented because Undivided++ tries to segment the wrods into the minimum root form due to this many times word lost it's original meaning.
साम्राज्यवादियों सा म्राज्यवादियों द
Some words are oversegmented. The reasons for this is due to large number of prefixes and suffixes.

सामर्थ्यशाली सा म र्थ ्य शाली
Some words are undersegmentation.

स्‍वादिष्‍ट
NOTE : Many word are not present in the training set.


Result

 Precision = 39 %
Recall = 38 %
F-score = 38 %

Link

 Test set : test_set
Manually Segmented Test set : test_set.
Training set by undivided++ : training set.
Code : code.
groundtruth-morph : code.
problems-morph : code.