Assignment-1
CS671(NLP)
Morphological Structure Discovery

---

Results

a) Corpus Pruning Trials and Effects

I merged all the three Corpus(Heliohost, data from blogs and data from newspapers) which contained alphanumeric characters and some other non-useful special characters which I pruned using a Python Script. Then I removed words with less than 3 characters(excluding मात्रा) using Python Script again.This pruned set was given as input to another Python Script which counted the frequency of each word. The output was in the format "Frequency Word" which was given as input to UnsupervisedWordSegmentation.cpp

Effects:

1) Removing less frequent words slightly improved the result as certain NonHindi words were removed and certain other words which created a noise in segmentation results.
2) Removing words with lesser number of characters did not have much effect as was expected because they might have led to over segmentation. 3) Making PROMOTE_LONG_SEGMENTATION=0 avoided over-segmentation and hence improved the result a lot(roughly, every 2 out of 10-15 words were corrected)

Undivided++ Parameters

#define SMALL_ROOT_LENGTH 3
#define LOW_FREQUENCY_DROPOUTS 1
#define LOW_FREQUENCY_DROPOUTS_LEARNING 1
#define SUFFIX_CUTOFF_THRESHOLD 70
#define PREFIX_CUTOFF_THRESHOLD 50
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 8
#define WRFR_PREFIX_THRESHOLD 1.5
#define SLS_NORMALIZATION_CONSTANT 5
#define ALLOMORPH_REPLACEMENT_THRESHOLD 3
#define ALLOMORPH_DELETION_THRESHOLD 3
#define ALLOMORPH_ADDITION_THRESHOLD 3
#define PROMOTE_LONG_SEGMENTATION 0
#define PROMOTE_LONG_SEGMENTATION_LENGTH 15
#define INDUCE_OUTOFVOCABULARY_ROOTS 0
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5

The parameters passed to UnsupervisedWordSegmentation.cpp were:
./a.out input 1 1 0 1 0

Statistics:

Precision-74.53
Recall-76.54
F-Score-75.52

Discussion on Error Cases:

Unidentified Rules
Some words couldn't be correctly segmented because of certain rules of "Sandhi" used in Hindi which inserts certain additional characters in the word. e.g.
व्युत्पन्न:व्युत्पन्न (wrong)
व्युत्पन्न:वि+उत्पन्न (correct)
"Sandhi" inserts an extra य which was not observed by our code

Over-Segmentation
The reasons for over segmentation is due to large number of prefixes and suffixes which lead to undesirable segmentation of certain words and due to certain Non-Hindi words present in the corpus.Example:

हाइपरटेंशन:हाइप+र+टेंशन
विश्‍वविद्यालय:विश्‍व+विद्य+ालय (wrong)
विश्‍वविद्यालय:विश्‍व+विद्या+आलय (correct)
Here PROMOTE_LONG_SEGMENTATION creates a huge impact on results.

Under-Segmentation
Makin PROMOTE_LONG_SEGMENTATION=0 leads to undersegmentation. Example:

महत्वाकांक्षाओं:महत्वाकांक्षा+ओं (wrong)
महत्वाकांक्षाओं:महत्व+आकांक्षा+ओं (correct)

---

b) Hand Annotated Test Corpus download
Statistics:
* -> 70
% -> 16
Total -> 257

* marked words are those which were incorrectly segmented by Undivided++
% marked words are those which are either wrongly written or are names or are English words written in Hindi

Important:
Words are less than 300 because I have removed words with frequency less than 3 and words with number of characters less than 3. Also some words were not there in the corpus(as informed by MS Ram) (excluding मात्रा)

---

c) Code download

---

Submited By:
Pranjal Singh
10511