Home Work #1
Morphological Structure Discovery
Part A
Corpus Pruning Trials and Effects
First, I merged the data from blogs and newspapers from Heliohost, in a one large corpus file using the following command.
cat Hi_Newspapers.txt >> corpus
cat Hi_Blogs.txt >> corpus
Then to convert the corpus in a clean frequency word file I wrote a Python Script
In initial trials, there was lot of junk words like dates,alphanumeric words,special character were coming but then on improving the python script I was able to prune all non-hindi characters and successfully created a frequency word file. The file is sorted depending upon the frequency of each word. This file is named as "freq_all_4".Then this file was used as an input to the program "UnsupervisedWordSegmentation.cpp".
UnDivide++ parameters used
#define SMALL_ROOT_LENGTH 3
#define LOW_FREQUENCY_DROPOUTS 2
#define LOW_FREQUENCY_DROPOUTS_LEARNING 2
#define SUFFIX_CUTOFF_THRESHOLD 70
#define PREFIX_CUTOFF_THRESHOLD 55
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 6
#define WRFR_PREFIX_THRESHOLD 1.5
#define SLS_NORMALIZATION_CONSTANT 6
#define ALLOMORPH_REPLACEMENT_THRESHOLD 3
#define ALLOMORPH_DELETION_THRESHOLD 3
#define ALLOMORPH_ADDITION_THRESHOLD 3
#define PROMOTE_LONG_SEGMENTATION 0
#define PROMOTE_LONG_SEGMENTATION_LENGTH 12
#define INDUCE_OUTOFVOCABULARY_ROOTS 0
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5
Run command used ./a.out file_name 1 1 1 1 0
Reason for using above values :
- for LOW_FREQUENCY_DROPOUTS and LOW_FREQUENCY_DROPOUTS_LEARNING : When tried with default ie 1(for both) the execution gave error core dumped. And therefore the total matched words in finalSegmentation file and given list were much less. Also the result obtained were not to precise.I increased these two parameters sequentally and the best results were obtained for the above values.
- for SUFFIX_CUTOFF_THRESHOLD : Not much changes were seen on changing this therefore it is nearly kept same.
- for WRFR_SUFFIX_THRESHOLD : On increasing this, many wrong suffixes were used and reducing it much was leaving many valid suffixes unaatended.Therefore, this value is the optimum.
- for PROMOTE_LONG_SEGMENTATION : Maximum improvment was seen on switching this off. It was found that on forcefull segmentation large words were segmented in many parts rather then just relevant parts.
Precision/Recall/F-score results
hits(H) = 305 , insertions(I) = 148 , deletions(D) = 76
- Precision = H/(H+I) = 0.67
- Recall = H/(H+D) = 0.80
- F-score = 2H/(2H+I+D) = 0.73
Discussion of error cases
Error were mainly due to 4 reasons:
- Proper Noun: Proper Noun present in the corpus should not be segmented but these are often segmented by the program. For example words like क्रिस्टिना
- English words written in hindi: These words often causes letters like स(s),िस(es),िंग(ing) as suffixes which leads to problem for partitioning normal hindi words.
- Over-Segmentation :Some words were segmented very poorly. like राष्ट्रगान (र+ा+ष्ट+्र+गान), बुद्धिमतापूर्ण(ब+ुद्ध+िम+ता+पूर्ण) This is mainlt as in hindi many word contains multiple affixes leading to problem.
- Under Segmentation :There were some cases where segmentation ws not done. These were mainlythe ones where either suffix or prefix exist but their frequency is too less that they were discarded in relative frequency step.For example:समीक्षाधीन(समीक्षा+धीन) but neither धीन nor समीक्षा were present much and therefore not recognised as suffix.
Part B
Hand Annotated Test corpus.
For the hand-annotated test corpus, the link is Download
Content:- freq_all_4: frequency words in main corpus
- hindi_pre_suff_hand_clean.txt: hand annotated test corpus
- output.txt: resultant file (segmented by undivide++)
Part C
Code for handling the corpus and the test data
The code(all python scripts used) for handling the corpus and data, the link is Download