Ayush Mittal
CS671- Home work Assignment 1
Morphological Analysis of Hindi using Undivide++.
I also used Linguistica files for which can be found at :
Linguistica But I was not able to run it for finding prefixes and suffixes together.
Corpora used for the analysis consisted of Hindi Blogs and Newspaper from heliohost.
Hindi Corpus was also taken from CFILT.
Filtration and Cleaning Data:
I filtered out all the one or two letter words, all non-hindi alphabets and all the special characters
except '-',':'. Finally using sort and uniq commands of linux I sorted the corpora on the basis of the count.
I experimented with different parameters available with the Undivide++ to achieve the highest accuracy of 57.77%.
Final values with respect to initial values are :
SMALL_ROOT_LENGTH 3^M
LOW_FREQUENCY_DROPOUTS 1^M // no need to change this as I have already removed 1 and 2 letters from corpus.
LOW_FREQUENCY_DROPOUTS_LEARNING 2^M
SUFFIX_CUTOFF_THRESHOLD 70 // left unchanged
PREFIX_CUTOFF_THRESHOLD 60 // left unchanged
COMPOSITE_SUFFIX_THRESHOLD 0.65^M // left unchanged
WRFR_SUFFIX_THRESHOLD 6^M // Reduced to 6 from 10.
WRFR_PREFIX_THRESHOLD 1.5^M // left unchanged
SLS_NORMALIZATION_CONSTANT 5^M // left unchanged
ALLOMORPH_REPLACEMENT_THRESHOLD 3^M // left unchanged
ALLOMORPH_DELETION_THRESHOLD 3 ^M // left unchanged
ALLOMORPH_ADDITION_THRESHOLD 3 ^M // left unchanged
PROMOTE_LONG_SEGMENTATION 0^M // Reduced to 0 from 1.
PROMOTE_LONG_SEGMENTATION_LENGTH 11^M // left unchanged
INDUCE_OUTOFVOCABULARY_ROOTS 0 // left unchanged
INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5// left unchanged
Links:
Test data:
here
Undivided++ Output:
here
Undivided++ Output with default parameters:
here
Discussion of Errors:
- Many words from english and other languages were present in the corpus like Helicopter,Electrician, Cenamatographer etc. They segmented wrongly.
- Some compound words were also present which segmented incorrectly.
- Some words didnot segmented at all while some others segmented excessively.
Results:
Out of total 300 words in the test data 20 were not found in the corpus. Total hits : 252 and Total number of insertions: 95 and TOtal deletions 46.
- Precision: 72.62%
- Recall: 84.5%
- F-Score:77.2%
Link to scripts:
scripts