Devanshu Arya (10229)
| Serial No. | SUFFIX_CUTOFF_THRESHOLD | PREFIX_CUTOFF_THRESHOLD | PROMOTE_LONG_SEGMENTATION | Precision | Recall | F-Score |
|---|---|---|---|---|---|---|
| 1 | 50 | 40 | 0 | 0.74 | 0.49 | 0.62 |
| 2 | 55 | 45 | 0 | 0.72 | 0.46 | 0.59 |
| 3 | 40 | 40 | 0 | 0.74 | 0.48 | 0.62 |
| 4 | 60 | 50 | 0 | 0.70 | 0.40 | 0.54 |
| 5 | 65 | 45 | 1 | 0.25 | 0.44 | 0.32 |
SMALL_ROOT_LENGTH 3
LOW_FREQUENCY_DROPOUTS 1
LOW_FREQUENCY_DROPOUTS_LEARNING 5
SUFFIX_CUTOFF_THRESHOLD 60
PREFIX_CUTOFF_THRESHOLD 50
COMPOSITE_SUFFIX_THRESHOLD 0.65
WRFR_SUFFIX_THRESHOLD 4
WRFR_PREFIX_THRESHOLD 0.1
SLS_NORMALIZATION_CONSTANT 7
ALLOMORPH_REPLACEMENT_THRESHOLD 4
ALLOMORPH_DELETION_THRESHOLD 4
ALLOMORPH_ADDITION_THRESHOLD 4
PROMOTE_LONG_SEGMENTATION 0
PROMOTE_LONG_SEGMENTATION_LENGTH 15
INDUCE_OUTOFVOCABULARY_ROOTS 0
INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5
SMALL_ROOT_LENGTH,LOW_FREQUENCY_DROPOUTS and LOW_FREQUENCY_DROPOUTS_LEARNING were kept at their respective default values otherwise the words were oversegmented. WRFR_SUFFIX_THRESHOLD and WRFR_PREFIX_THRESHOLD was initialised at 10 but it was observed that on decreasing these values the incorrect attachments of suffixes and prefixes kept on decreasing. WRFR_PREFIX_THRESHOLD was set to its minimum possible value to avoid oversegmentation of words to its prefixes.WRFR_SUFFIX_THRESHOLD, on decreasing improved the segmentation from 10-4 and then on further decrement resulted in undersegmentation of some words. PROMOTE_LONG_SEGMENTATION was the parameter that resulted in obtaining high level of accuracy. Initially it was set to 1 which when set to 0 gave good results. As written in the documentation "Setting PROMOTE_LONG_SEGMENTATION is particularly fruitful for morphologically complex languages (e.g., Turkish or Finnish) where the words tend to be long and concatenative", so by setting the flag for forceful segmentation off for hindi corpus good results were obtained. SUFFIX_CUTOFF_THRESHOLDand PREFIX_CUTOFF_THRESHOLD values were manipulated in hit and trial basis. For some combination of these, the code gave better results which are tabulated above.The corpus formed by merging texts from blog , newspaper and IITB corpus had lot of bugs especially blog and newspaper. It had lot of garbage datas including alphanumeric letters,urdu words, english words and many special characters which had to be removed. For removing these words and making a list of useful words with their frequencies a shell was used. The final output (the frequency of the word, followed by the word itself) was given as input to Undivided++ for segmentation. Initially the results were screwed up which got better with parameter initialisations. Some of the good results were जनमतसंग्रह -जन+मत+संग्रह ; चतुर्वेद -चतुर्+वेद ; सुखसुविधाओं -सुख+सुविधा+ओं ; अतिकुपोषित -अति+कु+पोषित .
One of the reasons of not obtaining very good results was the corpus used for training. The text copied from blog and newspaper were not clean. A single word had 3-4 spellings(आवश्कताएं आवश्कतायें आवश्यकताएंॅ आवश्यकताऐ आवश्कयताएं ) and at some places there were unnecessary spaces between the letters of same word which was considered as two words. Undivided++ is not very well optimised for hindi morphology segmentation. This is because unlike english, hindi words are complex to handle because of their rules of merging with other words ('sandhi') such as वस्त्राभूषण and लक्ष्यार्थ . One of the problem is the frequent use of english words in hindi which makes it possible to handle those words (डिजिटाइजेशन, वॉरंट ).For words with simple addition of prefixes and suffixes with the root word (प्रेरणास्रोत :प्रेरणा+स्रोत ,दुर्गापूजा :दुर्गा+पूजा) the code works perfectly well. Other than adding prefixes and suffixes some of the hindi words are derived from words which are neither a prefix or suffix . These are some of the reasons which makes 'Hindi' a tough language to deal, for morphological analysis.