Morphological Structure Discovery

Language: Hindi

Tool: Undivide++

Section #0: Groundthruth and problems(Convenor)

1. The groundtruth: GroundTruth-morph.txt
2. The Problems: problem-morph.txt

Section #1: Creating the file for Undivide++

1. First we created a large corpus by clubbing the given corpora, HC corpora and the hindi corpus from CFLIT, IIT Bombay. Here is the python code for combining the CFLIT files, here.
2. The we selected all the terms from the corpus that were a part of the hindi corpus. We simultaneously calculated and then printed the frequency of all the hindi words as undivide++ needs it. The python code for which is here.
3. We also encountered a hardware limitations as the memory was not sufficient to run the entire corpus. We edited the number of words with different values till we reached a limit, the code is here.

Next I ran the obtained frequency through Undivide++. I selected the words and their sandhi viched corresponding given list(list 02) using python code(here). Then I compared each sandhi viched to the correct sandhi viched of each word list02.The number of words found in corpora from list02 were 248 and they are here.

The initial precision was ~16%.

Section #2: Removing Infrequent words

Infrequent word were removed manually or by settings in the parameters. Setting LOW_FREQUENCY_DROPOUTS and LOW_FREQUENCY_DROPOUTS_LEARNING to 3 accomplished this, but only marginal change in efficiency was observed, keeping other parameters constant. Also setting these to 3 avoided a core dump that occured otherwise due to inadequate hardware(memory).

Section #3: Working with parameters in Undivide++

I used all major steps in the computation, for eg: ./a.out example.hin 1 1 1 1 1

After some experimentation, the best results were obatined with the following parameters:

• SMALL_ROOT_LENGTH 3
• LOW_FREQUENCY_DROPOUTS 3
• LOW_FREQUENCY_DROPOUTS_LEARNING 3
• SUFFIX_CUTOFF_THRESHOLD 80
• PREFIX_CUTOFF_THRESHOLD 45
• COMPOSITE_SUFFIX_THRESHOLD 0.65
• WRFR_SUFFIX_THRESHOLD 7
• WRFR_PREFIX_THRESHOLD 1.5
• SLS_NORMALIZATION_CONSTANT 5
• ALLOMORPH_REPLACEMENT_THRESHOLD 4
• ALLOMORPH_DELETION_THRESHOLD 4
• ALLOMORPH_ADDITION_THRESHOLD 4
• PROMOTE_LONG_SEGMENTATION 0
• PROMOTE_LONG_SEGMENTATION_LENGTH 10
• INDUCE_OUTOFVOCABULARY_ROOTS 0
• INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5

1. PROMOTE_LONG_SEGMENTATION set to 0
2. SUFFIX_CUTOFF_THRESHOLD
3. PREFIX_CUTOFF_THRESHOLD
4. WRFR_SUFFIX_THRESHOLD

Section #4: Results and Errors

The Results were:
1. Total Words from the given list, found in the corpus: 248
2. Total Words Correctly segmented: 121
3. Precision: 61.1%
4. Recall: 84.2%
5. F-Score: 72.11%

The error in the text had three basic classes:
• Over-Segmentation: Many words were overly segmented in the text. for eg: व्यंग्यकारों: व+्य+ंग+्य+कार+ों should have been व्यंग्यकारों: व्यंग्यकार+ों. We can control this by adjusting Prefix/Suffix threshold. Setting the SUFFIX_CUTTOFF_THRESHOLD greater than preffix returns better results.
• Under-Segmentation: Many words were not segmented at all or insufficiently segmentented. for eg: निस्तब्धता: निस्तब्ध+ता should have been निस्तब्धता: निस्+तब्ध+ता. We can control this by increasing PREFIX_CUTOFF_THRESHOLD. However increasing it beyond a limit leads to incorrect segmentation at random place in the word.
• Words From Different Languages: Many words from other languages, like english written in hindi, gave erroneous results due to different constructions of words in other languages.

Section #5: Links In Assignment

• The test corpus: CFLIT HC Corpora
• Manual segmentation done for the given file: list02
• Segmentation of the words in the list by Undivide: list02_undivide

  The link to the codes: codes