Assignment 1

Morphological Structure Discovery

Language: Hindi

Corpus Pruning trials and effects

Corpus from corpora.heliohost.org was taken and prune_corpus.bash was run on it to convert corpus into a form taken as input by Undivide++. Initially around 200000 distint words were present. They were pruned to around 88000 words as follows:

• Keep only hindi letters.
• Remove words with frequency lesser than 4.
• Remove words with less than 3 characters.

Results

F-Score: 0.754545
Precision: 0.786730
Recall: 0.724891

finalOutput.txt contains the final output of the code contains comparisons made by code between actual segmentation and segmentation made by Undivide++.
For each word it gives:

• Correct Segmentation
• Undivide++ Segmentation
• Insertions, deletions and hits for that word
• Cumulative insertions, deletions and hits till that word

Undivide++ Parameters Used

Undivide++ was run with parameter: 1 1 1 1 0. Following are the various parameters used and how they were tuned:

• LOW_FREQUENCY_DROPOUTS_LEARNING: 16. Uncessary prefixes and suffixes were being learned with low frequency words. Also meaningless root words were being formed with low frequency data. Increasing this parameter gave significant improvements in results.
• SUFFIX_CUTOFF_THRESHOLD: 300. Suffixes were tracked from topSuffixes.txt file generated by Undivide++. A lot of unwanted suffixes were generated at a low frequency. This threshhold was increased till most of the suffixes in topSuffixes.txt were meaningful.
• PREFIX_CUTOFF_THRESHOLD: 400. Prefixes were tracked from topPrefixes.txt file generated by Undivide++. A lot of unwanted Prefixes were generated at a low frequency. This threshhold was increased till most of the prefixes in topPrefixes.txt were meaningful.
• WRFR_SUFFIX_THRESHOLD: 8. Thresholds for various words were monitored from freqFile.txt and chosen after carefully studying them.
• WRFR_PREFIX_THRESHOLD: 0.2. No. of spurious prefixes were very large compared to suffixes, so this threshold was kept low so as to filter other prefixes.
• PROMOTE_LONG_SEGMENTATION: 0. Undivide++ often segmented long words when infact they can't be. Hence setting this parameter was an immediate improvement.
• Rest of the parameters were kept at their default value

Discussion of Error Cases and Results

• Undivide++ surprisingly learnt a lot! It produced some nice segmentations like:
  • परिपूर्णता:परि+पूर्ण+ता
  • प्रदर्शनकारी:प्र+दर्शन+कारी
  • प्रेरणास्त्रोत:प्रेरणा+स्त्रोत
  • विघ्नविनाशक:विघ्न+वि+नाश+क
  • अनुशासनात्मक:अनु+शासन+ात्मक
  • प्रतिनिधिमंडलों:प्रति+निधि+मंडल+ों
  and much more.
• Undivide++ was not able to learn any of the hindi "sandhi" rules like:
  प्रत्याशियों : प्रति+आशियों (correct)
  प्रत्याशियों : प्र+त्या+शि+यों (Produced by Undivide++)
• Over-segmentation: Over segmentation was not too much. But in some cases it segmented each letter of the word Eg.
  प्रश्नोत्तर:प्रश्+न+ो+त+्+त+र
• Under-segmentation: It was also not too much but Undivide++ was not able to segment words if part of them doesn't appear in the vocabulary or when 2 words are merged by sandhi rules eg. प्रदेशाध्यक्ष:प्रदेशाध्यक्ष (words merged by sandhi rule) मोतियाबिंद:मोतियाबिंद (no part of this word comes independently in the vocabulary)
• Sometimes treats "halant" seperate from the consonant like in:
  पूर्वोत्तार:पूर्व+ो+त+्+तार

Code and other Files

• Hand annotated file: hand_annotate.txt
• README
• Download hw1_code.zip