Unsupervised Morphological Analysis

Edits:

18 Aug

• Uploading the spelling mistakes in Hindi. Here.

17 Aug -

• Changed the Run.zip file made available for testing the list using a different dictionary. Specifically, changed the precision, recall, as per the definition used by Dr. Dasgupta.
• Changed the results tab as previously, precision and recall indicated something else completely.
• Modified the hindi manual word segmentation a bit (removed some more English words).
• Removed the less relevant files which were just the extra work done and should not be paid much attention to.
• I have not changed the parameters. The old values are now giving better precision, accuracy and recall.

Introduction

The work started with combining the heliohost corpus and word tokenization. First, a list of all sentences were found (which could later be used to find n-grams), following which the words were extracted from the sentences. Occurence of a newline character, full stop or purna viram marked the end of a sentence. The final word list was obtained in a file named "inp.txt" which was passed to the Undivide program.

An Interesting Observation

The list of words given to us contained both Hindi words and English words (English words written in Hindi, obtained primarily due to the Newspaper corpus). To understand the working of unsupervised morphological analysis model, results were taken once for all the words in the list and once for only the Hindi words.

Corpus pruning trials and effects

• All words whose count were less than 3 were removed.
• Garbage words, such as words beginning with a bindu or chandra-bindu were removed.

UnDivide++ parameters used

The parameters used for Undivided++ which gave the best accuracy are:

#define SMALL_ROOT_LENGTH 3
#define LOW_FREQUENCY_DROPOUTS 1
#define LOW_FREQUENCY_DROPOUTS_LEARNING 1
#define SUFFIX_CUTOFF_THRESHOLD 90
#define PREFIX_CUTOFF_THRESHOLD 90
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 12
#define WRFR_PREFIX_THRESHOLD 5
#define SLS_NORMALIZATION_CONSTANT 5
#define ALLOMORPH_REPLACEMENT_THRESHOLD 4
#define ALLOMORPH_DELETION_THRESHOLD 4
#define ALLOMORPH_ADDITION_THRESHOLD 4
#define PROMOTE_LONG_SEGMENTATION 0
#define PROMOTE_LONG_SEGMENTATION_LENGTH 15
#define INDUCE_OUTOFVOCABULARY_ROOTS 0
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5 

The compiling command:

./a.out inp.txt 1 1 1 1 0 

Precision/Recall/F-score results

Proper nouns and English words were removed manually from the test cases.

Result when including only words:

Total no. of correct calculations - 19
Total no. of incorrect calculations - 81
Exact accuracy = 19.0
Precision = 45.5
Recall = 76.5
F-Score = 57.1
	

Result - Hindi words.

Format = word : manual_segmentation : unsupervised_segmentation

List of wrongly segmented words.

Procedure to test the list in a different dictionary

1. Download this zip file.
2. Use this vocabulary list.
3. Paste the segmented word list with the name "finalSegmentation.txt" into the folder.
4. Type the following commands:

   python result.py
   python score.py
   	

Hand-annotated corpus

Only hindi words.

Code

Download here

Programming language: Python, bash scripting