Homework 1

This method deals with unsupervised approach to segment the word morphologically. This work [1] gives a good performance on languages which are morphologically less complex like English. Also the author claims that it gives a significantly good performance with the language "Bangla". But in my opinion this method is not designed to give considerably good results with the complex languages like "Hindi". The problem with the "Hindi" is its grammatical complexity like "Sandhi Vichchheda" ( उपरि + उक्त = उपर्युक्त, तेज: + मय = तेजोमय etc. like this a large no. of rules which is not easy to learn) and also in "Hindi" it is possible to write a word in many possible ways. Along with it, no. of suffixes or prefixes in a word occur more than one or two very frequently (अर्थनीतिशास्त्र :अर्थ+नीति+शास्त्र , अर्द्धपारदर्शक :अर्द्ध+पार+दर्शक , अवसरवादियों :अवसर+वादि+यों etc). Apart from adding prefixes and suffixes hindi words are also derived by insertig words which are neither prefix nor suffix. All these complexity make the morphological anlysis of "Hindi" tough. Getting a clean corpora is also a challange.

. In this assignment all text from newspaper, blog and iitb corpora is taken in a single file and segmented using shell script. The accuracy of this mehod can be improved by using a non noisy corpora for learning. As metioned in the paper[1] this method is better designed for detecting prefix attachment rather than suffix attachment. That's why WRFR prefix cutoff is kept quite lower than that of suffix cutoff. Instruction were followed as given on Guidelines.txt(Undivided++ homepage) to run the code. WRFR_SUFFIX_THRESHOLD is varried in the range of 10-3. Initially it was 10 and the words are over segmented because of the relative frequency idea used in the paper , so this parameter is started to be decresed . But it is observed that after a certain value this is not much affecting the segmentation so it is kept constant at 3. Same is applied with the WRFR_PREFIX_THRESHOLD and finally in the last it is kept constant at 0.1. Some more parameters which have been kept constant are listed below.

SMALL_ROOT_LENGTH 2

LOW_FREQUENCY_DROPOUTS 1

LOW_FREQUENCY_DROPOUTS_LEARNING 4

COMPOSITE_SUFFIX_THRESHOLD 0.65

The parameter which affected the result most is PROMOTE_LONG_SEGMENTATION. Making this boolean constant to 0 gives a huge improvement in results(precision get improved from 0.35 to 0.7). This constant is 1 by default. Other parameters are also varied but they were not showing much changes in the results so for final observation they are kept constant at their defualt value. Some results which were good are shown here in the table below :

Serial No.	PROMOTE_LONG_SEGMENTATION	SUFFIX_CUTOFF_THRESHOLD	PREFIX_CUTOFF_THRESHOLD	Precision	Recall	F-Score
1	0	50	60	0.69	0.34	0.46
2	0	30	40	0.75	0.50	0.6
3	0	35	30	0.70	0.46	0.55
4	0	35	25	0.67	0.46	0.54
5	1	50	50	0.34	0.51	0.40