CS671: Natural Language Processing

Department of Computer Science & Engineering, IIT Kanpur

Jul - Nov 2013

Morphological Structure Discovery

Observations

Part A

Corpus Pruning Trials and Effects

The initial Telugu corpus (from blogs and newspapers) had both the alphanumeric and other special characters, which needed to be removed.
For the pruning process a python script was used which replaced both the corrupted text and many other alphanumeric characters awith the newline character.Then the resultant words are sorted by the number of occurrences, which was then used as an input to the program's Undivide++ "UnsupervisedWordSegmentation.cpp".

For cleaning up the corpus,the parameters like Droping the words with frequency less than or equal to 1 (second and third parameter) and The smallest-root length of words to be taken as 5 (first parameter). These parameters were changed, and it was observed that increasing the minimum frequency pruned the dataset but did not significantly improve the segmentation accuracy. And also for the comparison purpose another code was implemented which calculated the HIT,Insertion and deletion frequency to give the precision , recall and F-score .

The dataset of 300 for comparison had to be chosen by us,so care was taken not to include translated words from english to telugu and also the words which are significantly longer in length from others ,and words which are proper names,and this showed a significant improvement than randomly picking the words from the dataset.Initially, the trials with randomly picking the words to compare were giving around 23 % perfect accuracy, but when used with the default parameters given in Undivided++ and all the above said cases were taken care of the accuracy went significantly, Accuracy-40 % , Precision - 82.3 % , Recall -96.5 % and F-score - 88.8 %

Undivided++ Parameters Used

The parameters used for Undivided++ which gave the best accuracy were.
#define SMALL_ROOT_LENGTH 3
#define LOW_FREQUENCY_DROPOUTS 2
#define LOW_FREQUENCY_DROPOUTS_LEARNING 3
#define SUFFIX_CUTOFF_THRESHOLD 70
#define PREFIX_CUTOFF_THRESHOLD 50
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 6
#define WRFR_PREFIX_THRESHOLD 1.5
#define SLS_NORMALIZATION_CONSTANT 5
#define ALLOMORPH_REPLACEMENT_THRESHOLD 3
#define ALLOMORPH_DELETION_THRESHOLD 3
#define ALLOMORPH_ADDITION_THRESHOLD 3
#define PROMOTE_LONG_SEGMENTATION 0
#define PROMOTE_LONG_SEGMENTATION_LENGTH 15
#define INDUCE_OUTOFVOCABULARY_ROOTS 0
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5

The parameters used while compiling were
./a.out file_name.csv 1 1 0 1 0
As mentioned before,trying to ignore longer strings improved the performance,especially the parameter which had the maximum positive effect was setting PROMOTE_LONG_SEGMENTATION to 0, which reduced a lot of cases of segmentation of long words.
And also the parameter LOW_FREQUENCY_DROPOUTS and LOW_FREQUENCY_DROPOUTS_LEARNING largely reduced the number of words considered and improved the performance.

Discussion of Error Cases

All errors can be categorized into mainly two types of errors (or a combination of these) :-

Over-segmentation - The long words are prone to be over-segmented , often words like కమ్యూనిస్టుల:క+మ్+యూ+ని+స్టు+ల (translation:communist)a word which got translated to telugu with no changes,tend to be segmented into almost each letter. But over-segmentation is reduced to a great extent when PROMOTE_LONG_SEGMENTATION is set to 0, as expected.

-- ముస్లిముల:ము+స్+లి+ము+ల (translation:Muslim)

అవకాశాలుంటాయి:అవకాశ+ా+ల+ుంట+ాయి
In most cases, many extra alphabets are to be introduced, it fails, like in the above. (The correct segmentation would be)
-- అవకాశాలుంటాయి:అవకాశాల+వుంటాయి

Under-segmentation - Some words are prone to under-segmentation, when the parameter PROMOTE_LONG_SEGMENTATION is set to 0,as the suffixes are not recognized sufficiently.

డబ్బులు:డబ్బులు అభ్యున్నతికి:అభ్యున్నతికి

where the correct segmentation would be

డబ్బులు:డబ్బు+లు అభ్యున్నతికి:అభ్యున్నతి+కి

Incorrect segmentation -
మూడేళ్లు:మూ+డే+ళ్లు

Here, it is segmented at the wrong place, and not segmented where it was supposed to be.Increasing/Decreasing the threshold of SUFFIX_CUTOFF_THRESHOLD adversely affects the whole test data, and create more false positives/negatives so this was the optimum amount.

Part B

Hand Annotated Test Data

For the hand-annotated test data, the link is Download Hand annotated test data.
And here is the full list of segmented data returned by Undivide++ .
This is the initial data set after basic pruning and cleaning for junk.

Part C

Code for handling the corpus

For the code for handling the corpus (pruning the corpus), the link is Download all code