HA 1:CS671 Natural Language Processing

Vishal Kumar Gupta 10817

Test Data Set Language: Hindi

PART A : Corpus Pruning Trials and Effects.

There were a number of parameters but all of them were not equally significant. It basically depended on the morphological complexity of language. So in order to learn the segmentation only those some parameters were varied as listed in the table and rest of the parameters were kept fixed as mentioned below.

SMALL_ROOT_LENGTH 2

LOW_FREQUENCY_DROPOUTS 1

LOW_FREQUENCY_DROPOUTS_LEARNING 5

COMPOSITE_SUFFIX_THRESHOLD 0.65

WRFR_SUFFIX_THRESHOLD 3

WRFR_PREFIX_THRESHOLD 0.1

The above thresholds detects incorrect attachments using relative frequency. The less they are the more better is as it prevents more and more over-segmentation.

The table below is constructed in the following manner:

1. First segmentation file is prepared using the corpus as mentioned at the bottom of the page.

2. Then a test data set, containing 300 words is hand segmented.

3. Then this hand segmented test data set is cross-checked with those of the segmentation in the corpus and correspondingly all the values are then calculated.

The two parameters ofprime importance are "SUFFIX_CUTOFF_THRESHOLD" and "PREFIX_CUTOFF_THRESHOLD".

During training phase it has been observed that when the parameter "PROMOTE_LONG_SEGMENTATION" is set to "1", the segmentation result is very poor as compared to when its value is set to "0". The reason being that when it is set to "1", it helps in segmenting long words efficiently but since in our corpora, most of the words are medium length or small length (to say less than 10), setting it to 0 works better. The results also clearly demonstrate this fact.Precision is too low in this case as well as recall and F-score.

PART B : Hand Annotated(Segmented) Test Corpus.

PART C.1 : Code for merging corpus and test dataset

PART C.2 : Code for calculating "Precision", "Recall" and "F-Score"

Spelling Errors