Observations

Corpus Pruning Trials and Effects

The Hindi corpus used from corpora.heliohost.org contains corpora from Hindi Blogs and Newspapers having approximately 15mn words. Another hindi corpus used is provided by CFILT containing approximately 3mn words.

This dataset needs to preprocessed to remove words having :

They need to be arranged in a (frequency word) manner. Also further preprocessing is required to remove short words and words having less frequency. A python script is written to do all the preprocessing described above.

Words having : Undivided++ Parameters Undivided++ gives the freedom to change various parameters to imrpove results depending on the laguage and the corporus used. Following are the parameters that were used to produce the best results :

#define SMALL_ROOT_LENGTH 3
#define LOW_FREQUENCY_DROPOUTS 1
#define LOW_FREQUENCY_DROPOUTS_LEARNING 1
#define SUFFIX_CUTOFF_THRESHOLD 70 //CRITICAL
#define PREFIX_CUTOFF_THRESHOLD 60 //CRITICAL
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 5
#define WRFR_PREFIX_THRESHOLD 1.5
#define SLS_NORMALIZATION_CONSTANT 5
#define ALLOMORPH_REPLACEMENT_THRESHOLD 3
#define ALLOMORPH_DELETION_THRESHOLD 3
#define ALLOMORPH_ADDITION_THRESHOLD 3
#define PROMOTE_LONG_SEGMENTATION 0
#define PROMOTE_LONG_SEGMENTATION_LENGTH 15
#define INDUCE_OUTOFVOCABULARY_ROOTS 0
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5

Undivided++ was run using the following parametes : ./a.out hindicorpus.txt 1 1 0 1 0

Results

From the given list of 300 words to test after removing words that did not exist in the Heliohost corporus, having frequency less that 3 or words having wrong spelling the test corporus reduced to 262 words. Different Evaluation Metrics are as follows :

Accurracy : 72.51 % (190/262 * 100)
Precision : 80.89 %
Recall : 86.055 %
F-Score : 83.39%

Discussion of Error Cases

  1. Over-Segmentatation Errors : There are many cases of over segmentation. Prime reason for this is the prefixes and suffixes in the hindi language are very varied, and the prefixes/suffixes that generally occur in words sometimes don't occur as prefixes/suffixes in words but rather are a part of the word itself. But on probabilistic measures, Undivided++ segments them and segments the word. A few examples for the same are :
    • संपत्तियां:सं+पत्तिया+ं : सं being a common prefix gets segmented here.
    • फ्रैंचाइजी:फ्रैंचाइज+ी :ी being a common suffix to denote feminine terms gets segmented here.
  2. Under-Segmentatation Errors : Some words are prone to under-segmentation. This mainly happens when the parameter PROMOTE_LONG_SEGMENTATION is set to 0. This parameter cannot be set as 1 as it helps a lot in the case of over segmentation. Some examples of this kind of error are :
    • प्रतियोगियों:प्रति+योगियों : योगियों should have been segmented here.
    • आनुवांशिकी:आनुवांशिक+ी : आनुवांशिक should have been segmented here.
  3. Addition of Letters : In the hindi language when root words are joined by prefixes and suffixes 2 or more letters club together and form a single letter or 'matra'. When segmenting Undivided++ does not generally add these letters to the root words and prefixes/suffixes. Hence giving a wrong segmentation. A few examples of the following type are :
    • चिकित्सालयों:चिकित्सा+लय+ों: Here आलय would be the correct segmentation.
    • पूर्वाग्रहग्रस्त:पूर्वा+ग्रह+ग्रस्त: Here आग्रह would have been the correct segmentation.
Links to Hand-Annotated Corpus, Results and Codes

Group Work Links