The original punjabi language corpus
of size ~200k words was obtained from Heliohost.
Due to presence of a lot of stray characters, it had to be pruned using sed/awk and grep to a wordlist
containing ~74k words. Following cleaning operations were done:
Remove entries with alphanumeric characters.
Remove words consisting of two or less characters.
Remove other characters e.g. /,-, ø, : etc.
The corpus was formatted apropriately so as to be accepted by the Undivide++ executable.
A hand annotated test set of 475 sufficiently long words, taken from the cleaned corpus, was created employing personal knowledge and understanding of the Punjabi language. It can be found here
.
Varying the runtime arguments given to undivide++ only slightly affected the resultant Precision, Recall and F-score values. Amongst internal parameters, promote_long_segmentation seemed to have the most pronounced effect. Low_frequency_dropouts and corresponding learning parameters ere kept to a minimum (1) as higher values were not giving better results. This could have been because of the small size of the corpus due to which many 'good' words were appearing only once or twice.
The parameters in Undivided++ which gave the maximum f score were:
#define SMALL_ROOT_LENGTH 3
#define LOW_FREQUENCY_DROPOUTS 1
#define LOW_FREQUENCY_DROPOUTS_LEARNING 1
#define SUFFIX_CUTOFF_THRESHOLD 70
#define PREFIX_CUTOFF_THRESHOLD 50
#define COMPOSITE_SUFFIX_THRESHOLD 0.65
#define WRFR_SUFFIX_THRESHOLD 6
#define WRFR_PREFIX_THRESHOLD 1.5
#define SLS_NORMALIZATION_CONSTANT 5
#define ALLOMORPH_REPLACEMENT_THRESHOLD 3
#define ALLOMORPH_DELETION_THRESHOLD 3
#define ALLOMORPH_ADDITION_THRESHOLD 3
#define PROMOTE_LONG_SEGMENTATION 1
#define PROMOTE_LONG_SEGMENTATION_LENGTH 15
#define INDUCE_OUTOFVOCABULARY_ROOTS 1
#define INDUCE_OUTOFVOCABULARY_ROOTS_THRESHOLD 5