Part A
Corpus Pruning and Test Corpus
To increase the performance of the code, the dataset was preprocessed after extracting all the (clean) words(~200k) from the dataset provided
here. All the words with frequency less than 3 and all the words with less than 3 characters were discarded. The worlist reduced to ~70k size.
Wordlist used to run code:
wordlist.pun
A test corpus was prepared by extracting words from the wordlist above that are: a) greater than 10 unicode characters in length, b) occur more than 3 times in the corpus, using the python script
extract.py. These words were randomly sorted and 306 out of them were segmented morphologically by hand using the basic knowledge of the language.
The Test Corpus resides here:
test_cases.txt
Part B
Results
Bash Script to run the analysis:
pipelining_script.sh
Auxillary pyhton scripts:
find_compare.py and
prec_rec_f.py. These reside in the same directory as the executable.
Experiments
The output was compared for different parameters in the UnDivided++ code. The results of which are shown in the table below.
Precision and Recall: Precision and Recall were computed using the formulae:
Precision = (H) / (H+I)
Recall = (H) / (H+D)
where H is the number of Hits (i.e., correctly placed boundaries), and I, D represent the
number of morpheme boundaries needed to be inserted into and deleted from correct segmentation to make it look like proposed segmentation.
[Dasgupta et al.]
Observations from Parameter Tuning in Undivided++
| SMALL
ROOT LENGTH |
LOW FREQUENCY
DROPOUTS |
LOW FREQUENCY
DROPOUTS LEARNING |
SUFFIX CUTOFF
THRESHOLD |
PREFIX CUTOFF
THRESHOLD |
COMPOSITE
SUFFIX THRESHOLD |
WRFR SUFFIX
THRESHOLD |
WRFR PREFIX
THRESHOLD |
SLS
NORMALIZATION CONSTANT |
ALLOMORPH
REPLACEMENT THRESHOLD |
ALLOMORPH
DELETION THRESHOLD |
ALLOMORPH
ADDITION THRESHOLD |
PROMOTE LONG
SEGMENTATION |
PROMOTE LONG
SEGMENTATION LENGTH |
INDUCE
OUTOFVOCABULARY ROOTS |
INDUCE OUT OF
VOCABULARY ROOTS THRESHOLD |
Options |
Precision |
Recall |
F-score |
| 10 |
2 |
3 |
70 |
70 |
0.4 |
5 |
1.2 |
6 |
3 |
3 |
3 |
0 |
15 |
1 |
15 |
11010 |
0.705667 |
0.66782 |
0.68622 |
| 10 |
2 |
3 |
70 |
70 |
0.4 |
21 |
0 |
6 |
3 |
3 |
3 |
0 |
15 |
0 |
15 |
11010 |
0.707182 |
0.66436 |
0.6851 |
| 10 |
2 |
3 |
70 |
70 |
0.4 |
10 |
1.2 |
6 |
3 |
3 |
3 |
0 |
15 |
0 |
15 |
11010 |
0.705882 |
0.66436 |
0.68449 |
| 3 |
2 |
3 |
70 |
70 |
0.5 |
5 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
11010 |
0.704587 |
0.66436 |
0.68388 |
| 3 |
2 |
3 |
70 |
70 |
0.5 |
5 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
11111 |
0.704587 |
0.66436 |
0.68388 |
| 3 |
2 |
3 |
70 |
70 |
0.5 |
5 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
10000 |
0.694946 |
0.66609 |
0.68021 |
| 3 |
2 |
3 |
80 |
70 |
0.4 |
4 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
11010 |
0.696364 |
0.66263 |
0.67908 |
| 3 |
2 |
3 |
80 |
70 |
0.4 |
4 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
11111 |
0.696364 |
0.66263 |
0.67908 |
| 3 |
2 |
3 |
70 |
70 |
0.4 |
4 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
11010 |
0.69708 |
0.6609 |
0.67851 |
| 10 |
2 |
3 |
70 |
70 |
0.4 |
5 |
1.2 |
6 |
3 |
3 |
3 |
1 |
30 |
0 |
5 |
11010 |
0.713193 |
0.64533 |
0.67757 |
| 3 |
2 |
3 |
70 |
50 |
0.65 |
6 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
15 |
11010 |
0.709924 |
0.6436 |
0.67514 |
| 3 |
2 |
3 |
70 |
50 |
0.65 |
6 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
15 |
11111 |
0.709924 |
0.6436 |
0.67514 |
| 3 |
2 |
3 |
70 |
70 |
0.4 |
4 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
10000 |
0.684588 |
0.6609 |
0.67254 |
| 3 |
2 |
3 |
70 |
50 |
0.65 |
6 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
15 |
10000 |
0.701128 |
0.64533 |
0.67207 |
| 3 |
2 |
3 |
80 |
70 |
0.4 |
4 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
10000 |
0.683363 |
0.6609 |
0.67194 |
| 10 |
2 |
3 |
70 |
70 |
0.4 |
5 |
1.2 |
6 |
6 |
6 |
6 |
0 |
15 |
0 |
5 |
11010 |
0.791768 |
0.56574 |
0.65994 |
| 10 |
2 |
3 |
70 |
70 |
0.5 |
5 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
11010 |
0.791563 |
0.5519 |
0.65036 |
| 3 |
1 |
1 |
70 |
40 |
0.75 |
10 |
1.5 |
5 |
3 |
3 |
3 |
1 |
15 |
0 |
5 |
11010 |
0.532919 |
0.74221 |
0.62039 |
| 3 |
1 |
1 |
70 |
40 |
0.75 |
10 |
1.5 |
5 |
3 |
3 |
3 |
1 |
15 |
0 |
5 |
11111 |
0.532919 |
0.74221 |
0.62039 |
| 3 |
1 |
1 |
70 |
40 |
0.75 |
10 |
1.5 |
5 |
3 |
3 |
3 |
1 |
15 |
0 |
5 |
10000 |
0.531599 |
0.74221 |
0.61949 |
| 3 |
1 |
1 |
70 |
60 |
0.65 |
10 |
1.5 |
5 |
3 |
3 |
3 |
1 |
15 |
0 |
5 |
11010 |
0.515588 |
0.74394 |
0.60907 |
| 3 |
1 |
1 |
70 |
60 |
0.65 |
10 |
1.5 |
5 |
3 |
3 |
3 |
1 |
15 |
0 |
5 |
10000 |
0.514354 |
0.74394 |
0.6082 |
| 10 |
2 |
3 |
70 |
70 |
0.2 |
5 |
1.2 |
6 |
6 |
6 |
6 |
0 |
15 |
0 |
5 |
11010 |
0.5 |
0.77163 |
0.6068 |
| 15 |
2 |
3 |
70 |
70 |
0.5 |
5 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
11010 |
0.816054 |
0.42215 |
0.55644 |
| 15 |
2 |
3 |
70 |
70 |
0.5 |
5 |
1.5 |
5 |
3 |
3 |
3 |
0 |
15 |
0 |
5 |
11010 |
0.79902 |
0.28201 |
0.41688 |
| 10 |
2 |
3 |
70 |
70 |
0.4 |
5 |
1.2 |
6 |
3 |
3 |
3 |
1 |
15 |
0 |
5 |
11010 |
0.502564 |
0.3391 |
0.40496 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0.6862 |
Maximum f-score |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0.7716 |
Maximum Recall |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0.8161 |
Maximum Precision |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Almost all the parameter tuning and its results can be found here, but they may not be reader friendly
Discussing Results
- The highest f-score was: 0.6822
- The highest precision and recalls were 0.816 and 0.771 respectively.(in case we want just one of them optimized)
- Over Segmentation: This gives an increase to the recall value but adversely affects precision. With lower suffix/prefix thresholds and the PROMOTE_LONG_SEGMENTATION turned ON the words are segmented at almost every character. e.g. ਨੌਕਰ+ਸ਼+ਾ+ਹ+ਾਂ instead of ਨੌਕਰ+ਸ਼ਾਹ+ਾਂ because ਸ਼ਾਹ is a not a frequent suffix in the corpus.
- Under Segmentation: This gives an increase to the precision value but adversely affects recall. e.g. ਕਾਬਿਲੇਜਿਕਰ remains as it is , instead of becoming ਕਾਬਿਲ+ੇ+ਜਿਕਰ because the word ਕਾਬਿਲ is not present in the vocabulary in this form and hence doesnt become a root. This can be controlled by modifying the values of SUFFIX(PREFIX)_CUTOFF_THRESHOLD to a bit low.
- Root Word not in Vocabulary: Most of the root words itentified during the and segmentation are out of vocabulary, because 2 sounds from root-suffix or prefix-root combine to form one phoneme and hence just one unicode character, so the code while learning those stems will not identify them as their vocabulary versions and hence make an error. So, unlike learning all words like contains, containing and containedbeing derived from contain, but words ਕਹਿੰਦੇ (kehnde= kehna+de) , ਕਹਿੰਦਿਆਂ (kehndeya=kehna+deya) will not be identified as derivatives of ਕਹਿਣਾ (kehna) because they have a character missing from the root in the joined word.