This readme on how to run the code and what is the purpose of each file.


CODE FILES:

wordlist.pun : final prunned corpus of the punjabi language used in this testing.

handsegmented : file containing segmentation of ~475 words, compiled by hand.

comparison.py : python script to display side by side human version and undivide++ version of segmentation for each word in file 'handsegmented'.

side-by-side.txt : output of comparison.py is directed to this file.

prf.py : this python script is to calculate and display precision, recall and f-score using the just generated side-by-side.txt.

scri.h : bash script to compile undivide++, run it with '$./a.out wordlist.pun 1 1 0 1 0', compare results of undivide++ with 'handsegmented' using comparison.py, and finally calculate precision etc using prf.py.

*rest of the files are either data, or undivide++ files or intermediate versions of the corpus while pruning.

RUNNING:

1. Run ./scri.h 
2. Editing the parameters inside UnsupervisedWordSegmentation.cpp is possible in an editor of your choice. 
3. Similarly parameters given to the executable can also be edited in scri.h, the default being as in this command: '$./a.out wordlist.pun 1 1 0 1 0'


RESULT:

 Precision, recall and f-score are displayed.


DATA FILES:

The tabulated results of all test runs are stored in data.ods


PRUNNING THE CORPUS:

a). remove words of 2 or less character length 

awk '{
if( length($2) >= 3 )
	print
}' pun200k > punlg3


b). remove words occuring just once

awk '{
if ($1 >=2)
	print
}' punlg3 > punfg1

c). remove words containing -, / etc. etc.

grep -v '[\/\øæ]' punfg1 | grep -v '[\-]' | grep -v '[\:]' | grep -v '[î]' > wordlist.pun

