Corpus Pruning Text and Trials. The list assigned to me was List28.The language chosen was hindi.
I have removed the words with frequency less than 4(frequency:1-3).I also removed the words separated by hyphen("-") and words separated by full stop(".") and poorn viram("|") so that we work on words present in Hindi only.I first combined the two corpora(blogs and news) together in one large single corpus.Also,some use of shell commands like sed and awk was made to refine the corpora further.The programming language used is python otherwise.
The link to the code can be found here.
The file arpit_ground_truth.txt contains the hand segmented data including words separated by hyphen(as they were given in the list assigned),removing the words having incorrect spellings.
The file segregate.py is used to create the file "output.txt"(in the format Dasgupta's code expects it) from the corpus to be fed to the code by Dasgupta.The file The file obtained is finalSegmentation.txt inside the output folder.
The main calculation is done in the file "final.py"
Please note that use python3 for running the code as python2.x is shown to show some errors for the same code which is working absolutely fine for python3. I attribute the reason behind such discrepancy as python3 has much better support for utf-8 characters than python2.x . The parameters I used while segmenting the training dataset are:The threshold have been shifted too as per the language corpus requirements.Long segmentation is set to 0 to improve precision. The parameters passed are 1 1 0 1 0.
Case:When we don't remove words with foreign origin(i.e. we use entire list of 300 words).The precision and recall is found to be pretty small
Regarding the assignment of error cases,and the calculation of recall,precision and f-score of the data,I have taken error whenever two segment boundaries disagree in the processed output.The file link to make the calculations is here.
The first column consists of the number of words fragmented by the C++ file(calculated as number of '+' + 1).
The second column consists of the number of words fragmented by us(calculated again as number of '+' + 1).
The third column consists of the words/word fragments common to both of these lists.
Retrieved values=572
Relevant values=260
Intersection of retrieved and relevant values=97
Calculation of precision:
Precision=Intersection of Relevant and Related values / Retrieved values.
Thus,Precision=97/572=0.16958042
Calculation of recall
Recall=Intersection of Relevant and Related values / Relevant values.
Thus,recall=97/260=0.373076923
Calculation of f-score
f-score= (2 * precision *recall)/(precision+recall)=2*0.17*0.373/(0.373+0.17)=0.233554328
Thus,f-score can be approximated as 0.235
Discussion on error cases
Many cases were found where there were either spelling mistakes or the words themselves had foreign origin.A lot of English words were found in the dataset while some words had their origin from other languages like Urdu and Persian.Such words,when removed had a great impact on the precision and recall values.The final list of words with such words removed can be found inside the zip file as arpit_ground_truth.txt . Some of the examples of such 'non hindi' words are:CaseWhen we include the words separated by hyphen(which were otherwise removed during the time of formation of the corpus) and removing the words having spelling mistakes,the precision and recall values increase.
For ex:The dataset now contains words like सुरक्षा-व्यवस्था and words like खरीदते-खरीदते
New precision with these words removed=0.48253
New recall value=0.68161
New F-score=0.6562/1.1635=0.563987967=0.564
The link to file for doing this calculation is results.txt
Another interesting fact noticed was the program speed increases when we rename the corpus to .hin instead of .txt!!!Similarly,I was trying to get the code running with python 2.x version initially but it had little(read it as no) support for UTF-8 characters.However,this problem was removed when I used python3.Henceforth,the compilation of code should also begin with the command python3 instead of python alone!