Corpus Used: Hindi HC Corpora at www.corpora.heliohost.org/
The raw corpus had english alphabets, numeric characters and other special characters.
Used a python script (given in Part-C) to prune out the non-Hindi characters, tokenize Hindi words, manage white spaces and then to count the number of occurences of each such tokenized word and sort the words according to their frequency in decreasing order.
The resultant output file had the format required by undivided++ as input ("frequency of the word", "word itself" in each new line).
Two files were created out of the corpus. First file with word and number of occurences of that word in the corpus. This count will help me in my unigram modelling. The second file was for bigram modelling in which I count number of occurences of each pair of words. I start my sentence with start tags and end tags. Hence my two files for unigram and bigram are ready which the python script uses to build up the language model.
Experimented with the parameter splits.length. Finally it was raised from 20 to 30 leading to an increase of f-score from around 87% to 96%.
I ran codes both for unigram and bigram models and results are reported in hw2A.html. Created new validation set and result on validation sets are also reported in mentioned webpage.
NOTE: All Punctuations marks should be cleared and then only defragmenter should be run. While checking precision, recall and F-Score, lines in two files being compared should be aligned (corresponding texts on corresponding line numbers).
Codes for creating frequency counts, defragmenter_unigram, defragment_bigram and checking accuracy is submitted in codes.zip. New validation sets, ground truths, input files and output of unigram/bigram models are provided.