Submitted by: Prashant Jalan

CORPUS ANALYSIS -

make_corpus.sh -
This is a simple shell script to take all the text files in a folder and concatenate them into a single file.

get_sentence.py -
This python code reads the corpus file and takes out the sentences to store them in another file. Algorithm - It scans and allows any Hindi character but in case of anything else, a space is inserted. For a full stop or purna viram or newline, a newline is inserted.

get_sentence2.py -
This is the same as get_sentence.py except that for a newline detection it won't insert a newline, rather replace it with a space. It was made for the processing of the IITB corpus. In that corpus, a newline doesn't indicate the end of a sentence.

get_words.py -
This python code takes any sentence file as input and removes all the white space to get simple linguistic units (words). Then it counts the frequency, sorts them and prints them.

NOTE: For running any python file, the input file name and output file name should be changed accordingly. I did two analysis, one where Hindi numerical digits were included and one where they were excluded. For morphological analysis, the numerical digits were excluded.


MORPHOLOGICAL ANALYSIS -

test.py - 
This python code tests if the given list of words are present in the corpus file and if there count is greater than three.

result.py -
This python code is used to get the mophological segmentation from the Undivide output and print it into a file. It will take the manual segmentation as input, then print in one file the output of the undivide and in one file the manual+unsupervised segmentation.

score.py -
This python code takes the file which has both manual and automatic segmentations and then calculate the recall, precision and f-score.

makeList.py -
A small program which I had written to remove the English words from the list.

NOTE: First, I tried doing the morphological on words extracted from all the three corpora. But given the fact that Undivide can't handle large memory size, I did the analysis for the Hi_Blogs and Hi_Newspapers corpora excluding the words having a frequency count of less than 3.

NOTE: Renaming the input of undivide program from .txt to .hi, increases the word segmentation count obtained in output.
