CS671 Homework #2
Saurabh Srivastava
Roll No. 13111164
The details of submission, along with links are given below
-
Step 1: Choosing the Corpora
There were three Corpuses available for Hindi. One was extracts from Newspapers, another from Blogs, and one more set of corpuses from IIT-B. I have chosen all the three Corpuses for the analysis combined with each other after cleaning.
-
Step 2: Preprocessing the Corpora
The Corpura contained some headers which were removed. They are then written, line-by-line to one large copus file. The Corpus file created after merging them is given here.
-
Step 3: Finding Unigrams, Bigrams and Trigrams (Trigrams may be required for part b of the Homework)
The Corpus file is then read line-by-line to extract frequency data. The files which contain this information in a reverse sorted order can be found at the following links
unigram.txt
bigram.txt
trigram.txt
-
Step 4: Preparing Validation Sets
In order to make sure that the validation is done on documents that are not a part of the corpus, I have taken some validation data from 4 hindi articles published today (13-Sep-2013). The links of the articles are also given below.
Article 1
Article 2
Article 3
Article 4
I then created input files and ground truth files for the text in these articles, and ran it on the code to fine tune the word max length parameter. The links to the input files and their corresponding ground truths are
input1.txt gt1.txt
input2.txt gt2.txt
input3.txt gt3.txt
input4.txt gt4.txt
-
Step 5: Running the Sentence Segmentation Code on Validation and Test Sets
The code was then run on Validation sets and consequently on the Test Set. The input file for the Test Set is here, and the corresponding output file produced by the program is here
[NOTE: I have removed all the extra lines from the input file and also aligned a few sentences on new lines in accordance to the lines in the Ground Truth file].
The Output for the Validation Sets are also available here:
output1.txt
output2.txt
output3.txt
output4.txt
Comments
The Precision and Recall results can be found here.
Code
The code for the homework (part a) is uploaded here in a Zip File along with a README.
UPDATE
Part b: I could not complete the Part b of the assignment. Whatever I could complete is there in this file.
NOTE: The Files in this Zip File shall be extracted to a folder, and the files unigram.txt and bigram.txt must also be placed in the same folder before running the code.