Homework 2

N-gram based Language Models

Language: Hindi

Language data

vocab.txt: Hindi words with their frequency (vocab.txt)

vocab_big.txt: Hindi 2-word bigrams with their frequencies (vocab_big.txt)

Validation data

wordans_val.txt: Sentences on which segmentation is tested. Part of corpus taken from http://homepages.inf.ed.ac.uk/miles/babel.html.
wordseg_val.txt: Validation file used (obtained after removing spaces from above file).

Results

Using only unigram counts:

Using given test file

scores.txt: Scores for the given file(wordseg.txt)
final_output.txt: Detailed output for the above input file. Tells the correct and predicted segmentation along with hits, insertions and deletions for each sentence.

Using validation file

Using bigram counts too:

Using given test file

scores_big.txt: Scores for the given file(wordseg.txt)
final_output_big.txt: Detailed output for the above input file. Tells the correct and predicted segmentation along with hits, insertions and deletions for each sentence.

Using validation file

scores_big_val.txt: Scores for the validation file(wordseg_val.txt)
final_output_big_val.txt: Detailed output for the above input file. Tells the correct and predicted segmentation along with hits, insertions and deletions for each sentence.

code