Homework 2 A
Anant Raj Roll No - 10086
Word Boundary Segmentation
Unigram Model
Word segmentation in this assignment is done on the basis of Unigram Model for Language. First corpus data which is the combined version of Newspaper, Blog and IITB corpus is cleaned up and word frequency is being counted. Probability for each word is calculated on the basis of word frequency and total no. of tokens. Results are quite good for the given test set. We have also checked it with two other self annotated sample txt files randomly taken from the internet and results are very good on that also. Changing the value of L(parameter which is used for controlling the split length) effect the word boundry segmentation. Keeping L large gives a bit bad result. We have varied L in the range of 200 to 20 and we got good enough result for L = 20. All the codes and corpus used are attached in the link given below. Following are the results obtained for the Word Segmentation.
| Precision | Recall | F-Score |
|---|
| 0.98745 | 0.9393 | 0.9628 |
Another small test sample is created from a child's hindi story website and results for them are :
| Precision | Recall | F-Score |
|---|
| 0.99 | 0.91 | 0.95 |
Third test sample is also taken from the internet which is a small part of some blog, some newspaper text and some stories. Results on them are given below :
| Precision | Recall | F-Score |
|---|
| 0.988 | 0.946 | 0.966 |
Bigram Model
Also analysis has been carried out for segmenting word using Bigram Language model. For getting the bigram count the code given by "Prof Katrin Erk" is used. Pair of words which have frequency less than or equal to 3 has been removed from the file. Results are not very different(improved by small factor). Unigram is also giving the almost the same accuracy.
Result for the Given Sample test Data(on the web) using bigram model :
| Precision | Recall | F-Score |
|---|
| 0.98 | 0.97 | 0.98 |
Result for 2nd sample test using bigram model :
| Precision | Recall | F-Score |
|---|
| 0.99 | 0.934 | 0.96 |
Result for 3rd sample test using bigram model :
| Precision | Recall | F-Score |
|---|
| 0.98 | 0.98 | 0.98 |
Conclusion
As can be seen froom the above analysis the results using the biagram model is slightly better than unigram model because it takes care of the probabliy of words occuring together.
All The Generated Test Samples and Codes Here
Link to download all the files