Given a statement, we needed to find out the word boundary segmentation of that statement
The results obtained after finding out the unigram data was
|
UNIGRAM |
HITS |
1088 |
MISSES |
62 |
PRECISION |
0.946086956522 |
RECALL |
0.539130434783 |
Discussion of Error Cases
The errors were mostly occuring when there was a "matra" in the middle of the word. That made it segment there. We need to remove words which start with matra.
Code for calculating word boundary segmentation
For the code for handling the corpus (pruning the corpus), the link is Download all code