The task was to separate the word boundaries.
Word corpus provided in assignment1 was used i.e. Hi_blog.txt and Hi_newspaper.txt. Words with frequency less than 2 were removed.
Parameters used for tuning were:
1. L i.e. maximum length of the word
2. X where the function avoid_long_words(key, N) returns X./(N * 10**len(key))
3. N i.e. Number of tokens
Following table illustrates the results:
(Precision and Recall are in percentage )
| L | X | N | Precision (error1) | Recall (error1) | Precision (error2) | Recall (error2) |
| 10 | 10 | 10000 | 6.0 | 11.9 | 44.7 | 89.0 |
| 10 | 1 | 10000 | 6.0 | 11.9 | 44.7 | 89.0 |
| 10 | 10000 | 10000 | 6.0 | 11.9 | 44.6 | 89.0 |
| 20 | 10 | 10000 | 4.4 | 6.3 | 62.4 | 88.9 |
| 20 | 1 | 10000 | 4.4 | 6.3 | 62.4 | 88.9 |
| 20 | 10000 | 10000 | 4.1 | 6.0 | 62.1 | 88.9 |
| 20 | 10000 | 100000 | 3.2 | 4.2 | 67.0 | 88.9 |
| 20 | 10000 | 10 | 4.3 | 50.7 | 8.1 | 98.5 |
Best results were obtained at L = 20, X = 10 and N = 10000 where Precision = 62.4 and Recall = 88.9.