Word Boundary Segmentation

Language: Hindi

Tool: Norvig Code

Section #1: Structure on Work Done

  1. First we created a large corpus by clubbing the given corpora, HC corpora and the hindi corpus from CFLIT, IIT Bombay. Here is the python code for combining the CFLIT files, here.
  2. The we selected all the terms from the corpus that were a part of the hindi corpus. We simultaneously calculated and then printed the frequency of all the hindi words as the code provided by norvig needs. The python code for which is here. Then created the Code to genetrate bigrams as the norvig code need it.
  3. Next I ran the obtained frequency and words through the norvig code for unigrams and then for bigrams
  4. The precision and recall was checked using the following code given.
  5. The maximum length of word was set to 28.

    Section #2: Results

    The Results were(given set):
      For Bigram functions used
    1. Precision: 92.3%
    2. Recall: 96.3%
    3. F-Score : 93.7%

      For Unigram functions used
    1. Precision: 92.4%
    2. Recall: 95.1%
    3. F-Score : 94.2%
    The Results were on an external test set:
      For Bigram functions used
    1. Precision: 95.7%
    2. Recall: 98.9%
    3. F-Score : 97.2%

      For Unigram functions used
    1. Precision: 95.7%
    2. Recall: 97.5%
    3. F-Score : 96.6%

    It was observed on the given data set, as you increase L, the precision increases however to aboid overfitting 28 was set as L

    Section #3: Links In Assignment