HomeWork II: N-gram based Language Models

Vempati Anurag Sai
Y9227645

Y9227645_hw2.zip

PART A:

Word boundary segmentation -

Two Telugu corpora (Blog Texts and Newspapers) are merged for braoder content in the corpus. First the source, type and topic arguments provided in the corpus are removed. All the alpha-numerics, special symbols and foreign language words (present in the corpus due to imperfection of the language detector, as mentioned) are removed. Finally, the occurences of all the unique Telugu unigrams and bigrams in the corpus are extracted and these are used to train the probabilities of Norvig's ngrams code.

The entire code is written in bash script using following commands:

Test Set -

A test set containing space removed sentences (tel_wordseg.txt) is prepared to evaluate the performance. This contains roughly 1000 words in total. These sentences were not used to update unigram and bigram word counts during the training phase. tel_wordseg_gt.txt is the corresponding ground-truth file.

ngrams Parameters Used -

L (maximum length of first word that is considered while segmenting a sentence) is set as 40.
Maximum recursion limt in python is set to 3000 so that lengthy sentence segmentation is feasible.

Results -

Using just unigram probabilities:
Precision:   91.620%
Recall:       68.106%
F-Score:     78.132%
Using bigram probabilities:
Precision:   97.765%
Recall:       84.622%
F-Score:     90.720%


PART B:

Spell Checker -

A new function "update_edit_count" is added to ngrams distribution to get the count of single edits from the spelling errors file spellin_errors.txt. The occurences of various edits can be seen in tel_count_1edit.txt. As I had only about 300 (word,misspelling) pairs for obtaining the edits count file, the frequencies are very small. Would be much better if more data is available.


def update_edit_count(wrong, right, d=2): 
    "Update a dict of {edit: count} pairs where count is the number of times that particular edit has been applied" 
    count_dict = {}
    def editsR(hd, tl, d, edits): 
        def ed(L,R):
		return edits+[R+'|'+L] 
	global flag
        C = hd+tl 
        if C == right:
	    for e in edits:
		    if e not in count_dict: count_dict[e] = 2 
		    else: count_dict[e] += 1
	    return
        if d <= 0: return 
        extensions = [hd+c for c in tel_alphabet if hd+c in PREFIXES] 
        p = (hd[-3:] if hd else '<') ## previous character 
        ## Insertion 
        for h in extensions: 
            editsR(h, tl, d-1, ed(p+h[-3:], p)) 
        if not tl: return 
        ## Deletion 
        editsR(hd, tl[3:], d-1, ed(p, p+tl[0:3])) 
        for h in extensions: 
            if h[-3:] == tl[0:3]: ## Match 
                editsR(h, tl[3:], d, edits) 
            else: ## Replacement 
                editsR(h, tl[3:], d-1, ed(h[-3:], tl[0:3])) 
        ## Transpose 
        if len(tl)>=6 and tl[0:3]!=tl[3:6] and hd+tl[3:6] in PREFIXES: 
            editsR(hd+tl[3:6], tl[0:3]+tl[6:], d-1, 
                   ed(tl[3:6]+tl[0:3], tl[0:6])) 
    ## Body of edits: 
    editsR('', wrong, d, []) 
    return count_dict

The function "edits" is altered to accomodate telugu script. +1 Smoothing is applied on the unigram frequencies to get the corresponding probabilities. Phonet data was not provided with the Telugu language distribution of aspell. So, I haven't implemented that part yet!


def edits(word, d=2): 
    "Return a dict of {correct: edit} pairs within d edits of word." 
    results = {} 
    def editsR(hd, tl, d, edits): 
        def ed(L,R): return edits+[R+'|'+L] 
        C = hd+tl 
        if C in tel_Pw: 
            e = '+'.join(edits) 
            if C not in results: results[C] = e 
            else: results[C] = max(results[C], e, key=Pedit) 
        if d <= 0: return 
        extensions = [hd+c for c in tel_alphabet if hd+c in PREFIXES] 
        p = (hd[-3:] if hd else '<') ## previous character 
        ## Insertion 
        for h in extensions: 
            editsR(h, tl, d-1, ed(p+h[-3:], p)) 
        if not tl: return 
        ## Deletion 
        editsR(hd, tl[3:], d-1, ed(p, p+tl[0:3])) 
        for h in extensions: 
            if h[-3:] == tl[0:3]: ## Match 
                editsR(h, tl[3:], d, edits) 
            else: ## Replacement 
                editsR(h, tl[3:], d-1, ed(h[-3:], tl[0:3])) 
        ## Transpose 
        if len(tl)>=2 and tl[0:3]!=tl[3:6] and hd+tl[3:6] in PREFIXES: 
            editsR(hd+tl[3:6], tl[0:3]+tl[6:], d-1, 
                   ed(tl[3:6]+tl[0:3], tl[0:6])) 
    ## Body of edits: 
    editsR('', word, d, []) 
    return results 

To compare the my results against those recommended by aspell-0.60, I have compiled the aspell's telugu distribution with the telugu wordlist I have used in the assignment. A test set with around 100 sentences with misspelled words is collected from the arhives of "AndhraBhoomi" newspaper. The test set contains text from various categories like sports, politics etc. The performance of our spell checker, aspell and the ground truth can be seen in "output_on_testset.txt"

Observations:

Misspelled sentence: తెల్లవారు జామున మైదానంలోకి వెళ్లి మూత్ర విసర్జన చేస్తూ కేరింతలు కిట్టారు
Unigram Output: తెల్లవారు జామున మైదానంలోకి వెళ్లి మూత్ర విసర్జన చేస్తూ కేరింతలు కట్టారు
Bigram Output: తెల్లవారు జామున మైదానంలోకి వెళ్లి మూత్ర విసర్జన చేస్తూ కేరింతలు కొట్టారు
Misspelled word: కిట్టారు (This isn't a Telugu word)
Unigram correction: కట్టారు (కేరింతలు కట్టారు is equivalent to 'tied cheers' in English. Though the individual words make sense, the sentence as a whole doesn't)
Bigram correction: కొట్టారు (కేరింతలు కొట్టారు is equivalent to 'cheered' in English.)
The clear advantage of bigram model being able to capture the context can be seen in this example.

Resources -

Test Set
Ground-Truth corresponding to the test set
Word Segmentation using unigram frequencies
Word Segmentation using bigram frequencies
Single edits count file
Performance of spell checker

References -

Chapter 14 of _Beautiful Data_ by Peter Norvig: Natural Language Corpus Data [http://norvig.com/ngrams/ch14.pdf]. Related code and data: [http://norvig.com/ngrams].