Two Telugu corpora (Blog Texts and Newspapers) are merged for braoder content in the corpus. First the source, type and topic arguments provided in the corpus are removed. All the alpha-numerics, special symbols and foreign language words (present in the corpus due to imperfection of the language detector, as mentioned) are removed. Finally, the occurences of all the unique Telugu unigrams and bigrams in the corpus are extracted and these are used to train the probabilities of Norvig's ngrams code.
The entire code is written in bash script using following commands:
A test set containing space removed sentences (tel_wordseg.txt) is prepared to evaluate the performance. This contains roughly 1000 words in total. These sentences were not used to update unigram and bigram word counts during the training phase. tel_wordseg_gt.txt is the corresponding ground-truth file.
L (maximum length of first word that is considered while segmenting a sentence) is set as 40. Maximum recursion limt in python is set to 3000 so that lengthy sentence segmentation is feasible.
Using just unigram probabilities:Precision: 91.620%
Using bigram probabilities:Precision: 97.765%
A new function "update_edit_count" is added to ngrams distribution to get the count of single edits from the spelling errors file spellin_errors.txt. The occurences of various edits can be seen in tel_count_1edit.txt. As I had only about 300 (word,misspelling) pairs for obtaining the edits count file, the frequencies are very small. Would be much better if more data is available.
def update_edit_count(wrong, right, d=2):
"Update a dict of {edit: count} pairs where count is the number of times that particular edit has been applied"
count_dict = {}
def editsR(hd, tl, d, edits):
def ed(L,R):
return edits+[R+'|'+L]
global flag
C = hd+tl
if C == right:
for e in edits:
if e not in count_dict: count_dict[e] = 2
else: count_dict[e] += 1
return
if d <= 0: return
extensions = [hd+c for c in tel_alphabet if hd+c in PREFIXES]
p = (hd[-3:] if hd else '<') ## previous character
## Insertion
for h in extensions:
editsR(h, tl, d-1, ed(p+h[-3:], p))
if not tl: return
## Deletion
editsR(hd, tl[3:], d-1, ed(p, p+tl[0:3]))
for h in extensions:
if h[-3:] == tl[0:3]: ## Match
editsR(h, tl[3:], d, edits)
else: ## Replacement
editsR(h, tl[3:], d-1, ed(h[-3:], tl[0:3]))
## Transpose
if len(tl)>=6 and tl[0:3]!=tl[3:6] and hd+tl[3:6] in PREFIXES:
editsR(hd+tl[3:6], tl[0:3]+tl[6:], d-1,
ed(tl[3:6]+tl[0:3], tl[0:6]))
## Body of edits:
editsR('', wrong, d, [])
return count_dict
The function "edits" is altered to accomodate telugu script. +1 Smoothing is applied on the unigram frequencies to get the corresponding probabilities. Phonet data was not provided with the Telugu language distribution of aspell. So, I haven't implemented that part yet!
def edits(word, d=2):
"Return a dict of {correct: edit} pairs within d edits of word."
results = {}
def editsR(hd, tl, d, edits):
def ed(L,R): return edits+[R+'|'+L]
C = hd+tl
if C in tel_Pw:
e = '+'.join(edits)
if C not in results: results[C] = e
else: results[C] = max(results[C], e, key=Pedit)
if d <= 0: return
extensions = [hd+c for c in tel_alphabet if hd+c in PREFIXES]
p = (hd[-3:] if hd else '<') ## previous character
## Insertion
for h in extensions:
editsR(h, tl, d-1, ed(p+h[-3:], p))
if not tl: return
## Deletion
editsR(hd, tl[3:], d-1, ed(p, p+tl[0:3]))
for h in extensions:
if h[-3:] == tl[0:3]: ## Match
editsR(h, tl[3:], d, edits)
else: ## Replacement
editsR(h, tl[3:], d-1, ed(h[-3:], tl[0:3]))
## Transpose
if len(tl)>=2 and tl[0:3]!=tl[3:6] and hd+tl[3:6] in PREFIXES:
editsR(hd+tl[3:6], tl[0:3]+tl[6:], d-1,
ed(tl[3:6]+tl[0:3], tl[0:6]))
## Body of edits:
editsR('', word, d, [])
return results
To compare the my results against those recommended by aspell-0.60, I have compiled the aspell's telugu distribution with the telugu wordlist I have used in the assignment. A test set with around 100 sentences with misspelled words is collected from the arhives of "AndhraBhoomi" newspaper. The test set contains text from various categories like sports, politics etc. The performance of our spell checker, aspell and the ground truth can be seen in "output_on_testset.txt"