CS671 Homework #1

The details of submission, along with links are given below

• Step 1: Choosing the Corpuses

  There were three Corpuses available for Hindi. One was extracts from Newspapers, another from Blogs, and one more set of corpuses from IIT-B. I have chosen all the three Corpuses for the analysis combined with each other after cleaning.

• Step 2: Cleaning the Corpuses

  The Corpuses contained a number of Header Lines. I first removed them. I then attempted cleaning the corpus further, by removing ASCII only words, and words with less than 3 characters. Since we are doing Morphological analysis only, removing these words reduces the time of the analyzing code significantly. Also, the IIT-B corpus was divided into a number of smaller file. I merged them all into a single corpus file first. The link to the cleaned Corpuses is given here [Link Removed].

• Step 3: Merging the Corpuses

  As explained, in order to take advantage of the words in all the corpuses, I merged them together into one, single, monolithic hindi file. It may not make any sense straightaway (since a number of small words, punctuations etc. are cleaned), but it is good for Morphological Analysis. The link to the merged Corpus is given here [Link Removed].

• Step 4: Running Linguistica on the Corpus

  I have used Linguistica to do the Morphological Analysis in this Homework. Linguistica produces a number of files with the statistics such as stems, suffixes, affixes etc. Out of these, two files are of our use. One that lists down all the suffixes that can be applied to a stem (X_Mini1_Suffixes.txt), and one, which contains, among other detailed statistics, the mini words or the unbreakable words(X_Mini1_Words.txt) in the corpus. I used this file to check for words, for which no suffixes could be found (Linguistica treated them as one unseparable entity). The link to the Linguistica Output for the merged Corpus is given here.

• Step 5: Pre-processing Linguistica Output

  Since I used Java as my Programming Language, I could not take advantage of any features like those in NLTK. As a result, I had to do some pre-processing on the Linguistica Output, in order to make the matching of the words easy. Essentially, I removed all the extra information, like headers, suffix count etc. from these files, so that only relevant data remains for processing. The link to the Processed Linguistica Output files is given here.

• Step 6: Manual Segmentation of words in the List

  Unfortunately, a good number of words in my Word List were not suitable for Morphological Analysis of Hindi terms. There were a good number of non-hindi words, written in Devnagri, which is almost impossible for Linguistica to know. Some words are formed by amalgamation of two entirely different hindi words, which doesn't provide any help in the Morphological Analysis as well. Also, there are certain words, which do not appear at all in the Corpus. These words have been kept out for the analysis. The link to the Manual Segmentation of the words (as per my inference) is given here.
  NOTE: The words with an "*" after them are those which will be ignored while doing the analysis for one reason or the other.

• Step 6: Matching the Segmentations

  I have written code in java, to find out the segmentation boundaries for a word decided by Linguistica, as compared to the Manual decomposition. Essentially, I pickup the starting stem from the manual decomposition, and keep adding suffixes to it, to form larger morphemes. I then check if Linguistica was able to start with the same stem, and proceed further, choosing the same suffixes at each step or not. If no stem is found for a word, we check the mini word file to see if it appears there with no suffix count. The link to the Output Files is given here. There are two files here. The file raw_final_output.txt just tells whether a match was found or not. In case it was not found, it also tells, what segmentation Linguistica did. The other File, final_output.txt, writes the output in tab separated text, mentioning Match or Mis-match, both versions of segmentation, and other parameters for the operation (Hits / Insertions / Deletions / Precision / Recall / F-Score).

Comments

Code