Vishal Kumar Gupta 10817
@article{silberermodels,
title={Models of Semantic Representation with Visual Attributes},
author={Silberer, Carina and Ferrari, Vittorio and Lapata, Mirella}
year={2013},
annote = { Grounded Language Acquitision is defined as extracting representations for the meaning of natural language tied to
physical world. The basic motivation is exploting the visual modality of langauge to learn semantics. Various
approaches employed include semantics parsing, cross-mapping of NL and actions, association of language and perpeutal
data and learning based on linguistic and perpeutal input. The basic goal is construction of distribution models
which are perpetually grounded. Two prime motivation behind the goal are: (1) There exists a direct relation between
language interpretation and physical world interactions(Cognitive aspect) and (2) The model can very well learn
representations for multimodal data and its applications are diverse(Engineering Aspect).
Visual modalities can be considered as intrinsic representation of language. This can also include auditory, motor
and haptic modalities. The main reason for using visual modalities include (1) easy availability of data and 2) images
in the database are hierarchial according to wordnet synsets. Previously lower layer features(SIFT) were used. This
paper presents the use of high leval features in the form of visual attributes were used which offers many advantages.
This include: (1) Using attributes offers generalization over unseen objects and (2) attribute-centric approach for
representing images helps build a distributioanl model for lexical meaning also.
For the purpose of dataset images for the concepts in McRae et al.’s (2005)production norms were used. This data set
possesed a nominal hierarchial of WordNet. Data was divided into training, development and testing sets. Development
set is used to develop and refine attribute annotation scheme. Attribute annotation is conducted on per-concept basis
rather than per-image basis. For each of the attribute a L2 regularized SVM(Fan et al., 2008) classifier is learned.
Since a single concept is visible in many images and we were entitled to learn a concept w based on its visual attributes
a single representation comprising of those attributes was needed. For the purpose the following approach is followed: for
each input image a vector is constructed which has prediction scores for the attributes of that image as entries for a
given concept. Based on these vectors a normalized probability distribution of attribute w is estimated.
For evaluating the result coorelation coefficients were calculated in order to observe inter-model relation. In most
of the cases two modalities are found to perform better than one. The model also performed quite well on unseen data
showing generalization as expected. In totalty, the results are summarised as follow:
=> Distributional models perform better with visual attributes.
=> Attribute-centered representation certainly have advantages over conventional approaches based raw features.
}
}