Prof. Mukherjee is on medical leave. If you need to contact him, please contact his brother Mr. Ashis Mukherjee at +919868948570.
![]() |
![]() |
![]() | |
(a) | (b) | (c) | |
(a) Visuo-motor learning: i. take a sample of images at random poses spanning the motor space. ii. find similar images and compute local tangent neighbourhoods. iii. stitch these into a visuo-motor map - a manifold. (b) gross motion planning: i. from this map, remove nodes that overlap with body or obstacle. ii identify goal poses as images that capture target object (white circle) in palm. plan gross motions within non-colliding nodes (click image to see). Fine-motion : at contact regions. (c) Unknown robot modeling: Same process based on unannotated images of an unknown robot, without any prior information on geometries or kinemantics, without camera calibration, modeling both robot and its task space. (click to see motion) A part of the semantics of the target object (shape and location) is represented as the contact region in this visuo-motor map. |
![]() | |
the symbol for "in". At its semantic pole is an image schema (a generative classifier) - a function with two arguments based on a distribution on the visual angle. it has learned the association of this schema with the linguistic unit "in" from co-occurring language. |
![]() |
![]() |
![]() |
|
phrase structure discovered from untagged corpus, using the [Solan/Edelman:2002] algorithm ADIOS. | symbolic composition: symbolization for "in the box", with one argument slot free, for the trajector |
![]() |
A sample frame from a 2D video (from Tversky group, Stanford U.). The parallel commentary for this scene may say: The big square is pushing the small square. |
Starting with just a video and a set of narratives describing events in the video, we develop a grounded cognitive grammar that learns to relate a perceptual structure with phrases such as "circle in a box". This is then used to learn metaphoric extensions of containment via selectional restrictions. Items showing up at the [container] position in such constructions mostly belong to the location category, but group and time are also very common, followed by state, cognition, and act.
Unsupervised temporal clustering is used to discover spatial activity from a 2D video. Perceptual attention restricts search to objects attended to sequentially. Learned temporal templates constitute a model for each activity, and are mapped to words. The fact that chase takes two arguments, or that these are commutative, are discovered in perception.
The entire process outlined above is agnostic to which particular language is being learned. Here we demonstrate this by learning words related to the containment situation (e.g. Tight and Loose) for a loosely inflected language (English) and a heavily inflected one (Telugu).
By changing the granularity of the classification in the above process, one may be able to learn that move-away-A-fixed, move-away-B-fixed, and move-away-both-simultaneously are different types of move-away.
![]() |
Frame from complex 3D video. Appearances of tracked foreground objects are clustered using several features. These noisy clusters are then associated with words from a commentary. Nouns for four of the ten object classes can be learned. Also, action descriptors such as "moving from left to right" can be learned. [Guha/Mukerjee:2008] |
Learning words from complex 3D video. Here foreground object blobs are first clustered based on appearance. Nouns and trajectory labels are learned.
Solutions to hard problems may lie in small regions of
the possible space. These regions can be characterized using much lower
dimensions than used in the original problem formulation. These lower
dimensional mappings may correspond to "chunks" which have been related to the
development of expertise in areas such as design. Eventually, some of
these "chunks" may map to "symbols" that associate a label with a meaning.
More: The "infant designer"
enterprise
Other interests: Hands-on learning in schools
publications | students | storytelling science | literary | iitk birds | book excerptise |
cse home page | center for robotics | iitk home |