Topics in Probabilistic Modeling and Inference
CS698X
2020-21 (even semester)


Instructor: Piyush Rai: (email: piyush AT cse DOT iitk DOT ac DOT in)
Q/A Forum: Piazza
TAs and email: Soumya Banerjee (soumyab AT cse), Dhanajit Brahma (dhanajit AT cse), Amit Chandak (amitch AT cse), Pratik Mazumder (pratikm AT cse), Rahul Sharma (rsharma AT cse)

Background and Course Description

Probabilistic models for data are ubiquitous in many areas of science and engineering, and specific domains such as visual and language understanding, finance, healthcare, biology, climate informatics, etc. This course will be an advanced introduction to probabilistic models of data (often through case studies from these domains) and a deep-dive into advanced inference and optimization methods used to learn such probabilistic models. This is an advanced course and ideally suited for student who are doing research in this area or are interested in doing research in this area.

Pre-requisites

Instructor's consent. The course expects students to have a strong prior background in machine learning (ideally through formal coursework), and ideally also some prior exposure/appreciation to basic principles of probabilistic modeling. The students are expected to have strong foundations in probability and statistics, linear algebra, and optimization, and must also be proficient in programming in Python.

Grading

There will tentatively be 4 homeworks (may include a programming component) worth 40% of the total grade, a mid-sem exam worth 15%, an end-sem exam worth 25%, and the remaining 20% will be allocated to one of the following options that you will have to choose: 4 quizzes OR a course project (to be done in group of 4 students). The exams and quizzes will be held online (details will be shared)

Reference materials

There will not be any dedicated textbook for this course. We will use lecture slides/notes, monographs, tutorials, and papers for the topics that will be covered in this course. Some recommended, although not required, reference books are listed below (in no particular order):

Schedule

The schedule and the course material will be posted weekly on the mooKIT website for this course. There will be two lectures per week (Tuesday and Thursday), made available in form of videos on mooKIT. In addition, there will be a weekly discussion session every Tuesday 6:15pm-7:30pm.

Schedule (Tentative)

Lecture No. Topics Readings/References/Comments Videos/Slides/Notes
1 Course Logistics, Intro to Probabilistic Modeling and Inference [Z15], [B14] (for now, up to sec 3), a brief prob-stats refresher, a basic tutorial on Bayesian inference will be posted on mooKIT
2 Basics of Probabilistic/Bayesian Modeling and Parameter Estimation Wikipedia entries (to be read in the same order) on Bayesian Inference, Prior, Likelihood, Posterior, Posterior Predictive, Credible Intervals (for now, these articles are meant for cursory reading; may safely skip the parts that seem too advanced to you), Additional Reading: MLAPP Section 3.1-3.3, Conjugate Priors will be posted on mooKIT
3 Bayesian Inference for Some Basic Models Lecture 2 suggested readings + MLAPP 3.3-3.5, Bayesian Inference for Gaussians, Wikipedia entry on Dirichlet distribution will be posted on mooKIT
4 Bayesian Inference for Gaussians (Contd) and Exponential Family MLAPP 4.3-4.6 (it is far more detailed than you probably need at the moment; you may skip very detailed proofs, can focus more on the examples and the standard results on Gaussian properties, inference, etc), PRML 2.3, Bayesian Inference for Gaussians, MLAPP 9.1-9.2, some notes on exp-family (if further interested, may skim through the Wikipedia article on exp-fam) will be posted on mooKIT
5 Probabilistic Linear Regression PRML 3.3, MLAPP 7.1-7.3, 7.6 (7.6.1-7.6.2), Recommended Readings: Bayesian Inference tutorial (with Bayesian linear regression as case study; as of now, may read up to section 3) will be posted on mooKIT
6 Probabilistic Approaches for Sparse Modeling Recommended Readings: Section 4 of Bayesian Inference tutorial, The Relevance Vector Machine paper (don't need to read all of it in detail; can just skim over to see the key ideas at a high level, and the experimental results), and the other references mentioned in the slides will be posted on mooKIT
7 (1) Probabilistic Models for Classification: Logistic Regression, (2) Laplace Approximation MLAPP 8.4 will be posted on mooKIT
8 (1) Generalized Linear Models, (2) Generative Models for Supervised Learning MLAPP 9.3 (for GLM), 3.5.1.2, 3.5.2, 3.5.5 (for some examples of generative classification, including the Bayesian way) will be posted on mooKIT
9,10,11 Gaussian Processes PRML Sec. 6.4, MLAPP Sections 15.1-15.2.5, (Optional: 15.3-15.4), Illustration of various kernels for GP, Some GP software packages: GPFlow (Tensorflow based), GPyTorch (PyTorch based), GPML (MATLAB based) will be posted on mooKIT
12 Probabilistic Approaches to Active Learning Bayesian Active Learning, BALD paper (sections 1-2, rest optional). An old but classic paper on this topic: Information-Based Objective Functions for Active Data Selection will be posted on mooKIT
13 Bayesian Optimization An introduction to Bayesian Optimization (with some code), Another Python Notebook on Bayesian Optimization will be posted on mooKIT
14 Multi-parameter Models, Conditional Posteriors, Local Conjugacy Highly recommended: Paper on Bayesian Matrix Factorization, and Gibbs Sampling for the Uninitiated (note: we will look at Gibb sampling again in more detail and formally when talking about MCMC but if you want to get a good and practical overview then this tuutorial is very nice and doesn't require you to understand MCMC in much detail beforehand) will be posted on mooKIT
15 Latent Variable Models and EM PRML Chapter 9 (has examples of EM for Gaussian Mixture Model and Bayesian Linear Regression), MLAPP 11.4, Optional readings: Original EM paper (technically very dense but lots of interesting insights), Another classic paper on EM (more accessible), Online EM (practically oriented), Online EM (theoretically oriented) will be posted on mooKIT
16 Introduction to Variational Inference Readings: PRML 10.1,10.2,10.3.10.4, Life after EM (shows the connection between EM, variational EM, and variational inference, through several examples), VI: A Review for Statisticians will be posted on mooKIT
17 VI (Contd) and Recent Advances in VI VI: A Review for Statisticians (Sec 4.3 on SVI), SVI paper (if you are interested in a more in-depth treatment of SVI), Advances in Variational Inference (a bit long but I would suggest skimming it over to get a sense of the various recent advances in VI) will be posted on mooKIT
18 Approximate Inference via Sampling Readings: PRML 11.1-11.3, MLAPP 24.1-24.3, Recommended: Intro to MCMC for Machine Learning, Monte Carlo for Absolute Beginners, Gibbs Sampling for the Uninitiated will be posted on mooKIT
19 Gibbs Sampling Examples, Some Aspects about MCMC Readings: PRML 11.1-11.3, MLAPP 24.1-24.4, Gibbs Sampling for the Uninitiated will be posted on mooKIT
20 MCMC with Gradient Information, Recent Advances in MCMC Recommended: SGLD paper and other papers referenced in the slides (not required to get into every technical detail but try skimming through some of these papers to get a high level idea), Survey paper on SGMCMC methods like SGLD and improvements, Patterns of Scalable Bayes (See sec 4.2 for parallel MCMC, though other parts are also useful for a general introduction to approximate inference methods, including scalable methods), No U-Turn Sampler (section 2 describes the basics of HMC), will be posted on mooKIT
21 Introduction to Nonparametric Bayesian Modeling Suggested Readings: For NPBayes modeling general intro - this tutorial survey paper, For NPBayes Clustering, this paper on Dirichlet Process will be posted on mooKIT
22 Probabilistic Models for Deep Learning Suggested Readings: Weight Uncertainty in Neural Networks (Blundell et al, 2015), and other papers references in the slides; this tutorial paper on Bayesian Deep Learning will be posted on mooKIT
23 Deep Generative Models Suggested Readings (also look at the references in the slides): For PPCA, FA, etc (classical models), see MLAPP Chap 12; for gamma-Poisson latent factor model and Dirichlet-multinomial PCA, see this paper and this paper; for LDA, see this tutorial paper; for VAE, see the VAE paper, and see this tutorial paper and this survey; for GAN, see the GAN paper, and also see this survey; will be posted on mooKIT

Useful Links and Softwares

Course Policies

Anti-cheating policy