@article{dahl2012context,
  title={Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition},
  author={Dahl, George E and Yu, Dong and Deng, Li and Acero, Alex},
  journal={Audio, Speech, and Language Processing, IEEE Transactions on},
  volume={20},
  number={1},
  pages={30--42},
  year={2012},
  publisher={IEEE},
annote = { 

This paper deals with deep learning and HMM in the application of large scale vocabulary recognition. A lot of work has been done previously in the area of Large scale Vocabulary recognition using GMM and there are some recent advances in this field((such as large-margin estimation, large-margin hidden Markov model (HMM), large-margin MCE, boosted MMI and CRF) which improves the result. But quest for the search of better features is still on because we can't reach up to human level accuracy till now. Deep Learning gives a hope in this area. Problem with the ANN is that after few layers back propagation doesn't work much and often results in sticking at local minima. Pre training (Greedy Layer Wise pretraining with unlabelled data) gives a good initialization of the weight. Pretraining acts as a data dependent regulariser. In this paper the author proposes a hybrid between pre trained deep neural networks and context dependent HMM. This paper starts with a brief introduction of deep learning and it's importance. After then it is built upon it and description about RBM, deep belief nets, features, context dependent HMM and results is provided. \\

\textbf{Previous work}\\
The current state of the art method for context dependent large scale vocabulary recognition is using combination of Gaussian mixture models and hidden Markov Modal. Though work has also been done to develop acoustic modals using ANN. Generally these works deals with the estimation of HMM state posterior probabilities using neurl network. The nodes of neural network are trained to get the posterior probabilities. Later ANN-HMM model is used for contexts dependent phone recognition in the Automatic Speech recognition task. The posterior probability of context dependent phone is modelled at each acoustic observaion as: \\
$p( s_{i}, c_{j}| x_{t}) =   p( s_{i}| x_{t})* p( c_{i}|s_{j}, x_{t})$ \\
or\\
$p( s_{i}, c_{j}| x_{t}) =   p( c_{i}| x_{t})* p( s_{i}|c_{j}, x_{t})$ \\

 and this model outperforms the state of the art GMM-HMM model. But as mentioned earlier various challenges were there with the ANN implementation. Few of them are less no. of labelled data, inefficient back propagation for deeper neural network, sticking at local minima etc. Also few years we don't have so much computation power to tarin the ANN-HMM model and GMM-HMM training is easier to parallelize and due to these GMM-HMM model has been used widely till now. So this work work differs from the previous works in many aspects. One is obviously it uses deeper neural network and unsupervised pretraining. Also the output of the neural network is posterior of senones (tied triphone HMM state), instead of context independent phones and it makes this techniques more useful in context dependent word recognition.

\textbf{Deep belief network and RBM}\\

Deep belief networks are probabilistic generative model with multiple layers of stochastic hidden unit above a single layer of observed variables that is data vector itself or some higher level data features which represent the data. Pretraining is unsupervised and greedy layerwise. Learning the connection weights in DBN is equivalent to training each adjaecent pair of layers on RBM. Once each layer is independently trained on RBM then finlly the backpropagation algorithm is being run on the whole system to fine tune the weights which is initialized by weights obtained from pretarining. Boltzman machines have the energy function linear in terms of it's parameters and Restricted boltzman machine resricts BMs to those without visible-visible and hidden-hidden connections. RBMs are undirected graphical models which is also an example of energy models. Energy function of RBM is defined as: \\
$E(h,v) = -b'v - c'h - h'Wv$ \\
where $W$ is the weight matrix of visible hidden connection, b is visible unit bias and c is hidden unit bias. Using this energy function it comes out that each hidden unit is indepenedent in itself given variable and that's why it's called RBM. $P(h_{i} = 1 | v)$ and vice versa is taken as sigmoid function.  
Once pretraining is completed then the weights are fine tuned using back propagation.

\textbf{CD-DNN-HMM}\\
HMM has been used very widely in the area of speech recognition. It's a generative model in which the accoustic features are assumed to be generated from hidden markov process. One need mainly three parameters(initial state probaility distribution, transition probability and observation probability) in HMM. In state of the art method the observation probabilty is modelled using Gaussian mixture model. The full architecture is described in the figure: \\
  
  \begin{figure}
  \centering
  \includegraphics[width=7.5cm, height=5cm]{image1.jpg}
  \caption{Diagram of the whole architecture}
  \end{figure}
  
Senones are directly modelled as the output of the neural network. So better we predict the probability of senones, better will be recognition result. The decoded word sequence $\hat{w}$ is determined as \\
\hat{w} =  \underset{w}{\operatorname{argmax}} p(w|x) = \underset{w}{\operatorname{argmax}} p(x|w)p(w)/p(x) \\
where $p(w)$ is the language model probaility and $p(x|w)$ is the acoustic model probability. In this method the observation probabilty is determinned using the output of the DNN instead of GMM which is used in GMM-HMM case. Training of CD-DNN-HMM method involves viterbi algorithm apart from deep unsupervised pretraing and fine tuning of weights. For it we have to first train a CD-GMM-HMM model to get label the senones. Using this labelled senones the DNN is trained.

\textbf{Results}\\
Results are evaluated on the dataset collected from the Bing mobile voice search application and there is significant amount of improvement in result as comapre to various GMM-HMM models. Reesults are also compared with context independent ANN-HMM model and the proposedmethod in this paper shows some improvemnts.
-------Anant Raj 10086---------
}

}