In addition to being seen as an
autoencoder neural network architecture, variational autoencoders can also be studied within the mathematical formulation of
variational Bayesian methods, connecting a neural encoder network to its decoder through a probabilistic
latent space (for example, as a multivariate Gaussian distribution) that corresponds to the parameters of a variational distribution.
Thus, the encoder maps each point (such as an image) from a large complex dataset into a distribution within the latent space, rather than to a single point in that space. The decoder has the opposite function, which is to map from the latent space to the input space, again according to a distribution. By mapping a point to a distribution instead of a single point, the network can avoid overfitting the training data.[2] Both networks are typically trained together with the usage of the
reparameterization trick, although the variance of the noise model can be learned separately.
A variational autoencoder is a generative model with a prior and noise distribution respectively. Usually such models are trained using the
expectation-maximization meta-algorithm (e.g.
probabilistic PCA, (spike & slab) sparse coding). Such a scheme optimizes a lower bound of the data likelihood, which is usually intractable, and in doing so requires the discovery of q-distributions, or variational
posteriors. These q-distributions are normally parameterized for each individual data point in a separate optimization process. However, variational autoencoders use a neural network as an amortized approach to jointly optimize across data points. This neural network takes as input the data points themselves, and outputs parameters for the variational distribution. As it maps from a known input space to the low-dimensional latent space, it is called the encoder.
The decoder is the second neural network of this model. It is a function that maps from the latent space to the input space, e.g. as the means of the noise distribution. It is possible to use another neural network that maps to the variance, however this can be omitted for simplicity. In such a case, the variance can be optimized with gradient descent.
To optimize this model, one needs to know two terms: the "reconstruction error", and the
Kullback–Leibler divergence(KL-D). Both terms are derived from the free energy expression of the probabilistic model, and therefore differ depending on the noise distribution and the assumed prior of the data. For example, a standard VAE task such as IMAGENET is typically assumed to have a gaussianly distributed noise; however, tasks such as binarized MNIST require a Bernoulli noise. The KL-D from the free energy expression maximizes the probability mass of the q-distribution that overlaps with the p-distribution, which unfortunately can result in mode-seeking behaviour. The "reconstruction" term is the remainder of the free energy expression, and requires a sampling approximation to compute its expectation value.[8]
Formulation
From the point of view of probabilistic modeling, one wants to maximize the likelihood of the data by their chosen parameterized probability distribution . This distribution is usually chosen to be a Gaussian which is parameterized by and respectively, and as a member of the exponential family it is easy to work with as a noise distribution. Simple distributions are easy enough to maximize, however distributions where a prior is assumed over the latents results in intractable integrals. Let us find via
marginalizing over .
where represents the
joint distribution under of the observable data and its latent representation or encoding . According to the
chain rule, the equation can be rewritten as
In the vanilla variational autoencoder, is usually taken to be a finite-dimensional vector of real numbers, and to be a
Gaussian distribution. Then is a mixture of Gaussian distributions.
It is now possible to define the set of the relationships between the input data and its latent representation as
Prior
Likelihood
Posterior
Unfortunately, the computation of is expensive and in most cases intractable. To speed up the calculus to make it feasible, it is necessary to introduce a further function to approximate the posterior distribution as
with defined as the set of real values that parametrize . This is sometimes called amortized inference, since by "investing" in finding a good , one can later infer from quickly without doing any integrals.
In this way, the problem is to find a good probabilistic autoencoder, in which the conditional likelihood distribution is computed by the probabilistic decoder, and the approximated posterior distribution is computed by the probabilistic encoder.
As in every
deep learning problem, it is necessary to define a differentiable loss function in order to update the network weights through
backpropagation.
For variational autoencoders, the idea is to jointly optimize the generative model parameters to reduce the reconstruction error between the input and the output, and to make as close as possible to . As reconstruction loss,
mean squared error and
cross entropy are often used.
As distance loss between the two distributions the Kullback–Leibler divergence is a good choice to squeeze under .[8][9]
is equivalent to simultaneously maximizing and minimizing . That is, maximizing the log-likelihood of the observed data, and minimizing the divergence of the approximate posterior from the exact posterior .
The form given is not very convenient for maximization, but the following, equivalent form, is:
where is implemented as , since that is, up to an additive constant, what yields. That is, we model the distribution of conditional on to be a Gaussian distribution centered on . The distribution of and are often also chosen to be Gaussians as and , with which we obtain by the formula for
KL divergence of Gaussians:
Here is the dimension of . For a more detailed derivation and more interpretations of ELBO and its maximization, see
its main page.
does not allow one to put the inside the expectation, since appears in the probability distribution itself. The reparameterization trick (also known as stochastic backpropagation[10]) bypasses this difficulty.[8][11][12]
The most important example is when is normally distributed, as .
Since we reparametrized , we need to find . Let be the probability density function for , then [clarification needed]
where is the Jacobian matrix of with respect to . Since , this is
Variations
Many variational autoencoders applications and extensions have been used to adapt the architecture to other domains and improve its performance.
-VAE is an implementation with a weighted Kullback–Leibler divergence term to automatically discover and interpret factorised latent representations. With this implementation, it is possible to force manifold disentanglement for values greater than one. This architecture can discover disentangled latent factors without supervision.[13][14]
The conditional VAE (CVAE), inserts label information in the latent space to force a deterministic constrained representation of the learned data.[15]
Some structures directly deal with the quality of the generated samples[16][17] or implement more than one latent space to further improve the representation learning.
^Dilokthanakul, Nat; Mediano, Pedro A. M.; Garnelo, Marta; Lee, Matthew C. H.; Salimbeni, Hugh; Arulkumaran, Kai; Shanahan, Murray (2017-01-13). "Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders".
arXiv:1611.02648 [
cs.LG].