机器学习中的神经网络Neural Networks for Machine Learning：Lecture 15 Quiz

Lecture 15 QuizHelp
Center

Warning: The hard deadline has passed. You can attempt it, but you will not get credit for it. You are welcome to try it as a learning exercise.

In
accordance with the Coursera Honor Code, I  certify that the answers here are my own work.

Question 1

The objective function of an autoencoder is to reconstruct its input, i.e., it is trying to learn a function f,
such that f(x)=x for
all points x in
the dataset. Clearly there is a trivial solution to this. f can
just copy the input to the output, so that f(x)=x for
all x.
Why does the network not learn to do this ?

Optimization
algorithms that are used to train the autoencoder are not exact. So the network cannot easily learn to copy the input to the output.
The
network has constraints, such as bottleneck layers, sparsity and bounded activation functions which make the network incapable of copying the entire input all the way to the output.
The
objective function used to train an autoencoder is to minimize reconstruction error if x lies
in the dataset but maximize it if x does
not belong to the dataset. This prevents it from copying the input to the output for all x.
Since
all the hidden units in an autoencoder are linear, the model is not powerful enough to learn the identity transform, unless the number of hidden units in each layer is not less than the number of input dimensions.

Question 2

The process of autoencoding a vector seems to lose some information since the autoencoder cannot reconstruct the input exactly (as seen by the blurring of reconstructed images reconstructed from 256-bit codes). In other words, the intermediate representation
appears to have less information than the input representation. In that case, why is this intermediate representation more useful than the input representation ?

The
intermediate representation has more noise.
The
intermediate representation actually has more information than the inputs.
The
intermediate representation loses some information, but retains what is most important. We hope that this retained information is "semantic". The intermediate representation will then be a more direct way of representing semantic content.
The
intermediate representation is more compressible than the input representation.

Question 3

What are some of the ways of regularizing deep autoencoders?

Using
large minibatches for stochastic gradient descent.
Using
a squared error loss function for the reconstruction.
Adding
noise to the inputs.
Using
high learning rate and momentum.

Question 4

In all the autoencoders discussed in the lecture, the decoder network has the same number of layers and hidden units as the encoder network, but arranged in reverse order. Brian feels that this is not a strict requirement for building an autoencoder. He insists
that we can build an autoencoder which has a very different decoder network than the encoder network. Which of the following statements is correct?

Brian
is correct. We can indeed have any decoder network, as long as it produces output of the same shape as the data, so that we can compare the output to the original data and tell the network where it's making mistakes.
Brian
is correct, as long as the decoder network has at least as many parameters as the encoder network.
Brian
is mistaken. The decoder network must have the same architecture. Otherwise backpropagation will not work.
Brian
is correct, as long as the decoder network has the same number of parameters as the encoder network.

Question 5

Another way of extracting short codes for images is to hash them using standard hash functions. These functions are very
fast to compute, require no training and transform inputs into fixed length representations. Why is it more useful to learn an autoencoder to do this ?

Autoencoders
have several hidden units, unlike hash functions.
Autoencoders
have smooth objective functions whereas standard hash functions have no concept of an objective function.
For
an autoencoder, it is possible to invert the mapping from the hashed value to the reconstruct the original input using the decoder, while this is not true for most hash functions.
Autoencoders
can be used to do semantic hashing, where as standard hash functions do not respect semantics , i.e, two inputs that are close in meaning might be very far in the hashed space.

Question 6

RBMs and single-hidden layer autoencoders can both be seen as different ways of extracting one layer of hidden variables from the inputs. In what sense are they different ?

The
objective function and its gradients are intractable to compute exactly for RBMs but can be computed efficiently exactly for autoencoders.
RBMs
are undirected graphical models, but autoencoders are feed-forward neural nets.
RBMs
define a probability distribution over the hidden variables conditioned on the visible units while autoencoders define a deterministic mapping from inputs to hidden variables.
RBMs
work only with binary inputs but autoencoders work with all kinds of inputs.

Question 7

Autoencoders seem like a very powerful and flexible way of learning hidden representations. You just need to get lots of data and ask the neural network to reconstruct it. Gradients and objective functions can be exactly computed. Any kind of data can be plugged
in. What might be a limitation of these models ?

Autoencoders
cannot work with discrete-valued inputs.
The
hidden representations are noisy.
If
the input data comes with a lot of noise, the autoencoder is being forced to reconstruct noisy input data. Being a deterministic mapping, it has to spend a lot of capacity in modelling the noise in order to reconstruct correctly. That capacity is not being
used for anything semantically valuable, which is a waste.
The
inference process for finding states of hidden units given the input is intractable for autoencoders.