CNN Behavior Analysis: Visualizing the activations and first-layer weights

Layer Activations 

 

The most straight-forward visualization technique is to show the activations of the network during the forward pass. For ReLU networks, the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse and localized. One dangerous pitfall that can be easily noticed with this visualization is that some activation maps may be all zero for many different inputs, which can indicate dead filters, and can be a symptom of high learning rates.

Convolutional/FC Filters

 

The second common strategy is to visualize the weights. These are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network. The weights are useful to visualize because well-trained networks usually display nice and smooth filters without any noisy patterns. Noisy patterns can be an indicator of a network that hasn't been trained for long enough, or possibly a very low regularization strength that may have led to over fitting.

Back propagation 

 

Primary reason we are interested in this problem is that in the specific case of Neural Networks, f will correspond to the loss function ( L ) and the inputs x will consist of the training data and the neural network weights. Training data is given and fixed so it is a constant factor in the equation. So here we are left with two variables which are weights and biases of each layer. In back propagation Convolutional Neural Network compute the gradient at every layer according to the loss function at output and these calculated new weights are then updated to converse the network for the final solution.

An Introduction to CNN: Carrying the Machine learning on its shoulders

Traditional neural network layers use a matrix multiplication to describe the interaction between each input unit and each output unit. This means every output unit interacts with every input unit. Convolutional networks, however, typically have sparse interactions. This is accomplished by making the kernel smaller than the input and using it for the whole image.

Parameter sharing concept is used in CNN. It refers to using the same parameter for more than one function (input values) in a model. In a convolutional neural net, each member of the kernel is used at every position of the input. The parameter sharing used by the convolution operation means that rather than learning a separate set of parameters for every location, we learn only one set. This is also called as sparse connectivity.

If the function that a layer needs to learn is indeed a local, translation invariant function, then the layer will be dramatically more efficient if it uses convolution rather than matrix multiplication. If the necessary function does not have these properties, then using a convolutional layer will cause the model to have high training error.

Pooling

First stage, the layer performs several convolutions in parallel to produce a set of presynaptic activations. In the second stage, each presynaptic activation is run through a nonlinear activation function, such as the rectified linear activation function. This stage is sometimes called the detector stage. In the third stage, we use a pooling function. A pooling function replaces the output of the net at a certain location with a summary statistic of the nearby outputs. For example, the max pooling operation reports the maximum output within a rectangular neighborhood. Pooling helps to make the representation becomes invariant to small translations of the input.

Zero-padding

Zero-padding setting is when just enough zero-padding is added to keep the size of the output equal to the size of the input. It calls same convolution, full convolution, in which enough zeroes are added for every pixel to be visited k times in each direction.

The CNN behavior analysis will be explained in the further articles from the R&D team at SiliconMentor working in Computer Vision, Biomedical Signal Analysis, VLSI and their associated domains.