MullOverThings

Useful tips for everyday

# Why does cautiously initializing deep neural networks matters?

## Why does cautiously initializing deep neural networks matters?

A neural network works best when input data is centered (have a mean of 0 and std of 1). So when input values get multiplied by weight values, their activation remains on scale 1. What does it do? It helps in graceful optimization of neural networks.

## How to initialize a 5 layer neural network?

Authors considered neural networks with 5 layers deep with standard initialization. They initialized random weights from a normal distribution (0 mean and 1 variance). 0 peak increases for higher layers (layer 4 and 5).

## Are there deep neural networks dramatically overfitted?

If you are like me, entering into the field of deep learning with experience in traditional machine learning, you may often ponder over this question: Since a typical deep neural network has so many parameters and training error can easily be perfect, it should surely suffer from substantial overfitting.

## Are there any proofs of two layer neural networks?

Zhang, et al. (2017) provided a neat proof on the finite-sample expressivity of two-layer neural networks.

## Why is ReLU activation important for deep neural networks?

ReLU activation is a threshold at zero which enables the network to have sparse representations. For eg, after uniform initialization of the weights, around 50% of hidden units continuous output values are real zeros. Relu loses a lot of information (get replaced by zero values) this effects into aggressive data compression.

## How does zero padding in convolutional neural networks work?

Zero padding is a technique that allows us to preserve the original input size. This is something that we specify on a per-convolutional layer basis. With each convolutional layer, just as we define how many filters to have and the size of the filters, we can also specify whether or not to use padding. What is zero padding?

## How to calculate w _ l of a neural network?

W_l = d by n weight matrix, where d is the number of filters. n is x’s length which is n=k ² c. Channels of current layer l are the same as filters of the previous layer (l-1). y_l = resulting vector after matrix multiplication of weights and inputs and addition of bias.