MullOverThings

Useful tips for everyday

# Are local minima a problem in neural network?

## Are local minima a problem in neural network?

Supervised learning of multilayered neural networks with conventional learning algorithms faces the local minimum problems. Using a gradient descent to adjust the weights involves following a local slope of the error surface which may lead toward some undesirable points, or the local minima.

## How can local minima be avoided?

How to get out of a local minimum ?

1. Use another activation function. Instead of using ReLu, you could try using Tanh.
2. Play with the learning rate of your optimizer.
3. Try using the BatchNormalization layer in Keras.

## How do I stop local minima on CNN?

Reduce your learning rate. Reduce your batch size. Stack more layers. Check if your model is actually learning : send random noise as your data, and the network loss should not be decreasing.

## What is local and global minima?

A function can have multiple minima and maxima. The point where function takes the minimum value is called as global minima. Other points will be called as local minima. At all the minima points, the first order derivative will be zero and related value can be found where the local or global minima occurred.

## How the false minima get reduced?

Explanation: Presence of false minima will increase the probability of error in recall. Explanation: Presence of false minima can be reduced by stochastic update.

## When does a series converge in machine learning?

The series is of course an infinite series only if you assume that loss = 0 is never actually achieved, and that learning rate keeps getting smaller. Essentially meaning, a model converges when its loss actually moves towards a minima (local or global) with a decreasing trend.

## Why are local minima so rare in deep learning?

Because the number of dimensions are so large with deep learning, the probability that an optimum only consists of a combination of minima is very low. This means ‘getting stuck’ in a local minimum is rare.

## Can a strictly converging model be used as convergence?

Its quite rare to actually come across a strictly converging model but convergence is commonly used in a similar manner as convexity is. Strictly speaking rarely exists practically, but is spoken in a manner telling us how close the model is to the ideal scenario for convexity, or in this case convergence.

## Do you need more hidden layers for convergence?

More or less hidden layers should not affect convergence though the generalization power of the two would be different. For a deep neural network that you mention, finding an effective local minima is the key. As per the paper, Gülçehre, Çağlar, and Yoshua Bengio.