In training of neural network, my loss over the epochs(using gradient descent) first decreases, then increases a bit and then continues to decrease smoothly. What could be the possible mistake in my implementation?
Loss not decreasing smoothly
I just compared my code to prateek bhaiya’s code, and realised that i had initialised the biases with random values from np.random.randn, while he had initialised with zeros. Now my code works fine. Why is it important to initialise biases with zeros?
Hey @ambika11, it is like a thumb rule we need to initialize all bias as zeros or very small values near to zero. The reason being that, while back propagation, our gradients may not suffer from vanishing gradients or from exploding gradients.
Hope now your doubt is cleared 
Plz mark the doubt as resolved in my doubts section.