Implementation of Back-propagation

#1

Hi! I have tried to implement the back-propagation with some flexibility, however the updation is not getting proper as due to high values of the weight matrices after the updation is causing the softmax function to provide NaN values. My code is available at https://ide.codingblocks.com/s/277136 ,please let me know how to overcome the scenario, what modification I need to introduce in the code.

#2

Hey @tisandas2011

First thing you should do is to try lowering the Learning Rate. Try values like 1e-3 or 1e-4. A high learning rate can make the model diverge and explode the gradients.

Let me know if this helps!

#3

Hi Vasu, after chaning the learning rate to 0.000001, it seems that the loss is steadily increasing. My hunch is it might be due to the weight update. Is there anything I missed while implementing this?

#4

Loss at itr 0 : 1577.4572639692792 Loss at itr 1 : 1578.0108886205794 Loss at itr 2 : 1578.5630795862435 Loss at itr 3 : 1579.113856051781 Loss at itr 4 : 1579.6632370543093 Loss at itr 5 : 1580.2112414833935 Loss at itr 6 : 1580.757888081897 Loss at itr 7 : 1581.303195446841 Loss at itr 8 : 1581.847182030272 Loss at itr 9 : 1582.3898661401372 Loss at itr 10 : 1582.9312659411685

#5

There must be some mistake in the implementation. The loss should never increase steadily for such a small Learning Rate. Why have you added these columns of ones to the weight matrices?
image

#6

Hey @tisandas2011,

I have made quite a few changes to your code. Your formulae were correct but your code was missing many conventions and practices used in Neural Nets (Look at the weight matrix initialization in the modified code). This is the training data after making the changes:
image

I’m attaching the modified the code. Study it carefully and try to find the corrected mistakes.
https://drive.google.com/file/d/1Og9_4Pp1gjKr9QH_mFFOkxZ9BMBjl-zb/view?usp=sharing

#7

Hi! currently I am going through the same code… However I have copied the modified NeuralNet code only into my notebook like as https://ide.codingblocks.com/s/281138 .I am seeing the same issue of increasing error. Not sure whether still I am missing something even after using the modification of NeuralNet class. Please let me know.

#8

Try making these two changes:

  1. Change the multiplier of weight initialization. Use 0.01 instead:
    w = 0.01*np.random.normal(size=(<required-size>))
  2. Scale your dataset.

Let me know if these measures help!

#9

I hope I’ve cleared your doubt. I ask you to please rate your experience here
Your feedback is very important. It helps us improve our platform and hence provide you
the learning experience you deserve.

On the off chance, you still have some questions or not find the answers satisfactory, you may reopen
the doubt.