NN Implementation doubts

#1

HI! I am facing the following doubts while going through the Neural Network Implementation:

  1. delta3 = y_ - y. This means that the loss function is used is MSE, as opposed to cross entropy as shown in the loss method definition. Is my assumption correct or am I missing something here?

  2. In relation to the previous query, derivative of softmax activation function is provided with as 1. It would be great if some document for the derivation is provided.

  3. Could not derive the derivative of tanh, it would be great if some document for the derivation is provided.

It would help me a lot to understand the NN in details. Thanks.

#2

hey @tisandas2011 ,
Answers :
-> 1. Yeah it correct , that MSE is implemented. Is was just to show a simple neural network. If you want to dig deeper you can surely try that.
-> 2. Here is a link to a great page to understand the derivative of softmax : https://www.mldawn.com/the-derivative-of-softmaxz-function-w-r-t-z/
-> 3. For tanh :
a. https://theclevermachine.wordpress.com/2014/09/08/derivation-derivatives-for-common-neural-network-activation-functions/
b. https://blogs.cuit.columbia.edu/zp2130/derivative_of_tanh_function/
c. https://www.coursera.org/lecture/neural-networks-deep-learning/derivatives-of-activation-functions-qcG1j

I hope this would have helped you to understand the required topics.

Thank You. :slightly_smiling_face:

#3

Thanks for the help!