Gradient Descent Variants

#1

In video , it is said that to optimise gradient descent algorithm, we have used mini batch , so now in mini batch, it is said that , instead of iterating over all the training data, we are iterating through no of batches (which is basically group of training examples if i m not wrong) , now my question is even if we are iterating through batches only, indirectly we are iterating through all the training examples, so how does it reduces time???

#2

Hello Shalini,
Mini-batch Gradient Descent basically loads the training data that we have in mini-batches (of 32,64,128 or such number of training examples per batch), computes loss for every training example but updates the weights only when the entire mini-batch has been iterated over.

The advantages of this approach are as below:

  1. The model update frequency is higher than batch gradient descent which allows for a more robust convergence, avoiding local minima.
  2. The batched updates provide a computationally more efficient process than stochastic gradient descent.
  3. The batching allows both the efficiency of not having all training data in memory and algorithm implementations because the RAM does not have to store the entire training data at once, it holds the mini batches.

The other two general approaches that are used in Gradient Descent:
Stochastic Gradient Descent
Stochastic gradient descent, often abbreviated SGD, is a variation of the gradient descent algorithm that calculates the error and updates the model for each example in the training dataset.

Batch Gradient Descent
Batch gradient descent is a variation of the gradient descent algorithm that calculates the error for each example in the training dataset, but only updates the model after all training examples have been evaluated Basically, it considers the entire training set as a batch.

…

I hope this gave you clarity in this topic.

#3

I hope this resolved your query?

#4

I hope I’ve cleared your doubt. I ask you to please rate your experience here
Your feedback is very important. It helps us improve our platform and hence provide you
the learning experience you deserve.

On the off chance, you still have some questions or not find the answers satisfactory, you may reopen
the doubt.