Interesting. I’m taking Nonlinear Optimization (taught by Steve Wright) and we give some theoretical analysis of these methods, but it’s nice to see the big picture with visualization.
Great. Possible to share the link to the notes or video lectures (if any). I have gone through vanilla grad. descent, SGD and Mini-bathGD, and currently going through the Stochastic Average Gradient solver. Would love to dig deep into the topic.