We propose a stochastic recursive momentum method for Riemannian non-convex optimization that achieves a near-optimal complexity of $\tilde{ \mathcal{ O}}(\epsilon^{ -3})$ to find $\epsilon$-approximate solution with one sample. That is, our method requires $\mathcal{ O}(1)$ gradient evaluations per iteration and does not require restarting with a large batch gradient, which is commonly used to obtain the faster rate. Extensive experiment results demonstrate the superiority of our proposed algorithm.
Categories: math.OC cs.LG stat.ML