Newtonian-Shampoo: Modified Newton-Schulz Adapted for Shampoo Preconditioners

Shampoo optimizer was proposed in [1]

References:

TODO:

• [] Shampoo won ML-Perf competition
• [] the best optimizer in the paper Beyond Outliers
• [] slow because of EVD and/or Coupled-Newton
• [] explain how we adapt Newton-Schulz for -1/p power
• [] provide implementation
• [] show results