https://openreview.net/forum?id=fZqMVTz7K5
AdaMeM: Memory Efficient Momentum for Adafactor | OpenReview
Adafactor is a memory efficient algorithm which does not maintain momentum and has near 0 memory overhead as compared to gradient descent. However it performs...
memoryefficientmomentumopenreview