https://openreview.net/forum?id=qrwe7XHTmYb
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding | OpenReview
Neural network scaling has been critical for improving the model quality in many real-world machine learning applications with vast amounts of training data...
gshardscalinggiantmodels