Wow. 5000 TPUs
Thinking about it, and looking at the graphs of learning curves etc, this paper is Google saying to everyone else: our hardware is so much better than yours & we have so much more of it that we can beat anything you do in AI without breaking sweat. You can’t touch us in other words.
Quantity has a quality all of it’s own & quantity counts more than anything else in current RNN training.