Hi, the author here. Thank you for your comment.

While I was aware of GoogLeNet, I didn't read the paper in enough detail to know that they used Polyak averaging too. Thank you for making me aware of it. I'm happy to cite it in the next paper version.

However, the only time they mention averaging is:

Polyak averaging [13] was used to create the final model used at inference time.

My goal with the paper was to study the empirical convergence speed-ups in more detail and be precise about how averaging is used, not claiming to be the first one who applies some sort of averaging to improve the model's final performance (there are plenty of papers that do that already, e.g., the SWA paper mentioned in the related work section).

​

EDIT: Added the citation to the new version!