kajladk t1_is8lqr0 wrote on October 14, 2022 at 2:16 AM

Reply to comment by ReginaldIII in [N] First RTX 4090 ML benchmarks by killver

But isn't this different? We are comparing raw metric (fps, images/sec) with an aggregate score which might already have ways to eliminate or regularize some outlier metrics in-built

ReginaldIII t1_isal2mt wrote on October 14, 2022 at 2:36 PM

Nvidia and Puget want to report lucky run. Lots of people do this. They're being fully transparent that they are reporting lucky runs. And it makes sense from their perspective to report their best theoretical performance.

It honestly just doesn't bother me to see them doing it because it's very normal and lots of people report this way. Even if we think an average with an error bar would be fairer.