Viewing a single comment thread. View all comments

kajladk t1_is8gtdt wrote

Reply to comment by ReginaldIII in [N] First RTX 4090 ML benchmarks by killver

Can anyone explain how taking the max value from 100 runs is a good benchmark when for most other benchmarks (gaming fps etc) the average fps across multiple runs gives a more realistic performance and eliminates any outliers

0

afireohno t1_isblrse wrote

>average fps across multiple runs gives a more realistic performance and eliminates any outliers

Thanks for the laugh. I'll just leave this here so you can read about why the mean (average) is not a robust measure of central tendency because it is easily skewed by outliers.

1

kajladk t1_isd009f wrote

Umm, I know mean is more skewed by outliers than median, but it's still "better" than taking the best value

1

ReginaldIII t1_is8j7x8 wrote

I would say there's precedent for lucky run benchmark scores. Consider 3dmark as an example.

https://benchmarks.ul.com/hall-of-fame-2/timespy+3dmark+score+performance+preset/version+1.0

All of those runs with different system configurations are peoples luckiest runs.

0

kajladk t1_is8lqr0 wrote

But isn't this different? We are comparing raw metric (fps, images/sec) with an aggregate score which might already have ways to eliminate or regularize some outlier metrics in-built

1

ReginaldIII t1_isal2mt wrote

Nvidia and Puget want to report lucky run. Lots of people do this. They're being fully transparent that they are reporting lucky runs. And it makes sense from their perspective to report their best theoretical performance.

It honestly just doesn't bother me to see them doing it because it's very normal and lots of people report this way. Even if we think an average with an error bar would be fairer.

0