Viewing a single comment thread. View all comments

AndromedaAnimated t1_j1pzd1b wrote

I like the linked info, please don’t misunderstand. Thank you for posting!

I just… see so many flaws in this experimentation.

  1. The example with numbers instead of pictures is much easier as it circumvents most of visual and spatial processing of the human eyes and brain - and then also the typical human output (writing, pressing buttons, speaking etc.)

(I was able to solve it in seconds - and I think every human could. The visual one needed a minute or something which is longer. And I am human! The speed difference is also partly due to visual processing of numbers being not necessary in GPT. As long as this factor is not accounted to the results are not clean.)

  1. LLM do not have fears of rank loss or punishment, they don’t care if they are perceived as stupid, while human test subjects do. This interferes with the processing and leads to worse results.

That’s not fair testing. Results as such not comparable to human results.

If anyone wants sauce I will try to find, it’s no problem. Just wanted to throw in this ideas first because maybe someone can use them.

5