Viewing a single comment thread. View all comments

pyepyepie t1_iywkmz6 wrote

Reply to comment by lemlo100 in [D] NeurIPS 2022 Outstanding Paper modified results significantly in the camera ready by Even_Stay3387

I was a software engineer for a few years (I would probably say I am a little more skilled as a coder than in DS), and I still find it difficult to not mess up experiments if I don't recheck myself. Mostly, I just assume my results are garbage and try to attack them until I come to the conclusion that it's actually real. It's even more important when the task is not supervised (i.e., difficult to implement, MARL, GANs...), for example (RL) - you might think you developed a nice algorithm just to find out you accidentally modified the rewards.

45

lemlo100 t1_iywnr89 wrote

Totally true. I also tend to believe my results are garbage and double- and triple-check. For my last project I implemented some tests in fact. It was a data augmentation approach for reinforcement learning so it was testable. My supervisor was not happy about is and considered it a waste of time. I also ran about 50 seeds after reading the Neurips best paper "On the edge of the statistical precipice" in my experiments as opposed to only five like my supervisor used to do. We were not able to work together and ended it early because he didn't want me junior interfering in him dashing out cooked results.

Edit: That same supervisor, by the way, had a paper published that contained a bug. Sampling was not quite implemented the way it was described in the paper. When I brought attention to this, since my project was based on this piece of code, instead of thanking me for spotting the bug he argued how in his opinion it shouldn't make a difference. That was shocking.

43

pyepyepie t1_iywowgo wrote

Thank you sir for making SIGNIFICANT contributions, it takes a lot to go against your supervisor's opinions, but it seems like you did the moral thing.

8

maxToTheJ t1_iywupll wrote

> Totally true. I also tend to believe my results are garbage and double- and triple-check.

The market doesnt reward that though. We cant really say for sure that the paper being discussed would have won Outstanding Paper with the less impressive gains so at the end of the day not checking could inadvertantly help your career.

8

pyepyepie t1_iyx0k1s wrote

True. Who am I to say what is good and what's not, but I tend to enjoy simple papers with good ideas much more than papers that contain many moving parts (I am 100% unable to get that kind of result but I can enjoy it :) ).

I kind of treat complicated papers without robust code as noise or maybe a source of ideas, but when I try to implement it it's mostly not working as well as expected - e.g., I had to implement a model for a task related to speech and I have no expertise in the field, most of the models I tried to use were really bad in comparison to a good, simple solution (inspired by ResNet), and I found a model that performs better only due to preprocessing. It's hard to come up with new ideas so I am happy there is so much information, but sometimes it's too much.

1

domestication_never t1_iyy7qc8 wrote

I am a manager that works both with scientists and engineers.Every new scientist gets sent to "coding bootcamp" and doesn't come back till they learn unit testing a a minimum.

Every engineer gets sent to machine learning bootcamp and doesn't come back till they can explain WAPE, MAPE, overfitting etc.

I do this as much for quality software as to stop the damn fights. At least they have an appreciation for the finer points of the others profession.

6