dojoteef t1_iyvxzsz wrote
See the author's explanation on OpenReview:
> We update the result tables in the camera-ready version. The revision is due to a different data version of query augmentation. Previously, the data is cooked by one of our co-authors while using a different train-test split to train the query generator, causing some data leakage issue. All experiments in the previous submission are based on this query augmentation version, so the performance is relatively higher. When preparing the camera-ready version, we review and reproduce the code end-to-end for official release. At that time, we realize the data leakage problem. So, we re-cook the query augmentation data and reproduce all the experiments again in the new table. After solving the data leakage problem, NCI still shows more than 15% improvement over the current best SOTA. We have released the complete open-source code at GitHub: > > https://github.com/solidsea98/Neural-Corpus-Indexer-NCI > > Welcome to follow and reproduce our work. Looking forward to further discussions and collaborations.
Even_Stay3387 OP t1_iyvyx5w wrote
are you sure this is allowed? data leakage... really funny. I can also post a very very good results to cheat the review and then say I am sorry there is data leakage here.
mil24havoc t1_iyw1c2s wrote
You should be a better scientist. Best papers shouldn't be awarded for performance, that would be bad science. Best papers are awarded for innovation and quality. Fixing your results is the responsible thing to do.
42gauge t1_izye18i wrote
> Best papers shouldn't be awarded for performance, that would be bad science. Best papers are awarded for innovation and quality
But exceptionally good performance, whether real or fake, is usually used as a predictor for innovation and quality. If the authors hadn't made this mistake, this paper would have obviously been of higher quality - and yet, do you really think it would have stood out even more without that error?
dojoteef t1_iyw254f wrote
Mistakes happen. In this case the authors report the issue publicly and should be commended for that.
The NeurIPS organizers can choose to address the issue in whatever way they deem appropriate, especially as the authors are not hiding the fact that their results were changed.
Of course you're free to assume it's malicious if you want (at least that seems to be the stance your taking, but if it's not then I might have misinterpreted your response).
dulipat t1_iyw3ycl wrote
Very good results with bad scientific method will be rejected anyway
42gauge t1_izye37i wrote
Did this paper have very good results with bad scientific method?
Viewing a single comment thread. View all comments