AdMassive9465

AdMassive9465 t1_j5aliqj wrote

It was not: in theory, it should have been a discussion that involved "only" the reviewers and the AC. However, none of the reviewers (besides me) participated---thereby resulting in a 1:1 discussion with the AC (whose remarks could not have been addressed in any "revision" of the paper, since this occurred in December).

Interestingly, there is even a "gap" between the Reviewer and AC guidelines:

  • according to the AC guidelines, the authors could "participate in the discussion" occurring between November 18th and December 12th...
  • ...but this was not stated in the reviewer guidelines, which stated that, after November 18th, the only role of the reviewer was to have the "virtual discussion with the AC for borderline papers".

I was unaware that the authors could still "participate" after November 18th, so I did not even think that it was possible to involve them in the discussion (had I known that the other reviewers would have done nothing, I would have certainly involved the authors)

3

AdMassive9465 t1_j59s6n6 wrote

Agreed. I don't know what happened behind the curtains, and I hope that whatever happened was done for a good reason.

Yet, I have a bad aftertaste: I cannot stop thinking that my effort was useless. Truly, I could have written shit-reviews such as "no novelty, the paper ultimately describes a method related with neural networks, which have been studied for decades" (jk) and forget about it.

6

AdMassive9465 t1_j59pyqi wrote

Personally, as a reviewer, this year's ICLR reviewing system was a trainwreck.

  • Reviewers had 12 days to review ~5 papers. Twelve days to review 5 papers submitted at (arguably) one of the best venues of this (complex) domain.
  • Of the 16 expected reviews (for these 5 papers), only 8 had been delivered before the deadline (5 were mine). 5 were submitted on the day of the deadline. 3 were submitted in the "buffer" period before the opening of the rebuttal phase.
  • A paper was withdrawn by the authors after the opening of the rebuttal phase. Surprisingly -- for this paper -- a reviewer proposed to reject the paper and, among the motivations, they stated that the paper should "compare" with a paper that had been accepted 1 month before the submission deadline of ICLR'23, and that was supposed to be presented at a conference held in mid 2023.
  • Of the other 16 reviewers assigned to the other papers I reviewed, only 3 wrote one post (aside from the review). I even explicitly asked the support of other reviewers w.r.t. an obscure point of a paper, and received no response (this paper was rejected).
  • I championed one paper, which was ultimately rejected by the AC who had great misconceptions of the paper as a whole. None of the reviewers chimed in, and my attempts at changing the AC's opinion were in vain (the AC was not "wrong", but they were just looking at the paper from the wrong perspective). I feel very sorry for the authors since they had very little they could do: they interacted just with myself, and I was pleased by the paper; and now they got a reject due to concerns expressed _after_ the closure of the rebuttal phase (and raised just between myself and the AC).
  • Out of the 5 papers I reviewed, only one was accepted.

I think that the worst part, however, is that all of the above will be "meaningless" given the ridiculous amount of submissions to ICLR'23.

Frankly speaking, this is discouraging as a reviewer who "tries their best" to help research.

^(Edit: fixed numbers after double-checking)

22