Jan 21 '23 02:00 AM UTC

Coming soon

I see, thanks

Decisions are out!

First first-authored paper here, got accepted with a spotlight!

Welp, rejected. To be fair, I was hoping for an AC stroke of luck, which by definition is rare. Guess I'll drink something and start working on the ICML submission.

I'm not seeing area chair meta reviews or decisions on OpenReview yet. Anyone see them on their papers? I got an email only.

Can you see your meta review?

Yes! (I meant to comment that decisions are out, not reviews - sorry.)

I can see decision and meta review comment on openreview

Ahh, I had to log in to see the meta review.

I found it by logging in. Maybe they aren't public yet?

Right, not public yet

I think everyone that is either top-5 or top-25 gets an oral. There are no specific spotlights this year.

Hope you have more luck next time!

congrats! spotlight here as well. I guess we'll see each other in Kigali :)

Ahh yes, you're right!

Congrats to you as well!! Hope to see you there! :)

I also have one :)

That's really wonderful, congrats!! :)

1

Do we know when are they going to get public?

Probably soon? Or at camera ready stage?

The email says "All papers will be de-anonymized within a week, and the last version of the submitted paper will remain available on the openreview website".

Spotlight vs notable-top-25% distinction - From the email:

This year, we are disassociating the presentation style (e.g., Accept (oral), Accept (spotlight)) from the AC recommendation. What this means is that your paper will receive 2 "tags":

Tag for AC’s recommendation

• notable-top-5% was traditionally called Accept (oral)
• notable-top-25% was called Accept (spotlight).
• Poster accept

Tag for presentation style

• In-person presentation
• Digital presentation

Why 2-tags? We decided to host all oral presentations only in-person. While all accepted papers will be offered digital presentation (i.e., video will be available for asynchronous viewing), notable-top-5% and notable-top-25% papers will also be offered an in-person presentation. Poster accept will be offered in-person poster session as well.

Personally, as a reviewer, this year's ICLR reviewing system was a trainwreck.

• Reviewers had 12 days to review ~5 papers. Twelve days to review 5 papers submitted at (arguably) one of the best venues of this (complex) domain.
• Of the 16 expected reviews (for these 5 papers), only 8 had been delivered before the deadline (5 were mine). 5 were submitted on the day of the deadline. 3 were submitted in the "buffer" period before the opening of the rebuttal phase.
• A paper was withdrawn by the authors after the opening of the rebuttal phase. Surprisingly -- for this paper -- a reviewer proposed to reject the paper and, among the motivations, they stated that the paper should "compare" with a paper that had been accepted 1 month before the submission deadline of ICLR'23, and that was supposed to be presented at a conference held in mid 2023.
• Of the other 16 reviewers assigned to the other papers I reviewed, only 3 wrote one post (aside from the review). I even explicitly asked the support of other reviewers w.r.t. an obscure point of a paper, and received no response (this paper was rejected).
• I championed one paper, which was ultimately rejected by the AC who had great misconceptions of the paper as a whole. None of the reviewers chimed in, and my attempts at changing the AC's opinion were in vain (the AC was not "wrong", but they were just looking at the paper from the wrong perspective). I feel very sorry for the authors since they had very little they could do: they interacted just with myself, and I was pleased by the paper; and now they got a reject due to concerns expressed _after_ the closure of the rebuttal phase (and raised just between myself and the AC).
• Out of the 5 papers I reviewed, only one was accepted.

I think that the worst part, however, is that all of the above will be "meaningless" given the ridiculous amount of submissions to ICLR'23.

Frankly speaking, this is discouraging as a reviewer who "tries their best" to help research.

^(Edit: fixed numbers after double-checking)

About the trainwreck review part, I totally agree with you. Even after the deadline, I was getting random (I am saying random because I had almost close to zero expertise in those fields, so I had to decline all those requests) emergency review invitations which made me understand the overall situation. One thing that made me really confused was: why did they take so much time (almost 2 months, after the author discussion period was over) to send out the decision? I was really not sure why the meta-review process deserved this much time (~60 days) when the actual review time was just 12 days! Any thoughts about this?

Agreed. I don't know what happened behind the curtains, and I hope that whatever happened was done for a good reason.

Yet, I have a bad aftertaste: I cannot stop thinking that my effort was useless. Truly, I could have written shit-reviews such as "no novelty, the paper ultimately describes a method related with neural networks, which have been studied for decades" (jk) and forget about it.

This is very interesting to read. For the paper you championed, was that discussion between you and the AC public, i.e. visible to authors? I had a similar experience from an author perspective. We had really constructive discussions with all the reviewers, and were fairly optimistic we'd get in, but then the meta-review was pretty disconnected from all of that. Unfortunately no idea what may have happened between AC and reviewers, so it's a bit of a mystery from our perspective. It's frustrating to have all these in-depth discussions, but then no opportunity to clear up even basic misunderstandings in the meta-review, where it actually matters. Having a discussion period with the AC would have made such a difference, and given the timeline it seems it would totally have been feasible.

It was not: in theory, it should have been a discussion that involved "only" the reviewers and the AC. However, none of the reviewers (besides me) participated---thereby resulting in a 1:1 discussion with the AC (whose remarks could not have been addressed in any "revision" of the paper, since this occurred in December).

Interestingly, there is even a "gap" between the Reviewer and AC guidelines:

• according to the AC guidelines, the authors could "participate in the discussion" occurring between November 18th and December 12th...
• ...but this was not stated in the reviewer guidelines, which stated that, after November 18th, the only role of the reviewer was to have the "virtual discussion with the AC for borderline papers".

I was unaware that the authors could still "participate" after November 18th, so I did not even think that it was possible to involve them in the discussion (had I known that the other reviewers would have done nothing, I would have certainly involved the authors)

Rejected after an 866 score (up from 856). the 8 score reviewer had given a 2 line review and the meta reviewer said their review was discounted (I expected this).The sad part is that in both neurips (score 656) and iclr (score 866), our paper was borderline/weak accept from all reviewers (and one 8 score this time), but the meta reviewer comes up with new problems with the paper, without giving us a chance to respond.

Any way, onto the grind for the next conferences, I hope I will get a first author top conference paper this year!

link?

Got rejected, due to a reviewer changing his score from 8 to 5.