It depends. Most conferences specifically state their policy in relation to preprints. For example NeurIPS states:

> What is the policy on comparisons to recent work? Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions. Authors are not expected to compare to work that appeared only a month or two before the deadline.

> Are arxiv papers also subject to the policy above? Yes, we do not distinguish arxiv papers and other published (conference & journal) papers, and the two-month rule applies in the same way. More nuanced judgements, including how to determine the date of publication, should be made by the area chair handling the submission.

and ICLR has a similar policy:

> Q: Are authors expected to cite and compare with very recent work? What about non peer-reviewed (e.g., ArXiv) papers? (updated on 7 November 2022)

> A: We consider papers contemporaneous if they are published (available in online proceedings) within the last four months. That means, since our full paper deadline is September 28, if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2022, authors are not required to compare their own work to that paper. Authors are encouraged to cite and discuss all relevant papers, but they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which includes papers exclusively available on arXiv. Reviewers are encouraged to use their own good judgement and, if in doubt, discuss with their area chair.

You should check the policy for the conference you are reviewing for/submitting to for more relevant instructions.

Thanks, I never knew this is written in reviewer guide.

Besides conferences policy as u/dojoteef mentioned, you may be reviewing for a journal that does not have an explicit rule for that. In this case, you may find some papers that have been in Arxiv for a long time and not published anywhere else, but they are considered as references to many, such as CPC for example. In this case, I guess it is Ok to ask to consider such papers.

However, if this is not the case and the paper is recent, it may not be a good idea. But eventually, it is left to your judgment as a reviewer; an expert in the domain.

You may also ask the authors to consider an Arxiv paper, but not make it the basis of your yes/no decision.

If that arxiv paper has high impact (i.e., lots of citations), I think yes.

Yes.

If an ArXiv paper beats you, you are still beaten. If a blog post beats you, you are still beaten.

Peer reviewed material does not have a special status within science. Some important results are in somebody's BSc thesis, or on website somewhere, or in some mathematicians talk, which somebody wrote up a lecture note about. That doesn't mean that they can just be ignored. This can be true even for big results. For example, the proof of Poincaré conjecture were simply TeXed up by the mathematician who proved it and put in his university webpage.

Here's an upvote for a strong stance against gatekeeping.

Thanks. It's nice to know that not everyone disagrees.