[P] searchthearxiv.com: Semantic search across more than 250,000 ML papers on arXiv

Submitted by universal_explainer t3_106hxgk in MachineLearning

I just launched searchthearxiv.com, a simple semantic search engine over virtually all ML papers published on arXiv since 2012. The site uses OpenAI's `text-embedding-ada-002` model to match the embedding of your query against each of the paper embeddings, retrieving the ones with the highest cosine similarity. It also allows you to insert an arXiv link to find similar papers.

This was mostly meant as a fun side project. However, if people find it useful, I'm happy to maintain it and keep the database up-to-date. I'd love to know what you think! ❤️

Update: Thanks to u/ml-research for pointing out that some papers were excluded from search results regardless of the search query. This was due to a bug in the way the database was queried, and should now be fixed.

37

Comments

You must log in or register to comment.

jakderrida t1_j3hsusc wrote

I looked up clown porn and I didn't find anything useful.

8

coumineol t1_j3hm4t4 wrote

Is that meaningfully better than just googling?

5

dreaming_geometry t1_j3hnyvd wrote

Data not yet collected. Why don't you try some side-by-side comparisons and report back?

5

universal_explainer OP t1_j3if5d5 wrote

Might be in some cases, maybe not in others. Anecdotally, a query like "model using only attention mechanism site:arxiv.org" on Google doesn't bring up "Attention Is All You Need", while it does here. Aside from that, it might be a useful resource for finding similar papers based on an arXiv link.

3

[deleted] t1_j3i3rwu wrote

[deleted]

2

universal_explainer OP t1_j3ij9jz wrote

Hey, thanks for trying it out!

First, do you mind sharing an example of different queries that return the same results? I have not been able to reproduce that (unless, of course, the queries are semantically similar, in which case that would be expected).

Also, of course exact search is far superior if you know the title of the paper you are looking for! In that regime, Google Scholar wins every time. However, semantic search might be better if you either a) can't remember the title but do remember some of the content or b) are simply looking to explore papers based on a handful of keywords.

Finally, the size of the database has no bearing on the quality of the embeddings, since I'm using the pretrained model by OpenAI. There is no notion of "popularity" except to rank the 10 papers with the highest cosine similarity to the query embedding according a citation score (if it's available).

3

fakesoicansayshit t1_j47vdrd wrote

Man, connect it to a chatbot after fine tuning it with the citations numbers as a human feedback input and you got yourself an uncensored, local, ML assistant!

Willing to share embeddings?

1

ml-research t1_j3l36cj wrote

Does it omit some papers if it fails to parse them? Because I cannot find some arXiv papers.

1

universal_explainer OP t1_j3les1h wrote

Are you talking about when inserting an arXiv link to find similar papers? In that case, it is important that the paper being referenced is already stored in the database. If it's a very recent paper (as in less than a week or two old), it won't work. This should be easy to fix, though, by simply scraping the abstract from arxiv.org and using it as the query.

If you're talking about searching for specific papers, I'd be interested to know the queries and the desired result. Feel free to post it here or in a DM 🙂

1