[D] Has anyone integrated ChatGPT with scientific papers?

Submitted by justrandomtourist t3_ztjw7j in MachineLearning

A guy on Twitter shared a ChatGPT that is aware of all the podcasts from Andrew Huberman, which is great (https://huberman.rile.yt/?query=)

Has anyone open sourced something like ChatGPT that it is easy to fine tune with external knowledge, potentially tested on scientific papers? It would be great for brainstorming, writing research proposal and exploring the literature in a different way. Maybe even integrating it with Zotero.

As of now I talked about finetuning the model, but let’s say I take the easier path of few shot learning instead. Is there a way to save the state of ChatGPT? In other words, if I open a new chat and feed it all the papers by copy and paste for example, is there a way I can use it next week? Sometimes I have found the session to expire, but recently it seems past chats are saved. Will this last indefinitely you believe?

TL;DR: best way to adapt ChatGPT to specific knowledge?

35

Comments

You must log in or register to comment.

myUser9999 t1_j1e5fq5 wrote

Check out Galactica, by (among others) Meta

12

Rioghasarig t1_j1ew6cy wrote

Are you sure this technology is linked to ChatGPT? It doesn't seem to say that anywhere on that webpage.

4

CriticalTemperature1 t1_j1g44a8 wrote

You can finetune gpt-3, but it will cost you a few dollars. I've found good success just copying the text of a paper into chatGPT and asking for a summary that a fifth grader can understand.

Another way is to just input the titles of relevant papers and ask it for more suggestions, or ask it for the most influential papers in topic X

2

pyepyepie t1_j1hi1tv wrote

Replying to idrajitsc (#1,031,066)

ChatGPT will do it too, it happily invented papers (with nice ideas! although it just merged two ideas most of the time) for me when I asked for it to write a literature review. Then again, we face the challenge of grounding correctly vs flexibility. My hypothesis is that the model was trained using feedback from non-domain experts as well, so unless we solve grounding fundamentally I would even go and say it is the expected behavior of the model. That is, it was probably rewarded to make facts that sound good even if incorrect, in comparison to facts that sound bad, which makes its hallucination trickier even if it happens less. No reason to think fine-tuning will solve it.

2

idrajitsc t1_j1i9pwf wrote

Replying to pyepyepie (#1,037,047)

Yeah it's purely a language model, if its training has anything to do with information content and correctness it's gonna be very ad hoc and narrowly focused. All any of these models are really designed to do is sound good.

3

Rosa-Amanita t1_j1r4ag4 wrote

  1. Yes, this has been done: https://elicit.org/
  2. Open source or not, you can't run this on your own machine. The GPU requirements are huge. So you would be looking at using an API. OpenAI has the most advanced models that can be used through an API. Would be GPT3-davinci-003 which is similar to ChatGPT but with less social skills but more flexibility.
  3. You don't save the state of ChatGPT. Modern language models have a memory of less than 6000 words. With every query you make with ChatGPT the last ~6000 words of your session get sent to give the model context. After every query the model forgets all about that query.
  4. To give language models awareness of more data, without fine-tuning them on that data, a process is used called "embedding". Your code looks through a database and finds the information that is most relevant to your query. It then feeds that information together with your prompt to the language model and the language model gives a response based on that info.
  5. Yes, you can implement this in a couple of days. Use this: https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb and, depending on the size of your data, this: https://www.pinecone.io/
6