Viewing a single comment thread. View all comments

michaelthwan_ai OP t1_jcxrcfu wrote

Reply to comment by BalorNG in [P] searchGPT - a bing-like LLM-based Grounded Search Engine (with Demo, github) by michaelthwan_ai

I agree with you. 3 thoughts from me

- I think one direction of the so-called safety AI to give a genuine answer, is to give it factual/external info. I mean 1) a Retrieval-based model like searchGPT 2) API calling like toolformer (e.g. check weather API)

- LLM, is essentially a compression problem (I got the idea in lambdalabs). But it cannot remember everything. Therefore an efficient way to solve so are retrieval methods to search a very large space (like pagerank/google search), then obtain a smaller result set and let the LLM organize and filter related content from it.

- Humans are basically like that right? But if we got a query, we may need to read books (external retrieval) which is pretty slow. However, humans have a cool feature, long-term memory, to store things permanently. Imagine if an LLM can select appropriate things during your queries/chat and store them as a text or knowledge base inside it, then it is a knowledge profile to permanently remember the context bonded between you and the AI, instead of the current situation that ChatGPT will forget everything after a restart.

3

BalorNG t1_jcxtq26 wrote

There is a problem with context length, but than given the fact that us humans have even less context length and can get carried away in conversation... I think 32kb context length is actually much greater leap in GPT4 than other metrics if you want it to tackle more complex tasks, but it is "double gated". Again, even humans have problems with long context even in pretty "undemanding" tasks like reading fiction, that's why books have chapters I presume :) Btw, anterograde amnesia is a good example how humans would look like w/o longterm memory, heh.

Anyway, I'm sure a set of more compact models trained on much more high-quality data is the way to go - or at least fine-tuned by high-quality data, coupled with APIs and other symbolic tools, and multimodality (sketches, graphs, charts) as input AND output is absolutely nessesary to have a system that can be more than "digital assistant".

5

michaelthwan_ai OP t1_jcxx6ib wrote

Yeah great summary related to the memory.

My next target may be related to compact models (which preserve good results), as I also believe it is the way to go :D

1

BalorNG t1_jcy0trr wrote

Yea, I'm sure that compact-ish distilled, specialised models trained on high quality, multimodal data is the way to go.

What's interesting, once generative models get good enough to produce synthetic data that is OF HIGHER QUALITY than laion/common crawl/etc, it should improve model quality which should allow to generate better synthetic data... not exactly singularity, but certainly one aspect of it :)

1

michaelthwan_ai OP t1_jcy73od wrote

Your idea sounds like GAN - maybe one model will generate high-quality synthetic data and another one try to 'discriminate' it, then they may output an ultra-high quality one finally (for another model to eat). And an AI model community is formed to self-improve...

2

BalorNG t1_jcy7l5d wrote

Yea, in a way something like this was already done with LLAMA-Alpaca finetune - they used chatgpt to generate instuct finetune dataset, what, while far from pefrect, worked pretty damn well.

2