Viewing a single comment thread. View all comments

michaelthwan_ai OP t1_jcxx6ib wrote

Reply to comment by BalorNG in [P] searchGPT - a bing-like LLM-based Grounded Search Engine (with Demo, github) by michaelthwan_ai

Yeah great summary related to the memory.

My next target may be related to compact models (which preserve good results), as I also believe it is the way to go :D

1

BalorNG t1_jcy0trr wrote

Yea, I'm sure that compact-ish distilled, specialised models trained on high quality, multimodal data is the way to go.

What's interesting, once generative models get good enough to produce synthetic data that is OF HIGHER QUALITY than laion/common crawl/etc, it should improve model quality which should allow to generate better synthetic data... not exactly singularity, but certainly one aspect of it :)

1

michaelthwan_ai OP t1_jcy73od wrote

Your idea sounds like GAN - maybe one model will generate high-quality synthetic data and another one try to 'discriminate' it, then they may output an ultra-high quality one finally (for another model to eat). And an AI model community is formed to self-improve...

2

BalorNG t1_jcy7l5d wrote

Yea, in a way something like this was already done with LLAMA-Alpaca finetune - they used chatgpt to generate instuct finetune dataset, what, while far from pefrect, worked pretty damn well.

2