khrisrino t1_j6mx49i wrote on January 31, 2023 at 1:53 PM

Reply to comment by shanoshamanizum in Why AI can not replace search index by shanoshamanizum

It depends on what the AI optimization function is. The AI has been trained on a large corpus of data (scraped from the internet I guess?). Search engine has also been trained on a large selection of data from the internet. So what’s the difference ultimately? The search engine ranking algorithm also has biases when you go deep into it. There is an infinite variety of queries you can come up with and a rather limited training set of data behind it. Google themselves said that even after all these years 15% of queries are completely new queries. So for those new queries it’s likely you’re just getting some random results which could very well be all wrong. Same issue with Reddit … if you ask some common questions you get great answers. But ask something nontrivial and all you get are random and misinformed answers. The Reddit ranking function is also inherently very susceptible to bias because it depends on people to upvote or downvote without knowledge of whether the people voting actually know anything about the topic.

shanoshamanizum OP t1_j6mxesx wrote on January 31, 2023 at 1:55 PM

>The AI has been trained on a large corpus of data (scraped from the internet I guess?). Search engine has also been trained on a large selection of data from the internet. So what’s the difference ultimately?

One is an index where ultimately you decide what to read while the other one is a crystal ball with no alternative options.

khrisrino t1_j6mzbk0 wrote on January 31, 2023 at 2:10 PM

You think you’re getting the full set of possible search results from which you then freely choose … but that’s not how it works. They have a ranker behind the scenes that decides what you get the see.

shanoshamanizum OP t1_j6mzfcy wrote on January 31, 2023 at 2:10 PM

We are comparing the concepts not the misbehavior of providers.

khrisrino t1_j6n01m9 wrote on January 31, 2023 at 2:15 PM

Yes conceptually the search engine is better in that way. But we don’t live in a conceptual world. We are very much exposed to the misbehavior of providers so we cannot ignore them in the evaluation.

shanoshamanizum OP t1_j6n0dp4 wrote on January 31, 2023 at 2:17 PM

Of course, I agree with that and new search engines without caveats are built such as - https://ipfs-search.com/#/ which is not centralized. If the concept is alive new providers will come.