Very interesting! What other ways of augmenting AI capabilities do you see will follow? Do you think effectiveness of this suggest that symbolic approaches will be connected with NNs, for example an AI agent will be equipped with a separate mathematical theorem checker, or other whiteboard to note down intermediate calculations?

Writing down intermediate calculations is not a concept we invented. In our paper we call this 'elicitive prompting' and mention chain-of-thought prompting and the scratchpad papers as previous examples of this.

​

I'm super excited about elicitive prompts (self-ask is in that category too)! I think they're going to enable us to get much more out of these models.

​

And yes, just like we can integrate Google Search we can also integrate lots of other systems, I'm really excited to see how this research direction develops!

As a hobbyist, I love it! This looks simple enough that I could implement it myself.

Except maybe I’d try changing out the google api with a database connection and have it perform sql queries as intermediate steps.

Could be fun! Eg put my family tree in the database and ask it “was my great uncle alive at the same time as my third cousin twice removed?” Etc.

Yup! We don't require any finetuning or special syntax, so it's super easy to extend this and play around with it!

Sorry if I don't completely follow, do you use self ask to finetune the model or do you use it before the model surfaces the result after a prompt? And how does the model come up with the self ask questions specifically?

As a lurker in ML field, I was like damn that's awesome.

I can imagine it being used in various domains, after all more questions are better than incorrect or partial answers.

Good Job :D

What if Google search has it wrong?

Query: https://www.google.com/search?q=Which+year+was+%22The+Lady+Is+a+Tramp%22+by+Frank+Sinatra+released+for+the+first+time%3F

First result from Wikipedia:

>1937 [WRONG] by Chappell & Co. The song appears in the film version of Babes in Arms (1939 [WRONG]) as an instrumental version only. ... Tony Bennett and Lady Gaga duet. "The Lady Is a Tramp" Released September 20, 2011 [WRONG]

Third result from Cafe Songbook:

>"The Lady Is a Tramp" was, however, added to the movie version of a Rodgers and Hart show for which it wasn't written. The song was interpolated into the 1957 [APPROXIMATELY RIGHT] ...

Discogs link for confirmation: https://www.discogs.com/de/master/544337-Frank-Sinatra-The-Lady-Is-A-Tramp

Curious - what are the primary differences between this, and the "information correction system" in LaMDA? https://arxiv.org/pdf/2201.08239.pdf figure 3

There's no finetuning, it's just a prompt. GPT-3 is just smart enough to learn how to decompose questions based on the one example in the prompt...

Yup, right now if Google is wrong, the model will still take it. I'm sure there are lots of ways to improve in this direction.

LaMDA doesn't do multi-hop questions, only singlehop. They have 2 different LMs that talk to each other whereas we have just one. They finetune their model on specially-made data, we just have a prompt.

​

Our approach is inspired by LaMDA and other related amazing previous papers, but our approach is much simpler and easier to implement.

Hi,

This is interesting, but when I tried the same questions you did, GPT3 gave me the correct response without asking any further questions.

Here is an example:

Question: What is the capital of the country where the Taj Mahal is at?
Answer: The Taj Mahal is in India, and the capital of India is New Delhi.

Yup it answered it correctly by 'talking things through'. Sometimes this happens automatically. A prompt like self-ask makes this happen with much much higher probability.

If you run an empirical evaluation on hundreds of questions you'll see that chain of thought and self-ask get much higher performance than not using a prompt or using a prompt that asks for the answer immediately.

make sense :)