Viewing a single comment thread. View all comments

cthorrez t1_j9x772h wrote

along with each prompt, just put: "And at the end of your response, state on a scale from one to ten how confident you are in you answer"

This works amazingly and is very accurate. source

It has the added bonus where you can get confidence intervals on your confidence intervals just by asking how confident it is in it's estimation of its confidence.

6

iidealized t1_jabkowf wrote

I’ve heard you can even ask the LLM: what fraction of your uncertainty is aleatoric vs epistemic, and how would the uncertainty estimates changed if you used bootstrap vs MC dropout :)

1