the true human in the loop.

is it though? how would you even do that? i think if you have that actually figured out, it's easily a $1b idea.

Sure, but the question is how often does it happen to get the right answer vs. the wrong answer and how would be measure that.

perfect, thank you!

this is true so far, it would seem.

you'd think there'd be some clever folks trying to quantify things better.

Thanks for this thoughtful answer.

Re: 2, are there solid numbers we would conceptual even be able to get? Are there known ongoing efforts?

Is there some alternative benchmark that measures factual accuracy of output?

Or is that impossible to use and create because any model would overfit that data?

everyone's scaling back assistant efforts, though, and cortana is basically dead, so, interesting idea, but i don't think so.

lots of implicit signals to look at based on what the user does after.