In case of such articles, I cannot escape the feeling that the authors do not interact with these models at length and mainly argue with their imagined form of interaction. Here, it is the premise of the significant part of the paper:

>a fictional question-answering system based on a large language model

...with imagined conversations and discussion of its imagined flaws, eg. the author criticizes it for lack of communicative intent, no awareness of the situation, no ability to "know anything", or that it "cannot participate fully in the human language game of truth" (self-citation a.d. 2010, in "Embodiment" presented as, roughly, everyday use of words and adjusting the use to the context). Thanks, I guess? How about interacting with actual models that beat you in the game of truth and are sometimes too nosy in their communicative intent?