Yeah it's purely a language model, if its training has anything to do with information content and correctness it's gonna be very ad hoc and narrowly focused. All any of these models are really designed to do is sound good.