I have a very basic understanding of how ML algorithms work — you feed them buckets of data, have them look for patterns in the data, and then attempt to generate new data based on those patterns. So I can see how you could give GPT-3 a topic and it could spit out a bunch of words commonly associated with that topic. What I understand less is how it combines those words into original sentences that actually make sense.

I know GPT-3 doesn’t have any sense of what it’s saying — like if I asked it to generate Tweets saying “Elon Musk is dumb”, it doesn’t know who Elon Musk is, what dumb is, or even what “is” is. But somehow it’s able to find information about Elon Musk, and formulate it into a sentence insulting his intelligence.

Can someone who knows more about the inner workings of GPT-3 or language models in general explain the “thought process” they go through when generating these responses?

Edit: I should also add that I understand how basic language models and sentence construction work. What I’m asking about specifically is how does it generate sentences that are relevant to a given topic, especially when there are modifiers on it (eg “write a song about Homer Simpson in the style of The Mountain Goats”)