Viewing a single comment thread. View all comments

dojoteef OP t1_jc4hwyw wrote

Reply to comment by rePAN6517 in [R] Stanford-Alpaca 7B model (an instruction tuned version of LLaMA) performs as well as text-davinci-003 by dojoteef

If you actually want the NPCs to meaningfully add to the game rather than merely being mouthpieces then your approach won't work. How do you ensure what they say is consistent with the game world? E.g. what if they make up the location of a hidden treasure, offer to give you an item, etc. All of that needs to be accounted for in the game logic as well, otherwise they'll say things that make no sense in the game world.

It's actually a challenging problem and requires research. As far as I know there a very few people actively researching this area; if they are, then they certainly aren't publishing it. Hopefully my next paper which investigates using LLMs in Disco Elysium will help change that.

18

generatorman_ai t1_jc5w4m9 wrote

The general problem of generative NPCs seems like a subset of robotics rather than pure language models, so that still seems some way off (but Google made some progress with PaLM-E).

LLMs and Disco Elysium sounds like the coolest paper ever! I would love to follow you on twitter to get notified when you release the preprint.

4

dojoteef OP t1_jc6om7a wrote

Thanks for the vote of confidence!

Unfortunately, I recently deleted my twitter account 🫣. I was barely active there: a handful of tweets in nearly a decade and a half...

That said, I'll probably post my preprint on this sub when it's ready. I also need to recruit some play testers, so will probably post on r/discoelysium recruiting participants in the next few weeks (to ensure high quality evaluations we need people who have played the game before, rather than using typical crowdsourcing platforms like MTurk).

1

rePAN6517 t1_jc4jkbt wrote

Honestly I don't care if there's not complete consistency with the game world. Having it would be great, but you could do a "good enough" job with simple backstories getting prepended into the context window.

3

v_krishna t1_jc4orxw wrote

The consistent with the world type stuff could be built into the prompt engineering (e.g., tell the user about a map you have) and I think you could largely minimize hallucination but still have very realistic conversations

2

PriestOfFern t1_jc6x37m wrote

Take it from someone who spent a long time working on a davinchi support bot, it’s not that easy. It doesn’t matter how much time you spend working on the prompt, gpt will no matter what, find some way to randomly hallucinate something.

Sure it might get rid of a majority of hallucinating, but not a reasonable amount. Fine tuning might fix this (citation needed), but I haven’t played around with it enough to comfortably tell you.

2

v_krishna t1_jc7wzmx wrote

I don't doubt it. I've only been using it for workflow aids (copilot style stuff, and using it to generate unit tests to capture error handling conditions etc), and now we are piloting first generative text products but very human in the loop (customer data used to feed into a prompt but the output then feeds into an editor for a human being to proof and update before doing something with it). The amount of totally fake webinars hosted by totally fake people it has hallucinated is wild (the content and agendas and such sound great and are sensible but none of it exists!)

1

mattrobs t1_jcs3vvo wrote

Have you tried GPT4? It’s been quite resilient in my testing

1

blueSGL t1_jc5rpta wrote

could even have it regenerate the conversation prior to the vocal synt if the character fails to mention the keyword (e.g. map) in the conversation.

You know, like a percentage chance skill check. (I'm only half joking)

1