Submitted by terserterseness t3_10fxryj in MachineLearning
DaLameLama t1_j4zhqqj wrote
Does ChatGPT actually get past the token limit? Codex supports ~8000 tokens. You might underestimate how much this is. Has anyone tested the limits?
Unfortunately, OpenAI aren't serious about publishing technical reports anymore.
xorbinant_ranchu t1_j4zmevh wrote
1 token ~= 4 chars in English 1 token ~= ¾ words 100 tokens ~= 75 words
https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
I've had it reference much more than ~6000 words so there's definitely something else going on
LetterRip t1_j56kpcq wrote
It probably does a summarize if it is longer than the allowed input.
andreichiffa t1_j50x4ky wrote
Reported token size is 2048, but they likely do a hard attention mask. In about 1/4th of words
EmmyNoetherRing t1_j510553 wrote
>Unfortunately, OpenAI aren't serious about publishing technical reports anymore.
Do OpenAI folks show up to any of the major research conferences? These days I mostly come into contact with AI when it wanders into the tech policy/governance world, and this seems like the sort of work that would get you invited to an OSTP workshop, but I'm not sure if that's actually happening.
OpenAI's latest not-so-technical report (on their website) has a few folks from Georgetown contributing to it, and since AAAI is in DC in a few weeks I was hoping OpenAI would be around and available for questions in some capacity, in some room at the conference.
DaLameLama t1_j519tns wrote
There was an OpenAI party at NeurIPS, but I wasn't there. No clue about AAAI :)
EmmyNoetherRing t1_j51cvjh wrote
yeah, as an uninformed guess it seems like IJCAI or NeurIPS would be a more natural home, but AAAI is actually in DC, which seems helpful for some categories of conversation. if the right people attend.
[deleted] t1_j51wpic wrote
[deleted]
EmmyNoetherRing t1_j50zesm wrote
I've heard a diverse variety of folks talk about leaving chatGPT tabs/sessions open for for days or weeks and maintaining context plausibly well throughout.
Viewing a single comment thread. View all comments