Eh, doesn’t chat gpt-4 have like 1.4 trillion parameters? It dwarfs this by almost 5 times.

Edit: turns out, I was wrong! :D

Gpt4s parameter counts aren't known yet

Ah okay, nvm then. Sorry

The context window is apparently massive though, more than 10 times the size of gpt3, it could potentially write whole novels at that scale

https://mobile.twitter.com/transitive_bs/status/1628118163874516992?s=46&t=Biiqy66Cy9oPH8c1BL6_JQ

Chatgpt is based on gpt3 a 175 billion parameters model.

Is anything about gpt4 known? It seems like just a bunch of rumors and not even a release date

A credible researcher had commented that ChatGPT can write code, and GPT-4 could write entire programs.

Wow, yeah, this looks amazing. My biggest issue with GPT-3 is the relatively small context window. This will open so many new possibilities.

32k tokens would mean approximately 150 kB of text. That is a decent sized code base! Also with this much context memory the known context saving tricks would work much better so this could be theoretically used to create code bases of virtually unlimited size.

This amazes me and also (being software dev) also scares me...

But, as they say, what a time to be alive!

There was that very popular but completely unfounded rumor about 100T param count. It was debunked by Sam Altman himself.

If you think about it for just 1 second you would realize that 100T param model would need at least 200 TB of VRAM or 2560 Nvidia A100s...

It was 5 Morbillion parameters

[removed]

Interesting

>Is anything about gpt4 known? It seems like just a bunch of rumors and not even a release date

I work with someone who has Beta access to GPT-4. He won't tell me much other than it's mind-blowing & that software development will never be the same. He confirms the rumors that it indeed can write an entire piece of software.

Number of parameters is not the whole story. Quality of training material and training time and training techniques matter as much or more.

The larger models require more resources for inference, as well. I'd be more impressed by a model smaller than GPT-3 that performed just as well.

Gesundheit

Great way to look at it, it’s much more important to squeeze the maximum out of your system. Efficiency over excess

Oh great another rumor

True, we've seen models a tad bit bigger than GPT3 which are so bad, even GPT 2 would blow them out the water.

Think AI21 Jurassic park or whatever they call their largest model. I hate how stupid it is

It's not about the size;)

Well... You asked if anything was known. I gave you info from a coworker with beta access. My apologies if my info didn't come with a boquet of roses and a handwritten card.

And no bias?

That’s exciting. Would be amazing if the next one didn’t just make shit up when it doesn’t know the answer

I understand you’re trying to help, but this being Reddit and all there’s no way we can trust what you are saying or take it officially as something “known”. No offense though.

>I'd be more impressed by a model smaller than GPT-3 that performed just as well.

from the article: "Aleph Alpha’s model is on par with OpenAI’s GPT-3 davinci model, despite having fewer parameters.", so... you're saying you would be even more impressed if it used even fewer parameters? Anyway I think anyone could guess gpt3 is poorly optimized so it shouldn't be surprising to anyone that plenty of models have matched its performance on some benchmarks with less parameters.

I for one welcome competition in the race to AGI

It's the new Space Race

I sure don't. We need to get it right. Not barrel ahead in an arms race.

Great, and is it open source?

latent space race

Check out Amazon’s multimodal chain of thought model, only 738 million and scores better on all question classes than ChatGPT. See table 4 on page 7 here - https://arxiv.org/pdf/2302.00923.pdf

[deleted]

Heh. I tried their rationalization step with ChatGPT, just with prompting. For their question about the fries and crackers it said the problem is flawed, because there's such a thing as crackers with low or no salt. Also correctly inferred that fries are usually salted, but don't have to be. (of course, it didn't have the picture to go by, which was the point of the research)

Great paper though. Thanks for sharing.

r/thatswhatshesaid

It's not a space race until Governments start pouring massive amounts of their GDP into it.

You obviously do not know my ex.

NEIN!

Are you aware of either the "training material" or "training time" or "training techniques" utilized?

You know the closer we get to AGI the more that will happen. Every government will want to be the first in control of an ASI which will basically make them the dominant superpower of the world. It will be as dystopian as it sounds.

paywalled though and likely will be just as censored. It's also currently not available. So.... who cares?

Achieving GPT 175B Level Accuracy with a 10x More Efficient Model

https://sambanova.ai/blog/achieving-gpt-175b-level-accuracy-with-a-10x-more-efficient-model/

Artificial German Intelligence

What the actual fuck is that paper? The thing performed better than a human at several different question classes.

At fucking less than one billion parameters. 100x less than GPT 3.5.

Edit: For clarity, I am impressed not angry lol.

I'm sure they've read those papers too, you know.

Agreed

All else being equal, number of model parameters does matter. Well funded startups can acquire the needed data, compute resources, and human talent to build the models. Just like how OpenAI beat Google at this game.

Germany saving Europe again ! No wait..

Are you angry or impressed ?

impressed lol

[deleted]

If these models are so smol and efficient, why are they not released ?? I just don't get it. I thought PaLM was kept private because it was too costly to run to be profitable...

Like what would happen if like 4% of global GDP was put into this?

Of course no. Unless the research is done within a University context (or publicly funded), you won't have the model open source. SD is maybe the exception, and it seems to me like they regret releasing it and are now doing whatever they can to regain control.

32k context window it seems.

https://mobile.twitter.com/transitive_bs/status/1628118163874516992?s=20

Hm? Care to elaborate on what they’re doing to “regain control?”

Unfortunate, but I figured. Something's up when the Russians are the only ones releasing LLMs

First they tried to take control of the DF subreddit (Source). Apparently it was solved on good terms.

Also, newer versions are much more controlled in term of what you can generate. No more NSFW allowed, no more "famous artists" based models. They was also rumors about new license terms (not sure if it did happen actually) that essentially provide them with legal power to force users to update to a newer version (as crazy as it sounds). There is a reason that the community is still using 1.5 version over the 2.0 version.

Honestly, the way I see it, Stability AI are not doing it with bad intentions (at least I hope), and are kind of forced to do that, as they are a legal entity and have to address all the threats of legislative actions regarding explicit sexual contents and living artists.

I had to do some research about this. I found that the transformers are mostly open source, but the trained models are not.

Boo

never ask AGI what is was doing in 1939

Fuck dude, that's clever

Static parameters are meaningless. Human brains are not static until after death. Besides modeling reality requires more than a bit of algebra.

Surely it won't be funny...

Im from germany and I know germany is incompetent anything related to IT, its all about old economy. Don’t get any hopes up.

There will be both good and bad that comes out as we get closer to AGI, and attain AGI, just like and other technological revolution. To paint it as "either" dystopian or utopian is naive.

Künstliche Allgemeine Intelligenz 🇩🇪

I'm going to make a genuine question because no one ever gave me a clear answer. When will these language models ever start to be useful?

That's because the smaller models are less useful. With neural networks (likely including biological ones) there's a hard trade off between specialized performance and general performance. If these 100+x smaller models were trained on the same data as GPT-3 they would perform 100+x worse on these metrics (maybe not exactly because in this case the model was multimodal which definitely gave a performance advantage). The big reason this model performed so much is because it was fine tuned on problems similar to the ones on this exam where as GPT-3 was fine turned on anything and everything. This means that this model would likely not be a great conversationalist and would probably flounder at most other tasks GPT-3.5 does well on.

That is really a nice one

I tried to use their system on their playground. It takes a lot of prompting to get anything sensible out of it.

China is pouring money into AI research.

He Kai, sag mal was gescheites.

It depends. We cannot see the other side of a singularity.

We could have an alignment issue and end up as paper clips.

AI could solve everything from climate change to social inequality by reducing the human race to 50 million Stone Age hunter gatherers.

Or, you could have the top 1% living in a utopia while everyone else is living in a dystopia.

Inb4 The Germans literally create Funny-Bot

Singularity is approaching fast.

People might not realize that a sufficiently advanced LLM can simulate Ai researchers and programmers. For example, "simulate a thousand of the top Ai researchers, discussing and then programming an AGI".

The GLM models are from China and open sourced.

I didn't know China was doing it too! I know Russia recently open sourced one. If their tactic is to undermine western power by open sourcing their next big products, they can keep doing it

Geopolitics cares

any sufficiently advanced LLM is indistinguishable from trueAGI™?

The German CovidApp was surprisingly solid

of there are many biases.

These neural networks mostly consistent of weights and biases.

They are useful now.

Two friend of mine use ChatGPT for work.

Not this bias, the other that reddit agrees with and downvote you if you mention, like a religious dogma.