Viewing a single comment thread. View all comments

soggy_mattress t1_jdl4zkg wrote

I think of the 100b parameter models as analogous to the first room-sized computers that were built in the 70s. Seems the pattern is to first prove a concept, no matter how inefficiently, and then optimize it as much as possible.

188

Vegetable-Skill-9700 OP t1_jdl680d wrote

That's an interesting analogy!

24

Short_Change t1_jdlp0cw wrote

Actually if his analogy is true, we will have 20 trillion parameters in the future for modern consumption.

37

Crystal-Ammunition t1_jdlpzsw wrote

At that point, the training data world have to almost completely be synthetic, right?

19

EmmyNoetherRing t1_jdma3em wrote

Introspection? Cog-sci/classical AI like to use the term, not always in the best justified fashion I think. But when you’re hallucinating your own new training data it seems relevant.

18

currentscurrents t1_jdmyjrb wrote

Bigger models are more sample efficient, so it should need less data.

But - didn't the Chinchilla paper say bigger models need more data? Yes, but that's only true because right now compute is the limiting factor. They're intentionally trading off more data for less model size.

As computers get faster and models bigger, data will increasingly become the limiting factor, and people will trade off in the opposite direction instead.

7

itshouldjustglide t1_jdoazux wrote

Don't bigger models need more data so that all of the neurons can be trained so as to reduce unnecessary noise and randomness?

1

ganzzahl t1_jdovu3h wrote

I'm also very interested in this – does anyone have papers similar to Chinchilla, but without the training FLOPs restriction, and instead comparing identical dataset sizes?

An aside: I feel like I remember some older MT papers where LSTMs outperformed Transformers for some low resource languages, but I think that's outdated – using transfer learning, multilingual models and synthetic data, I'm fairly certain Transformers always outperform nowadays.

1

PilotThen t1_jdpnoul wrote

I didn't find a paper but I think that is sort of what EleutherAI was doing with their pythia models.

You'll find the models on huggingface and I'd say that they are also interesting from an opensource perspective because of their license (apache-2.0)

(Also open-assistent seems to be building on top of them.)

2

hadaev t1_jdlym7s wrote

Idk, internet is big.

0

CacheMeUp t1_jdxvq8t wrote

Perhaps the challenge is not the size of the internet (it's indeed big and easy to generate new content), but rather the uniqueness and novelty of the information. Anecdotally, looking at the first page of Google results often shows various low-informativeness webpages, where only a few sentences provide information and the rest is boilerplate, disclaimers, generic advice or plain spam.

1

Thebadwolf47 t1_jdnbfya wrote

wasn't he rather comparing the parameters to the volume of the first computer and not their transistor count?

1

Puzzleheaded_Acadia1 t1_jdn6ugl wrote

I see a future where LLMs or llamas that are multimodels or any other new kind artificial intelligence run on esp32 level of hardware i don't know how that will work but I'm pretty sure we are heading there

1

atheist-projector t1_jdm7hw1 wrote

Especialy when considr that sgd is a local minima we can probably do a whole lot better if we find a niced optimizer.

0