what is the size of the opt model you are comparing with in that table?

It’s written in the 2nd column (params)

OPT 6.7B

It does need more parameters to compensate (For instance, it has nearly a billion more parameters than GPT-J-6B without substantial performance gains) while losing out on LAMBADA (Ignoring the weighted average as I don't really understand the point of weighing it, since it distorts the metrics).

Its an extremely interesting direction, but I fear as you scale this model the scaling plot might start to flatten out - much like other RNN rewrites/variants. Hope further research is able to pinpoint the underlying issue and fix it. Till then, best of luck to OP! 👍

I have to say, i really like that somebody is doing this, no matter the outcome.

How can it be used in a multi-label text classification task?

Really cool stuff!

I cannot say how much I love that you are doing this!

Great results! What is the meaning of the first row in your table?

How much faster is RNN on inference than GPTJ?

RWKV-3 1.5B on A40 (tf32) = always 0.015 sec/token, tested using simple pytorch code (no CUDA), GPU utilization 45%, VRAM 7823M

GPT2-XL 1.3B on A40 (tf32) = 0.032 sec/token (for ctxlen 1000), tested using HF, GPU utilization 45% too (interesting), VRAM 9655M

Moreover RWKV-4 is bf16 and faster than 16bit GPT models.

Training speed: RWKV-4 1.5B BF16 ctxlen1024 = 106K tokens/s on 8xA100 40G.

You can try the RNN hidden state

Could you also measure the performance on CPU?

How much smaller are the embeddings?

RWKV 7B is faster than GPT 6B, and RWKV scales great actually :)

If you check the table, RWKV is better than GPT-neo on everything at 3B (while smaller RWKV lags behind on LAMBADA).

But GPT-J is using rotary and thus quite better than GPT-neo, so I expect RWKV to surpass it at 14B.

Moreover RWKV 3B becomes stronger after trained for more tokens and I am doing it for the 7B model too.

Interesting work. Appreciate your effort. There are few works which use convolutions as well (referred as ConFormer). But I’m not sure if it has been tried in comparing with GPT works.

How do you train such large models (AWS, GCP, etc)? And how much is the estimated cost?

Great stuff, and much needed!! Transformer are expensive.

Is the RNN mode suitable for update-able-neural-net NNEU used in tree-search games? This is where the next tree node evaluation re-uses the previous node.

Efficiently updatable neural network

>An efficiently updatable neural network (NNUE, a Japanese wordplay on Nue, sometimes stylised as ƎUИИ) is a neural network-based evaluation function whose inputs are piece-square tables, or variants thereof like the king-piece-square table. NNUE is used primarily for the leaf nodes of the alpha–beta tree. While being slower than handcrafted evaluation functions, NNUE does not suffer from the 'blindness beyond the current move' problem. NNUE was invented by Yu Nasu and introduced to computer shogi in 2018.

^([ )^(F.A.Q)^( | )^(Opt Out)^( | )^(Opt Out Of Subreddit)^( | )^(GitHub)^( ] Downvote to remove | v1.5)

Why not writing a paper?

Plot twist: the model getting integrated in transformers lib ( ͡° ͜ʖ ͡°)

What we have at this moment:

https://github.com/huggingface/transformers/issues/17230

So again. What is the disadvantage with using your method?