How To Scale Transformers’ Memory up to 262K Tokens With a Minor Change?

Submitted by rezayazdanfar t3_11qfl2o in deeplearning

I just published my latest medium article.

This article is a fabulous attempt to leverage language models in memorizing information by transformers with the least required effort by inserting only an external memory near the last layer of the transformer.

We can use this to retrieve information that the transformer is trained with which helps the reliability of predictions.

Feel free to share this and/or contact me directly.

https://medium.com/towards-artificial-intelligence/extending-transformers-by-memorizing-up-to-262k-tokens-f9e066108777

15

Comments

You must log in or register to comment.

WallyMetropolis t1_jc564sn wrote

Calling your own work 'fabulous' is a little unusual.

13

rezayazdanfar OP t1_jc5ncpy wrote

True but i didn't call my own work fabulous, i meant the main work. :)

7

WallyMetropolis t1_jc6qi9w wrote

I see. Typically, when you say "this thing" you're referring to the most recent mention of that thing. So "Here's my article. This article is fabulous." is probably not the structure you want. You really should reference the paper or the researchers you're writing about straight away, if that's what you're doing here. Even after skimming your article, that isn't clear.

It's a cautionary tale about "this" really. A tip that has helped my writing is to accumulate a list of words that I tend to over-use that add nothing and search for them in the editing process. Words like "this, really, just, again," and so on.

2

rezayazdanfar OP t1_jcu1yfy wrote

:) yess I see, thanks. If you like we can talk more and give me more feedback or/and comments; thus I can improve. :)

1

PM_ME_ENFP_MEMES t1_jc5yjdv wrote

Own it bro, don’t listen to the haters. If your work is fabulous, call it fabulous, literally nobody can stop you and nobody that appreciates your work will care what you call it as long as it contributes to moving things forward.

1

DeepLearningStudent t1_jc7ly6x wrote

Holy crap, if this works that’s a huge game changer for a problem I’m working on.

2

rezayazdanfar OP t1_jcu24bm wrote

:) happy to hear it, hope you found it practical in your work. I also aim to use it in my future project. :)

2