ResourceResearch t1_iro8zof wrote
Reply to comment by DeepNonseNse in [D] - Why do Attention layers work so well? Don't weights in DNNs already tell the network how much weight/attention to give to a specific input? (High weight = lots of attention, low weight = little attention) by 029187
Afaik it is not clear. In my personal experience, the number of parameters is more important, rather then the layer size, i.e. a smaller number of wider layers does the same job as a large number of narrower layers.
Consider this paper for empirical insights for large models: https://arxiv.org/pdf/2001.08361.pdf
Viewing a single comment thread. View all comments