[D] Is there any reason hugging face GPT2 would behave (fundamentally) differently from GPT-Neo?

Submitted by GasZealousideal8691 t3_10c9287 in MachineLearning

Hey guys. I'm running some experiments as part of a research project; it was initially implemented for GPT-Neo 1.3B, but there is one baseline we want to use that only supports GPT2-XL, so I implemented that into our code (i.e., just included a clause that was like "if model_name='gpt2', model=GPT2LMHeadModel.frompretrained('gpt2-xl')").

The issue is, GPT2 is giving absolutely absurd results that are clearly very incorrect. It is difficult to explain this without an in-depth explanation of my code, but basically I have a bunch of functions that do things like, for example, computing the probability of certain labels in a multiple-choice test.

So my question is, is there any fundamental difference in how these two models are setup in hugging face, that would result in such errors? I myself am not too familiar with hugging face models, so I'm not entirely sure. But the fact that the code runs but produces bad errors is weird; I would think that if something was wrong, there would be some sort of tensor-size doesn't match error somewhere...

1

Comments

You must log in or register to comment.

CKtalon t1_j4enpew wrote

GPT2 was trained on a different dataset, with little code (other than those obtained from the CommonCrawl). GPT Neo uses The Pile which contains a lot of code.

3

GasZealousideal8691 OP t1_j4gs8gc wrote

Replying to WigglyHypersurface (#1,368,771)

But would it affect it to this extent? To be clear, this is not just "bad performance", or "horrendous performance". Our project is loosely investigating the performance of different editing methods on LMs given some datasets we made, and none of the editing methods, from fine-tuning to gradient-methods, change the performance at all.

Furthermore, GPT2 outputs an equal accuracy and specificity values (specificity is basically the degree to which it "remembers" other unrelated facts; the goal here is to minimize catastrophic forgetting), which makes absolutely 0 sense, because they aren't even measured on the same scale. Accuracy is usually >0, <1 and specificity is usually ~26 based on our measures.

It doesn't have anything to do with the way accuracy/specificity are computed, because the code for GPT-Neo is identical minus the model= and tokenizer= statements, and it works fine for GPT-Neo. So there is something fundamentally crazy going on with GPT2...

1

GasZealousideal8691 OP t1_j4hk6kz wrote

Replying to WigglyHypersurface (#1,369,148)

Im fairly certain it’s something with the model. Like even fine tuning is giving these weird errors, when it had no problems for GPT-Neo.

We also ran this stuff on T5, obviously had to configure the rest of the code differently but it was doing fine for that as well.

1