How to efficiently re-train a classification model with an addition of a new class?

Submitted by kingfung1120 t3_z2kb6r in deeplearning

Lets say I have a huge labeled dataset which contains 3 classes { class_a, class_b, anything_but_a_b} the class 'anything_but_a_b' is neither class_a or class_b. A model is already trained with this dataset.

If I want to add a new class called class_c with a dataset of classes { class_c, not_class_c } (note: not_class_c data could possibly belongs to class_a/class_b/anything_but_a_b, while there could be some class_c inside anything_but_a_b). And I want to build a 4-classes model to predict { class_a, class_b, class_c, anything_but_a_b_c }, how should I train the model without manually relabeling the original 3-classes dataset?

More info:

{ class_a, class_b, class_c, anything_but_a_b_c } these 4 classes are mutually exclusive

​

Edit: modify wordings

9

Comments

You must log in or register to comment.

suflaj t1_ixgrm1f wrote

You can add a new class to the classifier and then do model surgery to transfer old model weights onto part of your new model.

4

JH4mmer t1_ixhfmew wrote

My recommendation would be to drop the "other" class entirely. Thats a classic mistake that I've seen juniors do many times, and it doesnt really work out like you expect it to in the real world. The main problem with that approach is that a catch-all class like that has infinite variance (theoretically requiring infinite training data). Plus your labels often become massively unbalanced relative to the positive classes.

Instead, think of your model as having multiple tails, one for each class you actually care about (e.g. what is the probability that a dog is in this image?, what is the probability that a cat is in this image? Etc.) Each output has its own logistical activation that's independent of the other classes. Where before you might have had a softmax layer that returned [0.2, 0.3, 0.5] for (dog, cat, other), you might now have [0.8, 0.7] for (dog, cat). The labels will not sum to 1 because they are independent of one another.

Note that this is the approach you would take for multi-class classification as well, so you might want to read up on that pattern for more information.

Lastly, if you have a trained model in this format, adding a new class is very easy. The first N layers of the network are shared for all classes and so are already pretrained for you. You would add a new tail to the model using whichever weight initialization strategy you care about, add some samples of the new class, and then do some fine tuning on the new tail layer(s) to make sure that your network can effectively detect the new class.

Of course there are many variations to this training approach. You may choose to also do some fine tuning of the entire network with a dataset that includes samples of the new class, but hopefully you get the idea.

I hope this points you in the right direction! Cheers.

2

First_Bullfrog_4861 t1_ixhi8ki wrote

but finetuning requires the relabeling of the original dataset to include both old and the new label, which op specifically does not want to do.

i don’t think what op wants is doable. or is there some approach i’m missing? i think what op basically wants to do is retrain, but with only data from the new class, and still avoid catastrophic forgetting of the other labels.

is there a way to do this?

1

JH4mmer t1_ixhvl5k wrote

You may be making a different set of assumptions about the training data than I am, so let me clarify a bit. :-)

If you start with images that truly do contain just one class, the addition of a new class label wouldn't change anything. Your level vector for the exisiting images would migrate from [1, 0] to [1, 0, 0], something that can be done automatically without additional human intervention. Your new images (used for training the new class) would have a label of [0, 0, 1].

If, however, your images do already contain more than one possible class (which is far and away much more common in real-world data), the original labels would be already invalid, since the original labeling assumed that there was only one correct answer. Those images that do contain multiple classes would have to be relabeled, yes.

The process I'm describing is a mechanical one that doesn't involve a separate knowledge distillation step. It's a technique my team has used successfully in industrial retail applications, where the number of classes is truly an unknown, and we have to add or remove classes from our trained models frequently.

2

First_Bullfrog_4861 t1_ixjazoo wrote

ok, got it. however, in my experience the number of labels is far less obvious in real world datasets than one might expect. consider an example with images of bottles, cups and glasses, so three labels.

a model trained on these three labels will need revision if further down after the deployment process ‚bottles‘ need to be split in ‚plastic bottles‘ and ‚glass bottles‘. both label sets are perfectly valid, due to the hierarchical nature of things.

anyway, my point is actually another one: afaik this will require dataset relabeling and fully iterate the training process on the newly labeled dataset.

or is there a faster way to make the model aware of the more finegrained bottle labels?

i mean, without access to data of cups and glasses, basically inform it of more finegrained bottle types but let it still keep its knowledge of cups and glasses.

1

LinuxSpinach t1_ixiy1zy wrote

>main problem with that approach is that a catch-all class like that has infinite variance

Sometimes it doesn't, and I've seen an 'other' class work well in these cases. In cases where the data being fed to the model already constrains the variance, then an other class won't have infinite variance. Eg. you know that all of the data will be pictures of fruit, but you only want to label apples, bananas and oranges. In this case, there is a finite number of fruits to take pictures of.

If you are going to use an 'other' label, I think it should be ok in cases where you could label the data, but the labels that the other class comprises are unimportant to your application.

1

emad_eldeen t1_ixh95fx wrote

I think this falls under incremental learning, where you seek to learn from the new dataset without forgetting the old classes.

1

saw79 t1_ixiusbb wrote

Your model should output 3 logits, one for class_a, one for class_b, and one for class_c.

When you use data from the 1st dataset,

  • penalize class_a outputs for samples with class_b and anything_but_a_b labels
  • penalize class_b outputs for samples with class_a and anything_but_a_b labels
  • penalize class_c outputs for samples with class_a and class_b labels

When you use data from the 2nd dataset,

  • penalize class_a outputs for samples with class_c labels
  • penalize class_b outputs for samples with class_c labels
  • penalize class_c outputs for samples with not_class_c labels
1