There's been a ton of academic work exploring knowledge distillation techniques, sparsity in networks and many others, often with vast numbers of citations. I was wondering what the status of those in real-world ML was. Has any of you used it in a concrete situation? What did you find to work best for you?

Comments

You must log in or register to comment.

suflaj t1_j3igfzr wrote on January 8, 2023 at 8:23 PM

Yes, it's the only way to get high throughput high performance models ATM.

With KD and TensorRT you can get close to 100x throughput (compared to eager TF/PyTorch on full model) with 1% performance hit on some models and tasks.

nmfisher t1_j3l4ipq wrote on January 9, 2023 at 8:36 AM

Echoing this, KD is also very useful for taking a heavyweight GPU model and training a student model that's light enough to run on mobile. Small sacrifice in quality for huge performance gains.

fredlafrite OP t1_j3l65ju wrote on January 9, 2023 at 8:58 AM

Interesting! Echoing this, do you know which kind of companies one could work on this in an applied setting?

Think_Olive_1000 t1_j3qynuz wrote on January 10, 2023 at 1:56 PM

Neural magic does work in this space, not sure about KD specifically

xenotecc t1_j3v3gr0 wrote on January 11, 2023 at 6:51 AM

How small do you make the student, when a teacher is let's say ResNet101? How do you find a good student/teacher size ratio?

Are there any tricks to knowledge distillation? Or just standard vanilla procedure?

suflaj t1_j3vg5tm wrote on January 11, 2023 at 9:37 AM

I think it's a matter of trial and error. The best ratios I've seen were 1:25, but these concerned transformer networks, which are much more sparse than resnets.

There are some tricks, but it depends on the model. Ex. for transformers, it's not just enough to imitate the last layer. I suspect that it's the same for resnets, given they're deep residual networks just like transformers.

xenotecc t1_j3vp75c wrote on January 11, 2023 at 11:38 AM

Thank you for the reply!

NichtMarlon t1_j3ikrkw wrote on January 8, 2023 at 8:49 PM

Yes its very useful for text classification tasks. Big transformers get highest accuracy, but we can't deploy them because they are too slow. So we distil knowledge from bigger transformers into smaller transformers or CNNs. If you have a decent amount of unlabeled data to pseudo-label with the teacher, there is barely any loss in accuracy for the student model.

madmax_br5 t1_j3ioy0f wrote on January 8, 2023 at 9:13 PM

I don't know, but I found this awesome open source site for pruning and fine-tuning various models you may find interesting: https://sparsezoo.neuralmagic.com/

fredlafrite OP t1_j3l6b40 wrote on January 9, 2023 at 9:01 AM

Nice! There are 20MB versions of BERT, super interesting thank you!

gamerx88 t1_j3m0drc wrote on January 9, 2023 at 2:29 PM

Yes, we used DistilBERT (and even logistic regression) heavily in my previous startup where data volume was web scale.

Depending on the exact problem, large transformer models can be an overkill. For some straightforward text classification even logistic regression with some feature engineering can hit within 3% point of a transformer, and costs a negligible fraction of them.

[deleted] t1_j3ll2ys wrote on January 9, 2023 at 12:15 PM

[removed]

aaaasd12 t1_j3i3ea8 wrote on January 8, 2023 at 7:04 PM

It's like transfer learning?

In the company that I'm work only use the normal things like classification tasks/ segmentation with clusters.

Maybe the use case that i see is in NLP with topic modeling using bertopic and tuning the hyperparameters.

But in general simple models are perfect for the tasks that se have.