Submitted by HFSeven t3_10a8a14 in MachineLearning
Hi! So i am looking into literature for determining the usefulness of samples/datasets used for training ML model. Lets say DNN was trained with datasets A, B and C so after training is there way to quantify which of the partial triaining datasets contributed most to the useful learning by ML model at the end of training! Brute force strategy can be to remove samples and train and see how it performs but ofcourse it will not be viable!
Not exactly what you are describing with the A, B, C groups, but this recent paper examines ways to prune data by introducing a way to measure how useful samples are: https://openreview.net/forum?id=UmvSlP-PyV
If you look for the training data set quality and filtering that's another word for usefulness.
You need to clarify what you mean by "useful learning". Performance on some downstream task? You may be interested in meta-learning.
This paper on shapely values sounds similar to what you are asking for. Hope it is helpful.
Well depends on what usefulness is.
If you can prove that all of your samples belong to the same distribution, then simply looking up which have the greatest gradient norm will be a measure of how useful they are for the model. Another approach is looking at how much their contribution would be in improving the performance of other samples, but then your dataset becomes a dependent variable.
But obviously this is dependent on the current weights, the loss function and various other biases. This is because gradient norm is proportional to the error, and so the samples for which the model predicts the most erroneous result will end up being most useful, given the perfect LR for it.
Thanks! Yes active learning and curriculum learning seem to be related to this! Thanks
Yes eventually will be determined by downstream task performance!
Seems to be super relavant!!! Will read it now!
Interesting take! Thanks
Thanks... I'll read it now.
Data Shapely is one option but can be computationally expensive. If you’re looking for practical code to try running on real data, here are some tutorials to find the least useful data:
https://docs.cleanlab.ai/stable/tutorials/image.html
https://docs.cleanlab.ai/stable/tutorials/outliers.html
as well as the MOST useful data to label next (or collect an extra label for):
Try influence functions: https://arxiv.org/abs/1703.04730
Interesting! Will have look at it! Thanks
There is the idea of core sets in continual learning. Research on how to construct such core sets could be of your interest
Thanks! Will read about it!
HateRedditCantQuitit t1_j42ogtm wrote
Not exactly what you’re asking, but active learning has a lot to say on data point usefulness.