Viewing a single comment thread. View all comments

DreamyPen OP t1_ivvhmu0 wrote

Reply to comment by Ulfgardleo in [Discussion] Can we train with multiple sources of data, some very reliable, others less so? by DreamyPen

  1. There are two sources of data. One experimental measurements with small amount of scatter, so it is considered highly reliable data. The second source is data predicted using physics-based models. They are sometimes quite accurate, sometimes a bit off. So it is indeed a supervised problem, with unreliable outputs not labels.
  2. I'm learning material properties. Ideally able to learn from the experimental data (ground truth), while capturing the trends from the synthetic model-based data.
  3. The experimental data is always considered highly reliable. The model-based data can be accurate or not, so a fixed reliability score should be suitable without knowing with certainty whether the models prediction is reliable or not for given input.
  4. Answered previously.
  5. We are mainly interested in predicting material properties that are close to the experimental (reliable) data, while still picking some useful signal from the less accurate physics-based data.

I hope this helps clarifying my objectives. Thank you.

1

Ulfgardleo t1_ivvjvsd wrote

  1. you said "unreliable outputs". did you mean inputs? if you truely meant outputs (i.e., the material properties that you want to predict from some so far undefined inputs) then this is what in ml is called "label".
  2. Okay, i have the same issue here. typically ground-truth would be what we called label, but i can see that you would distinguish between simulated/measured ground-truth data.
  3. "model" here is the physics based model, not the ML-model, right?
  4. I don't see it answered. I ask it explicitely: is there any experimental measurmeent for which you also have the physics-model output?
  5. You lost me here.
1

DreamyPen OP t1_ivvm1k0 wrote

  1. Yes I did mean outputs/targets. The features are always known, they correspond to testing conditions (a certain temperature, a certain processing speed, etc.) Given these testing conditions (inputs / labels), can we predict the material properties (outputs/targets) Experimental measurements are very reliable.

  2. The physics based model can always output a prediction for any given labels (testing conditions). But it is not always reliable. We would still like to include them because it allows us to augment the small experimental data set, and, often times, it is quite good approximation from the ground truth. This will also answer 4. Indeed, since the physics based model can always make predictions, we will have in some instances both reliable and unreliable data.

  3. Correct! :)

  4. We do indeed.

  5. Hopefully my response to 1. clarified it.

Let me know if the goal is clearer, and thank you for your help.

1

Ulfgardleo t1_ivxodj0 wrote

  1. okay you completely confuse everyone in the ML community when you call inputs "labels". lets keep with inputs/outputs.

  2. This is good, because it allows you to estimate some crude measure for the quality of the physics model.

So, label noise is a broad field. I am mostly knowledgeable in the classification setting, where label noise has different effects. Moreover, you are not in the standard noisy label setting, because the noise is not independent of the label, so just using weights will be difficult. Similarly, if you have more than one output to predict, a single weight is difficult to compute.

The standard way to derive all of these methods is by noting that the MSE can be derived as the log-probability of the normal distribution p(y|f) where y is the ground truth and f is the mean, and variance is some fixed value. For the mse, the value of the variance does not matter, as long as it remains fixed, but with fairly little effort you can show that as soon as you give samples individual variances, this amounts to weighting the MSE.

So, the cheapest approach would be to give outcomes from the different sources a different variance and if you have more than one output, you will also have more than one variance. How do you guess the parameters? well, make them learnable parameters and train them together with your model parameters.

Of course you can make it arbitrarily complicated. Since your cheap labels come from a physics simulation, errors are likely correlated so you can learn a full covariance matrix. And from there you can make it as complex as you like by making the error distribution more complex, but you will likely not have enough data to do so.

1