Can you? Sure. Should you? Not sure about that.

If you think of neural networks as underlying probability distributions, then estimating the error of your regression as single values can seem misleading, as the true error actually lies within a distribution of predicted errors. Going off of that, it would be better to analyze the distribution of your errors in relation to your predictions.

I can do that during tests on labeled data but what about unlabeled data?

If you have the assumption that your test data and forecasting data are from the same distributions, which they should be unless data drift is occurring, then you can create the distribution of your errors when your model has been trained by looking at the errors for your test set. From there you could simply say, "This is my prediction and this is the general distribution of error, so therefore this is the distribution of my potential values," or you could take it a step further and look at the errors associated with the test data that are most similar.

There may be other techniques that work better, so further reading may help.

You might be able to recast the problem to assume the labels are acutally drawn from some distribution and put some simple liklihood function over it -- then learn the parameters of that distribution. This not theoretically sound, you wont capture any epistemic uncertainty, but most of the aleotoric, so dependend on your usecase, it might work.

In practice, use for example a Gaussian likelihood, and learn with a GaussianNLL Loss also the variance. As long as your samples stay within the same distribution, yadaya, this can work okish ..

Otherwise, there are plenty of recalibration techniques to get better results

> In practice, use for example a Gaussian likelihood, learn wicht GauddianNLL Loss also the variance. Ax long ad you stay eithin distri yadaya this can work okish ..

You ok?

Perhaps a Bayesian neural net would be what you're looking for

The paper Epistemic Neural Networks does this formally and efficiently. Much more than Bayesian networks, at the cost of slightly more than your standard forward pass.

If you don't want to go full Bayesian there's always the good old bootstrap. Retrain the model as many times as possible over N replicates of your original data sampled with replacement, then take the variance of your errors over the N errors.

Monte Carlo Dropout during the forward pass can be used for variance estimation.

Predict the distribution of the residuals. This will involve modelling a contribution from noise within the data and a contribution from uncertainty in the neural network weights. You can model the former with negative log-likelihood loss. And you can model the latter either by training multiple of the same model and taking the average and standard deviation of their predictions, or by using Monte Carlo dropout.

For more info, read this nice paper. This one is good too, and there's examples of code as well.

I feel like you're trying to re-invent regression residuals analysis in a NN setting? Where you're comparing your data points to the predicted values or class? There are a lot of tools on the regression diagnostics side, most of which are looking for things that don't really matter in an arbitrary non-linear curve fitting process like NN. So depends on what your need is for the error analysis.

My dropout for uncertainty coming from the model. Learning mean together with variance for uncertainty coming from the data. Assuming your target values are sampled from Normal distribution .

The network is already doing its best at minimising the distance. If your final goal is point estimates that minimise the distance, predicting the error is probably not a good way to go about improving performance.

However, if you care about the uncertainty / having a distribution over where the ground truth might be, then there are definitely various techniques that allow this.

For example, if you expect the errors to change depending on some conditioning variable, you could have the neural network output the locations (mean), and the standard deviations (uncertainty) of the positions, given the conditioning variables. In practice you would output log stds and exponentiate to ensure positivity. Then you could use a Gaussian likelihood function approximation, replacing the L2 norm with the negative log likelihood under the Gaussian assumption.

That's a cool reference! I hope the rest of the article is as exciting as the abstract promises.

What you want is a bayesian estimator. It gives you a probability distribution over all possible regression values (where the mode/expected value is the equivalent of the point estimate you are used to). The smaller the distribution the higher its estimated accuracy. You basicly get the value and its expected error all in one. No problem coding this into a neural network.

Jax only?

Just published a paper using MC dropout for uncertainty estimation - intro section includes some references OP may find helpful: https://www.nature.com/articles/s41467-022-34025-x

Two papers that provide foundational theory are by Gal and Sida

Do you have a recommended read?

The one book everyone practitioner of machine learning should have read: Bishop "Pattern Recognition and Machine Learning"

I bought that book during my masters and I couldn’t figure out what Bishop was saying. It’s years later and my math is way better so I’ll give it another shot. Thank you!

Wait you're one of the authors here?

You should check out what's known in the field as "uncertainty-aware learning". Definitely not the same as "getting NNs to estimate their own uncertainty" but certainly helpful for what you're wanting to do

I’m first author on the first paper, but wasn’t involved in either of the others.

Human Pose Regression with Residual Log-likelihood Estimation learns an error distribution using normalizing flows. The network predicts expected variance and this is used to train the flow model to learn the error distribution to reparameterize the loss function. The predicted variance can also be used at inference.

Mabye i dont quiet get the question. I am pretty new to ML. But doesnt it sound a little bit like neural processes? You get a uncertainty feedback by splitting your data into subsets and training several NN Models, thus getting some kind of distribution over possible functions.