Viewing a single comment thread. View all comments

newperson77777777 t1_j1r9oni wrote

Reply to comment by skn133229 in [D] Normalized images in UNET by skn133229

So based on my understanding, training fails when you try to normalize per image, right? When you don't normalize per-image, training is fine but you just get suboptimal validation performance - which, in this case, you are saying is because there may be more error during certain years, especially years that are not considered during training. Is that an accurate summary?

If this is the case, I would try to test two things independently (not at the same time): 1. standard normalization over the entire dataset and 2. standard normalization per year. I would also explore the data based on year year and try to see what are the differences to see how you can adjust your modeling. Additionally, I would try do a more thorough error analysis to understand patterns in the errors the model is making.

3

skn133229 OP t1_j1t9qgo wrote

I solved the problem with failure to train (see update in original post). It appears that implementing some random spectral shifts and random noise were necessary for training to take off. I will try your suggestions to see if I get better prediction results year over year.

1

newperson77777777 t1_j1tbh2m wrote

Weird, haven't encountered that issue before. If you do the feature-wise normalization that I discussed before, more likely than not that will result in, for most images, the image channel mean not being 0 as well, if that helps.

1