Viewing a single comment thread. View all comments

MUSEy69 t1_iyzzbxr wrote

Hi, you should always have an independent test split, and do whatever you want with the other, e.g. Crossvalidation visual sklearn reference

Why are you losing lots of datapoint in the test split? the idea is that distributions match so you can use the p-value criteria for this.

If you want to test lots of models try, optuna for finding the best hparams. No problem using the same metric, that's the one you care at the end.

Depending on your domain I would ignore step 5, because you can test disfribution shifts, and even new models in time and be able to compare them.

4

Visual-Arm-7375 OP t1_iz2e6z6 wrote

Thank you for the reply! Step 5 is because I have to submit the predictions for a separated from which I don't know the labels. So my idea was to use all the data.

1

MUSEy69 t1_iz4gkn2 wrote

Thank you for your question, it generated different points of view, from which I learned a lot.

2

killver t1_iz013vj wrote

> you should always have an independent test split

nope, this is not true

−7

[deleted] t1_iz07acr wrote

Please elaborate. Are you suggesting that we should hyperparameter-tune on the test set?

6

killver t1_iz0a6xk wrote

No the opposite. So why would you need a test set?

I am arguing that the test data is basically useless, because if you make a decision on it based on performance it is just another validation dataset, and if not you can better use the data for training.

−2