HiPattern t1_iycvpfp wrote on November 30, 2022 at 1:42 PM

Write a generator that feeds the data in batches:

https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly

somebodyenjoy OP t1_iycwvf4 wrote on November 30, 2022 at 1:52 PM

Very interesting, but I wanted the model to preprocess the data only once. This way, it’ll preprocess at every epoch

HiPattern t1_iycypm1 wrote on November 30, 2022 at 2:07 PM

You can preprocess, then write the data into a hdf5 file, and read the preprocessed data batch wise from the hdf5 file!

somebodyenjoy OP t1_iycyur2 wrote on November 30, 2022 at 2:08 PM

I do the same using numpy files, but they only let me load the whole data which is too big in the first place. Tensorflow let’s us load in batches huh, I’ll look into this

HiPattern t1_iyd91t0 wrote on November 30, 2022 at 3:24 PM

hdf5 files are quite nice for that. You can write your X / y datasets in chunks into the file. When you access a batch, then it will only read the part of the hdf5 file where the batch is.

You can also use multiple numpy files, e.g. one for each batch, and then handle the file management in the sequence generator.

somebodyenjoy OP t1_iydqmu0 wrote on November 30, 2022 at 5:20 PM

This is perfect, I won’t have to invest in additional RAM. Thanks for the tip!