nwatab t1_iy8bssy wrote on November 29, 2022 at 2:40 PM

Reply to comment by nwatab in [D] Simple Questions Thread by AutoModerator

Yes, it was cache that caused a problem. Now it works good. Somehow it didn't come up to me. Thanks!

nwatab t1_iy7yfr8 wrote on November 29, 2022 at 12:41 PM

Reply to comment by I-am_Sleepy in [D] Simple Questions Thread by AutoModerator

Thanks. My data is one CSV and a lot of jpgs. I'm using tf.data input pipelines. .cache() could cause a problem according to your insights. I'll check them.

nwatab t1_iy75dep wrote on November 29, 2022 at 6:14 AM

Reply to [D] Simple Questions Thread by AutoModerator

I was training 10GB dataset on AWS ec2 (AMI: Deep Learning AMI GPU TensorFlow 2.10.0 (Amazon Linux 2) 20221116). After half an epoch, ec2 is very slow due to lack of memory. Does anyone know why? I don't understand why "after about half an epoch (around less than 10 minutes)", it gets slow, instead of the beginning of training.