Viewing a single comment thread. View all comments

Zatania OP t1_ir5kskz wrote

Reply to comment by Top-Perspective2560 in [R] Google Colab alternative by Zatania

I'll try this solution if this works, will get back to you.

2

you-get-an-upvote t1_ir9978p wrote

FYI loading many small files from drive is very slow. If this applies to you, I recommend zipping the files, uploading to drive, copying the zipped file onto your colab machine, and unzipping.

from google.colab import drive

drive.mount('/content/drive')

!cp '/content/drive/My Drive/foo.zip' '/tmp/foo.zip'

os.chdir("/tmp")

!unzip -qq 'foo.zip'

Otherwise, if your dataloader is trying to copy files over from Drive one at a time it's going to be really slow.

Also I'd make sure you're not accidentally loading the entire dataset into RAM (assuming your crash is due to lack of RAM?).

2