ParanoidTire t1_j9hbd77 wrote on February 21, 2023 at 11:43 PM

Reply to comment by GaseousOrchid in [D] Simple Questions Thread by AutoModerator

I have years of grievances with io. It's really difficult to have something that is both flexible, performant, and can scale to terabytes of data with complex strucuture. As soon as you leave the nice cv or nlp domain you are on your own. Raw c type arrays loaded manually from disk in a separate Cuda stream can sometimes be really be your best shot.

GaseousOrchid t1_j9hhxdm wrote on February 22, 2023 at 12:30 AM

yeah, this has been my experience -- i'm working with a lot of custom data, and even though some of it is CV adjacent, it doesn't fit exactly (e.g., ~40 channels instead of 3 like RGB). would be nice, especially for research prposes, to have something to plug and play that just worked.