Submitted by TheButteryNoodle t3_zau0uc in deeplearning
Hey everyone. I'm building a new workstation for both personal and professional use and I need some help weighing the pros/cons of the different GPUs I'm looking at as well as general advice/recommendations.
Most of my professional work would fall within NLP and GNN models, however, I do occasionally dabble in image classifiers and stable diffusion as a hobby. The current GPUs that I was looking at are an RTX A6000 ADA, a used/refurbished A100 80GB (using PCIE instead of SXM4), or dual 4090s with a power limitation (I have a 1300watt PSU).
With the RTX A6000 ADA having 48gb of vram, it's definitely nice to be able to be able to load a whole new range of models that I wouldn't have been able to otherwise (without AWS or model parallelism), but it's harder to justify the current expected cost of $7,378-8210 when you could spend an additional $2-3k and get a used/refurbished A100 80GB GPU from ebay that provides almost double the vram and would likely outperform the new A6000 ADA card by a sizeable amount in FP16 and FP32 calculations.
However, you could also just get two RTX 4090s that would cost ~$4k and likely outperform the RTX 6000 ADA and be comparable to the A100 80GB in FP16 and FP32 calculations. The only consideration here is that I would need to change to a custom water-cooling setup as my current case wouldn't support two 4090s with their massive heatsinks (I'm assuming this change may cost in the range of $1.5-2k). Furthermore, I would likely need to put a power limitation on the GPUs with a 1300 watt PSU. The vram, at 24gb, would likely cover all of my professional use cases, but prevents me from loading in larger models without having to utilize model parallelism, which can be painful.
I also do like to play games casually. While this is not a major factor, it would be nice to not have to maintain two different rigs, with the A100 not really able to support games.
So, with all that being said, does it make sense to go for two 4090s, which would be ~4k plus a water cooling setup at ~1.5k, making it 5.5k in total? Or go for a RTX 6000 ADA at ~7.5-8k, which would likely have less computing power than 2 4090s, but make it easier to load in larger things to experiment with. Or just go for the end game with an A100 80gb at ~10k, but have a separate rig to maintain for games.
I do use AWS as well for model training for work. But with the recent AWS bills, my company has offered to pay a portion of the cost to build a new workstation. I will still be paying for most of the costs, but want to utilize the opportunity as personal PC upgrade. Any model training on AWS, that wouldn't be used for work, would obviously be billed to me (hence the interest in just getting a card with greater vram).
What do you all think makes the most sense here?
computing_professor t1_iynllw4 wrote
I'm far from an expert but remember the 4090s are powerful but won't pool memory. I'm actually looking into a lighter setup than you with either an A6000 or, more likely, 2x 3090s with nvlink so I can get access to 48GB of vRAM. While the 4090 is much faster, you won't have access to as much vRAM. But if you can make do with 24GB and/or can parallelize your model, 2x 4090s would be awesome.
edit: Just re-read your post and I see I missed you mention parallelizing already. Still, if you can manage, 2x 4090 seems incredibly fast. I would do that if it was me, but I don't care much about computer vision.