chuanli11
chuanli11 t1_ivhlelj wrote
Reply to comment by Zer01123 in [D] NVIDIA RTX 4090 vs RTX 3090 Deep Learning Benchmarks by mippie_moe
We used the recommended retail price from NVIDIA but OMG they are expensive on the street Lol
chuanli11 t1_ivhkx9p wrote
Reply to comment by learn-deeply in [D] NVIDIA RTX 4090 vs RTX 3090 Deep Learning Benchmarks by mippie_moe
Hey, Thanks for the comment. We made sure each GPU uses x16 PCIe 4.0 lanes. It is data parallel (PyTorch DDP specifically).
We look forward to the FP8/CUDA 12 update too.
chuanli11 t1_ivhsvnf wrote
Reply to comment by whata_wonderful_day in [D] NVIDIA RTX 4090 vs RTX 3090 Deep Learning Benchmarks by mippie_moe
BERT large did scales less well for 2x4090. You can find the exact numbers here:
https://github.com/lambdal/deeplearning-benchmark/blob/22.09-py3/pytorch/pytorch-train-throughput-fp32.csv
https://github.com/lambdal/deeplearning-benchmark/blob/22.09-py3/pytorch/pytorch-train-throughput-fp16.csv