norcalnatv
norcalnatv OP t1_j84wt52 wrote
Reply to comment by norcalnatv in The Inference Cost Of Search Disruption – Large Language Model Cost Analysis [D] by norcalnatv
If the ChatGPT model were ham-fisted into Google’s existing search
businesses, the impact would be devastating. There would be a $36
Billion reduction in operating income. This is $36 Billion of LLM
inference costs.
norcalnatv OP t1_j84wfs7 wrote
Reply to The Inference Cost Of Search Disruption – Large Language Model Cost Analysis [D] by norcalnatv
"Our model is built from the ground up on a per-inference basis, but it lines up with Sam Altman’s tweet and an interview he did recently. We assume that OpenAI used a GPT-3 dense model architecture with a size of175 billion parameters, hidden dimension of 16k, sequence length of 4k,average tokens per response of 2k, 15 responses per user, 13 million daily active users, FLOPS utilization rates 2x higher than FasterTransformer at <2000ms latency, int8 quantization, 50% hardware utilization rates due to purely idle time, and $1 cost per GPU hour. Please challenge our assumptions"
norcalnatv t1_j6drni3 wrote
Reply to comment by Biff_Malibu_69 in China’s Top Nuclear-Weapons Lab Used American Computer Chips Decades After Ban by hzj5790
Exactly. Relationships are what it’s all about.
norcalnatv t1_izshj9v wrote
If you own the laptop it’s always going to be cheaper to use that than going to the cloud.
norcalnatv t1_je3v5yw wrote
Reply to Should software developers be concerned of AI? [D] by Chasehud
If they're they're below average they should.