Viewing a single comment thread. View all comments

zero_for_effort t1_jdqvirs wrote

Reply to comment by ArcticWinterZzZ in Why is maths so hard for LLMs? by RadioFreeAmerika

Explain it like we're five?

15

0382815 t1_jdr6r9p wrote

The ELI5 is that it doesn’t actually calculate anything when you ask it to solve an equation. It guesses what the answer looks like—probably some combination of numbers—based on what it saw in the training dataset.

22

Ok_Faithlessness4197 t1_jdrt7xy wrote

That's not quite correct. While all it does is guess what the next token will be, it can intelligently infer that an equation (even one outside its training set) needs to be calculated, and then calculate it. The problem is it's inability to utilize it's understanding of mathematics to answer prompts in an efficient and accurate manner. Once a calculator is implemented (Probably in GPT 4.1 given the recent paper by Microsoft demonstrating its capability to use one), this problem will hopefully be resolved.

6

0382815 t1_jds9yxv wrote

It is quite correct, actually. When you prompt it to multiply, it does not run a multiplication algorithm.

4

Ok_Faithlessness4197 t1_jdsl2be wrote

Input: Solve 343 x 875 with long multiplication.

Output (ChatGPT): Sure! Here's how to solve 343 x 875 with long multiplication: 3 x 5 = 15 Write down 5, carry-over 1 3 x 7 = 21, plus the carried-over 1 = 22 Write down 2, carry-over 2 3 x 8 = 24, plus the carried-over 2 = 26 Write down 6, carry-over 2 3 x 5 = 15, plus the carried-over 2 = 17 Write down 7, carry-over 1 Add the carried-over 1 to the result of 3 x 4 = 12, which gives 13 Write down 3, carry-over 1 4 x 5 = 20 Write down 0, carry-over 2 4 x 7 = 28, plus the carried-over 2 = 30 Write down 0, carry-over 3 4 x 8 = 32, plus the carried-over 3 = 35 Write down 5, carry-over 3 4 x 5 = 20, plus the carried-over 3 = 23 Write down 3, carry-over 2 Add the carried-over 2 to the result of 3 x 8 = 24, which gives 26 Write down 6, carry-over 2 Add the carried-over 2 to the result of 3 x 3 = 9, which gives 11 Write down 1, carry-over 1 Add the carried-over 1 to the result of 3 x 8 = 24, which gives 25 Write down 5 So, the final result is 300125.

−4

0382815 t1_jdsn7o1 wrote

Once again, if you prompt it to multiply it does not run a multiplication algorithm.

3

robobub t1_jdst84e wrote

Why? Each of those tokens is O(1) and it is predicting each one incrementally, taking into account the ones it has just generated. So the full answer has taken O(m) where m is the number of tokens.

If it is possible for GPT to do 1+1, it can do a large number of them incrementally. It's not smart enough to do it all the time (you'll have more success if you encourage GPT to have a train of thought reasoning) but it's possible.

1

Ok_Faithlessness4197 t1_jdsqqgg wrote

Alright, go ahead and ignore the multiplication algorithm it just demonstrated.

−1

0382815 t1_jdsrl52 wrote

What you did was prompt it to multiply. For the third time this thread, I will tell you that what it is doing is not running a multiplication algorithm. It is guessing the next token based on the preceding tokens. The model is large enough to predict correctly in this case. It is still not running a multiplication algorithm the same way the calculator app on Windows does.

6

Ok_Faithlessness4197 t1_jdsskog wrote

I absolutely agree, it's multiplication algorithm is very slow, very inefficient, and very different from the way a calculator would handle it. I think it does differ too from how you're considering it, though. It's more than just a really good text predictor. It can use logic and solve novel problems in many unprecedented ways. Here, I would argue, it has a greater-than-superficial understanding of the math algorithm it used to multiply numbers. Can I ask how you'd define an algorithm, and what you'd consider "running a multiplication algorithm"?

−2

Ok_Tip5082 t1_jdtzd17 wrote

Chat GPT is not running the multiplication algorithm. You're being the human in the loop here by having it iterate through every step in the algorithm. You're manually executing a bunch of constant time operations and feeding the input back into itself.

You're basically writing and running code. If this qualified as being able to derive a multiplication algorithm then all CPUs are already sentient.

2

Ok_Faithlessness4197 t1_jdu12qm wrote

I make no claims about sentience. I will say however that this is far ahead of what was previously achievable by AI standards. In its current form, it has to be allowed enough time to satisfy the mathematical time requirement. In the future, once it's linked with WolframAlpha (A math AI) it will not make the simple mistakes it makes now.

0

Ok_Tip5082 t1_jdu2er4 wrote

Yeah, pragmatically I don't see any issues with arithmetic or using any math already proved. Imo it's still to be seen if LLMs can do novel thought, but even if not that's still ... what's a word signifying a greater change than revolutionary? Game changing?

I did see some AI coming up with independent models of physics that have no analog yet were able to properly model real physical systems and make valid predictions with a formula whose variables could not all be determined by the researchers, but idk if that was an LLM

2

MassiveIndependence8 t1_jdr6u2t wrote

It takes GPT the same amount time to do anything, and since it’s impossible to multiple say “18837678995747 x 29747778847678877” in the same amount of time as “2 x 2” due to the fact that it’s more complicated, we can confidently say that GPT will never be able to do math since it means that every hard problems out there is as easy as the easy ones.

10

ArcticWinterZzZ t1_jdt0plo wrote

GPT-4 always takes the same amount of time to output a token. However, multiplication has been proven to take more time than GPT-4 has available. Therefore, an LLM like GPT-4 cannot possibly "grow" the requisite structures required to actually calculate multiplication "instantly". There are probably quite a few more problems like this, which is why chain-of-thought prompting can be so powerful.

3