There's the issue of locally run LLMs. It's even possible low-scale with models like Pygmalion, but it would be an even bigger issue if there wasn't low-scale models, as nothing would stop the richer people from having a language learning model on the downlow, or as funny as it sounds, there might even emerge some sort of black-market of LLM

people are also seemingly very careless about what they put into LLMs

I had an enormous 10+ paragraph version of this very simple post discussing exactly some of those smaller LLMs, & while I'm not too familiar with Pygmalion, I know that the computing power necessary for the most successful models far outstrip what your average consumer is capable of generating. Effectively I argued that, because of economic & tech pressures, the AI industry is due for a contraction pretty soon, meaning that AI generated text would only come from an ever dwindling pool of sources as the less popular models die out.

I abandoned it before I got there, but I did want to touch on truly small scale LLMs & how fucked we could be in 3-5 years when any PC with a decent GPU can run a Russian Troll Farm.

Regarding privacy concerns, yeah. That's probably the best path to monetization this technology has at the moment. Training models on the business logic of individual firms & selling them an assistant capable of answering questions & circulating them through the proper channels in a company - but not outside it.

> I know that the computing power necessary for the most successful models far outstrip what your average consumer is capable of generating.

The training is resource intensive. The running is not, which is demonstrated by ChatGPT being able to support millions of users concurrently.

Even if you need a $3000 GPU to run it, that's a trivial cost for the help it can provide.

It’s kind of like trying to detect if a student used a calculator on a math test.

The key point to take away is that it’s no longer useful to teach students how to do complex calculations in their head. What’s far more important are the fundamental concepts. Do they understand the formulas and when to apply them, do they understand how to use the tools available to them to achieve a goal.

The end goal has never been to write an essay, it’s just to convey information. Far more important than the essay itself is information being conveyed. What idea is the student trying to communicate.

It will take academia a while to adjust though. For many years teachers stuck with the mantra of “you won’t always have a calculator on you”. I’m sure some will cry “you won’t always have access to an LLM generative text engine”, but we all know that’s simply not true.

Ehh no actually, that's not true. ChatGPT inferences are several times more expensive than your typical Google search, & utilize the same hardware resources used to train the model, operating at the same intensity, it seems.

>shouldn't the onus of delineating man from machine be on the side providing the AI chatbot?

It is.

Here's a very long read, but it will explain how OpenAI is building in watermarking for use by govt + themselves + maybe academia.

https://scottaaronson.blog/?p=6823

>'to watermark, instead of selecting the next token randomly, the idea will be to select it pseudorandomly, using a cryptographic pseudorandom function, whose key is known only to OpenAI. That won’t make any detectable difference to the end user, assuming the end user can’t distinguish the pseudorandom numbers from truly random ones. But now you can choose a pseudorandom function that secretly biases a certain score—a sum over a certain function g evaluated at each n-gram (sequence of n consecutive tokens), for some small n—which score you can also compute if you know the key for this pseudorandom function'

And why they wouldn't just stick it in a database of logs:

>'Some might wonder: if OpenAI controls the server, then why go to all the trouble to watermark? Why not just store all of GPT’s outputs in a giant database, and then consult the database later if you want to know whether something came from GPT? Well, the latter could be done, and might even have to be done in high-stakes cases involving law enforcement or whatever. But it would raise some serious privacy concerns: how do you reveal whether GPT did or didn’t generate a given candidate text, without potentially revealing how other people have been using GPT? The database approach also has difficulties in distinguishing text that GPT uniquely generated, from text that it generated simply because it has very high probability (e.g., a list of the first hundred prime numbers).'

That's not what I said lol. I said its manageable on hardware a consumer can buy.

> Am I missing something in my assessment?

Something you may be missing is that these models are being made available to collect data from users to make them better. Is data valuable to competitors commonly shared by competitors?

Also ever peek at charts showing how GPU power is growing vs time? In perhaps a decade our pocket devices may be locally running the current models. Anything done now will not help in the long run. I expect the various efforts to detect AI writing and AI art will add up to nothing because the error rates will be too high to be useful.

Oh, Sam's calculator shtick. Yeah I fundamentally disagree with that. Writing is not the same as Arithmetic. The goal of an essay is not to convey information, but for the student to internalize the concepts & present their interpretation & interaction with the ideas in a compelling & unique way. AI-assisted tools, at least with the strength of ChatGPT, negate this process to the detriment of most of academia. The struggle is the process.

> The end goal has never been to write an essay, it’s just to convey information. Far more important than the essay itself is information being conveyed

As someone who loathed writing essays/papers for school… if this is true, couldn’t we just write bullet points, a bulleted list?
There’s so much extra time needed, to check grammar, ensure sentences flow together nicely, good transitions between paragraphs, don’t use the same transition words every time, satisfying conclusion and introduction, etc etc.

Could save so much time, if the essay itself isn’t really the point anyway (outside of english/writing class).
Especially for those of us who aren’t gifted writers. So many late nights, all nighters in college finishing starting 5 or 10 page papers the day before they’re due.

I understand you now, my apologies.

“present their interpretation & interaction with the ideas in a compelling & unique way.”

So to convey information then…

Sure, NP, and you are partially right also lol. It may cost closer to $80,000 to have your own ChatGPT instance.

https://twitter.com/tomgoldsteincs/status/1600196988703690752

But then that sounds like a business opportunity lol.

Correct. Essays are a tradition, not a necessity.

Yes. The student's information. The AI cannot interpret, interact, nor can it, by definition, be unique. The AI cannot be used by a student as a crutch get out of developing their own assessments, as is done presently - & the essay is still an excellent medium to do this.

“the essay is still an excellent medium to do this.”

If that were true, then this wouldn’t be an issue.

I wonder why high school math students are still required to show their work when Wolfram Alpha can factor polynomials no problem.

Because they “won’t always have a calculator on them”…

Wolfram alpha can also show workings btw.

Then for the sake of people like me, I hope that that tradition goes away. I’d probably rather have a root canal or wisdom teeth out, than face an 8 page paper due in 12 hours, ever again.
(Except for writing classes, where the essay itself really is the point, of course)

Too late for me, but I’d be happy if future generations don’t have to go through that, while still learning.

Yeah I'm very aware. I'm also aware it's detrimental to getting the knowledge in your head. It's the same reason we don't teach kids basic arithmetic with 4 function calculators. An essay is "showing your work" about the topic of your essay.

Forgive me for not reading the entire post you linked, but is the plan that this watermarking would not be detectable by the general public out of concerns for “privacy”? Also, has this been implemented with ChatGPT (or do we know)?

Also, it surprises me that someone from OpenAI would acknowledge the shortcomings of their current measures for identifying AI-generated content.

To be fair, we're already a society built on cheating. All rich people already cheat. They used natural LLMs called "poor people" to do all their work for them and then just took credit for it. We call those people "CEOs".

Ultimately you can work around any detector by just having an AI learn solely of the cheater’s work. They’ll pick up on your mannerisms, which steps you skip, which route you’re more likely to take (convert to other units or just take it on as is), and what not.

In the end, there will be no way to detect AI’s work from humans. You could literally make the AI be wrong, just enough times not to trip the system but still get an A.

All this does is illustrate how stupid homework is and how important in class learning and work is.

Traditional education will always fail in a world with AI because traditional education barely works today. To put it more accurately, it barely works at the expense of the students who push themselves to work in an idiotic system.

Mostly thinking out loud here.

Education is already fundamentally busted on like 50 levels. Increased anti-cheating tech is so far from the original point of learning that it's not about learning anymore. Grading itself is already a necessary evil at best. And cheating being bad is a conclusion from that.

Okay, education, ideally, is for learning. In practice it has a double purpose of also being for certification. Which is to say, the examination and projects you're supposed to not cheat on are the point. This isn't really a good thing, but it may be a necessary one. Still, a lot of prestigious and/or exclusionary institutions are overwhelmingly about certification and not learning which clearly isn't.

More certification and stricter certification do not ensure more learning. In fact, they're often the opposite. In my own experience as a college student, I've had plenty of classes I cheated with super hard and ones I did honestly, and I've seen no correlation between classes I cheat on and classes where I don't learn much. I've had classes I did honestly, were challenging, got a great grade, and learned shit-all. I've also had classes I cheated on every single thing and learned a lot doing it.

In my experience, too, ChatGPT has been like a total godsend for learning. I ask questions to the professor, which needs to be done during the limited window of office hours, and they tell me to 'look over my notes and figure it out' cause they always do and they never help. I ask ChatGPT and they explain exactly what I want forever and I can ask about anything and I can do it at 2AM.

People in businesses will write out some bullet points and an AI will expand them into essays.

On the other end, people without the time to read much will have an AI summarize those essays in bullet points.

After a half century or so, everyone will get it through their thick heads that this is stupidly inefficient and just exchange the bullet points.

I disagree with some of this. "The struggle is the process" wafts of "I suffered, so so should you."

If our tools and technology progressed past the point of a certain test being useful, we move on and make new tests.

The math comparison is not useless. No one thinks writing and arithmetic are the same, so pointing it out isn't moving the discussion forward. The core point of the comparison is that when technology can perform part of the process, we change what it is we care about teaching. I don't know about you, but essays were often basically take-home busywork.

As far as writing essays go, LLM didn't exist when I was in school, but CliffNotes did. Sparknotes did. Enough internet to plagiarize with some clever editorializing. "Academia" has always had this problem. Some students will learn to the degree that they need to. And what industries are harmed by students fudging their essays? What jobs?

Won't those jobs also have access to the same tools? I'm in a very high level technical field and I now regularly use llm tools to get me started on templates for yaml files, terraform modules, etc. If anything, learning how to use it will be the skill.

The problem is these types of watermarks where the model layers are tweaked with a key to bend the output are easily obliterated by double dipping -- Chatgpt to generate then another paraphrase llm to rewrite. text canaries are brittle af.

Yeah, and I believe the author of that blog post acknowledges as much. I suppose being able to detect some text is better than being able to detect no text. Maybe that’s why watermarking is being pursued, but I can hardly speak for that author or for OpenAI.

>I know that the computing power necessary for the most successful models far outstrip what your average consumer is capable of generating.

And once upon a time a useful computer would never fit in an average person's home. Ignoring all the other ways your store -everything idea wouldn't be effective, the cost of compute and efficiency of these models is changing so fast that by the time your idea was implemented, it would already be obsolete.

Anything degree that relies on writing assignments is up to being automated anyway.

EDIT: I am so sorry this is long as shit & it ends on a downer. It's just a really morose & unpleasant read.

Later in the thread I used a better comparison: Wolfram Alpha is not used to teach pre-calculus. 4 function calculators are not used to teach basic arithmetic. We gate a student's "generative ability" based on the skills we want them to develop. Trigonometry does not measure a student's ability to draw a sine function, but rather their ability to represent, measure, & manipulate one. The robot can draw the line to match your function, that's the easy part. Making sure your function is correct is the part you need to learn.

The essay is the function, not the line. It is the proof of the struggle with something new that will produce necessary skills for development. At the very least, it's proof that the user can read a new thing & generate a cogent output from it, which is such an impressive accomplishment in nature that teaching it to machines has caused significant economic & social disruptions.

It's evidence of a user's ability to interrelate information - a process so complex it must be done essentially from scratch every time the user alters even one parameter of their data set. Where mathematical reasoning, at least elementary math, linearly grows in complexity, allowing students the ability to compress portions of the process generatively, no such linearity exists in any other discipline. No one in studying Faust is saying: "I learned about 17th century English Literature last year. I'll just plug Paradise Lost into the machine to return a comparison between Milton & Goethe's portrayals of aberrant desire"

Lastly, it's evidence of the user's ability to communicate, which can be considered a complex test of metacognition, a much simpler test of the arbitrary constraints of syntax, & a gauge for how fulfilling the experience was for the user. At the end of the day, that is what it's about.

We need people to have all of these skills. Many of them are difficult to learn. Most of them overlap with ChatGPTs advertised features. We are asking our education system to revolutionize itself in response to a new toy in an extremely short time while extremely underfunded & extremely overtaxed. This is a recipe for a goddamn catastrophe.

You asked what the actual fallout of the last several decades of neglecting liberal arts education has been, &, if I may be perfectly frank, I think it's produced a fucking wasteland. Our industries are corrupted by a revitalized mercenary fetish for cutting overhead & maximizing dividends at a human cost. Our public gathering places are being bulldozed & replaced with more profit-sucking real estate. Our actions are monitored, dissent is catalogued, & punishment is divvied out on an industrial scale. When it happens to us, so often we are incapable of placing it in a larger context. When it happens to others, we struggle with our incomplete grasp of empathy & susceptibility to reams of misinformation. All of this, helmed by engineers, computer scientists, lawyers, entrepreneurs, politicians, & citizens simultaneously over & under-educated.

I have a personal example. My dad held a degree in Nuclear Engineering & had nearly 30 year's experience in systems analysis, quality assurance, continuous improvement & adjacent managerial disciplines in the Energy, Aerospace, & Manufacturing industries. He died a year & a half ago. The disease was systematized ignorance. Delta variant was just a symptom.

Oh no that seems a bit silly to me. The last 15 years were literally about our global "store-everything" infrastructure. If we're betting on a race between web devs encoding tiny text files & computer engineers attempting to rescale a language model of unprecedented size to hardware so efficient it's more cost effective to run on-site than access remotely, I'm putting money on the web devs lmao

> I am so sorry this is long as shit & it ends on a downer. It's just a really morose & unpleasant read.

I read it all, though. You're probably tired of hearing it but I'm still sorry for your dad. That sucks.

Anyway... damn. Yeah. You've made excellent points.

I'm a technophile, I believe in progress, and I watch on the daily as outmoded corporate interests stymie or erase technical progress that could become cultural progress. I watch politicians flagrantly ignore scientific evidence for climate change or disease, while watching journalists encourage them.

So, perhaps much of my opinion surrounding AI is derived of those feelings and sympathies. But I think you're right on much, maybe all of what you've said.

So now, I don't know. I don't know the best way forward. I don't think AI is going away. Companies are working out tools to detect AI-generated text, but technological progress also demands the AI-generators get smarter.

Thank you for writing that all out, though.

Getting the AI to turn the essay into a podcast would be handy though.

>After a half century or so, everyone will get it through their thick heads that this is stupidly inefficient and just exchange the bullet points.

Naah, bullet point miss out on the language nuances that full text communicates, even in business

>In my experience, too, ChatGPT has been like a total godsend for learning.

I think you'd like this approach:

https://www.businessinsider.com/wharton-mba-professor-requires-students-to-use-chatgpt-ai-cheating-2023-1?op=1

Yes, but people in business tend to get a lot of emails, and if most of them resort to text summarization then the nuance is lost anyway. And it's mostly lost if the recipient skims.

Also, many senders of email aren't necessarily great communicators conveying valuable nuance anyway.

Ultimately, it's a cost-benefit calculation: get occasionally-valuable nuance on a bunch of emails, or keep emails simple and do something else that might be more valuable?