[D] GPT-3 is a DREAM for citation-farmers - Threat Model Tuesday #1

Submitted by TiredOldCrow t3_y7mwmw in MachineLearning

GPT-3 and the multitude of similar models that have come out over the last couple years likely represent a serious threat to scientific conferences. What can we do about it?

Some historical context: computer-generated papers have been showing up in major publications since a model called SCIgen released in 2005.

SCIgen uses a simple context-free grammar to produce templatized papers that are basically pseudo-scientific gibberish. People are still finding those papers many years later. These papers are generally churned out to inflate citation statistics, or by well-meaning researchers probing suspected low publication standards at existing conferences (which I generally don’t recommend, since it adds to the mountains of paper that we have to churn through as reviewers).

There’s a negligible chance that existing review processes will fare any better on 175B parameter generative Transformers.

Work has already been published that uses GPT-2 to “assist” in scientific writing. While using such a model while writing is not necessarily academically dishonest – such tools nevertheless greatly increase the ease of churning out fake papers.

Even if major conferences find creative solutions to this, the next rung of venues below them are likely to learn about these threat models the hard way. After all -- to an unethical researchers citing their own work at the bottom of a machine generated paper -- a citation is a citation, even in a low quality publication.

--

Open Questions

Threat severity: How serious is this threat? Is this really an old problem, and new generative models won’t make a huge difference?

Improving peer review: Does this just come back to reproducibility? Should we be reviewing certain papers by their code, rather than just their text? Do we need a way to weight the value of research citations by the quality of the work doing the citing?

Acceptability of machine text: How do you decide when machine generated text is unacceptable versus acceptable? Will detection models for machine generated text end up creating algorithmic biases targeting people who speak English as a second language and rely on translation models or writing assistants to help them write?

Legitimate AI-generated papers? Are there papers with real scientific value that could be entirely written by algorithms? E.g., could survey papers one day be produced almost entirely by a specialized model?

Defenses: What are some technical or social solutions that can help defend against this type of abuse?

--

This is Part 1 of a planned Tuesday discussion series on threat models, based on the things keeping us up at night since a recent survey paper focusing on the threat models of machine generated text.

7

Comments

You must log in or register to comment.

TiredOldCrow OP t1_isvei0e wrote

My own thinking is that large venues might implement automated filtering using detection models. Detection of machine generated text increases with sequence length, so papers with large amounts of generated text stand reasonable odds of detection.

That said, the results of this detection would likely need to be scrutinized by a reviewer anyways (especially if conferences don’t ban AI writing assistants altogether).

3

gravitas_shortage t1_isvkgcr wrote

Peer review is in a death spiral right now, it's not going to be a solution long-term. I expect the only viable option will be adversarial AIs trained to detect fake papers.

7

countably_infinite_ t1_iswvh51 wrote

A paper has merit based on the the academic contribution. Being clear and precise, easily understandable, makes it a better paper. If you manage to prompt a LLM so that the result excels in these criteria it should be accepted.

With the attitude some students and vanity-auithors have there is already an incentive to produce pseudo scientific mumbo-jumbo and see if you can get away with it. I mostly see a problem here in scalability, i.e. the pure mountain of noise that can be generated. Similar dangers are valid for journalism and democratic discourse in general imho.

One already decently working mitigation strategy is reputation (for authors, groups, conferences, ...) but of course this comes at the risk of overlooking some brilliant work that is coming from left field.

9

cadop t1_isxah1a wrote

I think there will have to be some collaboration from universities (and companies). Some not-so-refined ideas would be something like this (admittedly some problems with non-academic submissions):

- Submission requires a known affiliated email address (seems arxiv does this)

- Given a fake paper, the affiliated institute is warned and added to a known list, author(s) banned from submitting

- Second occurrence to appear on the list bans the university from submitting

"Submitting" referring to any network of conferences/journals that decide to opt into this.
This will very quickly stop authors from doing it, and universities will surely fire anyone who tries that.

For context, I am thinking of the github issue with (washington?) researchers putting bad commits/PRs

4

rw_eevee t1_isxukxi wrote

Bigger problem for non-ML. It’s one thing to generate non-sense about Shakespeare or LGBTQ glaciology, it’s another to generate a even a bad ML paper complete with derivations, proofs, and experiments. And if a model can produce compelling ML papers, it should be allowed in on principle (though the experiments would be an issue, e.g., if the model completely fabricated the graphs reviewers won’t easily catch this. But this could be treated the same as a researcher fabricating data.)

3

TheLastVegan t1_isycgq5 wrote

Replying to rw_eevee (#142,683)

Writing gibberish is a coveted skill in politics because an unreadable proposal is harder to criticize, and any logical fallacy can be supported by semanticism to give the illusion of substance! In identity politics, writing fluff is necessary to signal cultural affiliation, which adds emotional weight to the gibberish in an essay. If a grad student needs to cite 20 puff pieces to get approved by their thesis supervisor, then they're going to need the manifold hypothesis either way! In the social sciences, structuring data to fit or obfuscate a sketchy narrative will generally be more lucrative than writing an unbiased analysis.

3

the_mighty_skeetadon t1_iszprhg wrote

Replying to stevewithaweave (#143,169)

That can't be the only method, because if your model for generating fake papers differs significantly from somebody else's model, you will be both unable to detect those fake papers and unable to detect that you're failing.

Better is to have fake papers rejected from journals labeled thusly and to synthetically generate more fake papers with a wide variety of known approaches.

1