Earlier this year, a US radiologist published a paper written with the help of ChatGPT in a peer-reviewed journal.
Som Biswas of the University of Tennessee Health Science Center in Memphis wrote the article “ChatGPT and the Future of Medical Writing” for the journal Radiology.
He said he produced the article, which he edited, to help raise awareness about the usefulness of the technology.
“I am a researcher and I publish articles on a regular basis,” Dr Biswas told The Daily Beast.
“If ChatGPT can be used to write stories and jokes, why not use it for research or publication for serious articles?”
The Daily Beast reported he’s since gone on to publish 16 more journal articles in four months using the chatbot.
It also cited one journal editor who said they had experienced a “dramatic uptick” in articles.
Heather Desaire, a professor of chemistry at the University of Kansas can relate.
“The story rings true with my personal experience,” she told the ABC.
“I worry about journals being overwhelmed with paper submissions and, as a reviewer for those journals, being asked to do 10 times more reviews than I normally would.”
While Professor Desaire is no enemy of ChatGPT, she does think it’s important to keep an eye on unintended impacts, and she hopes her latest research might help.
A new AI-detector for scientific texts?
In today’s issue of the journal Cell Reports Physical Science Professor Desaire and colleagues reported they have developed a highly accurate method of detecting ChatGPT-generated writing in scientific texts.
Professor Desaire said it could help journal editors who find themselves deluged by material written with the help of the chatbot.
A detector might help editors prioritise what articles they send out for review, she said.
To develop their tool, the researchers first identified a set of “telltale signs” that differentiate AI-generated text from that written by human scientists.
They did this by carefully analysing 64 “perspective” articles from the journal Science — these are review papers that comment on current research and put it in context. Then they analysed 128 ChatGPT-generated articles on the same research topics.
From comparing the two they identified 20 characteristics that could help decide the authorship of scientific texts.
Covering paragraph complexity, diversity in sentence length, use of punctuation and vocabulary, some of these features appeared unique to scientists, the researchers found.
For example, while they found humans writing on Twitter might use punctuation like double exclamation points to express emotion, Professor Desaire and her team found scientists have different linguistic penchants.
“Scientists are special people,” she said. “They aren’t using double exclamation points, but they are using more parentheses and dashes than ChatGPT.”
And her research found scientists did like to go on a bit by comparison.
“The difference in paragraph length really jumps out at you,” said Professor Desaire.
Humans were more likely to write very short sentences and very long sentences, she added.
Another characteristic of human-generated scientific text was the use of “equivocal language” — words like “however”, “although” and “but”.
And scientists used more question marks, semicolons and twice as many capital letters as ChatGPT.
Training the AI-detector
The researchers then used these 20 features to train an off-the-shelf machine-learning algorithm known as XGBoost.
Known in the business as a “classifier”, the algorithm provides a mathematical way of deciding between two options — Professor Desaire and her team use it in their daily work to identify biomarkers for diseases such as Alzheimer’s.
