Chatbots, ChatGPT, and Scholarly Manuscripts: WAME Recommendations on ChatGPT and Chatbots in relation to scholarly publications

Journals have begun to publish papers in which chatbots such as ChatGPT are shown as co-authors. The following World Association of Medical Editors (WAME) recommendations are intended to inform editors and help them develop policies regarding chatbots for their journals, to help authors understand how use of chatbots might be attributed in their work, and address the need for all journal editors to have access to manuscript screening tools. In this rapidly evolving field, we expect these recommendations to evolve as well.

A chatbot is a tool ‘[d]riven by [artificial intelligence], automated rules, natural-language processing (NLP), and machine learning (ML)…[to] process data to deliver responses to requests of all kinds.’¹ Artificial intelligence (AI) ‘broadly refers to the idea of computers that can learn and make decisions in a human-like way’.² Chatbots have been used in recent years by many companies, including those in healthcare, for providing customer service, routing requests, or gathering information.

ChatGPT is a recently-released chatbot that ‘is an example of generative AI, because it can create something completely new that has never existed before,’³ in the sense that it can use existing information organized in new ways. ChatGPT has many potential uses, including ‘summarising long articles, for example, or producing a first draft of a presentation that can then be tweaked’.⁴ It may help researchers, students and educators generate ideas,⁵ and even write essays of a reasonable quality on a particular topic.⁶ Universities are having to revamp how they teach as a result.⁷

ChatGPT has many limitations, as recognized by its own creators: ‘ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers…Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended…While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behaviour.’⁸ And, ‘[u]nlike Google, ChatGPT doesn’t crawl the web for information on current events, and its knowledge is restricted to things it learned before 2021, making some of its answers feel stale.’⁹ OpenAI is currently working on an improved version that is ‘better at generating text than previous versions’ and several other companies are creating their own ‘generative AI tools’.⁷

Chatbots are ‘trained’ using libraries of existing texts. Consequently, in response to specific input from the human operator (a ‘question’ or ‘seed text’), chatbots respond with an ‘answer’ or other output. Ultimately, this output comprises a selection of the training materials adapted according to the algorithms. Since chatbots are not conscious,¹⁰ they can only repeat and rearrange existing material. No new thought goes into their statements: they can only be original by accident. Since chatbots draw on the library of existing texts on which they were trained, there is a risk that they might repeat them verbatim in some circumstances, without revealing their source. According to a recent preprint that used ChatGPT to generate text, ‘The percentage of correct references in the preliminary text, obtained directly from ChatGPT, was just 6%.’¹¹ Thus, if chatbot output is to be published in an academic journal, to avoid plagiarism, the human author and editor must ensure that the text includes full correct references, to exactly the same degree as is required of human authors.

More alarmingly, ChatGPT may actually be capable of lying intentionally—‘the intentionality is important, as the liar knows the statement they are making is false but does it anyway to fulfill some purpose…’ as demonstrated by Davis.¹² Of course, ChatGPT is not sentient and does not ‘know’ it is lying, but its programming enables it to fabricate ‘facts’.

Chatbots are not legal entities, and do not have a legal personality. One cannot sue, arraign in court, or punish a chatbot in any way. The terms of use and accepted responsibilities for the results of using the software are set out in the licence documentation issued by the company making the software available. Such documentation is similar to that produced for other writing tools, such as Word, PowerPoint, etc. Just as Microsoft accepts no responsibility for whatever one writes with Word, ChatGPT’s creator OpenAI accepts no responsibility for any text produced using their product: their terms of use include indemnity, disclaimers and limitations of liability.¹³ Only ChatGPT’s users would be potentially liable for any errors it makes. Thus, listing ChatGPT as an author, which is already happening^14,15 and even being encouraged,¹⁶ may be misdirected and not legally defensible.

While ChatGPT may prove to be a useful tool for researchers, it represents a threat for scholarly journals because ChatGPT-generated articles may introduce false or plagiarized content into the published literature. Peer review may not detect ChatGPT-generated content: researchers can have a difficult time distinguishing ChatGPT-generated abstracts from those written by authors.¹⁷ Those most knowledgeable about the tool are wary: a large AI conference banned the use of ChatGPT and other AI language tools for conference papers.¹⁷

Looked at in another way, chatbots help produce fraudulent papers; such an act goes against the very philosophy of science. It may be argued that the use of chatbots resembles papermills albeit with a small difference—though the latter clearly have an intention to deceive, this may not always be true for the use of chatbots. However, the mere fact that AI is capable of helping generate erroneous ideas makes it unscientific and unreliable, and hence should have editors worried.

On a related note, the year 2022 also saw the release of DALL-E 2,¹⁸ another ML-based system that can create realistic images and art from a description submitted to it as natural language text, by OpenAI, the same company that has made ChatGPT. More recently, Google has also released a similar product named Imagen.¹⁹ These tools too have raised concerns somewhat similar to those with ChatGPT. Interestingly, each image generated using DALL-E 2 includes a signature in the lower right corner, to indicate the image’s provenance;²⁰ however, it can be easily removed using one of several simple methods that are a web search away.

With the advent of ChatGPT and DALL-E 2, and with more tools on the anvil, editors need to establish journal policies on the use of such technology and require the tools to be able to detect content it generates. Scholarly publishing guidelines for authors should be developed with input from diverse groups including researchers whose first language is not English. This may take some time. In the meantime, we offer the following recommendations for editors and authors.