ChatGPT Presence in Academic Writing: Detecting AI-Generated Text in Undergraduate and Graduate Students’ Research Proposal Literature Reviews

With the rapid development of artificial intelligence (AI) tools, concerns emerge regarding students’ unethical uses of these tools to produce AI-generated research texts or their parts, and to present them as original writing. This issue is compounded by the lack of reliable tools for detecting machine-generated text. To address these concerns, the present study aimed to identify distinctive features of ChatGPT-generated research proposal literature reviews ( N = 45) and investigate the presence of these features in English-language literature reviews produced by undergraduate and graduate students from two Russian universities. During the first stage, an analysis of AI-generated texts and a small sample of graduate students’ ( N = 12) literature reviews was conducted. Findings revealed that many features typical of AI-generated texts were clearly present in student texts suggesting that these features may serve as indicators of machine-generated writing. One such feature was the unusually high recurrence of lexical items (predominantly with abstract meanings) in both AI-generated and student texts. Drawing on these insights, a frequency analysis was performed using AntConc to explore the occurrence of these items in AI-generated texts and compile a list of the most frequent items indicative of machine-generated writing (referred to in this study as “ChatGPT language”). At the second stage, findings on the initial indicators were validated, refined, and expanded based on an analysis of a larger sample of 47 English language literature reviews prepared by bachelor and master students. The study identified ten indicators of AI-generated writing pertaining to content, structure, and language use in literature reviews, which are detailed and illustrated in the paper. The study’s findings contribute valuable practical and research insights which may aid all those involved in teaching English language academic writing, reviewing students’ academic texts, and supervising research projects across diverse EAP contexts.