AI or Human? Linguistic Features in AI-Generated Versus Human-Written Texts

In recent years, the widespread usefulness of chat-based artificial intelligence has made the web-based Generative Pre-Trained Transformer (GPT) application popular among learners. The current methods for distinguishing AI-generated and paraphrased texts on various AI detector websites show limited effectiveness because of the rapid development of AI models. As a result, teachers might incorrectly assume that students’ original work was generated by AI. Therefore, this study aims to discover linguistic features used in AI-generated texts (AT) by utilizing the Tool for the Automatic Analysis of Cohesion (TAACO) and the Tool for the Automatic Analysis of Lexical Sophistication (TAALES) to identify distinctive linguistic features in AT compared with human-written texts (HT).

To distinguish the difference between AT and HT, we identified 77 topics from the EF-Cambridge Open Language Database (EFCamDat), categorized into 63 Image-with-Text topics and 14 Text-only topics. This study used students’ responses from the database across CEFR A1 to B2 as HT. The same topics were used to generate responses by the mini variant of GPT-4 Omni for comparison. An equal number of AT and HT were collected for each topic, resulting in a balanced dataset of 1000 texts per topic.

The cohesion analysis using TAACO revealed several distinctions between AT and HT. It revealed that AT had a higher token-type ratio and lemma-level moving average. It also showed stronger paragraph-level alignment and lower repetition rates than HT. However, HT exhibited stronger sentence-level overlap. Additionally, the lexical sophistication analysis using TAALES indicated that AT had higher vocabulary complexity and academic word usage than HT. It also demonstrated slightly more unique-word usage and a more formal tone than HT. In contrast, humans tended to use more straightforward and concrete vocabulary. In summary, the results uncovered linguistic differences between AT and HT, providing insight into pedagogical implications.