Empowering Students with AI: Using ChatGPT as a Writing Tool and Teaching Critical Evaluation of AI-Generated Text

Abstract

ChatGPT has the potential to enhance language skills for students studying English as a foreign language. The current study is an exploratory study of how students intuitively use the freely available version of ChatGPT and ChatGPT’s potential for use for feedback. In the first part of the study, we explored how undergraduate students intuitively use ChatGPT to produce an expository essay with a focus on the prompts they used and how they evaluated ChatGPT’s output. We found that novice users tended to use prompts that were vague, e.g., “Write an essay on …”, but more savvy users gave more specific prompts, e.g., “[…] Keep track of clarity and coherence of your arguments. In particular, the sub-themes you should address in the essay are […]”. However, none of our participants rated ChatGPT’s output highly, commenting on under-developed arguments, the lack of examples, and irrelevant material, though the overall essay structure and language use were good. Some of this may be due to poor prompt quality, but some of the issues may be inherent to ChatGPT’s limitations. In the second part of the study, we asked another group of undergraduate students to use ChatGPT to provide feedback on a student-produced argumentative essay and we compared that feedback to peer feedback and instructor feedback. Here, we found some overlap between all feedback, however ChatGPT focussed mostly on lower order concerns such as vocabulary, grammar and natural language use, but did not provide good feedback for higher order concerns such as quality of the arguments and paragraph structure. These results illuminate a need to mitigate common misconceptions of the extent to which generative language models can assist in teaching, producing, and evaluating texts. This research provides important insight into ChatGPT's potential in the classroom, informing instructors and students about its strengths and weaknesses.

Keywords

ChatGPT, AI, writing, classroom research