In AI We Trust?... Comparing Human Versus AI-Generated Scores for Academic IELTS Writing Task 1 Samples

Feedback is the breakfast of champions (Blanchard, 2009). This is quintessential in educational contexts, especially in English high-stakes testing. For example, Academic IELTS Writing Task 1, based on visual input that later constitutes a report (Lindeck et al., 2011), has four scoring criteria: task achievement, coherence and cohesion, lexical resource, and grammatical range and accuracy (IELTSTM, 2023a, 2023b). Understanding these elements and additional writing task particularities determines better test performance. However, prospective testees still struggle with this understanding. Personalized score feedback might target this situation, but it is time-consuming (Tsai, 2012) and subjective (Zhao et al., 2023). Moreover, many prospective testees lack access to feedback due to the restrictive costs associated with high-stakes tests (Balseiro-Barrios, 2022). Artificial Intelligence (AI) might prove useful against these issues (Zhao et al., 2023), even though it must still be finetuned on grading requirements (Ramesh & Sanampudi, 2022). Therefore, this study aimed to compare human versus AI-generated scores for Academic IELTS Writing Task 1 samples. Thirty essays were graded by two human raters as well as by three AI systems (ChatGPT, Gemini, and Claude), considering a modified checklist (TheIELTSCoach.com, 2014), expanded under IELTS scoring writing criteria (IELTSTM, 2023b). Interrater reliability was assessed with intraclass correlation (Koo & Li, 2016), while total scores and specific criteria grades were evaluated using one-way ANOVA (Grande, 2013) and one-way MANOVA (Grande, 2016a), respectively. Both human and AI-generated scores were moderately consistent, while there were no significant differences for general scores, with only a few specific differences in scoring criteria elements. While human scores were closer to the known essay grades, AI-generated results might prove advantageous due to their immediacy, helping tutors provide better writing insights and testees become more independent in their high-stakes test journey.