Do ChatGPT-Style AI ChatBots Help Students Learn? Yes, But There Are Caveats, Says Research
ChatGPT and other AI chatbots have become almost synonymous with conversations about education and edtech tools. But there’s an important question that so far remains unanswered: Do AI chatbots help students learn?
“Many empirical studies have been devoted to exploring the effects of AI chatbots on students’ learning outcomes. However, the findings from these studies were inconsistent,” says Yu Zhonggen, a professor of Foreign Studies at the Beijing Language and Culture University.
Some studies comparing students who learned with ChatGPT or other AI chatbots to students who didn't found a positive impact while others saw no effect. To try to understand these different findings, Zhonggen and colleagues researched all existing empirical studies — studies based on observed and measured phenomena — that looked at AI chatbots in education interventions from K-12 to college.
Ultimately, they found 24 appropriate studies and pooled the results together to answer a number of questions about ChatGPT-style chatbots and the impact on teaching and learning. The team’s findings were published in the British Journal of Educational Technology.
“Overall, the study found that AI chatbots had a significant positive effect on students’ learning outcomes,” Zhonggen says. “Specifically, AI chatbots were found to enhance students’ learning motivation, performance, self-efficacy, interests, and perceived value of learning. Additionally, AI chatbots could be helpful in alleviating students’ anxiety.”
However, there were limitations. These positive impacts were only seen in college students, with younger grade levels not seeing statistically significant improvement. Also, the longer a ChatGPT intervention went, the less effective it was.
Why Were AI Chatbots More Impactful for College Students?
Compared to other age groups, primary school students may not always have an effective interaction with AI chatbots due to lower language competency, self-directed learning ability, and digital literacy, says Zhonggen.
“They relied more on teachers’ guidance, which made it difficult for them to fully engage with learning through AI chatbots,” he says. “Secondary school students received more academic pressure from their parents and schools for good exam results. High academic pressure decreased their motivation to use AI chatbots for learning. By contrast, university students may be more proactive and capable of self-regulating their learning.”
Why Did The Impact Change Over Time?
The first time a student uses ChatGPT or a similar AI chatbot, there is a wow factor that seems to impact learning, except this novelty effect might wear off over time, rendering ChatGPT-style chatbots less effective.
“In the short interventions, students were excited and intrigued by the use of AI chatbots, which temporarily increased their learning interest, motivation, and performance,” Zhonggen says. “Once the novelty effect of AI chatbots has worn off, it would probably be difficult for students to sustain learning interests, have motivation, and perform well in chatbot-based learning.”
Other potential barriers to the sustained use of ChatGPT or similar tools exist. “Students were more likely to experience information overload when using AI chatbots for a long time,” Zhonggen says. “This may distract them from chatbot-supported learning, leading to poor learning performance.”
Future Research Implications
More research has to be done to identify the most effective interventions using AI chatbots as well as exactly why the tools may not be successful with younger students and why efficacy decreases over time. However, Zhonggen says the evidence is mounting in support of ChatGPT’s efficacy, particularly in higher education.
“When we wrote this paper, few empirical studies had explored the effect of ChatGPT on students’ learning outcomes,” Zhonggen says. Since he and his co-authors performed their literature review, there have been several new studies suggesting a benefit for ChatGPT with college students. “For example, through using ChatGPT for seven weeks, [one study found] there were significant improvements in university students’ self-efficacy, thinking skills, and motivation.”
As always, more research is needed, Zhonggen says. “We still call for more scholars to pay attention to the role of ChatGPT in different subject disciplines.”
