Are AI chatbots like ChatGPT aiding the learning process, or are they hindering the development of critical skills in future generations? A recent study from the University of Pennsylvania suggests that while AI tools have potential, they may not be as beneficial as hoped—at least not yet.
The study, which involved nearly 1,000 high school students in Turkey, found that students who practiced math problems using ChatGPT performed 17% worse on a subsequent test compared to those who practiced without the AI tool. Though ChatGPT users were initially more successful in solving practice problems, they faltered when it came time to apply their knowledge independently.
Study Highlights: The Experiment
Conducted in the fall of 2023, the study followed students from grades 9 to 11 through four rounds of math instruction. After each lesson, students were randomly assigned to one of three groups for a 90-minute practice session. The first group had to practice independently, the second could use ChatGPT freely, and the third was given a tutor version of ChatGPT, designed to provide hints rather than direct answers.
When tested afterward without any AI assistance, the group that practiced using ChatGPT performed significantly worse than their peers who studied without AI, despite solving 48% of practice problems correctly. This discrepancy suggests that ChatGPT’s assistance may inflate students’ confidence without building a solid understanding of the material.
Interestingly, students using the tutor version of ChatGPT performed 127% better during practice but scored no better than those without AI assistance in the test. This points to a troubling trend: students overestimated their abilities, believing the AI’s help had prepared them more than it actually did.
The AI Shortfall: Why Students Struggled
The core issue, according to the study, is that students often use ChatGPT as a shortcut rather than a learning tool. They focus on getting the right answer rather than understanding the underlying solution, which prevents the development of critical thinking and problem-solving skills.
Compounding the problem, ChatGPT itself struggled with math, delivering correct solutions only about half the time. In 42% of cases where a step-by-step solution was required, the chatbot got it wrong. Even with basic arithmetic, ChatGPT failed 8% of the time, calling into question its reliability in a learning environment. The tutor version performed better because it was pre-programmed with correct answers, but this, in turn, limited its effectiveness as a true learning aid.
Lessons for Educators and Parents
While this study is an early investigation and far from conclusive, it does raise valid concerns for educators and parents alike. The results suggest that, when not used thoughtfully, AI tools like ChatGPT could be counterproductive in educational settings. The ease with which students can access answers may encourage reliance on the tool, stunting their intellectual growth rather than fostering it.
However, this doesn’t mean AI tools have no place in education. With proper guidance and structured use, they could still offer valuable support in acquiring knowledge. The challenge lies in finding the right balance—using ChatGPT and similar tools not as shortcuts, but as aids to deepen understanding and encourage critical thinking.
For now, educators might need to rethink how they integrate AI into the classroom, ensuring that it complements, rather than replaces, traditional learning methods. The long-term potential of AI in education is vast, but this study is a clear reminder that, at least for now, it should be approached with caution.