ChatGPT in the Classroom: Promising Performance, Moderate Perceptions, and a Need for Caution

Summary of Key Points

• 📊 The Study: A meta-analysis of 51 experimental studies on ChatGPT's impact in education, published between late 2022 and early 2025.
• 📈 Learning Performance: The analysis found a large positive effect (Hedges’s g = 0.867) of ChatGPT on student learning performance. This effect was strongest in skills-based courses, in problem-based learning models, and when used for a duration of 4–8 weeks. [1][2] (Hedges' g is a statistical measure used to quantify the effect size, specifically the standardized mean difference between two groups.)
• 🤔 Learning Perception & Higher-Order Thinking: The study reported moderately positive effects on both students' perception of learning (g = 0.456) and their development of higher-order thinking (g = 0.457). [1][2]
• 🔬 Important Caveats: The authors explicitly state that the sample sizes for the perception and higher-order thinking analyses were small (19 and 9 studies, respectively), which calls for a cautious interpretation of these specific findings. [2]
• 🧑‍🏫 Context Matters: The effectiveness of the tool is not uniform. It changes based on the course type, the teaching model, the duration of use, and the role assigned to the AI (e.g., tutor vs. partner). [1][2]

A Cautious Look at the Evidence on ChatGPT in Education

As educators and researchers, we are positioned directly on the front lines of a rapid technological shift. The integration of generative AI, specifically tools like ChatGPT, into our classrooms is no longer a future hypothetical; it is a present reality. The debate surrounding its utility, however, is often polarized, oscillating between utopian promises of personalized learning and dystopian fears of cognitive decline. Anecdotes abound, but robust evidence has been harder to come by.

A recent meta-analysis by Jin Wang and Wenxiang Fan, published in Humanities and Social Sciences Communications, attempts to bring quantitative clarity to this discussion. [3] The study synthesizes the findings from 51 (quasi) experimental studies to measure ChatGPT's effect on three key areas: student learning performance, learning perception, and higher-order thinking. [1] The headline findings suggest a positive impact across the board. However, as with any academic work, the details and limitations are where the most valuable insights lie. It is premature to declare this a settled matter. A skeptical, precise examination is required before we overhaul our pedagogical strategies based on these numbers. This post will analyze the study's three main findings and consider their practical implications with the necessary caution.

Part 1: The Large Impact on Learning Performance

The most striking result from the meta-analysis is the large positive effect ChatGPT appears to have on student learning performance, with an effect size of g = 0.867. [2] This figure suggests that, on average, students using ChatGPT in a structured learning environment perform significantly better on assessments than their peers in control groups. The analysis further breaks down this effect, revealing that context is critical. The tool's impact was most pronounced in courses focused on skills and competencies development, within a problem-based learning (PBL) framework, and during interventions lasting between four and eight weeks. [4]

This is not entirely surprising. Skills-based courses and PBL often involve tasks with clear procedures or problems with identifiable solutions. ChatGPT excels at providing structured information, explaining concepts, and generating step-by-step guidance. In this capacity, it acts as a powerful information retrieval and organization tool, which can directly translate to better performance on related assessments. The "sweet spot" of a 4–8 week duration is also an interesting, if not cautionary, finding. The authors speculate that shorter periods may be insufficient for students to develop effective prompting skills, while longer periods might lead to over-reliance and a neglect of knowledge reinforcement. [4]

This is a critical point of skepticism for any educator. Are we measuring durable learning or sophisticated dependency? The study quantifies performance, but it cannot fully capture the nature of that performance. If students become adept at using the tool to generate correct answers without internalizing the underlying concepts, the "large positive effect" might be a short-term illusion. The finding that the effect is weakest in project-based learning—a model that requires more open-ended and complex problem-solving—further supports this concern. [4] It suggests that while ChatGPT is an effective assistant for defined tasks, its utility diminishes as the ambiguity and complexity of the work increase.

Part 2: The Moderate, and More Ambiguous, Effects

When the analysis turns to learning perception and higher-order thinking, the results become more modest and, frankly, require a higher degree of scrutiny. The study reports a medium positive effect for both learning perception (g = 0.456) and higher-order thinking (g = 0.457). [2] While still positive, these figures are substantially lower than the effect on performance.

More importantly, the authors themselves flag a significant limitation: the number of studies available for this part of the analysis was small. The finding for learning perception is based on 19 studies, and for higher-order thinking, a mere nine. [2] With such a limited sample, these conclusions should be seen as preliminary at best. They indicate a potential trend, but they are far from definitive proof.

The paper suggests that ChatGPT's limited "emotional intelligence" may explain its moderate impact on learning perception; it can provide information but cannot easily foster the human connection that deepens engagement. [5] For higher-order thinking, the authors note that AI models, being trained on existing data, may lack the ability for genuine critical analysis or creative problem-solving. This aligns with the experience of many educators who find that students can use AI to produce a competent summary but struggle to use it for generating a truly original or critical argument. The finding that ChatGPT's impact on higher-order thinking was strongest when used as an "intelligent tutor" in STEM courses is notable. [4] This suggests that when used within a structured framework (like a tutorial) to solve complex problems (common in STEM), it can scaffold the thinking process. However, this is a very specific application and should not be generalized to all forms of higher-order cognition.

Part 3: Final Remarks and a Call for Measured Action

So, what are we to make of this? The Wang and Fan meta-analysis provides valuable quantitative evidence that, under specific conditions, ChatGPT can be an effective tool for improving student performance on defined tasks. [3] It offers a useful counter-narrative to the purely negative discourse surrounding AI in education. However, its findings on more nuanced outcomes like higher-order thinking are built on a thin evidence base and must be treated with extreme caution. [2]

The study does not suggest that handing students ChatGPT will automatically improve learning. On the contrary, its findings on the importance of moderators like course type, learning model, and duration reinforce a core principle of educational technology: the tool is only as effective as the pedagogy that surrounds it.

A Call to Action for Fellow Educators and Researchers:

1. Prioritize Pedagogical Design: Instead of simply "allowing" or "banning" AI, we should focus on designing learning activities where it can be used effectively. Based on this study, this might involve using it as a tutor in PBL contexts for STEM or as a tool for skills development, with clear scaffolds and guidance.
2. Maintain a Healthy Skepticism: We must continue to question what we are measuring. Is it learning or dependency? Are we fostering critical thinking or merely improving the polish on unoriginal work? We should be wary of impressive performance metrics that may mask a decline in underlying cognitive processes.
3. Demand and Conduct More Nuanced Research: This meta-analysis highlights significant gaps in the literature, particularly concerning higher-order thinking and long-term knowledge retention. We need more high-quality, longitudinal studies that move beyond simple performance metrics to examine the deeper cognitive and affective impacts of these powerful tools.

Ultimately, this study should not end the conversation; it should refine it. It moves us away from asking if AI helps, toward the more precise and productive question of how, when, and for what purpose it can support student learning. I would also recommend you check Flintk12 tool, that allows students under teachers' supervision to chat with AI in a manner that is safe and ethical.

---

References

(Note: The following reference is for the primary article analyzed in this blog post. The original article contains an extensive list of the 51 studies included in its meta-analysis for further reading.)

Wang, J., & Fan, W. (2025). The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis. Humanities and Social Sciences Communications, 12(1), 1-21. https://doi.org/10.1057/s41599-025-04787-y [3]