Enhancing students’ learning experience in mathematics class through ChatGPT

ABSTRACT


INTRODUCTION
ChatGPT is the most recent, and at the time of writing, the most advanced chatbot developed by OpenAI.It has been designed to mimic human-like dialogue and engage with users naturally and intuitively.According to Rudolph et al. (2023) and our own search for literature, there are limited peer-review articles on the impacts of ChatGPT in higher education particularly within the domain of mathematics instruction (Alneyadi & Wardat, 2023;Shakarian et al., 2023;Wardat et al., 2023Wardat et al., , 2024)).Bonsu and Baffour-Koduah (2023) further note that there are few studies examining students' perceptions and usage of ChatGPT, primarily due to its novelty and newness.
One common concern when using ChatGPT in higher education is plagiarism (Dehouche, 2021;Khalil & Er, 2023;Mitchell, 2022).This concern regarding disruptive technology tools in education is not new especially when it comes to allowing students to use technology resources outside the classroom.However, with careful planning and clear guidance, students can greatly benefit from the available technology tools.Indeed, many studies have shown the effectiveness of using technology in mathematics classes to increase student achievement and engagement (Branch, 2015;Gouia-Zarrad & Gunn, 2016, 2017, 2018;Gouia-Zarrad et al., 2015;Higgins et al., 2019;Phoong et al., 2019).Despite the criticisms leveled by scholars, many studies indicate that there is a growing need for the acceptance of this technology (Hosseini et al., 2023;Hu, 2023;Yeo-The & Tang, 2021).
Another concern specific to the use and adoption of artificial intelligence (AI) technologies like ChatGPT is the possible reevaluation of the role of educators, as highlighted by Bozkurt (2023) and Firat (2023a).It is, therefore, crucial to strike a balance that capitalizes on the potential of AI while preserving the significance of interpersonal relationships and human interaction in the educational process, as mentioned by Javaid et al. (2023).

Supporting Student Engagement Using ChatGPT
ChatGPT offers the educational community a valuable tool that enhances student engagement, fosters critical thinking, and provides personalized assistance (Firat, 2023a;Shoufan, 2023).Previous studies have shown that students are motivated using ChatGPT.By leveraging the strengths of ChatGPT while recognizing its boundaries, educators can create more engaging and effective learning environments (Cotton et al., 2023).It allows students to practice and reinforce their understanding of the material, encouraging active participation and deeper engagement with the concepts.Furthermore, ChatGPT can function as a virtual tutor or a personal mentor, providing instant and personalized feedback to students in real-time (Gao, 2021;Javaid et al., 2023;WGMI Media, 2023).Stojanov (2023) stated that ChatGPT is "probably more efficient than it would have been with a human tutor because I would not have felt as free to ask the same questions and would have been concerned with reputation management".It can also act as a friend, guide, or philosopher (Chatterjee & Dethlefs, 2023).By facilitating collaboration and allowing students to post questions and engage in discussions, ChatGPT assists students in their learning process (Li & Xing, 2021).It addresses their prompts and provides explanations in a conversational manner, supporting students' comprehension and helping them grasp complex concepts.In addition, it can help students in writing tasks, such as generating ideas, writing essays (Bailey, 2020), and breaking writer's block (Duval et al., 2020).

Current Limitations of ChatGPT
While ChatGPT has demonstrated remarkable language generation capabilities, it is important to understand its limitations (Oviedo-Trespalacios et al., 2023;Stojanov, 2023).It operates based on patterns and examples it has been trained on without true understanding or consciousness (Hintze, 2023).It does not have access to real-time information or personal experiences.Instead, ChatGPT relies on machine learning algorithms and computational processes that leverage statistical patterns and probabilities derived from the training data.Consequently, ChatGPT can produce incorrect or nonsensical responses, especially when asked ambiguous or out-of-context prompts.The presence of potential biases in the training data patterns can also lead to biased or inappropriate responses.Additionally, ChatGPT can generate responses that appear plausible but lack understanding of the context or not coherent.This phenomenon is commonly called hallucination (Alkaissi & McFarlane, 2023;Bang et al., 2023).The use of ChatGPT also raises ethical considerations, such as ensuring transparency, accountability, and safeguards against malicious use.Ongoing research and development in ChatGPT aim to tackle these challenges through continuous improvement of the ChatGPT model, improving response quality, and addressing concerns related to biases, hallucination, and ethical implications.
Although there are some concerns about the use of ChatGPT in education and particularly in higher education, this paper aims to demonstrate its usefulness to assist students through a supervised activity carefully designed to enhance the students' learning opportunities in solving differential equations (DEs) numerically in an undergraduate mathematics class.

Research Questions
Understanding students' perspectives is crucial for educators to enhance the learning experience and better meet their needs.According to Freiberg (1999), students' perceptions are an important component in assessing the effectiveness of innovative tools and methods.Positive perceptions of the learning experience "influence student engagement and motivation, while negative perceptions can lead to disengagement and hinder academic success" (Jones & Carter, 2019).Therefore, it is essential to consider the students' views regarding the use of ChatGPT and the benefits it offers them.Although there are limited studies available on students' perceptions of using ChatGPT in higher education (Bonsu & Baffour-Koduah, 2023), we have found no previous studies examining students' perceptions of using ChatGPT in numerical implementation.In addition, research on ChatGPT has not been done in ... Therefore, exploring students' perceptions of the use of ChatGPT in … is highly important.To fill this research gap, this exploratory study focuses firstly on investigating students' perceptions of the effectiveness of ChatGPT in a specific course.By actively involving students, and including open-ended questions, the initial research question shown below was extended to include students' overall experience and their views of the advantages and disadvantages of utilizing ChatGPT as a learning tool.Three research questions are addressed: 1. How do students perceive the effectiveness of using ChatGPT in helping them to solve DE numerically?2. What advantages, if any, do students report for the use of ChatGPT as a learning tool?
3. What disadvantages, if any, do students report for the use of ChatGPT as a learning tool?
The paper is structured, as follows: Next section provides an explanation of the study's background and outlines the methods used.After that, we present a summary of the obtained results.Finally, we conclude the paper by discussing its limitations and offering some remarks.

METHODOLOGY Background to Study
DEs course (MATH 341) is a fundamental course within the curriculum that builds the mathematical foundations for engineering and other scientific fields.This course covers solving DEs and modeling real-world phenomena.It is a mandatory and essential course for engineering students.The lead author taught five sections of this course with a total enrollment of 90 mixedgender students.
DE course consists of two components.The first component is the analytical approach, which focuses on solving DE using algebraic techniques to find exact solutions in terms of elementary functions.The second component is the numerical approach, which focuses on solving DE using numerical techniques to obtain approximate solutions.While the analytical approach is typically the main component of our undergraduate introductory DE courses, the numerical approach holds greater relevance for engineering students.Numerical methods provide practical tools for solving DEs and analyzing real-world phenomena.Unfortunately, this component is often omitted or neglected due to time constraints and the inhomogeneity of students' coding skills.Since students come from diverse educational backgrounds, it is challenging to teach students who have various levels of coding experience.To address this challenge, we integrated ChatGPT into the course through project-based learning.
The lead author was teaching MATH 341, DEs for engineers during the spring, 2023 semester.From the beginning of the semester, students were informed that the course has a project component at the end of the semester to practice the learned concepts and provide a great opportunity for sharing and discuss.20% of the course grade is determined by performance on that project.

Choosing a project idea
During the initial phase, students actively engaged in selecting project ideas during class sessions, focusing on real-world phenomena that could be modeled with DEs within their areas of interest.With the assistance of ChatGPT, they explored a diverse range of potential topics, ranging from modeling population dynamics to examining neuron behavior or the spread of COVID-19.After deliberation, students selected a topic subject to instructor approval.This approach not only deepened their understanding of course concepts but also encouraged discussions.ChatGPT facilitated collaborative exploration, inspiring creative thinking as students engaged with various DEs applications.This strategy empowers students by giving them control over their learning journey while the teacher serves as a facilitator addressing inquiries and clarifying concepts as necessary.Additionally, we checked if the projects were suitable, especially if they were too complex for a first year DEs course.

Conducting research & gathering data
After approval of their project proposal, students conducted research and gathered relevant data to understand their project.This phase involved studying literature, collecting empirical data, and utilizing resources such as ChatGPT and online databases.

Implementing numerical methods & drawing conclusions
Using the gathered data, students implemented numerical methods such as Euler's method or Runge-Kutta method, depending on their coding proficiency, to solve the equations relevant to their chosen project.This step involved coding and simulation to obtain numerical solutions.Upon obtaining numerical solutions, students analyzed the results to interpret the implications for the real-world phenomena being modeled.They drew conclusions based on their analysis, discussing the significance of their findings and any limitations encountered during the process.Students were free to use any software, preferably Python or MATLAB.To facilitate their learning and stimulate their enthusiasm, students were allowed to use ChatGPT, which they could use to get codes and learn how to modify and adapt them to their individual needs.In fact, students received code assistance, enabling personalized and self-paced learning experiences as they were able to learn at their own pace and in their own way, which made the learning experience more effective and engaging.The students were no longer sitting passively.They were asking questions through an active engagement with ChatGPT, receiving instant feedback, and deepening their understanding of the material and the application of numerical methods.They become actively involved in the learning process and responsible for their learning through chatting with ChatGPT.In addition, they gained knowledge through these online discussions.This interactive and learner-centered approach not only enhances their understanding of numerical methods but also caters to the learning preferences of Generation Z, who are enthusiastic about leveraging and adopting innovative technologies.As mentioned by Shin and Lee (2021), "Generation Zs are the first-generation born and raised in the digital age".They quickly learn these new technological innovations and are excited to adapt and use them.

Class presentation & peer feedback
In the final phase, students presented their projects to the class, receiving constructive feedback from their peers.This interactive process of knowledge exchange and collaborative discussion enhanced comprehension and empowered students to take ownership of their learning.Each presentation, lasting approximately 10 minutes, typically followed this format: describing the real-world phenomena being modeled, deriving DE or system of DEs, solving DE numerically, and explaining the implications/limitations of the solutions for the modeled physical situation.
Presentations were assessed according to the following criteria: 1. Context: Did the presentation provide background information and clarify the problem's context and underlying assumptions?
2. Mathematics: Was the application of course methods clearly explained?While not exhaustive, did the presentation outline the key elements of mathematical theory?

Clarity & novelty:
Were all new terms and symbols defined for clarity?Did the presentation offer innovative insights?
4. Understanding: Did the presenter demonstrate a thorough understanding of the model's assumptions?Was the code effectively communicated and explained?

Data Collection
The study was conducted in Spring 2023 with five undergraduate mathematics classes.The research data was collected through student surveys designed by the authors.The surveys had a combination of yes/no questions and open-ended questions to collect both quantitative and qualitative data.To answer the research questions, the students were asked to fill out a short survey at the end of the course.27 male students and 21 female students filled out the survey.Data to answer the first research question, "How do students perceive the effectiveness of ChatGPT in helping them to solve DE numerically?" was collected through the first five close-ended questions.Data to answer the second and third research questions regarding the advantages and disadvantages of ChatGPT as a learning tool was collected through question 6 of the close-ended questions and through the open-ended question.

RESULTS
An analysis of the close-ended questions revealed a mostly positive view of the effectiveness of ChatGPT in helping them to solve DE numerically.From the students' responses, many students reported a positive user experience with ChatGPT and found it to be helpful in improving coding skills (91% agreement, see Figure 1) and understanding numerical solutions of DEs (72% agreement, see Figure 2).These findings are consistent with the existing literature, where students recognized that ChatGPT is useful not only for learning and study but also for professional applications (Shoufan, 2023).Other studies have also demonstrated that "chatbots can improve student performance and knowledge retention" (Alotaibi et al., 2020).
Additionally, students acknowledged that ChatGPT provided "a comprehensive understanding of complex topics in an easily understood manner" (Bonsu & Baffour-Koduah, 2023).This can be attributed to the integration of ChatGPT, which stimulates students' learning enthusiasm, makes students actively participate in class, and consequently deepens the understanding of numerical methods for solving DE (Gao, 2020).
As noted in Figure 3, 70% of the students felt that ChatGPT helped them solve problems or overcome challenges in the course, thus, further demonstrating that there are additional advantages to using ChatGPT.Furthermore, a substantial majority of students (80%) would recommend the use of ChatGPT to future students taking the course (see Figure 4).This strong recommendation can be explained by the transformative nature of the personalized and selfpaced learning experience facilitated by ChatGPT.Students are no longer dependent on traditional learning that involves seeking assistance from the professor, they receive immediate feedback according to their coding proficiency level.This aligns with Firat's (2023b) assertion that "students can more easily adapt their learning to their present level of understanding-and without being shy about a machine".These findings support the idea of the effectiveness of ChatGPT in promoting personalized learning and empowering students to take control of their educational journeys.
The next findings of this study are on students' perceptions of using ChatGPT to generate ideas.The results indicate that 65% of the students agreed that ChatGPT leads to new ideas or perspectives on modeling DEs or numerical methods (see Figure 5).These findings align with recent research, such as Dowling and Lucey (2023) that recently released that ChatGPT can significantly assist with idea generation.Additionally, AlAfnan et al. (2023), where conducting 30 tests on both theory-based and applicationbased tasks demonstrated ChatGPT's potential to replace search engines by consistently providing accurate and reliable information to students.As one student stated, "okay, so it's basically like searching Google, but this simplifies the information you need to the precise thing you want.It just makes things easy for me" (Bonsu & Baffour-Koduah, 2023).This supports the idea of the effectiveness of ChatGPT in promoting personalized learning.Furthermore, Rudolph et al. (2023) prompted a 'code red' due to ChatGPT potentially disrupting the dominance of Google search, with Microsoft already incorporating OpenAI technology to enhance its search engine, Bing (Tung, 2023).

Open-Ended Question
After completing the yes/no questions, the students were asked to respond to the following open-ended question ''is there anything else you'd like to share about your experience using "ChatGPT" in this course?''An analysis of the responses revealed seven themes related to the advantages and the disadvantages of using ChatGPT as a learning tool in a more general context than the previous questions.
Table 1 presents the themes found in the student's responses.The students' comments have not been edited.
Based on the analysis of student comments this study identified several key themes related to the use of ChatGPT in higher education.Four positive themes emerged, highlighting the advantages of ChatGPT in enhancing learning, serving as an effective learning tool, enabling personalized learning experiences, and offering practicality and ease of use.Students reported that ChatGPT helped them understand libraries, familiarize themselves with numerical methods, and generate new ideas.They perceive ChatGPT as an effective learning tool, providing detailed explanations and support in various educational tasks such as programming and writing.The personalized nature of ChatGPT was also highlighted, as it offered specific and relevant content tailored to individual needs, aiding students in overcoming challenges, particularly in writing.Furthermore, students appreciated its practicality and user-friendly interface, particularly in programming tasks.
However, the analysis also identified three negative themes concerning the use of ChatGPT in higher education.These themes emerged around limitations in math, course applicability, and concerns regarding limited reliability and coding mistakes.Students acknowledged the potential limitations of ChatGPT, particularly in terms of providing inaccurate information or experiencing hallucinations.Shoufan (2023) suggests that "ChatGPT, at least at the current time, should not be relied on as a sole resource for learning by students who do not have sufficient prior knowledge".The findings also indicated that students recognized ChatGPT to be more suitable for software courses rather than mathematics courses.This observation calls for further investigation and improvement to ensure the effective usage of ChatGPT in higher education, especially in mathematics courses.
It is noteworthy that students expressed concerns regarding the mathematical capabilities of ChatGPT, as evident in previous studies conducted by Azaria (2022), Davis (2023), Gao (2020), andFrieder et al. (2024).These concerns emphasize the importance of addressing the specific challenges and limitations related to ChatGPT's performance in mathematical contexts, thereby facilitating its effective integration in mathematics courses.
As seen in the qualitative data, the advantages of using ChatGPT outweigh the disadvantages.For the students in this study, they could see the value, as well as the limitations of using ChatGPT.When asked if they would be interested in using ChatGPT in future mathematics or engineering courses, 85% said they would (see Figure 6).This result, along with the thematic analysis supports the integration of ChatGPT in education.ChatGPT presents valuable opportunities to enhance students' learning experiences and empower students to take ownership of their learning, which supports the theme of personalized and customized learning, as discussed earlier in the analysis.

Limitations in mathematics
I would recommend in only implementing code but not for understanding mathematics concepts.
Yes, but you should understand code & manipulate it on your own.I would use it for coding, but in mathematics it has a lot of wrong answers so I would not recommend using it in mathematics.

Course applicability
Only software courses.Would not recommend using it in mathematics.Mathematics.I do not think so but other courses as coding course it's helpful I will use it.
I think it is beneficial for writing & it needs to be improved in engineering field.

CONCLUSIONS
In conclusion, this study has illustrated that the integration of ChatGPT in mathematics classes provides valuable opportunities to enhance students' learning and offer personalized learning.While students recognized the advantages of learning how to code and understand numerical solutions, they also recognized the potential limitations of ChatGPT in terms of providing inaccurate information or experiencing hallucinations.Higher education institutions may establish an effective learning environment that empowers students and enhances educational outcomes in mathematics education by addressing these issues and finding a balance between AI technology and human engagement.Moreover, beyond enhancing DEs understanding, students gained proficiency and confidence in utilizing AI-based applications that will constitute eventually an essential part of future engineering tool sets.
The findings can be generalized to diverse educational contexts and academic levels, enhancing their relevance and applicability.For instance, integrating ChatGPT into the learning process enables educators to implement project-based learning methodologies across different courses and educational levels.While project-based learning is renowned for enhancing student motivation and retention rates (Neji et al., 2023;Vesikivi et al., 2020), its adoption in first-year university courses often encounters challenges like diverse skill levels, time constraints, and the necessity for foundational concepts.By integrating ChatGPT into the learning process, educators can effectively overcome these obstacles and successfully incorporate project-based learning into their courses.ChatGPT's adaptive capabilities empower students to quickly grasp essential concepts and delve into complex projects, ensuring the effectiveness of project-based methods across a variety of educational settings and levels.
The relatively small sample size of the current study is one of its limitations.Although there was a satisfactory gender split among engineering students, a larger sample size would have allowed for more reliable data and quantitative analysis.Additionally, relying solely on self-reported data presents another limitation.The conclusions are based on participant views, which may be biased or inaccurate.Data could be inconsistent because of differences in how students interpret their ChatGPT experiences.Additionally, self-report metrics might not fully reflect ChatGPT's influence.Future research might consider adding observations or interviews to provide a more thorough analysis of ChatGPT's influence on mathematics education.
Finally, this study has shown that by leveraging the strengths of ChatGPT and addressing its limitations, higher education institutions can create an effective learning environment that encourages active student engagement, critical thinking, and creativity, ultimately leading to enhanced learning outcomes.Branch, L. J. (2015).The impact of project-based learning and technology on student achievement in mathematics.In Proceedings of the New Media, Knowledge Practices and Multiliteracies: HKAECT 2014 International Conference (pp. 259-268)

Figure 4 .Figure 5 .
Figure 4. Students' overall recommendation to continue using ChatGPT (Source: Authors' own elaboration) on ChatGPT as primary source, use it to open more research ideas.It helps but it also makes coding mistakes & has inaccurate explanations.Not always has correct answers (not that reliable).No, ChatGPT tends to be very misleading.It looks at problems from one perspective & I do not trust it with scientific content.Not always has correct answers (not that reliable).

Figure 6 .
Figure 6.Students' interest in using ChatGPT in future mathematics or engineering courses (Source: Authors' own elaboration)

Table 1 .
Thematic analysis Helped me understand libraries & their features.Yes, it made me familiarized with different numerical methods for solving differential equations.It did help me understand my subjects more, & it did help me a bit with programming.Absolutely I did find many ideas compared to my research on Google.ChatGPT as a learning tool is amazing!You just input your code & ask it about each part, very well explained.Yes, it gives me detailed explanation on implementation.Yes, ChatGPT provided me with a detailed explanation, which helped me understand DE faster.Yes, honestly ChatGPT helped me understand numerical implementation of DE when I was stuck.Yes, a bit, it only answered some of needed questions & made it easier to do MATLAB work since no one knows how to use it.Personalized learning It provides specific content related to given prompt, giving easier access to information in comparison to googling.It's amazing ability to comprehend what user is asking & provide an answer that most of time will be exactly what user needs.It provides you with various ideas & helps you start your essay writing, helping get rid of what we call 'blank page syndrome'.ChatGPT is useful for asking questions without having correct vocabulary or technical terms usually required to find answers on search engines like Google.Anyone could ask a question like they're talking to another human being & ChatGPT would usually be able to answer relatively correctly.