Investigating Chatbot-Assisted Self-Regulated Learning of Basic Electrical Engineering: Effectiveness and Variations of Students’ Learning Outcomes

1. Introduction

Educational chatbots are widely accepted tools for boosting student involvement in virtual learning environments. The research has received substantial attention because self-regulated learning can be developed among students through these methods (Guan et al., 2024). These conversational agents offer personalized, interactive experiences that can be particularly beneficial. In technical disciplines, basic electrical engineering often requires students to make multiple attempts to grasp complex concepts. The student will receive feedback that addresses their individual needs and aligns with their current level of understanding (Chang et al., 2023). AI-driven chatbots can perform the following functions. The implementation of Artificial Intelligence (AI) in educational settings leads to substantial improvements in student learning outcomes, engagement, and interest levels. The program can assist students interested in studying Science, Technology, Engineering, and Mathematics. Its ability to provide instant feedback and individualized guidance allows learners to take an active role in their educational path, developing a stronger connection to their learning process and a deeper understanding of the subject matter (Yin et al., 2025).

A personalized approach enables students to learn essential self-regulation techniques—goal setting, self-assessment, and metacognitive monitoring—that are vital to academic success. Lan and Zhou (2025) examine the factors that lead to success in higher education, stating that large language Model-based chatbots demonstrate substantial capability to improve various teaching methodologies, including flipped classrooms. The teacher can maintain active communication with students and promptly answer the students' questions (Vera, 2024). Nevertheless, technological progress can create various obstacles for society to overcome, and opportunities for educators, necessitating a re-evaluation of pedagogy. It is crucial to examine how effective the chatbot implementation is in driving students’ self-regulated learning, especially in basic electrical engineering, across schools of varying origins.

So far, there has been little research on students' learning outcomes across different types of schools and statuses when engaging in similar learning models. Electrical engineering education students in Indonesia come from various schools and backgrounds, possess different initial competencies, and have unique prior knowledge and learning preferences. Recognizing and understanding these differences, we develop a chatbot-assisted self-regulated learning to create learning environments that support diverse student needs and ensure fair learning opportunities. Research studies various participants to determine whether the methods lead to successful outcomes for specific student groups. According to Manzanares et al. (2023), students achieve better academic success through these strategies, which enhance their motivation and engagement. It became our foundation for evaluating the effectiveness of the learning model and students' achievement levels in chatbot-assisted self-regulated learning of basic electrical engineering.

The world has witnessed the debut of generative AI systems in recent times. Technologies such as ChatGPT further underscore the need to explore how University students can achieve independent learning with essential technological support. This exploration shows that the situation worsens because most undergraduate students currently use AI-based chatbots for skill development, knowledge acquisition, and motivation. Ma et al.'s (2024) research shows that these intelligent agents' ability to adapt to individual learning paces and styles makes them invaluable tools for fostering proactive learning habits (Ng et al., 2024). Therefore, this study examines how AI-powered chatbots support self-regulated learning of basic electrical engineering concepts and assesses their impact on students with diverse backgrounds.

In line with that, our research questions are: RQ1: Is the basic electrical engineering chatbot-assisted self-regulated learning environment effective? RQ2: Do students' diversities have an impact on the learning model? RQ3: What is the students' perception of the usefulness and usability of the multi-role chatbot in facilitating their learning of basic electrical engineering concepts?

2. Literature Review

2.1. Artificial Intelligence and Learning

The deployment of AI-based chat tools in educational settings leads to significant changes in students' interactions with materials and processes, requiring thorough evaluation against current educational theories (Bognár et al., 2024).  AI chatbots deliver learning support that aligns with Vygotsky's Zone of Proximal Development theory by providing appropriate assistance to students at their current learning level (Hauser et al., 2024). Scaffolding systems help students develop an understanding and acquire new skills by connecting their current abilities to their future potential under AI guidance (Darvishi et al., 2023). The framework of Self-Determination Theory enables the analysis of how AI chatbots impact intrinsic motivation, relatedness, competence, and autonomy, which drive self-regulated learning (Chiu, 2024).

Meeting these psychological requirements enables chatbots to create an educational environment in which students direct their learning journey and perceive AI assistance rather than authority (McGrath et al., 2024). Research supports the idea that interactive support systems with custom-configured AI chatbots reduce both internal and external mental workload and elicit positive emotional responses (Becker et al., 2025). AI chatbots generate customized educational content that keeps students focused while developing their problem-solving and innovative thinking skills (Wang & Xue, 2024; Xu et al., 2024). The systems provide instant feedback that adapts to each student's learning rate to achieve better educational outcomes (Chen, 2024).

Some research on the adoption of AI chatbot educational support requires identifying variables that influence students' acceptance of the technology (Roca et al., 2024). Evidence from a systematic review indicates that AI chatbots have become more common in educational environments, as advances in AI and natural language processing have expanded their capabilities (Lo et al., 2024). Generative AI chatbots have become more advanced, so scientists lack enough empirical studies that use proven learning theories to measure their effects on student learning and teaching methods. AI chatbots in education demonstrate the positive benefits of generative AI for personalized learning and student motivation (Wang & Li, 2024).

However, few studies provide empirical evidence of their use in electrical engineering applications. Further research is still needed to assess these tools using established pedagogical frameworks, including experiential, reflective, active, and self-regulated learning, as current studies do not. Information deficiency by studying the impact of a Self-Determination Theory-based multi-role chatbot on students' self-regulated learning behaviors and their basic electrical engineering academic performance is also crucial. It is necessary to examine, through scientific methods, whether GenAI systems with theoretical foundations lead to better educational outcomes than the initial positive user experiences with these tools.

2.2. Effectiveness of AI integration in learning

Academic professionals are increasingly interested in ChatGPT and similar generative AI chatbots, but further research is needed to establish their effects on student educational outcomes. Research indicates that AI tutoring systems and domain-specific chatbots yield positive learning outcomes (Gao et al., 2024; Liu et al., 2025). Nevertheless, as scientists increasingly study ChatGPT and similar generative AI chatbots, it is essential to conduct further studies to determine their effects on self-regulated learning in basic electrical engineering. The research addresses a knowledge gap by investigating how a multi-role chatbot system helps students develop self-regulated learning skills in basic electrical engineering, thereby achieving better academic results. The method follows the principles of Generative Learning Theory because it enables learners to build knowledge through active construction, a process that AI chatbots support through customized, interactive prompts, as Wu et al. (2024) said.

The educational generative AI architecture uses Retrieval-Augmented Generation to combine pre-trained language models with information retrieval systems, which produce answers through contextual understanding. The integration provides the chatbot with precise, up-to-date information, thereby improving instructional quality, according to Asghar et al. (2025).

2.3. Basic electrical engineering learning outcome through using AI-generated

Consequently, it is imperative to examine the impact of these resources on students' engagement, comprehension, and application of theoretical concepts within the electrical engineering curriculum. This study will also assess the efficacy of AI-powered conversational agents in enhancing deep learning and problem-solving skills, which are crucial for comprehending complex engineering principles (Caccavale, 2025). It entails assessing whether the chatbot's interactive features yield measurable improvements in students' analytical competencies and their ability to apply theoretical knowledge to practical circuit analysis and design challenges.

To gain a comprehensive understanding of electrical engineering, the chatbot must be able to interpret and generate responses from multimodal inputs, including circuit diagrams and graphical representations of electrical phenomena (Polverini & Gregorcic, 2024). Also, it is important to know how the chatbot's flexible learning paths can accommodate different learning styles and speeds, helping each student learn the basics of electrical engineering as quickly as possible. The students understand how integrating advanced machine learning and natural language processing algorithms into the chatbot can provide real-time, customized feedback, thereby reconciling the disparity between the acquisition of theoretical knowledge and the development of practical skills (Thway et al., 2024).

The personalized feedback is important for helping students learn and encouraging them to correct themselves, ultimately helping them better understand complex electrical engineering concepts. Furthermore, the chatbot's ability to foster adaptive expertise, enabling students to apply acquired problem-solving techniques to unfamiliar and ambiguous engineering contexts, is a competency frequently absent in conventional curricula. In electrical engineering, students often encounter circuit configurations unfamiliar to them, making it hard to recognize patterns (Ridgway & Cox, 2024). This ability to adapt and apply what they know is essential. The model was described in Figure 1.

3. Method

3.1. Learning model preparation

​‍​‌‍​‍‌ We created a learning model specifically for understanding basic electrical engineering by using large language models. We enable students to engage in deep, conversational interactions and use intelligent tutoring system features. The platform enables the development of complex AI-powered tools that deliver personalized learning materials and provide students with the feedback they need to encourage active learning and develop their critical thinking skills. We make the whole experience more user-friendly, so students can easily access learning materials and interact with teachers/staff on Microsoft Teams in a very comfortable, familiar way. The development process primarily involves building a detailed knowledge base from standard electrical engineering textbooks, research papers, and industry standards, ensuring the information provided is accurate and up to date.

3.2. Participants

We involved all students who took the Basic Electrical Engineering subject. There are six class groups (Group 1: 27 students; Group 2: 25 students; Group 3: 28 students; Group 4: 28 students; Group 5: 26 students; and Group 6: 25 students). Based on the census, the origin of school types is as follows: senior high school = 106 students; vocational high school = 53 students; public school = 96 students; and private school = 63 students. There are 109 males and 50 females (Table 1).

3.3. Research design

We assess the participants' initial ability before implementing the learning through a pre-test and administer a post-test on the learning outcomes at the end of the instruction (Figure 2). We measure learning effectiveness using the Normalized Gain Rules (Hake, 1998). Investigating the impacts of participants' diverse backgrounds on learning effectiveness. Furthermore, we trace the participants' attitudes and perceptions toward learning.

3.4. Instruments and Data Collection

We first developed all the instruments and then tested their validity and reliability. There are three instruments: an assessment sheet to validate the model, involving the teacher, students, and the expert; tests to collect initial data on basic electrical engineering ability and learning outcomes; and Questionnaires to assess participants' attitudes and perceptions toward learning.

The assessment sheet consists of 25 valid items with a reliability coefficient of .71. The test consists of 40 valid items, including 30 multiple-choice and 10 essay items, with a reliability coefficient of .70. The assessment score is 0-100, with 2 points for correct multiple-choice answers and 4 points for correct essay answers. Questionnaires consist of 50 valid items (Likert Scale, 1 = Strongly disagree; 2 = Disagree ;3 = Enough agree; 4 = Agree; 5 = Strongly agree) with a reliability coefficient of .72. Minimum score = 50, maximum 250.

To validate the learning model, data were collected and analyzed in two rounds: Round 1 involved 20 students, one lecturer, and one expert; and Round 2, 55 students, three lecturers, and two experts. A pre-test was used to measure the students’ initial abilities. Learning outcomes were collected after implementing the model for one semester, and students' perceptions of the learning model were also collected.

3.5. Data analysis

Quantitative data from pre- and post-intervention assessments were analyzed using Hake’s rule to determine learning effectiveness,

   (Hake, 1998)

Description:

g = Normalized gain

Post-test score = the score on the post-test

Pre-test score = the score on the pre-test

Maximum possible score = the maximum possible score on the test = 100

Measuring criteria of learning effectiveness based on Hake, namely High gain = g ≥ .7 (highly effective); Medium gain = .3 ≤ g < .7 (moderately effective); and Low gain = g < .3 (less effective), and additionally, analyzing student learning outcome gain based on the students’ school varieties by ANOVA at a .05 significance level. To describe students' attitudes and Perception trend levels regarding learning, descriptive statistics were used. The minimum ideal score is 50, and the maximum is 250. There are five categories of attitude and perception trend level determined: Very positive, 210 < Mean ≤ 250; Positive, 170 ≤ Mean < 210; Quite positive, 130 ≤ Mean < 170; Less favorable, 90 ≤ Mean < 130; Very less favorable, 50 ≤ Mean ≥ 90 —all statistical analyses using IBM SPSS.

4. Results

4.1. Learning model feasibility

Based on the Round 1 validation, which involved 20 students, one lecturer, and one expert, several aspects need improvement, including the learning flow, the learning guide, the scope of the learning materials, and the measurement of learning objective achievement. After revising the model, it was implemented with 55 students and validated by three lecturers and two experts (Round 2). The students', three lecturers', and two experts' assessments concluded that the model was suitable (Table 2). It means the model is ready to use.

4.2. Learning model effectiveness

Mean scores of pre-tests, post-tests, and gain (Table 3) are the basis for determining the learning effectiveness. The normalized gain (g) calculation is  . It describes a learning model that is highly effective in basic electrical engineering instruction.

4.4. The gain in students' learning outcomes by Background.  

Each group's student learning outcomes (gains) by school type, status, and gender (Table 4) showed that females outperformed males in all groups. The gain scores of participants' group learning outcomes serve as the basis for analyzing the role of students' existing diversity in learning.

It was found that students' variety of school backgrounds and gender did not significantly affect the learning outcomes of chatbot-assisted self-regulated learning (Table 5). It was decribed signifance of School Status (Sig.  = .595 > .05), School type (Sig. = 562 > .05), and gender (Sig. = .065 > .05). Likewise, their fellow interactions did not impact (School status and School type, Sig. = 820 > .05; School status and Gender, Sig. .950 > .05); School type and gender, Sig. = .402 > .05; School status, School type, and gender, Sig. = .744 > .05).

4.5. Students' Attitudes and Perceptions on the learning model

The trend in students' attitudes and perceptions of learning, based on school status, school type, and gender, is positive. Their means are in 170 ≤ Mean < 210 (Table 6). It describes that the students tend to accept the learning model positively. Based on interviews with several students, they generally said the learning model facilitates their learning.

5. Discussion

5.1. Chatbot-assisted self-regulated learning environment effectiveness

The developed learning model proved effective for use in basic electrical engineering instruction. This effectiveness was highlighted by its ease of use and practicality. However, it is important to emphasize that adaptive learning tools that accurately measure student understanding and provide the necessary support to correct misconceptions remain needed (Afzaal et al., 2021). Specifically, examining students' behavioral patterns when interacting with the chatbot could enable the computer to adjust teaching methods more quickly, as Caccavale (2025) suggested.

The insights gained from these exchanges inform the creation of more individualized learning paths, enabling the chatbot to guide students through remedial modules or offer additional challenges based on their demonstrated skills and areas of weakness. Furthermore, these interaction patterns can reveal the effectiveness of various chatbot responses, thus encouraging incremental adjustments to the chatbot's pedagogical design and its overall effectiveness in supporting self-regulated learning (Lin et al., 2024). Such an iterative improvement process, guided by continuous review of user interactions and learning outcomes, is crucial for intelligent tutoring systems to effectively promote self-regulation (Ng et al., 2024; Mejeh et al., 2024). Research has revealed that some students exhibit overconfidence in their understanding, particularly when rejecting chatbot guidance on complex problems, which can lead to incorrect corrections. This fact highlights the need to develop sophisticated AI mechanisms that can detect such metacognitive illusions and prevent them through adaptive feedback and corrective prompts (Kumar et al., 2024).​‍​‌‍​‍‌

5.2. Students' diversity's impact on learning

The findings from this study suggest that various demographic factors, such as school type, socioeconomic status, and gender, do not significantly influence learning outcomes within chatbot-assisted self-regulated learning environments. This outcome diverges from conventional educational research that frequently identifies these variables as significant predictors of academic achievement in traditional settings. Instead, the consistency observed across diverse student backgrounds highlights the potential of chatbot-assisted self-regulated learning to offer an equitable and universally effective pedagogical approach (Ng et al., 2024). This observed uniformity suggests that the scaffolding provided by educational chatbots in facilitating self-regulated learning processes may effectively mitigate pre-existing disparities arising from varied educational or socioeconomic contexts (Guan et al., 2024).

The mitigation underscores the capability of AI-driven tools to democratize access to effective learning strategies, enabling students from all backgrounds to develop critical metacognitive skills and improve academic performance (Chang et al., 2023). This aligns with broader research indicating that chatbot technology can exert a medium-to-high effect on overall learning outcomes, independent of such moderator variables (Deng & Yu, 2023). Specifically, metacognitive support delivered via chatbots has been shown to enhance self-regulated learning abilities, particularly in task strategy and self-evaluation, thereby optimizing the learning experience for all students (Xu et al., 2025). This further suggests that the inherent adaptability and personalized feedback mechanisms embedded in chatbot technologies effectively cater to individual learning needs, thereby neutralizing the differential impact typically associated with diverse student demographics (Manzanares et al., 2023).

5.3. Students' perception of the learning’s usefulness and usability

Students​‍​‌‍​‍‌ generally viewed the chatbot as an effective tool for personalized learning and quick feedback, which in turn would lead to a more efficient grasp of the course material, in line with Chen (2024). Nevertheless, this favorable reception did not extend to all aspects of the chatbot's functionality, as some students doubted its ability to handle complex or subtle inquiries.

The need for the chatbot to have adaptive response depth, which could adjust the complexity of the answer to the student's level of expertise, was repeatedly raised, suggesting that students wanted more intricate interactions. Moreover, examination of the data concerning the students' opinions about the chatbot revealed that their principal appreciation of the chatbot was to be used for clarification of doubts and understanding of concepts; however, they considered the chatbot to be less helpful in the areas of thorough content review and exam simulations (Vera, 2024). Although the chatbot successfully served as an on-demand tutor for certain concepts, its current configuration may not fully support broader learning goals or accommodate diverse learning strategies.

This highlights the need for future versions to include stronger systems for more thorough review, possibly through structured learning paths or connections to external knowledge, to increase their overall educational value (Lin et al., 2024). The chatbot was also seen as having potential to promote collaboration and collaborative learning, primarily through collaborative problem-solving and peer explanation. Moreover, features that allow students to record their thought processes or share interpretations with a virtual peer would foster higher-order thinking and understanding beyond mere memorization.

5.4. School background and demographic factors mediate effectiveness and engagement with a chatbot-assisted self-regulated learning.

Students' backgrounds and demographic factors, prior academic achievement, and learning styles may substantially influence their interactions with and perceptions of the chatbot, suggesting that a one-size-fits-all approach to integrating AI into education may not be practical (Stöhr et al., 2024). It also reiterates the need for personalized chatbot design that adapts to diverse learning profiles, thereby enhancing engagement and learning outcomes across a diverse student population (Sandu et al., 2024).

For example, students who prefer visual learning would benefit from chatbots with interactive diagrams and simulations in their interfaces. In contrast, students who prefer auditory learning benefit more from chatbots with audio explanation features. Students with a preference for kinesthetic learning may benefit from chatbot features that incorporate virtual experiments or active problem-solving scenarios (Sadegh‐Zadeh et al., 2023).

The effectiveness of these individualized approaches is further corroborated by findings that students' prior academic achievement was a significant predictor of how quickly they used the chatbot and of their performance improvement. For instance, students with high prior attainment frequently used chatbot assistance to ask advanced questions and broaden their knowledge. In contrast, students with lower prior attainment tended to use it only to build their foundational knowledge (Wambsganß et al., 2024). In contrast, students with lower prior academic achievement often require more structured, guided interactions with chatbots, relying on them as a primary resource to clarify key concepts and help address their near-complete lack of knowledge.

The findings point to the necessity of adaptive scaffolding tailored to individual academic experiences and learning preferences as a prerequisite for increasing chatbot efficacy and ensuring equitable access to their learning potential (Hauser et al., 2024). Differentiating support at the individual level through more granular learner profiling can improve the chatbot's utility for a broader range of students. Doing so would involve implementing advanced diagnostic techniques embedded in the chatbot to assess students independently and dynamically update their instructional approach (Stöhr et al., 2024). Having insight into user preferences and choices, including the number of questions or the time required to achieve the outcome, can enable the chatbot to engineer prompts that best suit individual student preferences and improve the efficiency of their learning experience.

6. Conclusion

The efficacy of chatbots depends on their personalized and scaffolding capabilities, which enhance reading comprehension and problem-solving, especially within a Retrieval-Augmented Generation architecture. The adaptability of this approach allows the chatbot to provide scaffolded responses and thorough steps, which are essential for constructing knowledge and developing understanding.

7. Limitation

Despite the promising findings, this study encountered several limitations that warrant consideration for future research, including the relatively short study horizon and the focus on a single academic discipline, which may limit the generalizability of the results.

Author Contributions: All authors contributed to this research.

Funding: Universitas Negeri Medan, Grant Number: 01994/UN33/KPT/2023.

Conflict of Interest: The authors declare no conflict of interest.

Informed Consent Statement/Ethics Approval: Not applicable.

Declaration of Generative AI and AI-assisted Technologies: This study has not used any generative AI tools or technologies in the preparation of this manuscript.