PROFESSIONAL ROLE DEVELOPMENT

PRIMARY OUTCOME AND PRIMARY HYPOTHESIS

Primary Outcome

To measure the effect of flipped classrooms on learning topics in nursing informatics as compared with classic education, a comparison was made between a control group (classic education) and an intervention group (flipped classroom). Toward this aim, the two different education methods were applied in two 6-week school terms, and the testing questionnaire was used to determine and compare the knowledge level outcomes of the students in both groups.

Primary Hypothesis

Flipped classrooms have a positive effect on knowledge level outcomes compared with classic education. The hypotheses in this context are as follows:

H1a: There is a statistically significant difference between students who are instructed and are not instructed in flipped classrooms in terms of pretest and posttest score differences.

SECONDARY OUTCOMES AND SECONDARY HYPOTHESIS

Secondary Outcomes

The control group (classic education) and the intervention group (flipped classrooms) were compared to measure the effect on students' readiness of learning the topics taught in the nursing informatics course in flipped classrooms. Two different educational methods were therefore applied in a 6-week period, and the levels of readiness were compared using an appropriate scale.

Secondary Hypotheses

The difference in the initial and final knowledge levels and readiness scores between students who received flipped education and those who did not is statistically significant. In this context, the hypotheses are as follows:

H1b: There is a statistically significant difference between students who are and are not instructed in flipped classrooms in terms of the difference between the first and last scores on the flipped learning readiness scale.

H1c: There is a statistically significant relationship between the differences in the pretest and posttest scores of those who are exposed to the flipped classroom to a lesser extent compared with those who are given the opportunity to make more use of the model.

METHODS

Study Design

The study was conducted in pretest-posttest quasi-experimental design. The reporting stages of the study were prepared according to the TREND guide (Transparent Reporting of Evaluations with Nonrandomized Designs). 19

Setting and Participants

The study was conducted with third-year students attending the nursing informatics course in the nursing department of a state university in the 2022-2023 academic year. The nursing informatics course is an elective course (European Credit Transfer and Accumulation System-ECTS 2 credits) in semester VI. Power analysis was used to determine the appropriate sample size and was performed with G*Power version 3.1.9.7 (Heinrich Heine University, Dusseldorf, Germany). 20

Sample Size

The primary outcome was to compare classic education and the flipped classroom model in terms of the differences in student achievement scores. Our aim was to test whether the differences between the pretest and posttest scores ([DELTA]c - [DELTA]f) in the classic education period and the flipped classroom period differed from 0. Assuming the practical significance of a minimum difference of 10 points (SD, 30), with an effect size of Cohen  d = 0.33, a 5% margin of error, and 80% statistical power, the minimum sample size was determined to be n = 76. When 10% was considered as a sample loss, this number increased to n = 84.

Eligibility Criteria

The students are study volunteers who were taking the nursing informatics course.

Data Collection Instruments

We used the personal information form and the System Usability Scale (SUS), Flipped Learning Readiness Scale (FLRS), and Knowledge Test in the study, all of which were drawn up based on a review of the literature.

Personal Information Form

The personal information form contains questions about the students' age, sex, Internet usage status, and what features they had access to on their mobile phones.

System Usability Scale

This scale was adapted into Turkish by Kadirhan et al, 21  having been developed by Brooke. 22  The score is multiplied by 2.5 with a possible score ranging between 0 and 100 on the scale. A high score indicates increased usability, and a score between 65 and 70 indicates that the system is acceptable. The scale consists of 10 items. The responses to the Likert-type evaluation correspond to "0" = strongly disagree," "1" = disagree," "2" = decided," "3" = agree," and "4" = strongly agree," and items 2, 4, 6, 8, and 10 are reversely scored. 23

Flipped Learning Readiness Scale

The scale was developed by Hao  23  and adapted into Turkish by Durak. 24  It is graded from "strongly disagree" (1) to "strongly agree" (5), and there are no items reversely scored. The original scale has a total of 27 items and five subscales. These five subscales are self-directed learning and student control, classroom communication self-efficacy, technology self-efficacy, motivation for learning, and prestudy. 24

Student Information Test

This form was created by the researchers according to the weekly course contents of the nursing informatics course (Supplemental Digital Content, Figure 1,  http://links.lww.com/CIN/A398 ). It consists of 24 questions derived from the weekly course program. The course topics were covered in a total of 12 weeks, with an introduction to the course taking place in the first week and a course evaluation implemented in the last week.

Intervention and Control Group Protocol

To achieve a more productive learning experience in the intervention group, the researchers prepared presentations in line with the weekly syllabus. With the adoption of the flipped classroom method, professional videos, each running no more than 30 minutes, were created. The nursing informatics course was structured in two stages-the first 6 weeks carried out using the classic education method and the next 6 weeks using the flipped classroom method. Students experiencing the classic education (online) process generally assumed a more passive role, and the lecturer explained the lectures during class time. 25  In the flipped classroom model, the course materials and videos were delivered to the students 1 day before class convened, and they were provided with relevant instructions. Thus, an asynchronous online learning environment was created through Microsoft Teams (Licensed by Akdeniz University) and WhatsApp groups, and the course content was shared with the students.

In the flipped classroom, students were given the opportunity to discuss in class the videos they had watched and ask questions about the parts of the videos they did not understand. Presentations were shared via the Microsoft Teams platform (Microsoft Inc., Redmond, WA, USA) in both the classic education and flipped classroom. The students' knowledge levels were measured by online questionnaires and tests conducted in the first week and the sixth week in both phases. The two-phase structure of the course was designed to gradually deepen the students' learning according to Bloom's Taxonomy. The steps in Bloom's Taxonomy consist of Create, Evaluate, Analyze, Apply, Understand, and Recall. 26  In the first 6 weeks (weeks 1-6), the presentation of basic knowledge and concepts, for example, the history of nursing informatics and nursing classification, is aimed at developing lower-level cognitive skills (recall and understanding) as designated by Bloom's Taxonomy. From the eighth week onward (weeks 8-14), the focus was on more complex programming languages such as algorithms, large data, and artificial intelligence, in an effort to develop the higher-level skills of analysis, evaluation, and creation. The implementation of flipped learning during this challenging period was preferred to ensure the in-depth development of these skills.

The control group in the study comprised students receiving classic education.

Data Collection

In the first week of the course, the research process was explained to the students in detail. In the first 6-week period, the classic education method was used, and a pretest (personal information form, Knowledge Level Test) was administered to the students at the beginning of this period, after which a posttest and midterm examination were administered. At the beginning of the second 6-week period in which the flipped classroom method was to be applied, students were administered another pretest (personal information form, SUS, Knowledge Level Test) and also the FLRS. At the end of this semester, a posttest, the FLRS, and a second midterm examination were administered. The tests were drawn up and collected using Google Forms. Each student was given an average of 10 minutes to answer the forms. The Knowledge Test was organized according to the weekly course topics; two multiple-choice, four-choice questions were included under each topic. The Knowledge Test thus consisted of 24 questions in total.

Variables

Dependent Variables

The dependent variables were mean scores of the Knowledge Test, the mean scores of the SUS, and the mean scores of the Flipped Classroom Readiness Scale.

Independent Variables

The independent variables were age, sex, electronic devices owned, mobile phone and smartphone usage skills, Internet connection status and usage, ability to install applications (via Google Play Store or Apple Store), duration of technology usage for educational purposes, experience in using computers (duration and frequency), evaluations of academic success, and level of engagement in the nursing informatics course

Data Analysis

Descriptive statistics were presented in terms of frequency, percentage, mean, standard deviation, median (MD), minimum, maximum, and Q1-Q3 values. Normality was assessed using the histogram, Q-Q plots, skewness, kurtosis, and the Shapiro-Wilk test. Due to the nonnormal distribution, Wilcoxon signed-rank test and Mann-Whitney  Utest were used for comparing dependent and independent groups, respectively. Effect sizes were evaluated with rank biserial correlation ( rb) in the following range interpretations:  rb < 0.10 (very small) to  rb >= 0.5 (large). 27  Spearman correlation test was applied due to the nonnormal data. Univariable analyses identified risk factors, followed by multivariable logistic and linear regression for significant variables ( P < .05). Analyses were conducted using SPSS version 23.0 (IBM Inc., Armonk, NY, USA) and JAMOVI version 2.3 ( https://www.jamovi.org ).

Ethical Considerations

Ethics committee approval (February 22, 2023: KAEK-177) and institutional permission were obtained for the study. The students' informed consent was also received. The informed consent form was accessible to students on the first page of the online sheet.

RESULTS

Females constituted the majority of the students (85.7%); the mean age was 21.3 (SD, 1.6) years. The average duration of smartphone and computer usage was 7.11 (SD, 2.3) years and 7.51 (SD, 4.39) years, respectively. Although every student had a mobile phone, 31% had a laptop, and 2.4% had a tablet. Of the students, 71.4% rated themselves as "good" or "very good" at using their mobile phones, and among those who had continuous Internet access, 83.3% had a home Internet connection, 10.7% had a dormitory Internet connection, and 8.3% were telephone Internet users. Those who recommended the flipped classroom model for the next academic year constituted 76.2%. Regarding their own academic achievement level, 49 students (58.3%) indicated that it was moderate or low, whereas 35 students (41.7%) indicated that it was good or very good (Supplemental Digital Content, Table 1,  http://links.lww.com/CIN/A399 ).  Table 1  shows that there was an increase in the level of knowledge from 41.67 to 70.83 points in total, but the amount of increase in the flipped classroom (MD: 33.3, Q1: 16.7, Q3: 50) was higher than the amount of increase in the classic education group (MD: 25, Q1: 8.3, Q3: 33.3) ( P < .001).

The SUS MD was 72.5 (Q1: 65, Q3: 81.25). There was an increase in the MD technology self-efficacy score (MD: 4, Q1: 3.89, Q3: 4.56) after the implementation of flipped classrooms compared with the MD of the subscale at the beginning of the semester (MD: 3.94, Q1: 3.56, Q3: 4.22) ( P = .001). An increase was observed in the MD readiness score (MD: 4, Q1: 3.71, Q3: 4.21) after the implementation of flipped classrooms compared with the MD of the subscale at the beginning of the semester (MD: 3.76, Q1: 3.42, Q3: 4.08) ( P = .002) ( Table 2 ).

Among the students who were divided into two groups according to their level of participation in the traditional and flipped classroom, the mean rank value of the SUS score (MR: 49.43) of the students who stated that they participated more in flipped classrooms (n = 8) or participated equally (n = 38) was higher ( P = .004) than the group (n = 46) who participated more in classic education (MR: 34.11). Similarly, the mean scores in technology self-efficacy ( P = .015, medium effect size), motivation for learning ( P = .035, low effect size), prestudy ( P = .007, medium effect size), and readiness ( P = .012, medium effect size) were also higher in the group that did not participate more in classic education ( Table 3 ).

Student control and self-directed learning ( P = .008, medium effect size), technology self-efficacy ( P < .001, high effect size), motivation for learning ( P= .041, medium effect size), prestudy ( P = .004, medium effect size), and readiness ( P = .003, high effect size) subscale MDs were found to be higher in the first measurement in those who used the Internet freely compared with the other group (Supplemental Digital Content, Table 2,  http://links.lww.com/CIN/A400 ). In the model obtained with the four variables found to be significant in the univariable analysis, the level of participation in the flipped classroom (odds ratio [OR], 3.37; 95% confidence interval [CI], 1.21-9.36;  P = .02) and home Internet use (OR, 4.27; 95% CI, 1.1-16.52;  P = .036) were found to be independent risk factors ( Table 4 ).

The factors predicting the readiness score are analyzed in Supplemental Digital Content 3 ( http://links.lww.com/CIN/A401 ). Students (n = 56) who indicated that they were sufficiently or very engaged in the course (regression coefficient = .529; 95% CI, 0.08-0.98;  P = .023) displayed a more positive and significant readiness score than students (n = 28) who were not sufficiently engaged in the course or not at all engaged (Supplemental Digital Content, Table 3,  http://links.lww.com/CIN/A401 ).

DISCUSSION

Our findings demonstrate that the flipped classroom education model for third-year students taking the nursing informatics course results in better academic achievement than classic education. The use of the flipped classroom approach resulted in the enhancement of students' learning levels, concurrent with favorable developments in their technological proficiency and general preparedness. In addition, it was discovered that factors that influence student achievement included motivation, engagement in the course, and home Internet connection. These findings have important implications for educational practice and policy in an era when digital education is becoming increasingly dominant. It was determined, according to the results of a systematic review in which 274 studies were included, that the flipped classroom method increased student performance in nursing education and led to the satisfaction of both students and faculty members. 8

A review of the literature demonstrates that the flipped classroom model has been successfully applied in a number of different courses. 8-11  In particular, Oh and colleagues' 10  study is an important reference in this regard. The application of the flipped classroom teaching method in the nursing informatics course succeeded in significantly increasing the knowledge level of the students. Moreover, the majority of the students found this teaching method extremely positive and recommended it to others. 10  A similar trend was observed in this study in that the majority of our students recommended that this course be taught with the flipped classroom approach in the next academic semester. Our findings are supported by the literature. According to a bibliometric analysis of 2194 articles in the literature published between 2007 and 2021, there is scientific evidence that the flipped classroom provides student satisfaction and an increase in motivation. 28

Although the level of participation in the course and having Internet access at home were effective in finding flipped classroom instruction useful, the level of engagement in the course came to the fore in terms of readiness. One of the basic prerequisites for accessing course videos for flipped classroom learning is Internet access. 29  The present study also reveals the importance of this factor. In addition, a study evaluating the impact of flipped classrooms in a medical school found that the method increased students' engagement in the subject matter. 30  Similarly, another study by Chen et al  31  demonstrated that gamified e-books used in flipped classroom instruction for teaching Chinese characters contributed positively to students' learning achievement and motivation. These findings suggest that the flipped classroom has a positive impact on the learning process, encouraging students to become more engaged and to participate more actively in classes. In studies analyzing the factors affecting the level of participation and readiness in flipped classroom, it has been determined that factors such as students' motivation, technology self-efficacy, ease of access to learning materials, and teacher support play an important role. Students' self-efficacy and technology self-efficacy increase their course performance and participation levels. 32

Our study results regarding student learning in technical courses such as nursing informatics make a positive contribution to the literature. Our findings also provide important evidence that it is possible to enrich the academic experiences of nursing students through flipped education and further support the positive findings of previously conducted research studies of different types at various nursing school class levels. For example, the effectiveness of the classic education model for fourth-year nursing students taking an evidence-based practice course was compared with the flipped classroom model. The study demonstrated that the flipped classroom model offered students more of an opportunity to make small improvements in their attitudes, skills, and total evidence-based practice global competence as compared with the classic education method. 33  Results indicate that flipped classrooms help students learn more and develop a more favorable attitude toward learning.

Studies indicate that students benefit more from their education when the flipped classroom model is implemented in nursing education. In a study evaluating the effect of flipped learning on nursing informatics courses in two different universities in South Korea, the self-perceived nursing informatics competencies of the flipped learning group were higher than those of the comparison group. 12  A systematic review explored students' experiences with the skills of discernment, coping with difficulty, course engagement, accepting responsibility, and gaining expertise. 34  The results of this study reveal the effectiveness of this cutting-edge approach to education, especially in areas such as nursing, where practical skills are fundamental. In the same type of research, the study by Chu et al  35  also reveals that the flipped classroom method positively impacts students' sense of self-efficacy.

CONCLUSION

This study aimed to investigate the effects of the flipped classroom in the nursing informatics course on the level of knowledge, readiness for the flipped classroom model, and the usefulness of the flipped classroom in the context of third-year nursing students. The findings suggest that the flipped classroom model could potentially be a more effective approach to use in nursing informatics education. It is especially useful to emphasize that students' technology readiness and motivation to learn can play a critical role in the learning process. Therefore, adopting and supporting such an educational model may be beneficial for student success. To increase the usefulness of the online flipped classroom, Internet facilities in student dormitories should be increased, and to enhance readiness, initiatives to raise engagement in the course itself can be recommended. This suggests that the initial prejudice and excuses set forth regarding this model of education have been abandoned in favor of a positive point of view. This experience demonstrates not only that the flipped classroom model applies to extraordinary situations such as the period of a pandemic, but that it also has the capacity to ensure a continuity of education in unexpected situations such as natural disasters. Nursing students can thus have the opportunity to access quality education under all circumstances. It is clear that flipped classrooms may occupy an important place in future education models.

Limitations

Two limitations can be mentioned in this study. First, the nursing informatics course is offered 2 hours a week and in a single classroom. Randomization therefore was not performed because the intention was to have all the students taking the course benefit from flipped classroom education. Second, our findings can be generalized only for nursing students taking the nursing informatics course due to the quasi-experimental nature of the study.

References

1. Preeti SK. Flipped classroom: an innovative learning model in digital era.  International Journal of Multidisciplinary Education and Research. 2021;6(4): 37-43.  [Context Link]

2. Ghezzi JFSA, Higa EFR, Lemes MA, Marin MJS. Strategies of active learning methodologies in nursing education: an integrative literature review.  Revista Brasileira de Enfermagem. 2021;74(1): e20200130. doi:.

· Full Text

· Bibliographic Links

[Context Link]

3. Barbour C, Schuessler JB. A preliminary framework to guide implementation of the flipped classroom method in nursing education.  Nurse Education in Practice. 2019;34: 36-42. doi:.

· Bibliographic Links

[Context Link]

4. Flores O, Del-Arco I, Silva P. The flipped classroom model at the university: analysis based on professors' and students' assessment in the educational field.  International Journal of Educational Technology in Higher Education. 2016;13: 1-12.  [Context Link]

5. Flipped Learning Network (FLN). The four pillars of F-L-I-P(TM). Reproducible PDF can be found at  www.flippedlearning.org/definition . 2014.  [Context Link]

6. Ogurlu N. Use of flipped classroom teaching method in nursing education.  Journal of Nursing Science. 2023;6(2): 123-129.  [Context Link]

7. Kraml M. Advantages and disadvantages of flipped classroom in adult education using distance learning for learning programming.  EPH-International Journal of Educational Research. 2024;8(1): 26-31.  [Context Link]

8. Barranquero-Herbosa M, Abajas-Bustillo R, Ortego-Mate C. Effectiveness of flipped classroom in nursing education: a systematic review of systematic and integrative reviews.  International Journal of Nursing Studies. 2022;135: 104327. doi:.  [Context Link]

9. Ilgaz A. Flipped classroom with interactive videos applied to nursing students in the teaching of biostatistics knowledge: a mixed-method study.  Journal of Nursology. 2022;25(4): 245-253.  [Context Link]

10. Oh J, Kim SJ, Kim S, Kang KA, Kan J, Bartlett R. Development and evaluation of flipped learning using film clips within a nursing informatics course.  Japan Journal of Nursing Science. 2019;16(4): 385-395.  [Context Link]

11. Yang C, Yang X, Yang H, Fan Y. Flipped classroom combined with human anatomy Web-based learning system shows promising effects in anatomy education.  Medicine. 2020;99(46): e23096. doi:.

· Ovid Full Text

· Full Text

· Bibliographic Links

[Context Link]

12. Oh J, Kim SJ, Kim S, Vasuki R. Evaluation of the effects of flipped learning of a nursing informatics course.  The Journal of Nursing Education. 2017;56(8): 477-483.  [Context Link]

13. Healthcare Information and Management Systems Society. What is the nursing informatics? 2022.  https://www.himss.org/resources/what-nursing-informatics .  [Context Link]

14. Cronenwett L, Sherwood G, Barnsteiner J, et al. Quality and Safety Education for Nurses.  Nurs Outlook. 2007;55(3):122-131. doi:.

· Bibliographic Links

[Context Link]

15. Keskin D. Implementation of flipped model in EFL reading classrooms.  Turkish Online Journal of Distance Education. 2023;24(3): 261-279.  [Context Link]

16. Aygin D, Y[latin dotless i]lmaz AC. The effect of technology in nursing education and current applications.  Izmir Democracy University Health Sciences Journal. 2022;5(1): 32-46.  [Context Link]

17. Schramm L, Friedrich P, Schuttler J, Lutcke B. Interdisciplinary interactive blended learning concept in times of a pandemic-pain medicine "totally digital".  Journal of Medical Education. 2022;39(1): Doc6. doi:.  [Context Link]

18. Schmid RF, Borokhovski E, Bernard RM, Pickup DI, Abrami PC. A meta-analysis of online learning, blended learning, the flipped classroom and classroom instruction for pre-service and in-service teachers.  Computers and Education Open. 2023;100142.  [Context Link]

19. Isherwood G, Taylor K, Burnside G, et al. Teaching orthodontic emergencies using the "flipped classroom" method of teaching-a mixed methods RCT.  European Journal of Dental Education. 2020;24(1): 53-62. doi:.  [Context Link]

20. Des Jarlais DC, Lyles C, Crepaz NTREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement.  American Journal of Public Health. 2004;94: 361-366.

· Bibliographic Links

[Context Link]

21. Faul F, Erdfelder E, Buchner A, Lang AG. Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses.  Behavior Research Methods. 2009;41: 1149-1160.  [Context Link]

22. Kadirhan Z, Gul A, Battal A. System usability scale: a validity and reliability study.  Educational Sciences and Practice. 2015;14(28).  [Context Link]

23. Brooke J. SUS-a quick and dirty usability scale.  Usability Evaluation in Industry. 1996;189(194): 4-7. doi:.  [Context Link]

24. Hao Y. Middle school students' flipped learning readiness in foreign language classrooms: exploring its relationship with personal characteristics and individual circumstances.  Computers in Human Behavior. 2016;59: 295-303.  [Context Link]

25. Durak HY. Turkish adaptation of the flipped learning readiness scale for middle school students.  Bartin University Journal of Faculty of Education. 2017;6(3): 1056-1068. doi:.  [Context Link]

26. Poloski Vokic N, Aleksic A. Are active teaching methods suitable for all generation Y students?-Creativity as a needed ingredient and the role of learning style.  Education Sciences. 2020;10(4): 87. doi:.  [Context Link]

27. Hay[latin dotless i]rsever F, Orhan A. A theoretical analysis of flipped learning.  Mersin University Journal of the Faculty of Education. 2018;14(2): 572-596.  [Context Link]

28. Lopez-Martin E, Ardura-Martinez D. The effect size in scientific publication.  Educacion XX1. 2023;26(1): 9-17. doi:.  [Context Link]

29. del Arco I, Mercade-Mele P, Ramos-Pla A, Flores-Alarcia O. Bibliometric analysis of the flipped classroom pedagogical model: trends and strategic lines of study.  Frontiers in Education. 2022;71022295. doi:.  [Context Link]

30. Besche HC, Onorato S, Pelletier S, et al. A hierarchy of needs for remote undergraduate medical education: lessons from the medical student experience.  BMC Medical Education. 2022;22(1): 423. doi:.

· Full Text

· Bibliographic Links

[Context Link]

31. Kaushik JS, Raghuraman K, Verma S, Arya V, Gehlawat VK. Online interactive flipped classroom teaching in pediatrics for medical undergraduates.  Cureus. 2023;15(4): e37603. doi:.  [Context Link]

32. Chen C, Jamiat N, Mao Y. The study on the effects of gamified interactive e-books on students' learning achievements and motivation in a Chinese character learning flipped classroom.  Frontiers in Psychology. 2023;14: 1236297. doi:.  [Context Link]

33. Sointu E, Hyypia M, Lambert MC, Hirsto L, Saarelainen M, Valtonen T. Preliminary evidence of key factors in successful flipping: predicting positive student experiences in flipped classrooms.  Higher Education. 2023;85(3): 503-520. doi:.  [Context Link]

34. Ruzafa-Martinez M, Molina-Rodriguez A, Perez-Munoz V, Leal-Costa C, Ramos-Morcillo AJ. Effectiveness of the flipped classroom methodology on the learning of evidence-based practice of nursing students: quasi-experimental design.  Nurse Education Today. 2023;128: 105878. doi:.

· Bibliographic Links

[Context Link]

35. Chu TL, Wang J, Monrouxe L, et al. The effects of the flipped classroom in teaching evidence-based nursing: a quasi-experimental study.  PLoS One. 2019;14(1): e0210606. doi:.

· Bibliographic Links

[Context Link]

KEY WORDS: Flipped classroom; Knowledge level; Nursing informatics; Readiness; Usability