Nursing Research

RESEARCH ARTICLE Open Access

A model six-month workshop for developing systematic review protocols at teaching hospitals: action research and scholarly productivity Hiraku Tsujimoto1*† , Yuki Kataoka1,2,3*†, Yukihito Sato1,4, Masahiro Banno2,5,6, Emi Tsujino-Tsujimoto7, Yukiyoshi Sumi8, Ryuichi Sada9, Takashi Fujiwara10, Yoichi Ohtake11, Junji Kumasawa12, Haruki Imura13, Yoshinobu Matsuda14, Ryuhei So2,15, Tomoko Kagawa16, Takashi Yoshioka17,18, Yu Uneno19, Hiroyuki Nagano20, Mai Akazawa21, Takunori Hozumi22 and Yasushi Tsujimoto2,17,23

Abstract

Background: Research engagement contributes to the improvement of patient care. A systematic review is a suitable first scholarly activity because it entails summarization of publicly available data and usually requires neither rigorous ethical review nor research funding.

Methods: This study aimed to develop a model workshop for healthcare staff to acquire skills in creating systematic review protocols based on their own clinical questions at teaching hospitals. We used an action research method to create a model workshop at four hospitals in Japan from April 2015 to March 2017. To improve the program, we solicited reflections using participant questionnaires for each lecture and examined the quality of homework submitted by participants after each lecture. We administered a revised final version of the workshop at five hospitals from April 2016 to March 2017. We evaluated the participants’ scholarly productivity related to these workshops. The observation period was a minimum of 2 years following the workshops.

Results: Most participants had never developed a formal clinical research protocol and voluntarily participated in the workshop. The action research was developed and implemented at nine teaching hospitals in Japan, including one university hospital. The study developed a model nine-step workshop curriculum: 1) Research question development, 2) Search strategy development, 3) Search strategy brush-up, 4) Exclusion and inclusion criteria development, 5) Risk of bias assessment planning, 6) Meta-analysis planning, 7) Subgroup and sensitivity analysis planning, 8) Planning the presentation of results, and 9) Presentation protocols. A total of 233 participants, including medical doctors and other health professionals, produced 414 research questions. Seventy-nine participants (34%) completed the workshop, and 47 review teams accomplished systematic review protocols. The participants published 13 peer-reviewed articles as a result of the workshop. (Continued on next page)

© The Author(s). 2021 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

* Correspondence: [email protected]; [email protected] †Hiraku Tsujimoto and Yuki Kataoka contributed equally to this work. 1Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Amagasaki, Japan Full list of author information is available at the end of the article

Tsujimoto et al. BMC Medical Education (2021) 21:98 https://doi.org/10.1186/s12909-021-02538-6

(Continued from previous page)

Conclusions: We developed a structured scholarly productive model workshop for healthcare staff working at hospitals. We found healthcare staff with clinical subspecialties were able to develop an unexpectedly high number of research questions through this workshop. Medical teachers at hospitals with prior systematic review experience could teach how to develop systematic review protocols using this model. Further research is needed to increase the academic productivity of such workshops.

Trial registration: UMIN (https://www.umin.ac.jp/ctr/), UMIN000017107 (4/15/2015), UMIN000025580 (1/10/2017).

Keywords: Systematic reviews, Action research, Scholarly activity, Continuing professional development

Background Research engagement is generally believed to contribute to the improvement of patient care [1, 2]. Residency pro- grams in several countries include support for residents in their scholarly activities [3]. However, these programs may not involve adequate supervision and appropriate training in research methods [4, 5]. Systematic reviews (SRs) in medicine are a type of research that attempts to collate all empirical evidence to answer a specific research question [6]. The research method is a summarization of publicly available electronic patient data and usually requires nei- ther rigorous ethical review nor research funding. This is an established and widely used method [7], allowing even research novices to perform SRs for their first clinical re- search projects. Quality SRs require guidance and support of experi-

enced researchers; however, these opportunities are rarely available for healthcare staff in their first SR. The first step in conducting an SR is to structure clinical questions for the research. Vague clinical questions often set too wide of a scope, requiring support from a senior with prior research experience to ensure the appropri- ateness of clinical questions. Unlike narrative literature reviews, SRs a priori specify eligibility criteria to be reviewed [7]. Authors of SRs usually consider eligibility criteria based on both clinical questions and specifica- tion of the types of studies that have addressed these questions [7]. Another feature of SRs is a comprehensive literature review to achieve unbiased knowledge synthe- ses [8]. If a modification of search strategies is required, the review authors are forced to repeat the entire time- consuming literature review process. Therefore, such a repetition can be avoided with a review of SR search strategies before running them. Protocol preparation for SRs requires an analysis of pre-specification of out- comes. Changes in SR outcomes observed in published SRs may lead to a biased interpretation of SR results [9, 10]. Therefore, step-by-step peer review of the protocols is essential for producing quality SRs. Although several universities and organizations have developed massive open online courses (MOOCs) for SRs [11–13], MOOCs involve large groups of people and render providing step-by-step feedback from teachers and advice tailored

to the participants’ level of understanding a major chal- lenge [14]. Furthermore, MOOCs are generally known for their high dropout rates [14–16]. In this study, we developed a model continuous work-

shop (WS) utilizing an action research method to de- velop SR protocols with step-by-step peer-review. We subsequently evaluated its scholarly productivity.

Methods We used an action research method to create a model WS and implemented it at nine teaching hospitals in Japan. The institutional ethics committees of each hos- pital approved our research. Action research is a re- search method that aims at both taking and creating knowledge or theory relating to that action [17–19]. The action research process involves a cyclical process of (1) planning, (2) implementing the plan (action), (3) obser- vation, and (4) critical and self-critical reflection on the results of (1–3) and making decisions for the next cycle of (1–4) to improve the action. As such, the model WS was developed through a cyclical progress in four phases: planning, action, observation, and reflection [17]. The reflection phase after each lecture focused on attempting to improve both the next lecture for the same partici- pants and a lecture on the same topic in the next hos- pital setting. The lectures were given by three teachers who were physicians with prior SR experience and one teacher who had been one of the participants of the third WS and completed an SR through the WS.

Phase 1: planning The goal of the WS was for healthcare staff to learn how to answer their own clinical questions through SRs. We recruited participants from each of the nine hospitals. Two teachers first constructed an overall framework for the WS based on prior experience with writing SRs, the graduate school syllabus at Kyoto University Graduate School of Medicine/School of Public Health, and an introductory textbook for SRs [7].

Phase 2: action We held the first WS from April to August 2015 at Hyogo Prefectural Amagasaki General Medical Center

Tsujimoto et al. BMC Medical Education (2021) 21:98 Page 2 of 8

(HPAGMC) with 21 healthcare staff participants and held the second WS from October 2015 to February 2016 at Tenri Hospital with 14 healthcare staff partici- pants. We held the third WS from April 2016 to Sep- tember 2016 at Shiga University of Medical Science Hospital with 30 healthcare staff participants, including medical students and held the fourth WS from October 2016 to March 2017 at Kameda General Medical Center with 45 healthcare staff participants.

Phase 3: observation After each lecture, participants developed their SR protocols as their homework based on the lecture they took part in. We then assessed the quality of their protocols in terms of whether the study could actually be conducted in a repro- duceable way and with reference to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) [20]. We also administered an open-response evaluation questionnaire and a Likert scale questionnaire (see Supplementary Table 1, Additional file 1) to participants after each lecture and developed a list of common questions and mistakes.

Phase 4: reflection Immediately after each lecture, we held a formal debrief- ing to evaluate the lecture and the homework. We read the questionnaire responses and discussed curriculum im- provement, and thenmodified the subsequent lecture. All modifications were shared among the core members (HT, YK, and YT) and the external institutional members (ETT, YS2, and RS1) by email and approved through an interactive process until consensus was reached. Finally, we discussed the framework of the model WS

sessions in August 2016 for external validation. We con- ducted a WS using the revised final version of the cur- riculum at three hospitals (number of beds) in Japan: Sakai City Medical Center (487), National Hospital Organization Kinki-Chuo Chest Medical Center (385), Amagasaki Iryoseikyo Hospital (199). In addition, teachers who had not previously lectured in previous WSs independently conducted the WS at Kurashiki Cen- tral Hospital (1166) and Seirei Mikatahara General Hos- pital (934). Two years after the last WS, we analyzed the scholarly productivity related to the WS. The following terms were used to describe the participants’ outcomes with the following definitions:

� Research questions: a structured approach for framing questions that uses five components—the patient population or the disease being addressed, the interventions or exposure, the comparator group, the outcome or endpoint, and the study design chosen [7, 21]

� Protocol: a document that presents an explicit plan for a systematic review [20]

� Systematic review: an attempt to collate all relevant evidence that fits pre-specified eligibility criteria to answer a specific research question

The key characteristics of a systematic review are (1) a clear objective with an reproducible methodology, (2) a com- prehensive search, (3) an assessment of the validity of the findings, and (4) systematic presentation and synthesis [20]. The observation period was a minimum of 2 years fol-

lowing completion of the workshop because a previous study showed that the median time from protocol regis- tration to publication was over 1 year [22].

Results The first WS included seven interactive lecture-series from April 2015 to July 2015 at HPAGMC. Participants were mainly medical doctors, but other healthcare pro- fessionals were also present (Table 1) and voluntarily participated in the workshop. This section discusses our results under three main headings: structure of the final model WS (Table 2), quantitative data of each WS, and themes identified from the observations and reflections. The differences between the initial and final models are outlined in Table 3.

Structure of the final model WS We set specific objectives for each lecture, with the teaching method including short lectures and homework-based discussions (Table 2). Participants were expected to develop a part of their SR protocols based on the previous lecture as homework. They sub- mitted the homework through an online platform (Goo- gle Forms, https://www.google.com/intl/ja_jp/forms/ about/). We gave feedback using the online platform within 48 h of the submission and accepted resubmis- sions to ensure that participants would not lose the re- search rhythm [14]. All participants could see other participants’ homework and feedback and could use it as a reference for their own homework. Learning and prac- ticing the items in Table 2 was essential for creating a research protocol that complies with the items in PRIS MA-P. The process of action research and evaluation of the participant-created protocols improved the initial model.

Themes identified from observations and reflections During the observation and reflection phases, we carried out the following evaluations and then made modifica- tions to develop the final model WS described above.

Tsujimoto et al. BMC Medical Education (2021) 21:98 Page 3 of 8

Participants’ readiness Some participants had difficulty coming up with research questions due to a lack of clinical experience and know- ledge. In such cases, the research scope tended to be overly broad and difficult to handle as an SR. On the other hand, participants with clinical subspecialties, such as spe- cialized nurses and specialists in physical therapists, occu- pational therapists, or speech-language-hearing therapists,

had many viable research questions. Therefore, we par- ticularly encouraged clinical staff with a subspecialty to join the WS. Most participants had never developed a formal clin-

ical research protocol before, so some participants found it difficult to grasp the overall picture of the SR protocol and the connection between each lecture. Ultimately, some of these participants quit the WS if they found it

Table 1 Participant demographics

HPAGMC TH SUMH KMC KCH AISH SGMC&KCCMC SMGH Total

Physician 14 11 13 39 15 23 31 18 164

Pharmacist 3 0 3 0 2 0 11 0 19

OT/PT/ST 0 2 7 6 1 3 0 0 19

Nurse 3 0 2 0 1 0 8 0 14

Dietician/CP/MLT 1 0 2 0 6 0 0 0 9

Medical student 0 0 3 0 2 0 0 0 5

Medical librarian 0 1 2 0 0 0 0 0 3

Total 21 14 32 45 27 26 50 18 233

OT occupational therapist, PT physical therapist, ST speech-language-hearing therapist, CP clinical psychologist, MLT medical lab technologist, HPAGMC Hyogo Prefectural Amagasaki General Medical Center, TH Tenri Hospital, SUMH Shiga University of Medical Science Hospital, KMC Kameda Medical Center, KCH Kurashiki Central Hospital, AISH Amagasaki Iryo Seikyo Hospital, SGMC Sakai City General Medical Center, KCCMC Kinki-Chuo Chest Medical Center, SMGH Seirei Mikatahara General Hospital

Table 2 The final WS model

No. Agenda/Topic Objective Due at the next lecture

1 • Introduction to the workshop • What kind of RQs can and cannot be addressed by SRs

• Preliminary search of existing SRs relevant to RQs

• How to structure RQs for SR • Time management for research

• Understand what SR is • Finding existing SRs

• Prepare some RQs

2 • How to develop a search strategy using the Cochrane Library

• Develop a comprehensive search strategy • Preliminary search using structured search formulas and controlled words

3 • How to develop a search strategy using MEDLINE

• How to use reference manager

• Develop a comprehensive search strategy • Develop a comprehensive search strategy

4 • How to develop inclusion and exclusion criteria

• Understanding review teams

• Develop inclusion and exclusion criteria for the review

• Develop inclusion and exclusion criteria

5 • How to assess risk of bias • How to use search formulas from previous SRs

• Understand how to assess risk of bias • Experience risk-of-bias assessment with examples

6 • How to perform meta-analyses • How to deal with heterogeneity • How to deal with publication bias

• Understand how to deal with publication bias • Understand how to perform data synthesis

• Perform meta-analysis using a prepared dataset

• Revise drafted protocol sections

7 • How to synthesize results in a narrative • Understand how to present SR results • Revise drafted protocol sections

8 • Understanding the principles of GRADE • How to prepare the Summary of Findings table

• How to register the protocol in a systematic review registry

• Understand how to assess the quality of evidence using the GRADE approach

• Understand how to present SR results

• Revise drafted protocol sections

9 • Presentation day for protocols • Revise drafted protocol sections

RQ Research question, SR Systematic review, GRADE Grading of Recommendations Assessment, Development, and Evaluation

Tsujimoto et al. BMC Medical Education (2021) 21:98 Page 4 of 8

challenging to develop a single section of the protocol, particularly the search formula development. To remedy this, we provided a model SR protocol in the first lecture and explained how each section of the protocol is related to each lecture. Some participants also dropped out after finding exist-

ing SRs that covered their research questions. Although there is no way to find existing SRs without conducting an SR, previous SRs can be found quickly by using clin- ical practice guidelines and the Cochrane library (https://www.cochranelibrary.com/). To address this, we added a short lecture on finding existing SRs in the first meeting of the WS starting with the second hospital.

Time management for research activity Most participants were full-time healthcare staff and had no time allocated for research within their work sched- ule. Unfortunately, lack of time was a consistent reason for participants dropping out of the WS. To address this, we provided an estimated time commitment for each homework assignment in advance and incorporated a time-management skill lecture in the first meeting of the WS [14]. In this lecture, we emphasized the importance of setting aside scheduled research time at least once a week. In addition, we extended the intervals between WSs from 2 to 3 weeks to allow more time for home- work assignments. In addition, as most participants were healthcare pro-

fessionals, their priority was their patients, sometimes making it difficult for them to attend a scheduled lec- ture. To remedy this, we implemented flipped class- rooms [23], where participants watched lectures online (https://youtu.be/OT5C9vbNYAc), collaborated in on- line discussions, and then carried out research activities as homework. Implementing flipped classrooms allowed participants to skip some WS meetings if necessary. To facilitate participation in the meetings without having questions to ask, some participants requested that the lecture be both provided on video and given at the WS meeting. However, it was not possible to do both

because of limited time available for the meetings in the evening after work.

Research team We identified one successful case where busy junior resi- dents were able to create an SR protocol. We believe the reason for this success could be the support of their team members. The members of this research team met every day in the hospital ward. As one participant in the WS at HPAGMC observed, “Every time they met, they reminded each other that they had a research project to tackle together.” On the other hand, teams with too many members

had trouble finding time when all members could meet. These issues were not considered in the initial planning; therefore, at the first meeting, we facilitated participants in setting up their review team with two or up to three members who could meet frequently [24]. We encour- aged choosing participants who could meet every day to facilitate quick peer support for their task. Since attending meetings was not required, review

teams were not always able to meet. To reduce commu- nication obstacles and to foster camaraderie among the WS participants, we changed the feedback platform from email-based to a shared web cloud-based message board that every participant could read and participate in.

Scholarly productivity A total of 233 participants produced 414 research ques- tions. Seventy-nine participants (34%) finished the WS. They wrote 47 protocols adhering to current standards [20]. If the participants were interested, we offered sup- port in implementing the protocols and with publication. As a result, they published 13 peer-reviewed articles. We searched PubMed for SRs associated with affiliations of the hospitals that conducted the workshops in the 3 years before and after the workshops and found a three- fold increase in the number of publications from before to after the WSs (Supplementary Tables 2, 3, Additional file 1). This is a larger number compared to about a 1.6

Table 3 Differences between initial and final models

Initial model Final model

Target Everyone Everyone, especially medical staff with a subspecialty

Method of teaching Traditional classroom Flipped classroom

Model SR protocol No Distributed on the first day of the workshop

Time management advice No Yes

Review team members Maximum possible Two or three members who meet frequently

Finding previous SR No Brief way to find previous SRs that cover RQ

Intervals between each lecture Two weeks Three weeks

Feedback platform Email based Shared web cloud-based message board

RQ Research question, SR Systematic review

Tsujimoto et al. BMC Medical Education (2021) 21:98 Page 5 of 8

times increase for SRs associated with Japanese affilia- tions, suggesting an indirect effect of the WSs’ imple- mentation. One of the early participants ultimately became a teacher of this WS. Although unanticipated at the time of planning, due to the success of this WS, some teachers were invited to several clinical practice guideline working groups, where experts teach SR re- search methods. In addition, the WS helped teachers develop a meta-epidemiological perspective on the process of SR creation, enabling them to publish 47 papers inspired by the WS as of May 2020 (Supplemen- tary Table 4, Additional file 1; Supplementary data, Additional file 2).

Discussion During this project, we used action research methods to build and refine a model WS on developing SR proto- cols. We conducted the WS at 9 hospitals for over 200 healthcare staff, improved the quality of the lectures based on feedback and self-reflection, and derived a workable model. We found that healthcare staff were able to develop an unexpectedly high number of clinical questions through this WS. The WS model has several strengths. Since the lecture

had already been video recorded for the flipped class- rooms, the WS was easily transferable between institu- tions in Japan. We also made the lecture materials available to everyone via the Internet (https://drive. google.com/drive/folders/1IfO9rNbGf2F_Tc7iJammVwl6 9vXxJ8P_?usp=sharing). The workshop eventually had a form of recruitment that could be easily transformed into a location-independent format, and we were able to start a crowdfunded online salon (https://community. camp-fire.jp/projects/view/187310). In the first year, this led to 252 participants, 16 protocols, and 1 paper ac- cepted. Although we did not anticipate this at the begin- ning, the salon has become a form of learning that aligns well with the “stay-at-home” situation of the COVID-19 pandemic [25]. In addition, the model requires only one or two tutors, a small room with a projector, internet ac- cess, desks, chairs, and no standardized participants; therefore, it can be easily used in locations where educa- tional resources are limited. Furthermore, the model al- lows participants easy access to peer reviewers through a peer discussion of their protocols. The participants may have played a role similar to that of teaching assistants in the previously reported university course, leading to the success of this workshop [26]. A notable success story is when one of the participants (MB), who was new to SR during the third WS, became a competent tutor in the eighth and ninth WSs, contributing greatly to the WS’s sustainability. Our WS also had a higher completion rate (34%) than most MOOCs, which gener- ally have a completion rate of less than 13% [15]. We

believe these results demonstrate that, rather than pro- viding uniform information to all learners, teachers must strive to support individual learners [16, 27]. Given the participants’ readiness in this WS, this appears to be par- ticularly important for learners at hospitals. Such indi- vidually catered education cannot be provided by MOOCs. In addition, we believe our online-based quick feedback system helped participants manage their time and maintain a rhythm of engagement in the study. There were three main limitations to this WS model.

First, the number of participants was limited to about 20–30 learners per teacher at a time, whereas MOOCs usually accept an unlimited number of applicants. Sec- ond, the proposed model WS was only validated in a limited number of institutions in one country in one lan- guage. We encourage further validation in different languages and settings. Finally, due to scheduling con- straints, we could not perform a formal focus group ses- sion for the groups that dropped out or those who successfully published a paper. We believe that future research comparing the results of such focus groups would improve the academic productivity of the WS.

Conclusion We developed a structured model for interactive lecture- series WSs at hospitals. We showed that medical teachers in hospitals with prior SR experience could use this model to teach health professionals how to develop SR protocols. Further research is needed to increase the academic productivity of our model.

Supplementary Information The online version contains supplementary material available at https://doi. org/10.1186/s12909-021-02538-6.

Additional file 1: Supplementary Table 1. Questionnaire for participants. A questionnaire given to workshop participants. Supplementary Table 2. Number of SRs in PubMed affiliated with the hospitals for three years before and after the WS. PubMed search results for SRs three years before and after the WS. Supplementary Table 3. PubMed search details related to Supplementary Table 2. Search details related to Table 2. Supplementary Table 4. Issues covered by participants' SRs. Research topics for participants’ SRs

Additional file 2: Supplementary data. Publications related to the systematic review workshop since 2015.4

Abbreviations WS: Workshop; SR: Systematic review; MOOCs: Massive open online courses; HPAGMC: Hyogo Prefectural Amagasaki General Medical Center

Acknowledgements The authors would like to thank Prof. Toshi A. Furukawa, Kyoto University, for providing the schedule of his lectures at Kyoto University Graduate School of Medicine/School of Public Health as well as all of the participants and supporters of SRWS. The authors give special thanks to Dr. Hidemichi Yuasa, National Hospital Organization, Toyohashi Medical Center, Aichi, Japan, and Dr. Eishu Nango, JCHO Tokyo Joto Hospital, Tokyo, Japan, who attended the presentation day of the first WS at HPAGMC and provided peer comments for participants’ protocols. Special thanks are also given to Akira Onishi, Kobe

Tsujimoto et al. BMC Medical Education (2021) 21:98 Page 6 of 8

University, Hyogo, Japan, who gave a lecture about meta-analysis in the first WS at HPAGMC. The authors also extend a thank you to Dr. Shin-Ichi Kita- moto, Awachiiki Iryo Center Chiba, Japan; Dr. Akira Sato, Kameda General Medical Center, Chiba, Japan; Dr. Satoshi Jujo, Kameda General Medical Cen- ter, Chiba, Japan; Ms. Yukiyo Tada, Kameda General Medical Center, Chiba, Japan; Dr. Ryohei Yamamoto, Kameda General Medical Center, Chiba, Japan; Dr. Hideto Yasuda, Kameda General Medical Center, Chiba, Japan; Dr. Yuda Adachi, Kameda General Medical Center, Chiba, Japan; Ms. Ritsuko Ohse, Kameda General Medical Center, Chiba, Japan; and Dr. Yukiko Okami, Shiga University of Medical Science for their support in running the workshop. The authors would also like to thank Cochrane Japan (https://japan.cochrane.org/ ja) for their support in promoting this WS. The authors further thank Dr. Hir- omu Kutsumi and Dr. Katsutaro Morino, Shiga University of Medical Science, for their support in running and promoting the second iteration of this WS. Finally, the authors thank Editage (www.editage.jp) for their English language editing services. Aspects of this article were presented as the following pos- ter presentations: “A Model Workshop for Developing Skills of Systematic Re- views at Hospitals” at the 2015 JEJU Cochrane Workshop, June 12, 2015, Seogwipo, Jeju, Korea [28]; “A Model Workshop for Developing Skills of Sys- tematic Review Protocols at Hospitals: Midterm Report” at the Asia Pacific Medical Education Conference, Jan. 16, 2016, Singapore [29]; “A Model Work- shop for Systematic Review Protocols at Teaching Hospitals: The Second Mid- term Report” at the 2016 conference of The Association for Medical Education in Europe (AMEE), Aug. 30, 2016, Barcelona, Spain [30]; “Develop- ing A Model Workshop for Systematic Review Protocols at Teaching Hospi- tals: Midterm Report of Action Research” at the 24th Cochrane Colloquium, Oct. 25, 2016, Seoul, Korea [31]; “A Model Workshop for Writing Systematic Review Protocols at Teaching Hospitals: The Final Report” at the 2019 confer- ence of the AMEE, Aug. 27, 2019, Vienna, Austria [32].

Authors’ contributions HT and YK: concept design, analysis and interpretation of data, drafting of manuscript. Implementing and improving the workshop: HT, YK, YS1, MB, ETT, YS2, RS1, TF, YO, JK, HI, YM, RS2, TK, TY, YU, HN, MA, TH, and YT. All authors read and approved the final manuscript.

Funding Transportation expenses to some hospitals were paid for by the Japan Primary Care Association (http://www.primary-care.or.jp/jpca_eng/).

Availability of data and materials The data used during the current study are available from the corresponding authors on reasonable request.

Ethics approval and consent to participate This study was approved by institutional ethics committees (institution, date, reference number): Hyogo Prefectural Amagasaki General Medical Center, Feb. 2015, July 2015 and Jan. 2017, 26–42, 27–6, and 28–129; Tenri Hospital, June 2015, 683; Shiga University of Medical Science, Apr. 2016, 27–238; Kameda General Medical Center, Aug. 2016, 16–062; Sakai City Medical Center, Feb. 2017, 30; National Hospital Organization Kinki-Chuo Chest Med- ical Center, Feb. 2017, 585; Kurashiki Central Hospital, Feb. 2017, 2441; Ama- gasaki Iryoseikyo Hospital, July 2017, 29–3; Seirei Mikatahara General Hospital, Aug. 2017, 17–23. Protocol registration: UMIN (https://www.umin.ac.jp/ctr/), UMIN000017107 (4/15/2015), UMIN000025580 (1/10/2017). Written consent was obtained from the participants.

Consent for publication Not applicable.

Competing interests Y. Kataoka, Y. Tsujimoto, M. Banno, and R. So are the founding members of SRWS-PSG, which is a non-profit, crowdfunded, and managed group. By using a portion of the framework of this WS, they help their supporters in producing papers on clinical research, including systematic reviews. Y. Tsuji- moto is one of the directors of Cochrane JAPAN (https://japan.cochrane.org/ ja, since Jun. 2019). The authors declare that they have no competing inter- ests. The authors alone are responsible for the content and writing of the paper.

Author details 1Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Amagasaki, Japan. 2Systematic Review Workshop Peer Support Group (SRWS-PSG), Osaka, Japan. 3Section of Clinical Epidemiology, Department of Community Medicine, Graduate School of Medicine, Kyoto University, Kyoto, Japan. 4Department of Cardiology, Hyogo Prefectural Amagasaki General Medical Center, Amagasaki, Japan. 5Department of Psychiatry, Seichiryo Hospital, Nagoya, Japan. 6Department of Psychiatry, Nagoya University Graduate School of Medicine, Nagoya, Japan. 7Department of General Internal Medicine, Kanai Hospital, Kyoto, Japan. 8Department of Psychiatry, Shiga University of Medical Science, Otsu, Japan. 9Department of General Internal Medicine, Kameda Medical Center, Kamogawa, Japan. 10Department of Otolaryngology Neck and Surgery, Kurashiki Central Hospital, Kurashiki, Japan. 11Department of Internal Medicine, Itami Seifu Hospital, Itami, Japan. 12Department of Critical Care Medicine, Sakai City Medical Center, Sakai, Japan. 13Department of Infectious Diseases, Rakuwakai Otowa Hospital, Kyoto, Japan. 14Department of Psychosomatic Internal Medicine, National Hospital Organization Kinki-Chuo Chest Medical Center, Sakai, Japan. 15Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine / School of Public Health, Kyoto, Japan. 16Department of Respiratory Medicine, National Hospital Organization Kinki-Chuo Chest Medical Center, Sakai, Japan. 17Department of Healthcare Epidemiology, Kyoto University Graduate School of Medicine / School of Public Health, Kyoto, Japan. 18Center for Innovative Research for Communities and Clinical Excellence (CiRC2LE), Fukushima Medical University, Fukushima, Japan. 19Department of Therapeutic Oncology, Graduate School of Medicine, Kyoto University, Kyoto, Japan. 20Department of Healthcare Economics and Quality Management, Graduate School of Medicine, Kyoto University, Kyoto, Japan. 21Department of Anesthesiology, Shiga University of Medical Science, Otsu, Japan. 22Department of Critical Care Medicine, Aichi Children’s Health and Medical Center, Ohbu, Japan. 23Department of Nephrology and Dialysis, Kyoritsu Hospital, Kawanishi, Japan.

Received: 11 September 2020 Accepted: 4 February 2021

References 1. Harding K, Lynch L, Porter J, Taylor NF. Organisational benefits of a strong

research culture in a health service: a systematic review. Aust Health Rev. 2017;41:45–53.

2. Boaz A, Hanney S, Jones T, Soper B. Does the engagement of clinicians and organisations in research improve healthcare performance: a three-stage review. BMJ Open. 2015;5:e009415.

3. Noble C, Billett SR, Phang DTY, Sharma S, Hashem F, Rogers GD. Supporting resident research learning in the workplace: a rapid realist review. Acad Med. 2018;93:1732–40.

4. Stehlik P, Noble C, Brandenburg C, Fawzy P, Narouz I, Henry D, et al. How do trainee doctors learn about research? Content analysis of Australian specialist colleges’ intended research curricula. BMJ Open. 2020;10:e034962.

5. Kataoka Y, Ikegaki S, Kato D, Takada T, Tsujimoto Y, Sasaki S, et al. Scholarly activity support systems in internal medicine residency programs: a national representative survey in Japan. Intern Med. 2019;58:1859–64.

6. Green S, Higgins J, Alderson P, Clarke M, Mulrow C. What is a systematic review? In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.

7. Higgins JPT, Green S. Chapter 5: Defining the review question and developing criteria for including studies internet source (Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from https://www.handbook.cochrane.org.

8. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

9. Kirkham JJ, Altman DG, Williamson PR. Bias due to changes in specified outcomes during the systematic review process. PLoS One. 2010;5:3–7.

10. Tricco AC, Cogo E, Page MJ, Polisena J, Booth A, Dwan K, et al. A third of systematic reviews changed or did not specify the primary outcome: A PROSPERO register study. J Clin Epidemiol. 2016:1–9.

Tsujimoto et al. BMC Medical Education (2021) 21:98 Page 7 of 8

11. Polanin JR, Pigott TD. The Campbell Collaboration’s systematic review and meta-analysis online training videos. Res Soc Work Pract. 2013;23:229–32.

12. Li T, Dickersin K. Introduction to systematic review and meta-analysis. Johns Hopkins University. https://www.coursera.org/learn/systematic-review. Accessed 1 June 2020.

13. Cochrane Sweden, The medical degree Programme and the medical faculty library & ICT unit at Lund University. Interactive Learinig. Cochrane Interactive Learning. https://training.cochrane.org/interactivelearning. Accessed 1 June 2020.

14. Nawrot I, Doucet A. Building engagement for MOOC students: Introducing support for time management on online learning platforms. In: WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web. Seoul: Association for Computing Machinery, Inc; 2014. p. 1077–82.

15. Onah DFO, Sinclair J, Boyatt R. Dropout rates of massive open online courses: behavioural pattern. In: 6th International Conference on Education and New Learning Technologies. Barcelona: IATED; 2014. p. 5825-5834. EDULEARN14 Proceedings.

16. Chen BY, Kern DE, Kearns RM, Thomas PA, Hughes MT, Tackett S. From modules to MOOCs: application of the six-step approach to online curriculum development for medical education. Acad Med. 2019;94:678–85.

17. Coghlan D, Casey M. Action research from the inside: issues and challenges in doing action research in your own hospital. J Adv Nurs. 2001;35:674–82.

18. Nishigori H, Masuda K, Kikukawa M, Kawashima A, Yudkowsky R, Bordage G, et al. A model teaching session for the hypothesis-driven physical examination. Med Teach. 2011;33:410–7.

19. Cohen L, Manion L, Keith M. Research methods in education. 6th ed. New York: Taylor & Francis; 2007.

20. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1.

21. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:e1–34.

22. Tsujimoto H, Tsujimoto Y, Kataoka Y. Unpublished systematic reviews and financial support: a meta-epidemiological study. BMC Res Notes. 2017;10:703.

23. Moffett J. Twelve tips for flipping the classroom. Med Teach. 2015;37:331–6. 24. Chase J, Topp R, Smith CE, Cohen MZ, Fahrenwald N, Zerwic JJ, et al. Time

Management Strategies for Research Productivity. West J Nurs Res. 2013;35: 155–76.

25. Gordon M, Patricio M, Horne L, Muston A, Alston SR, Pammi M, et al. Developments in medical education in response to the COVID-19 pandemic: A rapid BEME systematic review: BEME Guide No. 63. Med Teach. 2020;42:1–14.

26. Li T, Saldanha IJ, Vedula SS, Yu T, Rosman L, Twose C, et al. Learning by doing-teaching systematic review methods in 8 weeks. Res Synth Methods. 2014;5:254–63.

27. Setia S, Tay JC, Chia YC, Subramaniam K. Massive open online courses (MOOCs) for continuing medical education – why and how? Adv Med Educ Pract. 2019;10:805–12.

28. Tsujimoto H, Kataoka Y, Tsujino E, Hozumi T, Sada R, Sato Y. A Model Workshop for Developing Skills of Systematic Reviews at Hospitals. Jeju: 2015 JEJU Cochrane Workshop; 2015. p. p376.

29. Tsujimoto H, Kataoka Y, Tsujino E, Nagano H, Hozumi T, Sada R, et al. A model workshop for developing skills of systematic review protocols at hospitals: midterm report. Singapore: Asia Pacific Medical Education Conference; 2016. p. 410.

30. Tsujimoto H, Kataoka Y, Tsujimoto Y, Tsujino E, Nagano H, Sumi Y, et al. A Model Workshop for Systematic Review Protocols at Teaching Hospitals: The Second Midterm Report. Barcelona: AMEE 2016; 2016. p. 796.

31. Kataoka Y, Tsujimoto H, Tsujino E, Sada R. Developing A Model Workshop for Systematic Review Protocols at Teaching Hospitals: Midterm Report of Action Research. Seoul: The 24th Cochrane Colloquium; 2016. p. 84.

32. Kataoka Y, Tsujimoto H, Banno M, Tsujimoto Y. A Model Workshop for Writing Systematic Review Protocols at Teaching Hospitals: The Final Report. Vienna: AMEE 2019; 2019. p. 7GG.

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Tsujimoto et al. BMC Medical Education (2021) 21:98 Page 8 of 8

BioMed Central publishes under the Creative Commons Attribution License (CCAL). Under the CCAL, authors retain copyright to the article but users are allowed to download, reprint, distribute and /or copy articles in BioMed Central journals, as long as the original work is properly cited.