Annotated Bibliography

DOI:10.1093/jnci/dju258 First published online September 12, 2014

© The Author 2014. Published by Oxford University Press. All rights reserved. For Permissions, please e-mail: [email protected].

Vol. 106, Issue 10 | dju258 | October 8, 20141 of 3 Brief Communication | JNCI

Use of Crowdsourcing for Cancer Clinical Trial Development Amanda Leiter, Tomasz Sablinski, Michael Diefenbach, Marc Foster, Alex Greenberg, John Holland, William K. Oh, Matthew D. Galsky

Manuscript received December 24, 2013; revised March 28, 2014; accepted July 17, 2014.

Correspondence to: Matthew D. Galsky, MD, Mount Sinai School of Medicine, Tisch Cancer Institute, 1 Gustave L Levy Place, New York, NY 10029 (e-mail: [email protected]).

Patient and physician awareness and acceptance of trials and patient ineligibility are major cancer clinical trial accrual barriers. Yet, trials are typically conceived and designed by small teams of researchers with limited patient input. We hypothesized that through crowdsourcing, the intellectual and creative capacity of a large number of researchers, clinicians, and patients could be harnessed to improve the clinical trial design process. In this study, we evaluated the feasibility and utility of using an inter- net-based crowdsourcing platform to inform the design of a clinical trial exploring an antidiabetic drug, metformin, in prostate cancer. Over a six-week period, crowd- sourced input was collected from 60 physicians/researchers and 42 patients/advo- cates leading to several major (eg, eligibility) and minor modifications to the clinical trial protocol as originally designed. Crowdsourcing clinical trial design is feasible, adds value to the protocol development process, and may ultimately improve the efficiency of trial conduct.

JNCI J Natl Cancer Inst (2014) 106(10): dju258 doi:10.1093/jnci/dju258

The results of prospective clinical trials are critical to informing the care of patients with cancer. However, only 2% to 7% of adult cancer patients in the United States enroll in clinical trials, a percentage that has remained stable for decades (1). Commonly cited accrual barriers include lack of trial awareness, lack of enthusiasm for a par- ticular experimental question or approach, and unnecessarily restrictive eligibility cri- teria (2–4). Despite these barriers, the pro- cess of clinical trial protocol development has remained largely unchanged. Input on key elements of a trial’s design is typically provided by a small group of researchers. Patient feedback is usually limited to the few patient advocates involved in academic or industry protocol review committees.

Crowdsourcing is the completion of a task by obtaining input from a large num- ber of individuals. We hypothesized that through crowdsourcing, we could har- ness the intellectual and creative capacity of a large group of physicians, research- ers, patients, survivors, and advocates and

improve the clinical trial design process. Because, to our knowledge, this was the first attempt to integrate crowdsourcing into cancer trial development, we assessed the feasibility and utility of this approach.

We used a secure web-based plat- form (Transparency Life Sciences, New York, NY) that enabled participants to provide input (through closed- and open- ended responses) regarding important design elements of a planned clinical trial exploring metformin in prostate can- cer (Supplementary Materials, available online). We invited the “crowd” to com- ment on the study design by distributing hyperlinks (by electronic mail) to a net- work of clinicians, researchers, and coor- dinators of prostate support groups and blogs (Supplementary Materials, available online). We also posted hyperlinks facilitat- ing participation to social-networking sites (eg, LinkedIn). The crowdsourcing plat- form was available for six weeks.

The feasibility of crowdsourcing input on clinical trial design was prospectively

defined as obtaining input from at least 20 physicians/researchers and 20 patients/ advocates. Members of the Mount Sinai Genitourinary Oncology Research Team wrote a complete study protocol prior to initiation of the crowdsourcing effort. After the crowdsourcing platform was closed to further input, open-ended responses were systematically categorized accord- ing to themes, and data were summarized. Data were presented to the research team, and modifications to the original protocol were implemented based on consensus. The number of changes to the original protocol was recorded by three independ- ent reviewers to assess agreement. Utility was prospectively defined as crowdsourced input resulting in at least one major change (changes to eligibility, dose, primary end- point, or statistical plan) or three minor changes (all changes not meeting criteria as a major change) to the study protocol.

Crowdsourced input on trial design was collected from 60 physicians/research- ers, representing various specialties (67% medical oncologists), and 42 patients/ advocates. This collective input led to nine total changes (four major and five minor) to the original protocol, including modifi- cations to eligibility criteria and study pro- cedures (Table  1). Importantly, input from both physicians/researchers and patients/ advocates led to protocol modifications. Agreement among three independent reviewers scoring the number of protocol modifications was excellent (eight, nine, and nine changes). Notably, 91% of phy- sicians/researchers and 76% of patients/ advocates agreed or strongly agreed with the statement, “I would participate in a similar clinical trial crowdsourcing effort in the future.”

Our report describes, to our knowledge, the first effort to employ crowdsourcing to inform cancer trial design and demon- strates both the feasibility and utility of this approach. We selected a fixed num- ber of participants to define feasibility, as the denominator of potential participants was unknown given the social network- ing approach utilized to enable participa- tion. Though our definition of feasibility was somewhat arbitrary, we argue that the

Brief CommUniCaTion

October 8 dju258

JNCI | Brief Communication 2 of 3jnci.oxfordjournals.org

achieved participation rate represents a meaningful increase in the number of individuals, particularly patients, to pro- vide detailed input on clinical trial design compared with conventional practice. Importantly, the value of crowdsourcing clinical trial design exists not only in estab- lishing consensus (eg, extent of agreement

with study population) but also in discover- ing potentially project-altering “gems” of wisdom. Our protocol modifications were a result of these “gems,” and, while consensus data did not contribute to our utility meas- ure, such input was important for confirm- ing many aspects of our original trial design. We also discovered potential barriers to

clinical trial conduct through crowdsourc- ing. For example, several patients indicated that they had been prescribed the proposed study intervention by their oncologists, off label. Finally, we believe that the process of engaging stakeholders (both clinicians and patients) could have a positive impact on trial accrual by facilitating trial awareness

Table 1. Categories of input solicited through crowdsourcing and select examples of input and protocol modifications

Protocol category Examples of open-ended responses Select examples of changes to protocol

Rationale* “The epidemiology remains the strongest data.” (R) “The rationale is essentially epidemiologic.” (R) “Reduction in growth factor and cell cycle modulators should reduce cancer

cell proliferation. Epidemiologic data are strongest here.” (R) Intervention/dosing “My medical oncologist put me on this drug over a year ago. I’ve read a lot

about it and so has he.” (P) “My oncologist prescribed metformin for my prostate cancer about

6 months ago.” (P) “Recently began taking metformin as prescribed by (my oncologist).” (P) “A number of men in my support group have had metformin prescribed by

their oncologist.” (P) Eligibility “Need to set some limit for minimum (PSA) increase.” (R)

“Some accounting for PSA kinetics may be requested to determine who should be on ADT rather than metformin (ie, PSADT <4 months).” (R)

“Need to define/standardize the length of time for adjuvant ADT following XRT, and what happens when the rises occur while on adjuvant ADT.” (R)

Added specific range for PSA doubling time to eligibility criteria

Added eligibility criteria for patients treated with adjuvant ADT

Endpoints “One could also use time to PSA doubling or PSA change at a fixed time point—some sort of landmark analysis as an endpoint.” (R)

“If there are no side effects, then even a slight delay of PSA progression or a slight delay in pain onset is a desirable result. However, if there are many serious side effects, then a greater benefit must be shown.” (P)

“Patients should discuss their experiences in taking metformin and whether it had any negative influence on their lifestyles.” (P)

Added time to PSA doubling as an endpoint

Added patient reported outcomes

Safety “The side effects I have read about seem to be relatively minor, not life threatening, and reversible if the medication is stopped.” (P)

“I have taken it for 10 years with no ill effect.” (P) “I am always concerned about the risk for side effects and those are already

detailed for the diabetic user and should cause a non diabetic equal concern.” (P)

Study procedures “If your endpoint is a PSA endpoint, I am nervous about having PSA’s drawn and measured at random clinics/labs.” (R)

“Obtaining reliable measures of PSA across multiple study sites is difficult.” (R)

“Clinical research is one of the last bastions of unfounded conservatism, offering patients who agree to participate significant inconvenience and false perception of safety by asking them to come to a site once in a while.” (R)

“Prestudy exam and start must be in person. Depending on the study, some intermediate exams would be prudent. End of study exam must be in person.” (P)

“All patients should be evaluated at one of the study sites for the baseline entry evaluation, and at least one more time at the study site during the first 6 months of treatment.” (P)

“Not having to travel long distances to participate.” (P) “It would not inconvenience me by requiring frequent appointments

or travel” (P)

Added central laboratory testing for PSA analysis

Added option of remote monitoring (telemedicine) vs in person visits for certain study visits

Data analysis/ visualization

“Patients could be provided with a code to log in and see a summary data section.” (R)

“Provide interim and final results to all participants.” (P)

Added secure website for patient access to real time results to protocol

* See Supplementary Materials (available online). ADT = androgen deprivation therapy; PSA = prostate specific antigen; P = patient/advocate input; PSADT = PSA doubling time; R = researcher/physician input; XRT = radiation therapy.

Vol. 106, Issue 10 | dju258 | October 8, 20143 of 3 Brief Communication | JNCI

and acceptance though such endpoints could not be addressed in this study.

There are limitations to our approach. Participants were technology-savvy and may not be representative of the crowd at large. This may impact the consensus- building aspect of the input, limiting gener- alizability, though it does not diminish the value of highly informative individual con- tributions. The crowdsourcing platform was available for input for six weeks, which could theoretically delay the trial develop- ment process; however, this approach is ideally initiated very early, at the stage of concept formulation. Adequate recogni- tion of contributors, management of intel- lectual property, and concerns about early disclosure of concepts to other investiga- tors all represent substantial challenges. Given the “network of networks” approach employed to invite participation of “the crowd” (Supplementary Materials, avail- able online), we were unable to track how many individuals actually saw the invitation to participate or to calculate the partici- pation rate. Finally, whether the protocol changes resulting from the crowdsourcing effort ultimately improve the efficiency (eg, increased trial accrual, decreased pro- tocol amendments, etc.) of the resulting

clinical trial needs to be established in future research.

The “wisdom of crowds” was eloquently described in James Surowiecki’s book of the same name and use of the crowd for prob- lem solving has proliferated with the rise of the Internet and digital social networks (5). Examples of the power of crowdsourcing in the sciences have begun to emerge (6). Cancer impacts the lives of virtually every human being, yet the pace of progress in cancer care is slow. Novel approaches are required to ensure that the best ideas are brought forward and tested in an optimal and efficient manner. Though the current effort focused on input derived specifically from trial stakeholders, expanding the use of crowdsourcing to the general population to address broader aspects of cancer control can easily be envisioned. The problem of cancer is too great to relegate the develop- ment of solutions to a select few.

references 1. Murthy VH, Krumholz HM, Gross CP.

Participation in cancer clinical trials: race-, sex-, and age-based disparities. Jama. 2004;291(22):2720–2726.

2. Denicoff AM, McCaskill-Stevens W, Grubbs SS, et  al. The national cancer institute-amer- ican society of clinical oncology cancer trial

accrual symposium: summary and recommen- dations. J Oncol Pract. 2013;9(6):267–276.

3. Lara PN Jr, Higdon R, Lim N, et  al. Prospective evaluation of cancer clinical trial accrual patterns: identifying poten- tial barriers to enrollment. J Clin Oncol. 2001;19(6):1728–1733.

4. George SL. Reducing patient eligibility cri- teria in cancer clinical trials. J Clin Oncol. 1996;14(4):1364–1370.

5. Surowiecki J. The Wisdom of Crowds. 1st ed. New York, NY: Anchor; 2005.

6. Cooper S, Khatib F, Treuille A, et al. Predicting protein structures with a multiplayer online game. Nature. 2010;466(7307):756–760.

funding MDG is supported by a Prostate Cancer Foundation Young Investigator Award.

notes JH is an employee of AMC Health. AG is an employee of Transparency Life Sciences. TS and MF are co-founders of Transparency Life Sciences.

Affiliations of authors: Department of Medicine, Division of Hematology/Oncology, Tisch Cancer Institute, Icahn School of Medicine at Mount Sinai, New York, NY (AL, WKO, MDG); Platform Development Department, Transparency Life Sciences, New York, NY (TS, MF, AG); Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY (MD); Clinical Trials Department, AMC Health, New York, NY (JH).