AAPM WORK GROUP ON GRAND CHALLENGES REPORT Samuel G. Armato III, PhD, Chicago, IL, Jinzhong Yang, PhD, Houston, TX

AAPM hosted two successful "grand challenges" in conjunction with the 2017 Annual Meeting. These challenges provided a unique opportunity for participants to compare their algorithms with those of others from academia, industry, and government in a structured, direct way using the same data sets, the same set of rules, and the same performance assessment metrics.

The first was a grand challenge on the auto-segmentation (AUTOSEG) of organs-at-risk (OARs) relevant to thoracic radiation treatment planning using computed tomography (CT) images. Rapid advances in radiation therapy call for the fast and accurate segmentation of medical images for treatment planning. The overall objective of the AUTOSEG Challenge was to provide a platform for the comparison of various auto-segmentation algorithms, a guideline for the selection of auto-segmentation algorithms for clinical use, and benchmark data for the evaluation of auto-segmentation algorithms. The AUTOSEG Challenge invited participants from around the world to apply their developed algorithms to perform auto-segmentation of OARs (including esophagus, heart, lung, and spinal cord) on real patient CT images. The AUTOSEG Challenge consisted of two phases: an off-site contest and a real-time contest. The website for the off-site contest opened in early May 2017 and closed on July 17, 2017, with 31 submissions from 11 participants. The real-time contest was held on August 2, 2017, at the AAPM Annual Meeting. Seven participants competed in the real-time contest, with four participants competing on-site and three participants competing remotely. Winners were chosen from among the on-site participants and were announced during the Annual Meeting at the Challenge Symposium, Auto-segmentation for Thoracic Radiation Treatment Planning: A Grand Challenge. The winners of the AUTOSEG Challenge were Dr. Xiao Han from Elekta Inc. (1st place), Dr. Xue Feng from University of Virginia (2nd place), and Dr. Dao Lam from Washington University in St. Louis (3rd place).

The AUTOSEG Challenge was sponsored by AAPM and the TCIA. Without support from AAPM Science Council (through the Working Group on Grand Challenges of the Technology Assessment Committee) and a team of volunteer organizers from MD Anderson Cancer Center, Massachusetts General Hospital, Mirada Medical, Memorial Sloan Kettering Cancer Center, Maastro Clinic, and Harvard University, the AUTOSEG Challenge would not have been a success.

The second was a "part 2" grand challenge (the PROSTATEx-2 Challenge) on the development of quantitative multi-parametric magnetic resonance imaging (MRI) biomarkers for the determination of Gleason Grade Group in prostate cancer hosted jointly by AAPM, the SPIE (the international society for optics and photonics), and the National Cancer Institute (NCI). The PROSTATEx-2 Challenge followed from the successful PROSTATEx Challenge on the development of quantitative image analysis methods for the diagnostic classification of clinically significant prostate lesions that was conducted in conjunction with the 2017 SPIE Medical Imaging Symposium. In mid-May 2017, a training set of prostate MRI scans containing 112 prostate lesions with known Gleason Grade Group (ordinal values from 1-5) was released, and in early June 2017, a set of test MRI scans with 70 prostate lesions became available (with lesion location specified but with Gleason Grade Group not revealed). By the June 26 submission deadline, 21 participants had submitted results from 43 methods. Winners were recognized during the Awards Ceremony at the Annual Meeting, and they presented their methods at a dedicated joint imaging-therapy scientific symposium. The winners of the PROSTATEx-2 Challenge were Bejoy Abraham and his mentor Dr. Madhu Nair from Kerala University (1st place) and Dr. Saifeng Liu from The MRI Institute for Biomedical Research (2nd place).

The team of volunteer organizers for the PROSTATEx-2 Challenge came from The University of Chicago, Radboud University, University of Michigan, FDA, and the NCI; their contributions and the effort of all the participants led to a successful challenge.

The Technology Assessment Committee under Science Council recently created the Working Group on Grand Challenges (WGGC), co-chaired by Sam Armato, Keyvan Farahani, and Kristy Brock. The purpose of the WGGC is to promote the conduct of grand challenges in medical imaging by (1) developing recommendations for promotion and hosting of computational challenges through AAPM activities (including the Annual Meeting) to help assess or improve the use of medical imaging in diagnostic and/or therapeutic applications, (2) vetting proposals from groups that wish to host a challenge in conjunction with AAPM Annual Meeting, (3) facilitating the execution of such proffered challenges, and (4) generating ideas for challenges to advance the field.Please watch for future calls for challenge proposals to help the WGGC achieve its goal of fostering a variety of grand challenges in the spirit of open science.