Room: AAPM ePoster Library
Knowledge-based DVH prediction systems vary in their approach to uncertainty estimation. This work establishes a framework for empirical quantification of DVH prediction accuracy, so that different knowledge-based algorithms can be fairly compared, and clinicians can properly interpret their results.
A natural method to quantify uncertainty of different knowledge-based predictions is to calculate the root-mean-square (RMS) deviation from the actual clinical DVHs of a large validation set. In such a case, ˜68% of clinical DVHs should lie within the prediction uncertainty band (i.e., 1 s from mean); this provides an intuitive standard by which to test prediction uncertainties provided by the knowledge-based model within the treatment planning system. RMS-uncertainties were compared for both ORBIT-RT and RapidPlan™ knowledge-based prostate models, built with the same training cohort. Prediction accuracy was expressed as a census of a subset of clinical DVHs (ORBIT-RT: 73 plans, RapidPlan: 26 plans) falling within the uncertainty band. Finally, the calculated RapidPlan RMS-uncertainty was compared to the provided RapidPlan uncertainty.
Both ORBIT-RT and RapidPlan predictions exhibit comparable mean- and RMS-deviation in the 80-105% dose region. Below 80%, differences emerged: the ORBIT-RT mean-deviation from clinical (prediction bias) was consistently lower than RapidPlan, while the RapidPlan RMS-deviation (prediction uncertainty) was often lower than ORBIT-RT. As expected, the RMS uncertainty bands of both models’ predictions encompassed ˜68% of clinical DVHs. However, for the provided RapidPlan uncertainty band, the frequency of clinical DVHs within the band fell far below 68% for most OARs.
The RMS-deviation method is an intuitive and testable means of quantifying DVH prediction accuracy. Provided uncertainty bands checked in this manner ensure that they comport with the expected meaning of a 1-s band. In this prostate cohort, RapidPlan RMS-uncertainty was consistently larger than the provided RapidPlan uncertainty, implying a systematic underestimate of the true modeling error.
Funding Support, Disclosures, and Conflict of Interest: K. Moore reports income for personal consulting and speaker honoraria from Varian Medical Systems. This work was supported in part by the Agency for Healthcare Research and Quality (AHRQ R01HS025440).