Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics

Introduction

The advent of large language models (LLM) has completely transformed natural language processing (NLP) . The traditional paradigm of NLP follows the typical pipeline of creating customized solutions for downstream applications through supervised training. For example, a pre-trained BERT model must be modified with additional network layers and then fine-tuned on labeled training data to perform tasks such as sequence classification or question answering. In some situations, it might also be beneficial or necessary to further pre-train such models on domain specific data to attain acceptable performance . For example, AgriBERT was pre-trained on agriculture-related text data, to properly address NLP tasks in the food and agriculture domain. However, the expansive size and exceptional few-shot learning capabilities enable LLMs to solve NLP problems through in-context learning, which reduces or even eliminates the need for annotated training samples . During in-context learning, LLMs generalize from a few examples (or no examples at all) based on prompts, which typically are descriptive user inputs that characterize desired responses from LLMs . For example, "summarize the following text" is a straightforward prompt that asks the language model to produce a summary for the input text. In general, LLMs provides a novel and simplified workflow for NLP that could potentially do away with supervised fine-tuning and its associated intricacies such as hyper-parameter tuning and model architecture modification. Furthermore, in-context learning significantly reduces the need for expensive and time-consuming human annotation . It is especially desirable in medicine and science due to the limited data available in these domains .

In recent months, the world has witnessed the rise of of ChatGPT https://openai.com/blog/chatgpt, which has enjoyed significant global popularity given its unprecedented language capabilities and accessibility to the general public through a chatbot interface. ChatGPT is based on the powerful GPT-3 model , one of the first large language models in history. The 175-billion-parameters GPT-3 was trained on a large data collection that encapsulated diverse Internet data (including the Common Crawl http://commoncrawl.org/ and Wikipedia https://www.wikipedia.org/). It demonstrates exceptional performance in a variety of NLP tasks spanning from text summarization to named entity recognition (NER) through its text generation objective (indeed, many NLP tasks can be translated to some forms of text generation). ChatGPT inherits these capabilities from GPT-3, along with the massive knowledge on diverse topics stored in the parameter space. More importantly, ChatGPT was trained through Reinforcement Learning from Human Feedback (RLHF), a reinforcement learning process that incorporates human preferences and human ranked values through user feedback. This process tunes the model to generate outputs that are most appealing and relevant to human users. The capabilities of ChatGPT empowers diverse practical applications ranging from essay writing to code generation .

One of the most powerful LLM to date is GPT-4 https://openai.com/research/gpt-4, a successor to GPT-3. While OpenAI has not revealed much about its technical details yet, GPT-4 has demonstrated superior performance over the GPT-3.5-based ChatGPT in various scenarios . In fact, as of March 2023, GPT-4 is powering Microsoft’s search engine, Bing , which demonstrates the potential of LLM-based search. In addition, unlike its predecessors, GPT-4 is a multi-modal model that accepts image inputs, which undoubtedly leads to more interesting applications.

GPT-4 has been shown to perform exceptionally well on various academic and professional benchmarks . For example, GPT-4 passes the USMLE exam with a $>$20% margin . In fact, GPT-4 scores at over the 90th percentile on the SAT, the Uniform Bar Exam and the verbal section of the GRE (see Figure 4 in the "GPT-4 Technical Report" ), where almost all of them included a multiple-choice component. Indeed, multiple-choice examinations are common for evaluating LLMs . Most multiple-choice exams that have been used to evaluate LLMs are based on topics that are among the most well represented in academics. For example, in 2022, the AP physics exam had 144,526 test-takers , the LSAT had 128,893 test-takers , the GRE had approximately 342,000 test-takers . As a result of the large numbers of test-takers taking these exams as well as the importance placed on scores in determining university admittance, there exists an exceeding amount of resources (including text data accessible on the internet). Regardless of the particular LLM under evaluation, the ease of access and overall ubiquity of these tests and relevant test preparation materials effectively preclude a high performance when evaluating LLMs on these tests. It is therefore important to also study LLMs on more obscure and specialized topics where the size of the training data is likely much smaller. In 2022, there were only 162 medical school graduates, who applied for radiation oncology residency programs . Radiation oncology physics therefore represents a topic that is relatively unknown to the general population and may therefore be a more fair test in evaluating LLMs as compared to highly represented knowledge-bases. Obscure topics may represent the greatest educational opportunity and also the greatest risk for the general population in the context of LLMs, as the responses may be more relied upon while being less accurate and with mistakes being less likely to be noticed.

An important factor in evaluating the accuracy of LLMs is to ensure that the test questions are left out of the training data , i.e. not contaminated. The best way to ensure this is to create new questions for testing. In this study, a multiple-choice examination has been created for this purpose. Four transformer-based LLMs have been chosen for evaluation: ChatGPT (GPT-3.5) , ChatGPT (GPT-4) , Bard (LaMDA) , and BLOOMZ . These results are compared to radiation oncology experts as well as non-experts. Additionally, ChatGPT (GPT-4) is further explored on how to improve its answers and on its deductive reasoning capabilities.

Related Work

Transformer-based pre-trained language models (PLMs), such as BERT and GPT , have revolutionized natural language processing. Surpassing previous methods (e.g., RNN-based models) in numerous tasks, they have promoted interest in and accessibility of language models . Generally, PLMs can be categorized into three types: autoregressive models (like GPT), masked language models (such as BERT), and encoder-decoder models (e.g., BART and T5 ). More recently, there is a rise of very large language models, including GPT-3 , Bloom , PaLM , and OPT . Rooted in the transformer architecture, these models draw inspiration from the likes of BERT and GPT but are developed at much larger scales.

The objective of large language models is to accurately learn contextual and domain-specific latent feature representations from input text . For example, the vector representation of "discharge" might vary considerably between medical and general domains. Smaller language programs often require continual pre-training and supervised fine-tuning on downstream tasks to achieve acceptable performance . However, very large language models could potentially eliminate the need for fine-tuning while maintaining competitive results .

Besides the progress in model architecture, scale and training strategies, large language models can be further aligned with human preferences through reinforcement learning from human feedback (RLHF) . This approach has been implemented in various LLMs, such as Sparrow , InstructGPT , and ChatGPT. InstructGPT was based on GPT-3 and was trained through a process during which user preferences were prioritized through human-generated ranking feedback. As a successor to InstructGPT, ChatGPT also employs RLHF, focusing on adhering to prompts and generating comprehensive responses. OpenAI also implemented guardrails to prevent the generation of biased and undesirable outputs . ChatGPT has become a highly successful AI chatbot, capitalizing on GPT-3.5’s capabilities to facilitate human-like interactions.

RLHF incorporates human feedback into the generation and selection of optimal results by training a reward model based on human annotators’ rankings of generated outcomes . This reward model then rewards outputs that best correspond to human preferences and values. We believe these groundbreaking innovations make ChatGPT the perfect candidate for this study.

The recent development of GPT-4 has significantly advanced the state-of-the-art of language models. GPT-4 demonstrates enhanced reasoning abilities, creativity, image comprehension, context understanding, and multi-modal abilities, leading to more sophisticated and diverse responses. The success of large GPT models spurs exploration into specialized variants for specific fields, such as dedicated large language models for medical and healthcare applications, which could potentially revolutionize these domains.

2 Language models and examination

Large language models have exceptional natural language comprehension abilities. In addition, they are trained on massive data that supplies substantial knowledge. These characteristics make large language models ideal candidates for academic and professional benchmarks.

OpenAI recently released the first study in the literature that evaluates large language models on academic and professional exams designed for educated humans . The results indicate that GPT-4 performs extremely well on a wide variety of subjects ranging from the Uniform Bar Exam to GRE. In addition, a study from Microsoft indicates that GPT-4 can pass USMLE, the professional exam for medical residents, by a large margin .

This study is the first evaluation of large language models in the realms of radiation oncology and medical physics, and we believe it can inspire future research in evaluating LLMs on highly-specialized branches of medicine.

3 Prompt engineering

Collecting and labeling data for training or fine-tuning NLP models can be resource-intensive and costly, especially in the medical domain . Recent studies suggest that by employing prompts, large-scale pre-trained language models (PLMs) can be adapted to downstream tasks without the need for fine-tuning .

A prompt consists of a set of instructions that customizes or refines the LLM’s response. Prompts extend beyond merely describing the task or specifying output formats. Indeed, they can be engineered to create novel interactions. For example, it is possible to prompt ChatGPT to emulate a cybersecurity breach with simulated terminal commands . In addition, prompts can also be used to generate additional prompts through a self-adaptation process .

The emergence of prompt engineering signifies the start of a new era in natural language processing . There is no doubt that carefully crafted prompts have much potential for diverse and sophisticated applications. However, determining the ideal prompt poses a unique challenge in the age of large language models. Currently, prompts can be designed manually or generated automatically . Although automatically produced prompts may outperform manual prompts in certain tasks , they often suffer from poor human-readability and explainability . Consequently, manual prompt generation may be favored in domains where interpretability is crucial, such as clinical practices and research. In this study, we design a suite of prompts and chain-of-thought prompts based on our experience in radiation oncology and medical physics and evaluate their impact on large language models.

Methods

A 100-question multiple-choice examination on radiation oncology physics was created for this study by an experienced medical physicist, designed to match the curriculum for radiation oncology resident education as recommended by the American Society for Radiation Oncology (ASTRO) . This exam includes questions on the following topics: basic physics (12 questions), radiation measurements (10 questions), treatment planning (20 questions), imaging modalities and applications in radiotherapy (17 questions), brachytherapy (13 questions), advanced treatment planning and special procedures (16 questions), and safety, quality assurance (QA), and radiation protection (12 questions). Of the 100 questions, 17 require numeric calculation (math-based). The exam questions are listed in the Appendix, Section B.

The 100-question multiple-choice test on radiation oncology physics was inputted to each LLM in 5 separate trials (Trial 1 - Trial 5), except BLOOMZ, which was only tested in one trial. Each trial, beginning on a new thread or after reset, began with an initialization prompt notifying the LLM that it was about to be tested. Next, the LLM was prompted with instructions and 20 questions in batches until the exam was complete. In each trial, the instructions indicated to the LLM that it should only return the correct answer with no justification. The instructions were included in each batch since it was observed that the LLMs were less likely to follow the instructions otherwise. In cases where the LLM could not accept 20 questions at a time, batches of 10 questions were used instead (Bard). In cases where not all the answers were returned by the LLM, the next batch would include the non-answered question(s) as well as the entire next batch. These occurrences were rare. In each test trial, the global prompt and instructions prompt were phrased differently in order to account for response-noise due to prompt-noise. The initialization prompts and instructions prompts are given in Table 1.

LLM test scores and their distributions were compared between each other as well as with scores from two human groups, medical physicists and non-experts. The medical physicists group included four experienced board-certified medical physicists, three medical physics residents, and two medical physics research fellows. The non-expert group included six individuals with advanced degrees in either electrical engineering, computer engineering, or computer science, but with no known prior experience or education on radiation oncology physics. Each human test-taker was allowed 3 hours to take the exam, closed book, also permitting the use of a basic calculator. In comparing the LLM scores and human scores, the mean scores, consistency in scores, and confidence in answers were evaluated.

To quantify the overall consistency of scoring success, the standard deviation and average correlation between trials, defined as the average of the upper values of the Pearson correlation matrix between trials, were calculated. The average correlation indicates how consistent the correct scores were between trials where 1 is interpreted as the distribution being identical, 0 is equivalent to the distribution being purely random, and -1 is interpreted as the distribution being perfectly anti-correlated.

In order to quantify the degree of confidence in the answers given by the LLMs and human groups, the number of correct answers for each question were counted across all trials. For example, if each LLM answered the same question correctly 5 times, then the percentage of questions where all 5 answers were correct was incremented by 1% (since there are 100 questions). Additionally, the test results were compared to the expected distribution that would occur if the test-taker were guessing at random. The expected number of correct answers in 5 trials, when randomly guessing, is approximately $0.25\times 5=1.25$ on average (98/100 questions have 4 choices, 2/100 have 5 choices). Using this value, the number of correct answer occurrences for each question can be estimated following the resultant Poisson distribution.

Finally, ChatGPT (GPT-3.5 and GPT-4) and Bard scores were compared to human scores where the scores were calculated based on majority vote.

2 Improving ChatGPT (GPT-4) accuracy - explain first, then answer

Due to the nature of transformer-based LLMs predicting the next word based on the prior context, it has been shown that the accuracy of responses can be improved if a sufficiently large LLM is prompted to develop the answer in a step-wise manner . ChatGPT (GPT-4) was evaluated using this strategy to see if its score could be improved by prompting it to explain first, then answer. The initialization prompt was the same as in Trial 1, however the instructions prompt for Trial 1 was changed to the following: "Instructions: For each multiple choice question, first give an explanation for the answer followed by the correct answer (letter choice)." These test results were then compared with the original non-justified test results.

3 Testing ChatGPT (GPT-4) on its deductive reasoning ability

In a multiple-choice question, an LLM will be most successful when the question and answer are often used in the same context. However, what happens if the correct answer has no shared context with the question, such as when the answer is "None of the above"? In this case, the LLM must deduce the correct answer by rejecting all the other answers, all of which likely share context with the question. This scenario would seem to be especially challenging for an LLM. To study the deductive reasoning ability of ChatGPT (GPT-4), each question of the 100-question multiple-choice exam was modified. Each correct answer was removed and replaced with "None of the above choices is the correct answer." Such a context-reduction transformation cannot be used on a human since a human would notice the pattern. Because of this, there are likely to be no examples of this sort of transformation to be found for tests that were designed for humans and were subsequently used in the training data for LLMs. It is assumed, then, that an LLM would not notice this pattern. The modified exam was given to ChatGPT (GPT-4) using the Trial 1 prompts and was subsequently tested for improving accuracy by explaining first, then answering as described in Section 3.2.

Results

The raw marks and mean test scores are shown in Figures 1 and 2 respectively, where the LLM mean test scores represent the mean of 5 trials (except for BLOOMZ - 1 trial) and the mean test scores for humans represent the mean of their respective groups (see Section 3.1). Each LLM was able to outperform the non-expert human group overall while only ChatGPT (GPT-4) outperformed the medical physicist group. For math-based questions, the medical physicists outperformed ChatGPT (GPT-4).

As can be observed in the raw marks shown in Figure 1, each LLM and human group showed variability between trials, not only in terms of uncertainty in the overall score, but also in terms of the number of times each question was answered correctly. The standard deviation and average correlation between trials are reported in Figures 3(a) and 3(b). The LLMs were much more consistent in their scores and answers as compared to the human groups, showing both a low standard deviation in scoring and a high average correlation between trials.

From the results shown in 4, Bard slightly outperformed the non-expert group, however both groups performed similarly to a random guesser. ChatGPT (GPT-3.5 and GPT-4) and the medical physicists showed no similarity to random guessing. ChatGPT (GPT-3.5) was either confident, getting 35% of answers correct in each trial, or confused, getting 28% of answers incorrect. ChatGPT (GPT-4) was even more confident, getting 67% of questions correct in each trial, however it also showed a propensity for confusion, getting 14% of questions incorrect in each trial. As a group, the medical physicists were neither extremely confident, nor confused, however tending towards agreement in selecting the correct answers.

Although ChatGPT (GPT-3.5 and GPT-4) scored well overall, their scoring distributions, shown in Figure 4, suggested that if the LLMs could work together, there would be very little improvement in scoring, since they tended to be either confident or confused with low variability. Bard (LaMDA) and the non-expert groups would also likely show little improvement in working together as their answers tended towards random success. However, because medical physicists tended towards agreement on correct answers, it would be expected that their score would improve considerably when working together. To test for this, the answers for each group were combined using a "majority vote". For each question, the most common answer choice was chosen as the group answer. In the case of a tie, one answer among the most common answer choices was chosen randomly. Figure 5 shows the scoring results when utilizing a majority vote. As shown, ChatGPT (GPT-3.5 and GPT-4) improved very slightly, 1%. Bard (LaMDA) and the non-expert group improved by 4% and 3% respectively. However, the medical physicist group improved greatly, by 23%.

2 Improving ChatGPT (GPT-4) accuracy - explain first, then answer

Figure 6 shows the results for having prompted ChatGPT (GPT-4) to explain first, then answer, therefore allowing the answer to develop. ChatGPT’s (GPT-4) overall score improved by 5%, exceeding each prior trial. The greatest improvement was in the brachytherapy and math-based questions categories. These results are in agreement with prior studies that found this capability to be an emergent characteristic for sufficiently large LLMs . Sample responses from ChatGPT (GPT-4) are given in the Appendix, Section A.

3 Testing ChatGPT (GPT-4) on its deductive reasoning ability

Figure 7 shows the results for the deductive reasoning test where the correct answer was replaced by "None of the above choices is the correct answer" in all 100 questions. Overall, ChatGPT (GPT-4) performed much more poorly as compared to the original test. Although the performance was generally worse, the explain first, then answer method was especially important in improving its ability to deductively reason through the questions. Without explaining first, ChatGPT (GPT-4) got 0% of math-based questions correct, which improved to 65% after incorporating the explain first, then answer method, only one question less accurate than the original trial also using the explain first, then answer method.

Discussion

The aim of this study was to evaluate LLMs on a highly-specialized topic, radiation oncology physics, based on a 100-question multiple choice exam that was specifically designed for this study. The exam can be found in the Appendix, Section B. The scoring results from the non-expert group suggest that the general population knows very little about radiation oncology physics as their scores were similar to random guessing. Bard (LaMDA) slightly outperformed the non-experts while BLOOMZ and ChatGPT (GPT-3.5 and GPT-4) greatly outperformed the non-experts. Amazingly, GPT-4 was able to outperform the average medical physicist in nearly all subcategories and improved its answer accuracy when prompted to explain its reasoning before answering (Figures 2 and 6). ChatGPT (GPT-4) showed a surprising ability to deductively reason in answering all 100 questions where each correct answer was modified to be "None of the above choices is the correct answer.", particularly when it was prompted to explain first, then answer, scoring 55% overall. This result is somewhat perplexing and could potentially be an emergent property. Emergent properties are known to occur as the number of parameters is increased in LLMs . This novel method may be a useful method in determining whether deductive reasoning improves with the number of parameters going forward.

While ChatGPT (GPT-4) outperformed medical physicists overall, this study has also provided evidence that individual LLMs cannot compete with a small number of medical physicists working together (Figure 5). The likely reason is that humans vary significantly in capabilities and knowledge from individual to individual, even when their professional backgrounds are similar. Additionally, while an answer in a multiple-choice question will either be correct or incorrect, the scoring count distributions shown in Figure 4 indicated that the medical physicists were far less likely to be confused, which, when aggregated over the whole group of medical physicists, allowed them to select the correct answer at a much higher rate in a majority vote. When ChatGPT (GPT-3.5 and GPT-4) was wrong, it was confidently wrong (confused). Similarly, when it was correct, it was confidently correct. Our results indicated that humans with expertise on a highly-specialized topic knew when to guess, how to guess intelligently, and were less likely to be wrong in their reasoning, even when the correct answer was not chosen. This comparison may not be completely fair as it is possible that if the exact same human could be tested repeatedly in the same manner as ChatGPT (GPT-3.5 and GPT-4), they might also repeat answers and show a degree of confusion individually. That point is arguably irrelevant, however, as there are many experienced medical physicists and only few LLMs as capable as GPT-4. The high degree of consistency in correct and incorrect answers for ChatGPT (GPT-3.5 and GPT-4) may be a sign of over-fitting (or memorization) in regards to radiation oncology physics knowledge. Regardless, being that radiation oncology physics is a highly-specialized topic, the performance of ChatGPT (GPT-4) was extraordinary and will likely continue to improve in the near-future. Practically speaking, this study suggests a great potential for radiation oncology experts to work alongside ChatGPT (GPT-4), using it as a highly knowledgeable assistant.

A weakness in evaluating LLMs using exams such as the one presented in this study is that this exam is not representative of the detailed and nuanced daily clinical work being performed by medical physicists and radiation oncology specialists. The relative performance between LLMs and medical physicists on radiation oncology physics exams reported in this study may therefore misrepresent the degree of equivalency between LLMs and individual medical physicists. Furthermore, GPT-4’s high performance on this certification-like exam, covering a highly specialized topic, suggests a degree of superficiality in the knowledge being assessed. Otherwise, we would have to entertain the possibility of GPT-4 being competent enough to fulfill the role of a medical physicist, which seems highly improbable. The radiation oncology community, and possibly the wider medical community, may therefore need to reevaluate certification procedures, as the necessity for humans to invest significant effort in acquiring such superficial knowledge will diminish as LLMs continue to advance. With this in mind, LLMs could potentially be used as a test for superficiality. Perhaps a greater focus on knowledge not known by the LLM should be more greatly emphasized.

This study is a continuation of a line of research that applies state-of-the-art NLP methods to radiation oncology. For example, Rezayi et al. trained BioBERT on a large corpus of radiation oncology literature and a curated and anonymized text dataset from a hospital to build ClinicalRadioBERT, a specialized language model for radiation oncology. Liao et al. proposed a framework of directing the attention of transformer-based language models to more important input tokens that significantly affect classification decisions. This method is particularly important for few-shot learning with few annotated samples, which is a common challenge in radiation oncology where it is difficult to collect and curate large amounts of multi-institution patient data that match certain requirements due to the concern of patient privacy. On a related note, ChatGPT has demonstrated superior performance as an effective text data augmentation approach over state-of-the-art text data augmentation methods in terms of testing accuracy and distribution of the augmented samples , which can also be used to address the few-shot learning challenge.

In addition, LLMs can be employed for innovative applications such as data de-identification. For example, GPT-4 outperforms ChatGPT and other language model competitors in de-identifying clinical notes with a 99% accuracy . This is of extreme importance to radiation oncology and all medicine specialities in general, since it is often cumbersome to anonymize data for cross-institution clinical collaboration and academic research. Some other applications of language models include building domain-specific knowledge graphs for oncology without manual annotation from clinicians or other domain experts.

2 Multi-modal models in radiation oncology

Multi-modal models are the future of language models and are important in medical diagnosis . Some early LLM studies with multi-modal data include ChatCAD , a framework to integrate images and texts for computer-aided diagnosis. It supports various diagnosis networks such as those for lesion segmentation and report generation. In this framework, ChatGPT can be used to enhance the outputs of these networks.

GPT-4 supports multi-modal inputs such as images, which further unlocks the potential of large language models in radiation oncology. It is necessary to investigate future models and applications that integrate text, images, dosimetric data, and other modalities into the diagnosis and treatment pipelines. We believe such multi-modal models display inherent affinity to the human brain and future LLM models for medicine can receive inspirations from advances in both neuroscience and NLP.

Conclusion

This study demonstrated that LLMs, particularly ChatGPT (GPT-4), can excel in answering questions on a highly-specialized topic like radiation oncology physics as well as exhibit surprising capabilities in deductive reasoning. While GPT-4 outperformed individual medical physicists on average, a small group of medical physicists working together still achieved better results, demonstrating the importance of diversity in knowledge and expertise among human professionals - and perhaps for LLMs as well. This further highlights the potential benefits of combining LLMs’ capabilities with human insight and experience to enhance knowledge and decision-making. Overall, the findings suggest a promising future for LLMs in specialized fields.

Acknowledgments

We would like to thank the many individuals (Yuzhen Ding, Yiran Cui, Yonghui Fan, Yikang Li, Riti Paul, Nupur Thakur, Sachin Chhabra, Chenbin Liu, Yang Li, Man Zhao, Yuchao Hu, Shumei Jia, Lian Zhang, Yao Xu, Hongying Feng, and Yunze Yang) that volunteered to take the radiation oncology physics exam.

References

Appendix A Explain first, then answer responses from ChatGPT (GPT-4)

In order to show how the explain first, then answer prompt works in ChatGPT (GPT-4), we selected 10 questions here for demonstration, starting from the 61${}^{\text{st}}$ question of the 100-question found in the Appendix, Section B.

1 Curie (Ci) is equal to $3.7\times 10^{10}$ Becquerels (Bq). Therefore, 10 Ci of 192Ir is equivalent to $37\times 10^{10}$ Bq or 370 GBq. Answer: C. Its decay rate is 37 GBq.

The equation for effective half-life ($T_{\text{eff}}$) is $\frac{1}{T_{\text{eff}}}=\frac{1}{T_{\text{phys}}}+\frac{1}{T_{\text{bio}}}$, where $T_{\text{phys}}$ is the physical half-life and $T_{\text{bio}}$ is the biological half-life. Solving for $T_{\text{bio}}$ gives $T_{\text{bio}}=\left(\frac{1}{T_{\text{eff}}}-\frac{1}{T_{\text{phys}}}\right)^{-1}=\left(\frac{1}{2}-\frac{1}{6}\right)^{-1}=3\,\text{hours}$. Answer: D. 3 hours

The initial dose rate ($D_{0}$) can be calculated using the formula $D_{0}=\frac{D_{\text{total}}}{(T\times\ln(2))}$, where $D_{\text{total}}$ is the total dose and $T$ is the half-life of the isotope. For 125I, the half-life is approximately 60 days. $D_{0}=\frac{120\,\text{Gy}}{(60\,\text{days}\times\ln(2))}\approx 5.9\,\text{cGy/h}$. Answer: B. 5.9 cGy/h

According to AAPM TG-43, 125I has the greatest anisotropy correction among the listed isotopes. Answer: D. 125I

AAPM TG-43 does consider the absorption and scattering in the source, so this statement is not a limitation of the protocol. Answer: A. The absorption and scattering in the source is not considered.

The radial dose function, $g(r)$, is approximately constant for a 192Ir source for values of $r$ less than 5 cm because the attenuation is compensated by the increased scatter. Answer: C. Attenuation is compensated by the increased scatter.

The dose rate at 5 mm (0.5 cm) from the cylinder surface is 700 cGy. The dose at the cylinder surface (r = 1.5 cm) can be estimated using the inverse square law: $\text{Dose}_{\text{surface}}=\text{Dose}_{\text{5mm}}\times\left(\frac{5\,\text{mm}}{15\,\text{mm}}\right)^{2}\approx 700\,\text{cGy}\times\left(\frac{1}{9}\right)=77.8\,\text{cGy}$. Answer: D. 500 cGy (closest to the calculated value)

To get a uniform dose at 5 mm from the cylinder surface in HDR vaginal treatment, the source dwell times are longer at the ends of the cylinder. Answer: B. are longer at the ends.

The prescription dose for permanent prostate implant seeds is lower when using 103Pd as opposed to 125I because the half-life for 103Pd is shorter. Answer: A. The half-life for 103Pd is shorter.

Source calibration is not performed for HDR daily QA, as it is typically done less frequently (e.g., monthly or quarterly). Answer: D. Source calibration.

Appendix B 100-question multiple-choice exam on radiation oncology physics

Correct answers = C, A, B, A, B, D, B, A, C, A, D, A, D, C, A, A, E, C, C, D, D, C, A, A, D, C, B, D, C, A, D, D, A, B, C, A, D, C, A, C, D, C, A, C, D, B, A, B, C, B, B, D, C, A, C, A, C, B, C, C, A, D, B, D, A, C, B, B, A, D, C, D, C, C, A, B, D, C, A, B, C, D, C, A, A, B, C, A, D, C, B, C, A, C, A, A, C, D, C, B

Which of the following particles cannot be accelerated by an electric field?

The mass of Carbon-12 (12C) is than the sum of the masses of 6 protons and 6 neutrons.

The binding energies for tungsten’s K, L, and M shells are 70, 11, and 2 keV, respectively. A 50 keV photon interacts with tungsten through the photoelectric effect. What are the possible energies of the emitted photoelectron?

a continuous spectrum from 2 keV to 70 keV

Which of the following statements is true regarding the x-ray spectrum of an x-ray tube?

The maximum photon energy is determined by the kVp.

The maximum photon energy is determined by the target material.

The characteristic photon energies are determined by the tube current.

The characteristic photon energies are determined by the filtration.

The half value layer (HVL) for an x-ray beam is 2 mm aluminum. What is the minimum thickness of aluminum required to reduce the x-ray exposure by 50%?

Which of the following component provides the high-power RF fields to accelerate electrons in a LINAC?

What is the purpose of the “tongue and groove” design of multileaf collimator (MLC)?

When a LINAC is used to treat patients in electron mode, which of the following hardware is not needed?

Compared to a flattened beam, the flattening filter free beam .

The mass attenuation coefficient has the highest dependence on the atomic number of the medium and the energy of the incoming photon energy for .

Which of the following is not an interaction between photons and a medium?

When the energy of the scattered photon is the minimum in a Compton interaction, what is the angle of the emitted Compton electron relative to the direction of the initial photon?

What is the relationship between the mass attenuation coefficient $\mu/\rho$, mass energy transfer coefficient $(\mu_{\text{tr}}/\rho)$, and the mass energy absorption coefficient $(\mu_{\text{en}}/\rho)$?

$\mu_{\text{tr}}/\rho\geq\mu_{\text{en}}/\rho\geq\mu/\rho$

$\mu_{\text{en}}/\rho\geq\mu_{\text{tr}}/\rho\geq\mu/\rho$

$\mu/\rho\geq\mu_{\text{en}}/\rho\geq\mu_{\text{tr}}/\rho$

$\mu/\rho\geq\mu_{\text{tr}}/\rho\geq\mu_{\text{en}}/\rho$

How does the KERMA compare to the absorbed dose for a 15 MV photon beam at 10 cm depth in a water phantom?

The KERMA is greater than the absorbed dose.

The KERMA is less than the absorbed dose.

The relationship between KERMA and the absorbed dose can’t be determined.

Which of the following beams will not produce a buildup effect in the percent depth dose curve?

Which device below is the most suitable for measuring lateral beam profiles?

Four exposed films have the same optical density of 1.25. If all four films are placed on top of each other, what is the fraction of light that can transmit through?

Which of the following devices is not suitable for in-vivo dose measurements?

Optically stimulated luminescence dosimeters (OSLDs)

According to TG-51, which of the following correction factors is not needed when converting the measured charge to the generated charge in the ionization chamber?

For a farmer chamber calibrated at the standard temperature and pressure, which is the PTP factor for a local temperature of 23∘C and pressure of 720 mmHg?

How is the beam quality defined in TG-51?

According to TG-51, a LINAC should have dose calibration in water tank .

Why is TMR always larger than PDD at depths larger than dmax?

PDD includes both beam attenuation and beam divergence.

PDD includes beam attenuation but does not include beam divergence.

TMR includes both beam attenuation and beam divergence.

TMR does not include beam attenuation but does include beam divergence.

Which of the following changes will result in a larger geometric penumbra?

Increasing the source to collimator distance.

Move the patient closer to the collimator.

The output of a 6 MV beam is 1 cGy/MU at SSD=100 cm, depth=1.5 cm and field size 10 cm x10 cm. What is the output at SSD=98.5 cm, depth=1.5 cm and field size 10 cm x10 cm?

For a single field SAD treatment, the output at dmax is 1.02 cGy and the TMR at depth 10 cm is 0.7. What is the MU needed to delivery 100 cGy at depth 10 cm?

What wedge angle will produce the most uniform dose distribution for two treatment fields separated by 100∘?

A patient’s spine is treated by two adjacent fields with SSD=100 cm. The field lengths (symmetrical fields) in the superior-inferior directions are 20 cm and 24 cm, respectively. What is the gap in the skin to match the two spine fields at depth 10 cm?

For a craniospinal treatment treated with symmetrical fields, the field length for the upper spine field is 25 cm, and the field size for the lateral cranial fields are 20 cm x 20 cm. What is the collimator rotation angle for the cranial fields to align with the divergence of the spine fields?

The dose-volume-histogram (DVH) contains the following information except:

What dose D5%=20 Gy to the heart on a DVH curve means?

5% of the heart receives less than 20 Gy.

95% of the heart receives more than 20 Gy.

95% of the heart receives less than 20 Gy.

Which of the following parameters is sometimes changed by the dosimetrist during treatment planning?

For electron radiation therapy, which electron energy has the minimum PDD on the surface?

Electron PDD has a low dose tail after the practical range. Which physical process is related to the generation of this tail?

Which of the following treatments is suitable for SSD setup?

Electron boost to chest wall of breast cancer

Which of the following statements is true for electron effective SSD?

It increases with increasing electron energy.

It is equal to the distance from the x-ray sources to the isocenter.

It is larger than the distance from the x-ray sources to the isocenter.

What is the range of a 6 MeV electron beam in the lung (0.25 g/cm3)?

Choosing appropriate beam modifiers such as wedges and compensators.

Optimizing beam fluence based on the desired dose constraints.

Adding beam block to protect organs at risk.

What is the purpose of the leaf sequencing algorithm in IMRT planning?

It converts optimized fluence to the achievable fluence by MLC.

It converts cost functions to the optimized fluence.

It converts desirable constraints to cost functions.

The dose gradient achievable in an IMRT plan is approximately .

Which of the following devices is not appropriate for IMRT patient QA?

Which parameter below is not typically set by a dosimetrist during IMRT treatment planning?

For material with HU=10, what is the ratio of linear attenuation coefficients between this material to water?

Which of the following methods can improve the spatial resolution of CT?

Why can’t MR images replace CT for accurate dose calculations in radiation therapy?

MR images have inferior soft tissue contrast than CT.

MR images have worse spatial resolution than CT.

MR images deliver more ionization doses to the patient than CT.

MR images do not have information about electron density.

The velocity of ultrasound in soft tissue is 1500 m/s. The time to receive a signal after sending a pulse is 0.1 ms. What is the depth of the anatomical interface?

Which of the following imaging modalities utilizes non-ionizing radiation?

Which of the following statements is true about the spatial resolution of PET images?

PET images spatial resolution is better than CT and MRI.

PET images spatial resolution increases when the average energy of the emitted positrons decrease.

PET images spatial resolution increases when the number of detectors in the ring decreases.

PET images spatial resolution is better than 1 mm.

Which information below is not stored in a DICOM RT plan file?

A 4D CT has 10 phases, 200 slices in each phase. Each slice has 512x512 pixels, where each pixel uses 2 bytes. How long does it take to transfer this 4D CT over the internet with a transfer speed of 2 GB/second?

As compared to diagnostic CT, which statement below is not true for CT simulator?

CT simulator usually has a larger bore size than diagnostic CT.

CT simulator gives better image spatial resolution and contrast than diagnostic CT.

CT simulator needs extra laser for localizing the treatment isocenter.

CT simulator needs accurate geometry and HU for the whole body.

Which techniques below do not result in a reduced setup margin for planning target volume (PTV)?

To get the best soft tissue contrast for a head and neck cancer treatment, which IGRT method is preferred?

Which of the following IGRT methods is not used for IMRT treatment?

Implanted fiducial markers and fluoroscopy imaging.

Which of the following is not an advantage of free-breathing gating for treating a lung tumor?

Patient is comfortable in free breathing.

is not a method for management of respiratory motion in radiation therapy.

The half-life for a radioactive nuclide is 69 days. How much does it decay in one day?

Which statement is correct for an HDR source, 192Ir, with activity 10 Ci?

Its decay rate is equivalent to about 10 grams of 226Ra.

Its decay rate is equivalent to about 1 gram of 226Ra.

The physical and effective half-lives for an unsealed isotope are 6 hours and 2 hours, respectively. What is the biological half-life?

What is the initial dose rate for a prostate 125I seed implant to deliver a total dose of 120 Gy?

According to AAPM TG-43, has the greatest anisotropy correction.

Which of the following is not a limitation of AAPM TG-43?

The absorption and scattering in the source is not considered.

The dose delivered while the source is in transit is usually ignored.

Why is the radial dose function, g(r), approximately constant for an 192Ir source for values of r less than 5 cm?

There is no attenuation for the first 5 cm.

Attenuation is included in the geometry factor.

Attenuation is compensated by the increased scatter.

Attenuation is included in the anisotropy factor.

An HDR vaginal cylinder case with a 3 cm diameter is planned to deliver 700 cGy to vaginal tissue at 5 mm from the cylinder surface. What is the approximate dose to the surface of the cylinder?

To get a uniform dose at 5 mm from the cylinder surface in HDR vaginal treatment, the source dwell times .

are longer at the superior end than the inferior end.

Why is the prescription dose for permanent prostate implant seeds lower when using 103Pd as opposed to 125I?

The energy of emitted photon for 103Pd is larger.

The radial dose function g(r) for 103Pd is larger.

The anisotropy function for 103Pd is larger.

is the most appropriate device to find a lost radionuclide seed in the operation room.

Which of the following tests is not a typical procedure for HDR brachytherapy?

Calibrating the source when a new source is used.

Changing the source at a frequency of 3-4 months.

Performing daily QA in the days only when patient treatment is scheduled.

Performing patient surveys only before the treatment.

Which statement below is true for TBI treatment?

Lung block is always needed when lateral opposed beams are used.

FFF mode is used to increase the dose rate.

The extended SSD increases the dose uniformity.

The treatment is faster than it is for regular patients.

The maximum field size for a LINAC is 40 cm x 40 cm, and the height of a TBI patient is 180 cm. What is the minimum SSD required for the TBI treatment with a collimator angle of 0o?

In total skin electron therapy (TSET), why is the gantry angle about 20o such that the central axis is above patient’s head and below patient’s feet?

To minimize the dose from x-ray contaminations.

To maximize the dose from x-ray contaminations.

is not an appropriate delivery method for intraoperative radiotherapy (IORT).

Which of the following statements correctly describe the lateral penumbras of therapeutic proton beams?

They are always larger than penumbras of MV x-rays in LINAC.

They are always smaller than penumbras of MV x-rays in LINAC.

is not needed for a dedicated nozzle for proton pencil beam scanning.

Which of the following statements is correct regarding range straggling in proton therapy?

It is due to the statistical uncertainty in energy loss.

It can be minimized by robust optimization.

Why do proton machines require a large gantry to bend protons, but LINACs don’t?

Protons at 250 MeV have a higher speed than electrons at 6-20 MeV.

The mass of protons is about 1800 times more than the mass of electrons.

Protons have positive charge while electrons have negative charge.

Protons do not generate x-rays by Bremsstrahlung radiation.

Which of the following statements is incorrect regarding intensity-modulated proton therapy (IMPT)?

An IMPT plan can be robust with robustness optimization.

IMPT is more conformal than double scattering.

IMPT is less sensitive to motion than double scattering.

IMPT by pencil beam scanning needs less hardware than double scattering.

Which of the following statements about RBE for protons is correct?

It is always a constant 1.1 for all depths.

It is the highest at the center of spread-out Bragg peak.

It is the highest at the distal fall off depth.

Which device is not appropriate to measure the lateral profile of small fields in SRS?

What is the purpose of using tertiary collimators in SRS technique?

To merge multiple isocenters to a single isocenter.

Which of the following statements is not true about SRS?

A flattening filter free (FFF) can’t be used in SRS technique.

A Winston-Lutz test is required to verify the coincidence between radiation and mechanical isocenters.

The SRS field may not have lateral charge particle equilibrium (CPE).

Both frame-based and frame-less systems are feasible for SRS.

When planning for SRS/SBRT, which of the following statements is not typically a priority?

Gradient index to be as small as possible.

Compared to regular fractionated treatment, is not required for SRS technique.

a higher precision in the coincidence of beam and gantry/couch isocenters.

Which SRS delivery technique can treat two brain metastases simultaneously using a single isocenter treatment plan.

According to TG-142, which test does not need to be performed daily?

Light field and radiation field coincidence

Which of the following parameters is the most important for cone beam CT (CBCT) used in IGRT?

Coincidence between the CBCT isocenter and the radiation beam isocenter

According to TG-142, which is the tolerance for monthly output constancy?

The quality assurance test for is not addressed in TG-142.

Which of the following parameters is not needed in shielding calculations for the 2nd barriers?

What is the occupancy factor used in shielding design for the nurse station next to a LINAC room?

The proper order of materials for LINAC treatment room doors (from inside to outside) is .

How many TVLs are needed to reduce the exposure rate to 1/5 of the original value?

has the highest radiation weighting factor.

Which of the following errors is not considered a Radiation Medical Event for a 5-fraction SBRT prostate treatment?

Which of the following factors is not needed for a FMEA process map in CT simulation of lung cancer?

Documenting immobilization devices used in CT sim.

Verifying HU accuracy of CT Sim in daily QA.