While there are currently multiple active project in our laboratory, they all have a common theme which is utilization of machine learning, and particularly deep learning, and computer vision for improvement of medical practice and education with focus on medical imaging. The scope of our research includes applications to cancer (breast, brain, thyroid) and orthopaedics, as well as more basic research on training and evaluation of convolutional neural networks and other machine learning models. The application projects have been performed in close collaboration with radiologists, oncologists, surgeons, and other colleagues in clinical departments.
Radiogenomics and prognosis of outcomes in breast cancer based on MRI using radiomics and deep learning
Magnetic resonance imaging shows a high potential for various cancer-related problems including prognosis of outcomes, identification of tumor genomics, and prognosis. We used various computer vision, traditional machine learning, and deep learning tools to make a step toward reaching this potential. We have approached various problems related to the us of MRI in breast cancer. On the radiogenomic front, we were among the first groups to discover that gene expression-based intrinsic subtype, is related to enhancement dynamics in breast cancer (Mazurowski et al., Radiology 2014). We confirmed the association of imaging with genomics/pathology in multiple follow up studies inluding a radiogenomic study of 922 patient and 529 features (Saha et al., BJC 2018). Furthermore, we showed that MR imaging could be a predictor of patient outcomes and specifically distant recurrence-free survival (Mazurowski et al., 2019). Other addressed problems include prediction of Oncotype DX status (Saha et al., JCRCO 2018), response to neoadjuvant therapy (Cain et al. BCRC 2019), upstaging in DCIS (Harowicz et al., JMRI 2017, and prediction of risk of cancer in normal patients (Grimm et al., AR 2018).
- J. Zhang, A. Saha, Z. Zhu, M. A. Mazurowski, Hierarchical Convolutional Neural Networks for Segmentation of Breast Tumors in MRI with Application to Radiogenomics, IEEE Transactions on Medical Imaging 38, 435-447 (2019)
- Z. Zhu, E. Albadawy, A. Saha, J. Zhang, M. R. Harowicz, M. A. Mazurowski, Deep Learning for identifying radiogenomic associations in breast cancer, Computers in biology and medicine 109, 85-90 (2019)
- M. A. Mazurowski, A. Saha, M. R. Harowicz, E. H. Cain, J. R. Marks, P. K. Marcom, Association of distant recurrence‐free survival with algorithmically extracted MRI characteristics in breast cancer, Journal of Magnetic Resonance Imaging 49, 231-240 (2019)
- A. Saha, M. R. Harowicz, L. J. Grimm, C. E. Kim, S. V. Ghate, R. Walsh, M. A. Mazurowski, A machine learning approach to radiogenomics of breast cancer: a study of 922 subjects and 529 DCE-MRI features, British journal of cancer 119, 508-516 (2018)
Assessment of knee osteoarhritis severity based on knee radiographs.
Knee osteoarhritis is a condition of the knee where the spacing between the bones decreases due to wearing out of cartilage (the tissue in the spacing). It is a common cause of knee pain. Precise determination of the extent of knee osteoarthritis is cruacial in order to make treatment decisions. We are developing and evaluating deep learning-based software that makes this determination.
Brain tumors: prognosis of outcomes and radiogenomics using computer vision and deep learning methods
Glioblastoma (GBM) and low grade gliomas (LGG) are common primary brain tumors associated with a significant mortality. We have developed computer vision tools for analysis of brain MRIs. In one of our early study (Mazurowski et al., Neuro-Oncology 2013), we demonstrated that imaging features extracted from standard MR images by radiologists improve predictive accuracy of survival models as compared to clinical features only. While the features extracted by radiologists are useful in survival prediction, the burden of assigning a set of features for every patients is significant. Furthermore, interobserver variability exists between the radiologists in terms of the assigned features.
In response to this limitation, we have developed a set of tools for automated analysis of brain tumors including segmentation (Buda et al., CBM 2019), shape analysis (Czarnek et al. JNO 2078), and deep learning-based classification (Buda et. al, Radiology AI 2020).
- M. Buda, A. Saha, M. A. Mazurowski, Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm, Computers in biology and medicine 109, 218-225 (2019)
- N. Czarnek, K. Clark, K. B. Peters, M. A. Mazurowski, Algorithmic three-dimensional analysis of tumor shape in MRI improves prognosis of survival in glioblastoma: a multi-institutional study, Journal of neuro-oncology, vol. 132, pp. 55-62, March 2017
- M. A. Mazurowski K. Clark, N. M. Czarnek, P. Shamsesfandabadi, K. B. Peters, A. Saha, Radiogenomics of lower-grade glioma: algorithmically-assessed tumor shape is associated with tumor genomic subtypes and patient outcomes in a multi-institutional study with The Cancer Genome Atlas data, Journal of Neuro-Oncology, vol. 133, pp. 27-35, May 2017
Investigating the impact of class imbalance on training and evaluation of machine/deep learning models
Class imbalance is a property of data where there is a significantly diferent number of cases from different classess. This is a very common occurence in machine learning. An example is cancer screening data where the cancer cases may constitute 1% or even less of all data while normal cases may constitute 99% of the data.
In our work we examined the impact of class imbalance on training traditional perceptrons as well as on training of modern convolutional neural networks. This resulted in two of the most cited papers in the journal Neural Networks in their respective years (Mazurowski et al., Neural Networks 2008), (Buda, Maki, Mazurowski, Neural Networks 2018).
- M. Buda, A. Maki, M. A. Mazurowski, A systematic study of the class imbalance problem in convolutional neural networks, Neural Networks 106, 249-259 (2018)
- M. A. Mazurowski, P. A. Habas, J. M. Zurada, J. Y. Lo, J. A. Baker, and G. D. Tourassi, Training Neural Network Classifiers for Medical Decision Making: The Effects of Imbalanced Datasets on Classification Performance, Neural Networks, vol. 21, pp. 427-436, March-April 2008
Harmonization of medical imaging data using deep learning
Medical images of one patient acquired using different equipment or acquisition parameters may have a very different appearence. This is a challenge when visually examining the image, performing a quantitative analysis of the images (e.g. radiomics) or training and evaluating deep learning models. It is a common knowledge, corraborated with one of our recent studies (AlBadawy et al., 2018), that when an algorithm is trained using data from one institution and tested on data from another institution, performance may decrease.
We developed two methods to harmonize breast MRI data. The first methods uses deep learning degmentation of parts of a breast MR image and then a peacewise linear pixel transformation in order to bring the pixel intensities to a common scale where the same tissue types are represented by the same pixel values in different images. The methods is described in (Zhang et al, 2018) and the code is availble at https://github.com/MaciejMazurowski/breast-mri-normalization.
More recently, we developed a harmonization method that uses cycle-consistent generative adversarial networks. Similarly as generating oil paintings from photographs, the method is capable of transforming images from one vendor/scanner to appear as if they were acquired using a scanner from a different vendor. We proposed some technical innovations in order to address limitations of Cycle GANs. The method is described in (Modanwal et al., 2019).
- E. A. AlBadawy, A. Saha, M. A. Mazurowski, Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing, Medical Physics 45, 1150-1158 (2018)
- J. Zhang, A. Saha, B. J. Soher, M. A. Mazurowski, Automatic deep learning-based normalization of breast dynamic contrast-enhanced magnetic resonance images, arXiv (2018)
- G. Modanwal, A. Vellal, M. A. Mazurowski, Normalization of breast MRIs using Cycle-Consistent Generative Adversarial Networks, arXiv (2019)
Adaptive computer-aided education in radiology
In this project, we aim to apply machine learning, computer vision, and recommender systems algorithms to improve education in radiology with focus on resident training. Specifically, we hypothesize that if challenging imaging cases can be identified for each trainee individually before they are seen and those cases are presented to the trainee instead of randomly selected cases, the educational outcomes will be improved. Toward this goal, we have constructed user models that use previous interpretations made by each trainee and are able to capture their strengths and weaknesses and predict challenging cases. For this purpose, we utilize both human-assigned features of images as well as features extracted automatically using computer visions algorithms. Those features are used by a machine learning algorithm trained on prior interpretations of a given radiologist-in-training in order to predict future cases that would be challeing for the trainee.
Within this direction of research, we also systematically investigated (through reader studies and statistical analysis) related aspect of trainee behavior such as the impact of perceiving a case as difficult on interpretation and error making.
The specific focus of this direction of our research shifted from mammography in the early stages of this project toward digital breast tomosynthesis in more recent work. Identifying eficient ways of educating radiologists to interpret digital breast tomosynthesis exams is of high importance due to the rapid shift of breast cancer screening toward this relatively new modality.