While there are currently multiple active project in our laboratory, they all have a common theme which is utilization of machine learning, and particularly deep learning, and computer vision for improvement of medical practice and education with focus on medical imaging. The scope of our research includes applications to cancer (breast, brain, thyroid) and orthopaedics, as well as more basic research on training and evaluation of convolutional neural networks and other machine learning models. The application projects have been performed in close collaboration with radiologists, oncologists, surgeons, and other colleagues in clinical departments.

Radiogenomics and prognosis of outcomes in breast cancer based on MRI using radiomics and deep learning


Segmentations of breast lesions using convolutional neural networks.

Magnetic resonance imaging shows a high potential for various cancer-related problems including prognosis of outcomes, identification of tumor genomics, and prognosis. We used various computer vision, traditional machine learning, and deep learning tools to make a step toward reaching this potential. We have approached various problems related to the us of MRI in breast cancer. On the radiogenomic front, we were among the first groups to discover that gene expression-based intrinsic subtype, is related to enhancement dynamics in breast cancer (Mazurowski et al., Radiology 2014). We confirmed the association of imaging with genomics/pathology in multiple follow up studies inluding a radiogenomic study of 922 patient and 529 features (Saha et al., BJC 2018). Furthermore, we showed that MR imaging could be a predictor of patient outcomes and specifically distant recurrence-free survival (Mazurowski et al., 2019). Other addressed problems include prediction of Oncotype DX status (Saha et al., JCRCO 2018), response to neoadjuvant therapy (Cain et al. BCRC 2019), upstaging in DCIS (Harowicz et al., JMRI 2017, and prediction of risk of cancer in normal patients (Grimm et al., AR 2018).


Assessment of knee osteoarhritis severity based on knee radiographs.

Knee osteoarhritis is a condition of the knee where the spacing between the bones decreases due to wearing out of cartilage (the tissue in the spacing). It is a common cause of knee pain. Precise determination of the extent of knee osteoarthritis is cruacial in order to make treatment decisions. We are developing and evaluating deep learning-based software that makes this determination.

The figure shows results of deep learning-based automated detection of knee joints and segmentation of the bones.

Brain tumors: prognosis of outcomes and radiogenomics using computer vision and deep learning methods

The figure shows tumor segmentation generated by a human observer (reference standard) in the first column, tumor segmentation generated by our algorithm in the second column, and the original image in the third column. The two rows correspond to two different MRI slices.
This figure shows the response of a random forest classifier that was used for segmentation of the magnetic resonance sequences into 5 classess: 4 tumor class and 1 normal class. In each of the images above, red corresponds to a high likelihood of a pixel being in the class represented in that image. Class 1 is normal tissue and the remaining classes are different compartments of the tumor (e.g., class 2 is enhancing tumor and class 3 is necrosis).

Glioblastoma (GBM) and low grade gliomas (LGG) are common primary brain tumors associated with a significant mortality. We have developed computer vision tools for analysis of brain MRIs. In one of our early study (Mazurowski et al., Neuro-Oncology 2013), we demonstrated that imaging features extracted from standard MR images by radiologists improve predictive accuracy of survival models as compared to clinical features only. While the features extracted by radiologists are useful in survival prediction, the burden of assigning a set of features for every patients is significant. Furthermore, interobserver variability exists between the radiologists in terms of the assigned features.

Investigating the impact of class imbalance on training and evaluation of machine/deep learning models

Class imbalance is a property of data where there is a significantly diferent number of cases from different classess. This is a very common occurence in machine learning. An example is cancer screening data where the cancer cases may constitute 1% or even less of all data while normal cases may constitute 99% of the data.

In our work we examined the impact of class imbalance on training traditional perceptrons as well as on training of modern convolutional neural networks. This resulted in two of the most cited papers in the journal Neural Networks in their respective years (Mazurowski et al., Neural Networks 2008), (Buda, Maki, Mazurowski, Neural Networks 2018).

The figure shows performance curves for different methods addressing class imbalance given a different number of minority classes (out of 10). The top rows (a-c) show results for MNIST and the bottom rows (d-f) show results for CIFAR-10.


Harmonization of medical imaging data using deep learning

Medical images of one patient acquired using different equipment or acquisition parameters may have a very different appearence. This is a challenge when visually examining the image, performing a quantitative analysis of the images (e.g. radiomics) or training and evaluating deep learning models. It is a common knowledge, corraborated with one of our recent studies (AlBadawy et al., 2018), that when an algorithm is trained using data from one institution and tested on data from another institution, performance may decrease.

We developed two methods to harmonize breast MRI data. The first methods uses deep learning degmentation of parts of a breast MR image and then a peacewise linear pixel transformation in order to bring the pixel intensities to a common scale where the same tissue types are represented by the same pixel values in different images. The methods is described in (Zhang et al, 2018) and the code is availble at

More recently, we developed a harmonization method that uses cycle-consistent generative adversarial networks. Similarly as generating oil paintings from photographs, the method is capable of transforming images from one vendor/scanner to appear as if they were acquired using a scanner from a different vendor. We proposed some technical innovations in order to address limitations of Cycle GANs. The method is described in (Modanwal et al., 2019).

The figure shows MRI scans acquired using scanners (a: GE, b: Siemens) from different vendors illustrating both different intensity and texture.


Adaptive computer-aided education in radiology

Diagram showing an overview of our adaptive educational system for radiology.

In this project, we aim to apply machine learning, computer vision, and recommender systems algorithms to improve education in radiology with focus on resident training. Specifically, we hypothesize that if challenging imaging cases can be identified for each trainee individually before they are seen and those cases are presented to the trainee instead of randomly selected cases, the educational outcomes will be improved. Toward this goal, we have constructed user models that use previous interpretations made by each trainee and are able to capture their strengths and weaknesses and predict challenging cases. For this purpose, we utilize both human-assigned features of images as well as features extracted automatically using computer visions algorithms. Those features are used by a machine learning algorithm trained on prior interpretations of a given radiologist-in-training in order to predict future cases that would be challeing for the trainee.

Our research workstation for reader studies
Our research workstation for reader studies

Within this direction of research, we also systematically investigated (through reader studies and statistical analysis) related aspect of trainee behavior such as the impact of perceiving a case as difficult on interpretation and error making.

The specific focus of this direction of our research shifted from mammography in the early stages of this project toward digital breast tomosynthesis in more recent work. Identifying eficient ways of educating radiologists to interpret digital breast tomosynthesis exams is of high importance due to the rapid shift of breast cancer screening toward this relatively new modality.