Jun Kong

BackgroundPathological complete response (pCR) is associated with favorable prognosis in patients with triple-negative breast cancer (TNBC). However, only 30–40% of TNBC patients treated with neoadjuvant chemotherapy (NAC) show pCR, while the remaining 60–70% show residual disease (RD). The role of the tumor microenvironment in NAC response in patients with TNBC remains unclear. In this study, we developed a machine learning-based two-step pipeline to distinguish between various histological components in hematoxylin and eosin (H&E)-stained whole slide images (WSIs) of TNBC tissue biopsies and to identify histological features that can predict NAC response.MethodsH&E-stained WSIs of treatment-naïve biopsies from 85 patients (51 with pCR and 34 with RD) of the model development cohort and 79 patients (41 with pCR and 38 with RD) of the validation cohort were separated through a …

In digital pathology, the multiple instance learning (MIL) strategy is widely used in the weakly supervised histopathology whole slide image (WSI) classification task where giga-pixel WSIs are only labeled at the slide level. However, existing attention-based MIL approaches often overlook contextual information and intrinsic spatial relationships between neighboring tissue tiles, while graph-based MIL frameworks have limited power to recognize the long-range dependencies. In this paper, we introduce the integrative graph-transformer framework that simultaneously captures the context-aware relational features and global WSI representations through a novel Graph Transformer Integration (GTI) block. Specifically, each GTI block consists of a Graph Convolutional Network (GCN) layer modeling neighboring relations at the local instance level and an efficient global attention model capturing comprehensive global information from extensive feature embeddings. Extensive experiments on three publicly available WSI datasets: TCGA-NSCLC, TCGA-RCC and BRIGHT, demonstrate the superiority of our approach over current state-of-the-art MIL methods, achieving an improvement of 1.0% to 2.6% in accuracy and 0.7%-1.6% in AUROC.

Motivation Deep learning attained excellent results in digital pathology recently. A challenge with its use is that high quality, representative training datasets are required to build robust models. Data annotation in the domain is labor intensive and demands substantial time commitment from expert pathologists. Active learning (AL) is a strategy to minimize annotation. The goal is to select samples from the pool of unlabeled data for annotation that improves model accuracy. However, AL is a very compute demanding approach. The benefits for model learning may vary according to the strategy used, and it may be hard for a domain specialist to fine tune the solution without an integrated interface. Results We developed a framework that includes a friendly user interface along with run-time optimizations to reduce annotation and execution time in AL in digital pathology. Our solution …

Background Neoadjuvant chemotherapy (NAC) is the standard treatment for early-stage triple negative breast cancer (TNBC). The primary endpoint of NAC is a pathological complete response (pCR). NAC results in pCR in only 30–40% of TNBC patients. Tumor-infiltrating lymphocytes (TILs), Ki67 and phosphohistone H3 (pH3) are a few known biomarkers to predict NAC response. Currently, systematic evaluation of the combined value of these biomarkers in predicting NAC response is lacking. In this study, the predictive value of markers derived from H&E and IHC stained biopsy tissue was comprehensively evaluated using a supervised machine learning (ML)-based approach. Identifying predictive biomarkers could help guide therapeutic decisions by enabling precise stratification of TNBC patients into responders and partial or non-responders. Methods Serial sections from core needle biopsies (n = 76) were stained with H&E and immunohistochemically for the Ki67 and pH3 markers, followed by whole-slide image (WSI) generation. The serial section stains in H&E stain, Ki67 and pH3 markers formed WSI triplets for each patient. The resulting WSI triplets were co-registered with H&E WSIs serving as the reference. Separate mask region-based CNN (MRCNN) models were trained with annotated H&E, Ki67 and pH3 images for detecting tumor cells, stromal and intratumoral TILs (sTILs and tTILs), Ki67+, and pH3+ cells. Top image patches with a high density of cells of interest were identified as hotspots. Best classifiers for NAC response prediction were identified by training multiple ML models and evaluating their performance by accuracy …

The Banff Digital Pathology Working Group (DPWG) was established with the goal to establish a digital pathology repository; develop, validate, and share models for image analysis; and foster collaborations using regular videoconferencing. During the calls, a variety of artificial intelligence (AI)-based support systems for transplantation pathology were presented. Potential collaborations in a competition/trial on AI applied to kidney transplant specimens, including the DIAGGRAFT challenge (staining of biopsies at multiple institutions, pathologists' visual assessment, and development and validation of new and pre-existing Banff scoring algorithms), were also discussed. To determine the next steps, a survey was conducted, primarily focusing on the feasibility of establishing a digital pathology repository and identifying potential hosts. Sixteen of the 35 respondents (46%) had access to a server hosting a digital pathology repository, with 2 respondents that could serve as a potential host at no cost to the DPWG. The 16 digital pathology repositories collected specimens from various organs, with the largest constituent being kidney (n = 12,870 specimens). A DPWG pilot digital pathology repository was established, and there are plans for a competition/trial with the DIAGGRAFT project. Utilizing existing resources and previously established models, the Banff DPWG is establishing new resources for the Banff community.

For routine pathology diagnosis and imaging-based biomedical research, Whole-slide image (WSI) analyses have been largely limited to a 2D tissue image space. For a more definitive tissue representation to support fine-resolution spatial and integrative analyses, it is critical to extend such tissue-based investigations to a 3D tissue space with spatially aligned serial tissue WSIs in different stains, such as Hematoxylin and Eosin (H&E) and Immunohistochemistry (IHC) biomarkers. However, such WSI registration is technically challenged by the overwhelming image scale, the complex histology structure change, and the significant difference in tissue appearances in different stains. The goal of this study is to register serial sections from multi-stain histopathology whole-slide image blocks. We propose a novel translation-based deep learning registration network CGNReg that spatially aligns serial WSIs stained in H …

One major goal of spatial query processing is to mitigate I/O costs and minimize the search space. However, geometric computation can be heavy-duty for spatial queries, in particular for complex geometries such as polygons with many edges based on a vector-based representation. Many past techniques have been provided for spatial partitioning and indexing, which are mainly built on minimal bounding boxes or other approximation methods and are not optimized for reducing geometric computation. In this paper, we propose a novel vector-raster hybrid approach through rasterization, where rich pixel-centric information is preserved to help not only filter out more candidates but also reduce geometry computation load. Based on the hybrid model, we implement four typical spatial queries, which can be generalized for other types of spatial queries. We also propose cost models to estimate the latency for those …

MEETING REPORT published: 16 October 2023 doi: 10.3389/ti. 2023.117832 respondents that could serve as a potential host at no cost to the DPWG. The 16 digital pathology repositories collected specimens from various organs, with the largest constituent being kidney (n= 12,870 specimens). A DPWG pilot digital pathology repository was established, and there are plans for a competition/trial with the DIAGGRAFT project. Utilizing existing resources and previously established models, the Banff DPWG is establishing new resources for the Banff community.

Motivation Morphological analyses with flatmount fluorescent images are essential to retinal pigment epithelial (RPE) aging studies and thus require accurate RPE cell segmentation. Although rapid technology advances in deep learning semantic segmentation have achieved great success in many biomedical research, the performance of these supervised learning methods for RPE cell segmentation is still limited by inadequate training data with high-quality annotations. Results To address this problem, we develop a Self-Supervised Semantic Segmentation (S4) method that utilizes a self-supervised learning strategy to train a semantic segmentation network with an encoder–decoder architecture. We employ a reconstruction and a pairwise representation loss to make the encoder extract structural information, while we create a morphology loss to produce the segmentation …

The human body is made up of about 37 trillion cells (adults). Each cell has its own unique role and is affected by its neighboring cells and environment. The NIH Human BioMolecular Atlas Program (HuBMAP) aims at developing a 3D atlas of human body consisting of organs, vessels, tissues to singe cells with all 3D spatially registered in a single 3D human atlas using tissues obtained from normal individuals across a wide range of ages. A critical step of building the atlas is to register 3D tissue blocks in real-time to the right location of a human organ, which itself consists of complex 3D sub-structures. The complexity of the 3D organ model, e.g., 35 meshes for a typical kidney, poses a significant computational challenge for the registration. In this paper, we propose a comprehensive framework TICKET (TIssue bloCK rEgisTration) to support tissue block registration for 3D human atlas, including (1) 3D mesh pre …

BackgroundRetinal pigment epithelium (RPE) aging is an important cause of vision loss. As RPE aging is accompanied by changes in cell morphological features, an accurate segmentation of RPE cells is a prerequisite to such morphology analyses. Due the overwhelmingly large cell number, manual annotations of RPE cell borders are time-consuming. Computer based methods do not work well on cells with weak or missing borders in the impaired RPE sheet regions.MethodTo address such a challenge, we develop a semi-supervised deep learning approach, namely MultiHeadGAN, to segment low contrast cells from impaired regions in RPE flatmount images. The developed deep learning model has a multi-head structure that allows model training with only a small scale of human annotated data. To strengthen model learning, we further train our model with RPE cells without ground truth cell borders by …

Background: Pathological complete response (pCR) after neoadjuvant chemotherapy (NAC) isassociated with a favorable prognosis and low recurrence rate, particularly in patients with triple-negative breast cancer (TNBC), an aggressive breast cancer subtype. However, only 30%-40%of patients with TNBC show pCR after standard NAC, while the remaining patients have non-pCR or residual disease. Different histological components of the tumor, including tumor stroma,polyploid giant cancer cells, and immune cells, can influence response to NAC in patients withTNBC. However, it remains unknown which histological components contribute to pCR afterNAC in some patients with TNBC but not in others. To address this, we developed a machinelearning pipeline to identify the most significant histological components contributing to NACresponse in patients with TNBC. Different representations of histological …

Liver fibrosis staging is clinically important for liver disease progression prediction. As the portal tract fibrotic quantity and size in a liver biopsy correlate with the fibrosis stage, an accurate analysis of portal tract regions is clinically critical. Manual annotations of portal tract regions, however, are time-consuming and subject to large inter-and intra-observer variability. To address such a challenge, we develop a Multiple Up-sampling and Spatial Attention guided UNet model (MUSA-UNet) to segment liver portal tract regions in whole-slide images of liver tissue slides. To enhance the segmentation performance, we propose to use depth-wise separable convolution, the spatial attention mechanism, the residual connection, and multiple up-sampling paths in the developed model. This study includes 53 histopathology whole slide images from patients who received liver transplantation. In total, 6,012 patches derived from …

Background Oncotype DX Recurrence Score (RS) has been widely used to predict chemotherapy benefits in patients with estrogen receptor-positive breast cancer. Studies showed that the features used in Magee equations correlate with RS. We aimed to examine whether deep learning (DL)-based histology image analyses can enhance such correlations. Methods We retrieved 382 cases with RS diagnosed between 2011 and 2015 from the Emory University and the Ohio State University. All patients received surgery. DL models were developed to detect nuclei of tumor cells and tumor-infiltrating lymphocytes (TILs) and segment tumor cell nuclei in hematoxylin and eosin (H&E) stained histopathology whole slide images (WSIs). Based on the DL-based analysis, we derived image features from WSIs, such as tumor cell number, TIL number variance, and nuclear grades. The entire patient cohorts were divided into one training set (125 cases) and two validation sets (82 and 175 cases) based on the data sources and WSI resolutions. The training set was used to train the linear regression models to predict RS. For prediction performance comparison, we used independent variables from Magee features alone or the combination of WSI-derived image and Magee features. Results The Pearson’s correlation coefficients between the actual RS and predicted RS by DL-based analysis were 0.7058 (p-value = 1.32 × 10–13) and 0.5041 (p-value = 1.15 × 10–12) for the validation sets 1 and 2, respectively. The adjusted R2 values using Magee features alone are 0.3442 and 0.2167 in the two validation sets, respectively. In contrast, the adjusted R2 …

3D spatial data has been generated at an extreme scale from many emerging applications, such as high definition maps for autonomous driving and 3D Human BioMolecular Atlas. In particular, 3D digital pathology provides a revolutionary approach to map human tissues in 3D, which is highly promising for advancing computer-aided diagnosis and understanding diseases through spatial queries and analysis. However, the exponential increase of data at 3D leads to significant I/O, communication, and computational challenges for 3D spatial queries. The complex structures of 3D objects such as bifurcated vessels make it difficult to effectively support 3D spatial queries with traditional methods. In this article, we present our work on building an efficient and scalable spatial query system, iSPEED, for large-scale 3D data with complex structures. iSPEED adopts effective progressive compression for each 3D object …

Motivation Predicting pathological complete response (pCR) to neoadjuvant chemotherapy (NAC) in triple-negative breast cancer (TNBC) patients accurately is direly needed for clinical decision making. pCR is also regarded as a strong predictor of overall survival. In this work, we propose a deep learning system to predict pCR to NAC based on serial pathology images stained with hematoxylin and eosin and two immunohistochemical biomarkers (Ki67 and PHH3). To support human prior domain knowledge-based guidance and enhance interpretability of the deep learning system, we introduce a human knowledge-derived spatial attention mechanism to inform deep learning models of informative tissue areas of interest. For each patient, three serial breast tumor tissue sections from biopsy blocks were sectioned, stained in three different stains and integrated. The resulting comprehensive attention …

The analysis of high resolution whole slide tissue images is a computationally expensive task, which adversely impacts effective use of pathology imaging data in research. We propose runtime solutions to enable efficient execution of pathology image analysis applications on modern distributed memory hybrid platforms equipped with both CPUs and GPUs. Hybrid systems offer significant computation capacity, but taking advantage of this computing power is complex. An application developer may have to implement multiple versions of data processing codes targeted for different computing devices. The developer also has to tackle the challenges of efficiently distributing computational load among the nodes of a distributed memory machine and among computing devices on a node. This is particularly difficult in analysis of high resolution images because of irregular computing costs of processing different image …

Understanding cellular remodeling in response to mechanical stimuli is a critical step in elucidating mechanical activation of biochemical signaling pathways. Experimental evidence indicates that external stress-induced subcellular adaptation is accomplished through dynamic cytoskeletal reorganization. To study the interactions between subcellular structures involved in transducing mechanical signals, we combined experimental data and computational simulations to evaluate real-time mechanical adaptation of the actin cytoskeletal network. Actin cytoskeleton was imaged at the same time as an external tensile force was applied to live vascular smooth muscle cells using a fibronectin-functionalized atomic force microscope probe. Moreover, we performed computational simulations of active cytoskeletal networks under an external tensile force. The experimental data and simulation results suggest that …

This book constitutes revised selected papers from two VLDB workshops: The International Workshop on Polystore Systems for Heterogeneous Data in Multiple Databases with Privacy and Security Assurances, Poly 2020, and the 6th International Workshop on Data Management and Analytics for Medicine and Healthcare, DMAH 2020, which were held virtually on August 31 and September 4, 2020. For Poly 2020, 4 full and 3 short papers were accepted from 10 submissions; and for DMAH 2020, 7 full and 2 short papers were accepted from a total of 15 submissions. The papers were organized in topical sections as follows: Privacy, Security and/or Policy Issues for Heterogenous Data; COVID-19 Data Analytics and Visualization; Deep Learning based Biomedical Data Analytics; NLP based Learning from Unstructured Data; Biomedical Data Modelling and Prediction.

Although polyploid cells were first described nearly two centuries ago, their ability to proliferate has only recently been demonstrated. It also becomes increasingly evident that a subset of tumor cells, polyploid giant cancer cells (PGCCs), play a critical role in the pathophysiology of breast cancer (BC), among other cancer types. In BC, PGCCs can arise in response to therapy-induced stress. Their progeny possess cancer stem cell (CSC) properties and can repopulate the tumor. By modulating the tumor microenvironment (TME), PGCCs promote BC progression, chemoresistance, metastasis, and relapse and ultimately impact the survival of BC patients. Given their pro- tumorigenic roles, PGCCs have been proposed to possess the ability to predict treatment response and patient prognosis in BC. Traditionally, DNA cytometry has been used to detect PGCCs.. The field will further derive benefit from the development of …