Disease-Image Specific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-modal Neuroimages

Yongsheng Pan, Mingxia Liu, Chunfeng Lian, Yong Xia, Dinggang Shen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Incomplete data problem is unavoidable in automated brain disease diagnosis using multi-modal neuroimages (e.g., MRI and PET). To utilize all available subjects to train diagnostic models, deep networks have been proposed to directly impute missing neuroimages by treating all voxels in a 3D volume equally. These methods are not diagnosis-oriented, as they ignore the disease-image specific information conveyed in multi-modal neuroimages, i.e., (1) disease may cause abnormalities only at local brain regions, and (2) different modalities may highlight different disease-associated regions. In this paper, we propose a unified disease-image specific deep learning framework for joint image synthesis and disease diagnosis using incomplete multi-modal neuroimaging data. Specifically, by using the whole-brain images as input, we design a disease-image specific neural network (DSNN) to implicitly model disease-image specificity in MRI/PET scans using the spatial cosine kernel. Moreover, we develop a feature-consistent generative adversarial network (FGAN) to synthesize missing images, encouraging DSNN feature maps of synthetic images and their respective real images to be consistent. Our DSNN and FGAN can be jointly trained, by which missing images are imputed in a task-oriented manner for brain disease diagnosis. Experimental results on 1, 466 subjects suggest that our method not only generates reasonable neuroimages, but also achieves the state-of-the-art performance in both tasks of Alzheimer’s disease (AD) identification and mild cognitive impairment (MCI) conversion prediction.

Original languageEnglish
Title of host publicationMedical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings
EditorsDinggang Shen, Pew-Thian Yap, Tianming Liu, Terry M. Peters, Ali Khan, Lawrence H. Staib, Caroline Essert, Sean Zhou
PublisherSpringer
Pages137-145
Number of pages9
ISBN (Print)9783030322472
DOIs
Publication statusPublished - 2019 Jan 1
Externally publishedYes
Event22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019 - Shenzhen, China
Duration: 2019 Oct 132019 Oct 17

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11766 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019
CountryChina
CityShenzhen
Period19/10/1319/10/17

Fingerprint

Brain
Neural Networks
Neural networks
Magnetic resonance imaging
Neuroimaging
Model Diagnostics
Alzheimer's Disease
Incomplete Data
Voxel
Modality
Specificity
Synthesis
kernel
Prediction
Experimental Results

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Pan, Y., Liu, M., Lian, C., Xia, Y., & Shen, D. (2019). Disease-Image Specific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-modal Neuroimages. In D. Shen, P-T. Yap, T. Liu, T. M. Peters, A. Khan, L. H. Staib, C. Essert, ... S. Zhou (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings (pp. 137-145). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11766 LNCS). Springer. https://doi.org/10.1007/978-3-030-32248-9_16

Disease-Image Specific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-modal Neuroimages. / Pan, Yongsheng; Liu, Mingxia; Lian, Chunfeng; Xia, Yong; Shen, Dinggang.

Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings. ed. / Dinggang Shen; Pew-Thian Yap; Tianming Liu; Terry M. Peters; Ali Khan; Lawrence H. Staib; Caroline Essert; Sean Zhou. Springer, 2019. p. 137-145 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11766 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Pan, Y, Liu, M, Lian, C, Xia, Y & Shen, D 2019, Disease-Image Specific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-modal Neuroimages. in D Shen, P-T Yap, T Liu, TM Peters, A Khan, LH Staib, C Essert & S Zhou (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11766 LNCS, Springer, pp. 137-145, 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019, Shenzhen, China, 19/10/13. https://doi.org/10.1007/978-3-030-32248-9_16
Pan Y, Liu M, Lian C, Xia Y, Shen D. Disease-Image Specific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-modal Neuroimages. In Shen D, Yap P-T, Liu T, Peters TM, Khan A, Staib LH, Essert C, Zhou S, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings. Springer. 2019. p. 137-145. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-32248-9_16
Pan, Yongsheng ; Liu, Mingxia ; Lian, Chunfeng ; Xia, Yong ; Shen, Dinggang. / Disease-Image Specific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-modal Neuroimages. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings. editor / Dinggang Shen ; Pew-Thian Yap ; Tianming Liu ; Terry M. Peters ; Ali Khan ; Lawrence H. Staib ; Caroline Essert ; Sean Zhou. Springer, 2019. pp. 137-145 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{4cb215b7bdd6426bbef883da95ab41a7,
title = "Disease-Image Specific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-modal Neuroimages",
abstract = "Incomplete data problem is unavoidable in automated brain disease diagnosis using multi-modal neuroimages (e.g., MRI and PET). To utilize all available subjects to train diagnostic models, deep networks have been proposed to directly impute missing neuroimages by treating all voxels in a 3D volume equally. These methods are not diagnosis-oriented, as they ignore the disease-image specific information conveyed in multi-modal neuroimages, i.e., (1) disease may cause abnormalities only at local brain regions, and (2) different modalities may highlight different disease-associated regions. In this paper, we propose a unified disease-image specific deep learning framework for joint image synthesis and disease diagnosis using incomplete multi-modal neuroimaging data. Specifically, by using the whole-brain images as input, we design a disease-image specific neural network (DSNN) to implicitly model disease-image specificity in MRI/PET scans using the spatial cosine kernel. Moreover, we develop a feature-consistent generative adversarial network (FGAN) to synthesize missing images, encouraging DSNN feature maps of synthetic images and their respective real images to be consistent. Our DSNN and FGAN can be jointly trained, by which missing images are imputed in a task-oriented manner for brain disease diagnosis. Experimental results on 1, 466 subjects suggest that our method not only generates reasonable neuroimages, but also achieves the state-of-the-art performance in both tasks of Alzheimer’s disease (AD) identification and mild cognitive impairment (MCI) conversion prediction.",
author = "Yongsheng Pan and Mingxia Liu and Chunfeng Lian and Yong Xia and Dinggang Shen",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-3-030-32248-9_16",
language = "English",
isbn = "9783030322472",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer",
pages = "137--145",
editor = "Dinggang Shen and Pew-Thian Yap and Tianming Liu and Peters, {Terry M.} and Ali Khan and Staib, {Lawrence H.} and Caroline Essert and Sean Zhou",
booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings",

}

TY - GEN

T1 - Disease-Image Specific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-modal Neuroimages

AU - Pan, Yongsheng

AU - Liu, Mingxia

AU - Lian, Chunfeng

AU - Xia, Yong

AU - Shen, Dinggang

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Incomplete data problem is unavoidable in automated brain disease diagnosis using multi-modal neuroimages (e.g., MRI and PET). To utilize all available subjects to train diagnostic models, deep networks have been proposed to directly impute missing neuroimages by treating all voxels in a 3D volume equally. These methods are not diagnosis-oriented, as they ignore the disease-image specific information conveyed in multi-modal neuroimages, i.e., (1) disease may cause abnormalities only at local brain regions, and (2) different modalities may highlight different disease-associated regions. In this paper, we propose a unified disease-image specific deep learning framework for joint image synthesis and disease diagnosis using incomplete multi-modal neuroimaging data. Specifically, by using the whole-brain images as input, we design a disease-image specific neural network (DSNN) to implicitly model disease-image specificity in MRI/PET scans using the spatial cosine kernel. Moreover, we develop a feature-consistent generative adversarial network (FGAN) to synthesize missing images, encouraging DSNN feature maps of synthetic images and their respective real images to be consistent. Our DSNN and FGAN can be jointly trained, by which missing images are imputed in a task-oriented manner for brain disease diagnosis. Experimental results on 1, 466 subjects suggest that our method not only generates reasonable neuroimages, but also achieves the state-of-the-art performance in both tasks of Alzheimer’s disease (AD) identification and mild cognitive impairment (MCI) conversion prediction.

AB - Incomplete data problem is unavoidable in automated brain disease diagnosis using multi-modal neuroimages (e.g., MRI and PET). To utilize all available subjects to train diagnostic models, deep networks have been proposed to directly impute missing neuroimages by treating all voxels in a 3D volume equally. These methods are not diagnosis-oriented, as they ignore the disease-image specific information conveyed in multi-modal neuroimages, i.e., (1) disease may cause abnormalities only at local brain regions, and (2) different modalities may highlight different disease-associated regions. In this paper, we propose a unified disease-image specific deep learning framework for joint image synthesis and disease diagnosis using incomplete multi-modal neuroimaging data. Specifically, by using the whole-brain images as input, we design a disease-image specific neural network (DSNN) to implicitly model disease-image specificity in MRI/PET scans using the spatial cosine kernel. Moreover, we develop a feature-consistent generative adversarial network (FGAN) to synthesize missing images, encouraging DSNN feature maps of synthetic images and their respective real images to be consistent. Our DSNN and FGAN can be jointly trained, by which missing images are imputed in a task-oriented manner for brain disease diagnosis. Experimental results on 1, 466 subjects suggest that our method not only generates reasonable neuroimages, but also achieves the state-of-the-art performance in both tasks of Alzheimer’s disease (AD) identification and mild cognitive impairment (MCI) conversion prediction.

UR - http://www.scopus.com/inward/record.url?scp=85075697955&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85075697955&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-32248-9_16

DO - 10.1007/978-3-030-32248-9_16

M3 - Conference contribution

AN - SCOPUS:85075697955

SN - 9783030322472

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 137

EP - 145

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings

A2 - Shen, Dinggang

A2 - Yap, Pew-Thian

A2 - Liu, Tianming

A2 - Peters, Terry M.

A2 - Khan, Ali

A2 - Staib, Lawrence H.

A2 - Essert, Caroline

A2 - Zhou, Sean

PB - Springer

ER -