RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records

Bum Chul Kwon, Min Je Choi, Joanne Taery Kim, Edward Choi, Young Bin Kim, Soonwook Kwon, Jimeng Sun, Jaegul Choo

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

We have recently seen many successful applications of recurrent neural networks (RNNs) on electronic medical records (EMRs), which contain histories of patients' diagnoses, medications, and other various events, in order to predict the current and future states of patients. Despite the strong performance of RNNs, it is often challenging for users to understand why the model makes a particular prediction. Such black-box nature of RNNs can impede its wide adoption in clinical practice. Furthermore, we have no established methods to interactively leverage users' domain expertise and prior knowledge as inputs for steering the model. Therefore, our design study aims to provide a visual analytics solution to increase interpretability and interactivity of RNNs via a joint effort of medical experts, artificial intelligence scientists, and visual analytics researchers. Following the iterative design process between the experts, we design, implement, and evaluate a visual analytics tool called RetainVis, which couples a newly improved, interpretable, and interactive RNN-based model called RetainEX and visualizations for users' exploration of EMR data in the context of prediction tasks. Our study shows the effective use of RetainVis for gaining insights into how individual medical codes contribute to making risk predictions, using EMRs of patients with heart failure and cataract symptoms. Our study also demonstrates how we made substantial changes to the state-of-the-art RNN model called RETAIN in order to make use of temporal information and increase interactivity. This study will provide a useful guideline for researchers that aim to design an interpretable and interactive visual analytics tool for RNNs.

Original languageEnglish
Article number8440842
Pages (from-to)299-309
Number of pages11
JournalIEEE Transactions on Visualization and Computer Graphics
Volume25
Issue number1
DOIs
Publication statusPublished - 2019 Jan 1

Fingerprint

Electronic medical equipment
Electronic Health Records
Recurrent neural networks
Neural Networks (Computer)
Research Personnel
Artificial Intelligence
Cataract
Heart Failure
Guidelines
Artificial intelligence
Visualization

Keywords

  • Healthcare
  • Interactive Artificial Intelligence
  • Interpretable Deep Learning
  • XAI (Explainable Artificial Intelligence)

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design

Cite this

RetainVis : Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records. / Kwon, Bum Chul; Choi, Min Je; Kim, Joanne Taery; Choi, Edward; Kim, Young Bin; Kwon, Soonwook; Sun, Jimeng; Choo, Jaegul.

In: IEEE Transactions on Visualization and Computer Graphics, Vol. 25, No. 1, 8440842, 01.01.2019, p. 299-309.

Research output: Contribution to journalArticle

Kwon, Bum Chul ; Choi, Min Je ; Kim, Joanne Taery ; Choi, Edward ; Kim, Young Bin ; Kwon, Soonwook ; Sun, Jimeng ; Choo, Jaegul. / RetainVis : Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records. In: IEEE Transactions on Visualization and Computer Graphics. 2019 ; Vol. 25, No. 1. pp. 299-309.
@article{ba25794be6024083a2f35fb9cae5b42d,
title = "RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records",
abstract = "We have recently seen many successful applications of recurrent neural networks (RNNs) on electronic medical records (EMRs), which contain histories of patients' diagnoses, medications, and other various events, in order to predict the current and future states of patients. Despite the strong performance of RNNs, it is often challenging for users to understand why the model makes a particular prediction. Such black-box nature of RNNs can impede its wide adoption in clinical practice. Furthermore, we have no established methods to interactively leverage users' domain expertise and prior knowledge as inputs for steering the model. Therefore, our design study aims to provide a visual analytics solution to increase interpretability and interactivity of RNNs via a joint effort of medical experts, artificial intelligence scientists, and visual analytics researchers. Following the iterative design process between the experts, we design, implement, and evaluate a visual analytics tool called RetainVis, which couples a newly improved, interpretable, and interactive RNN-based model called RetainEX and visualizations for users' exploration of EMR data in the context of prediction tasks. Our study shows the effective use of RetainVis for gaining insights into how individual medical codes contribute to making risk predictions, using EMRs of patients with heart failure and cataract symptoms. Our study also demonstrates how we made substantial changes to the state-of-the-art RNN model called RETAIN in order to make use of temporal information and increase interactivity. This study will provide a useful guideline for researchers that aim to design an interpretable and interactive visual analytics tool for RNNs.",
keywords = "Healthcare, Interactive Artificial Intelligence, Interpretable Deep Learning, XAI (Explainable Artificial Intelligence)",
author = "Kwon, {Bum Chul} and Choi, {Min Je} and Kim, {Joanne Taery} and Edward Choi and Kim, {Young Bin} and Soonwook Kwon and Jimeng Sun and Jaegul Choo",
year = "2019",
month = "1",
day = "1",
doi = "10.1109/TVCG.2018.2865027",
language = "English",
volume = "25",
pages = "299--309",
journal = "IEEE Transactions on Visualization and Computer Graphics",
issn = "1077-2626",
publisher = "IEEE Computer Society",
number = "1",

}

TY - JOUR

T1 - RetainVis

T2 - Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records

AU - Kwon, Bum Chul

AU - Choi, Min Je

AU - Kim, Joanne Taery

AU - Choi, Edward

AU - Kim, Young Bin

AU - Kwon, Soonwook

AU - Sun, Jimeng

AU - Choo, Jaegul

PY - 2019/1/1

Y1 - 2019/1/1

N2 - We have recently seen many successful applications of recurrent neural networks (RNNs) on electronic medical records (EMRs), which contain histories of patients' diagnoses, medications, and other various events, in order to predict the current and future states of patients. Despite the strong performance of RNNs, it is often challenging for users to understand why the model makes a particular prediction. Such black-box nature of RNNs can impede its wide adoption in clinical practice. Furthermore, we have no established methods to interactively leverage users' domain expertise and prior knowledge as inputs for steering the model. Therefore, our design study aims to provide a visual analytics solution to increase interpretability and interactivity of RNNs via a joint effort of medical experts, artificial intelligence scientists, and visual analytics researchers. Following the iterative design process between the experts, we design, implement, and evaluate a visual analytics tool called RetainVis, which couples a newly improved, interpretable, and interactive RNN-based model called RetainEX and visualizations for users' exploration of EMR data in the context of prediction tasks. Our study shows the effective use of RetainVis for gaining insights into how individual medical codes contribute to making risk predictions, using EMRs of patients with heart failure and cataract symptoms. Our study also demonstrates how we made substantial changes to the state-of-the-art RNN model called RETAIN in order to make use of temporal information and increase interactivity. This study will provide a useful guideline for researchers that aim to design an interpretable and interactive visual analytics tool for RNNs.

AB - We have recently seen many successful applications of recurrent neural networks (RNNs) on electronic medical records (EMRs), which contain histories of patients' diagnoses, medications, and other various events, in order to predict the current and future states of patients. Despite the strong performance of RNNs, it is often challenging for users to understand why the model makes a particular prediction. Such black-box nature of RNNs can impede its wide adoption in clinical practice. Furthermore, we have no established methods to interactively leverage users' domain expertise and prior knowledge as inputs for steering the model. Therefore, our design study aims to provide a visual analytics solution to increase interpretability and interactivity of RNNs via a joint effort of medical experts, artificial intelligence scientists, and visual analytics researchers. Following the iterative design process between the experts, we design, implement, and evaluate a visual analytics tool called RetainVis, which couples a newly improved, interpretable, and interactive RNN-based model called RetainEX and visualizations for users' exploration of EMR data in the context of prediction tasks. Our study shows the effective use of RetainVis for gaining insights into how individual medical codes contribute to making risk predictions, using EMRs of patients with heart failure and cataract symptoms. Our study also demonstrates how we made substantial changes to the state-of-the-art RNN model called RETAIN in order to make use of temporal information and increase interactivity. This study will provide a useful guideline for researchers that aim to design an interpretable and interactive visual analytics tool for RNNs.

KW - Healthcare

KW - Interactive Artificial Intelligence

KW - Interpretable Deep Learning

KW - XAI (Explainable Artificial Intelligence)

UR - http://www.scopus.com/inward/record.url?scp=85052616093&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85052616093&partnerID=8YFLogxK

U2 - 10.1109/TVCG.2018.2865027

DO - 10.1109/TVCG.2018.2865027

M3 - Article

AN - SCOPUS:85052616093

VL - 25

SP - 299

EP - 309

JO - IEEE Transactions on Visualization and Computer Graphics

JF - IEEE Transactions on Visualization and Computer Graphics

SN - 1077-2626

IS - 1

M1 - 8440842

ER -