Inter-observer agreement on the interpretation of capsule endoscopy findings based on capsule endoscopy structured terminology

A multicenter study by the Korean gut image study group

Byung Ik Jang, Si Hyung Lee, Jeong Seop Moon, Dae Young Cheung, In Seok Lee, Jin Oh Kim, Jae Hee Cheon, Cheol Hee Park, Jeong Sik Byeon, Youn Sun Park, Ki Nam Shim, Yong Sik Kim, Kyung Jo Kim, Kwang Jae Lee, Ji Kon Ryu, Dong Kyung Chang, Hoon-Jai Chun, Myung Gyu Choi

Research output: Contribution to journalArticle

19 Citations (Scopus)

Abstract

Objective. Capsule endoscopy (CE) is a novel investigation for the diagnosis of small-bowel disease but its interpretation is highly subjective. We studied the inter-observer agreement and accuracy of the interpretation of CE findings based on capsule endoscopy structured terminology (CEST). Material and methods. Fifty-six CE video clips were collected from eight university hospitals in South Korea and were independently reviewed by 13 gastroenterology experts and 10 trainees. All investigators recorded their findings based on CEST. To determine the accuracy of individual viewers, we defined the 'gold standard' as a joint review by four experts. Results. The 56 CE video clips included five normal cases, 19 cases of protruding lesions, 21 cases of depressed lesions, three cases of flat lesions, one case of abnormal mucosa, six cases with blood in the lumen, and one case of stenotic lumen. The overall mean accuracies for the experts and trainees were 74.3% ± 22.6% and 61.7% ± 25.4%, respectively. The overall accuracy for the trainee group was significantly lower than that for the expert group (P<0.001), especially in normal, tumor, venous structure, and ulcer cases. The accuracies of the two groups varied with the CE findings. The accuracies were higher in cases with more prominent intraluminal changes (e.g. active small-bowel bleeding, ulcer, tumor, stenotic lumen). In contrast, subtle mucosal lesions, such as erosion, angioectasia, and diverticulum, had lower accuracies. The mean kappa values for the experts and trainees were 0.61 (range 0.390.97) and 0.46 (range 0.170.66), respectively. Conclusions. Our results showed that there was substantial agreement between experts and moderate agreement between trainees. In order to achieve higher accuracies and better inter-observer agreement, we need not only more experience with CE but also consensus regarding CEST terminology.

Original languageEnglish
Pages (from-to)370-374
Number of pages5
JournalScandinavian Journal of Gastroenterology
Volume45
Issue number3
DOIs
Publication statusPublished - 2010 Feb 26

Fingerprint

Capsule Endoscopy
Terminology
Multicenter Studies
Surgical Instruments
Varicose Ulcer
Republic of Korea
Diverticulum
Gastroenterology
Ulcer
Neoplasms
Mucous Membrane
Research Personnel
Hemorrhage

Keywords

  • Capsule endoscopy analysis
  • Inter-observer variability
  • Small-bowel lesions

ASJC Scopus subject areas

  • Gastroenterology

Cite this

Inter-observer agreement on the interpretation of capsule endoscopy findings based on capsule endoscopy structured terminology : A multicenter study by the Korean gut image study group. / Jang, Byung Ik; Lee, Si Hyung; Moon, Jeong Seop; Cheung, Dae Young; Lee, In Seok; Kim, Jin Oh; Cheon, Jae Hee; Park, Cheol Hee; Byeon, Jeong Sik; Park, Youn Sun; Shim, Ki Nam; Kim, Yong Sik; Kim, Kyung Jo; Lee, Kwang Jae; Ryu, Ji Kon; Chang, Dong Kyung; Chun, Hoon-Jai; Choi, Myung Gyu.

In: Scandinavian Journal of Gastroenterology, Vol. 45, No. 3, 26.02.2010, p. 370-374.

Research output: Contribution to journalArticle

Jang, BI, Lee, SH, Moon, JS, Cheung, DY, Lee, IS, Kim, JO, Cheon, JH, Park, CH, Byeon, JS, Park, YS, Shim, KN, Kim, YS, Kim, KJ, Lee, KJ, Ryu, JK, Chang, DK, Chun, H-J & Choi, MG 2010, 'Inter-observer agreement on the interpretation of capsule endoscopy findings based on capsule endoscopy structured terminology: A multicenter study by the Korean gut image study group', Scandinavian Journal of Gastroenterology, vol. 45, no. 3, pp. 370-374. https://doi.org/10.3109/00365520903521574
Jang, Byung Ik ; Lee, Si Hyung ; Moon, Jeong Seop ; Cheung, Dae Young ; Lee, In Seok ; Kim, Jin Oh ; Cheon, Jae Hee ; Park, Cheol Hee ; Byeon, Jeong Sik ; Park, Youn Sun ; Shim, Ki Nam ; Kim, Yong Sik ; Kim, Kyung Jo ; Lee, Kwang Jae ; Ryu, Ji Kon ; Chang, Dong Kyung ; Chun, Hoon-Jai ; Choi, Myung Gyu. / Inter-observer agreement on the interpretation of capsule endoscopy findings based on capsule endoscopy structured terminology : A multicenter study by the Korean gut image study group. In: Scandinavian Journal of Gastroenterology. 2010 ; Vol. 45, No. 3. pp. 370-374.
@article{f099e76e44634639be1beea9da1546a9,
title = "Inter-observer agreement on the interpretation of capsule endoscopy findings based on capsule endoscopy structured terminology: A multicenter study by the Korean gut image study group",
abstract = "Objective. Capsule endoscopy (CE) is a novel investigation for the diagnosis of small-bowel disease but its interpretation is highly subjective. We studied the inter-observer agreement and accuracy of the interpretation of CE findings based on capsule endoscopy structured terminology (CEST). Material and methods. Fifty-six CE video clips were collected from eight university hospitals in South Korea and were independently reviewed by 13 gastroenterology experts and 10 trainees. All investigators recorded their findings based on CEST. To determine the accuracy of individual viewers, we defined the 'gold standard' as a joint review by four experts. Results. The 56 CE video clips included five normal cases, 19 cases of protruding lesions, 21 cases of depressed lesions, three cases of flat lesions, one case of abnormal mucosa, six cases with blood in the lumen, and one case of stenotic lumen. The overall mean accuracies for the experts and trainees were 74.3{\%} ± 22.6{\%} and 61.7{\%} ± 25.4{\%}, respectively. The overall accuracy for the trainee group was significantly lower than that for the expert group (P<0.001), especially in normal, tumor, venous structure, and ulcer cases. The accuracies of the two groups varied with the CE findings. The accuracies were higher in cases with more prominent intraluminal changes (e.g. active small-bowel bleeding, ulcer, tumor, stenotic lumen). In contrast, subtle mucosal lesions, such as erosion, angioectasia, and diverticulum, had lower accuracies. The mean kappa values for the experts and trainees were 0.61 (range 0.390.97) and 0.46 (range 0.170.66), respectively. Conclusions. Our results showed that there was substantial agreement between experts and moderate agreement between trainees. In order to achieve higher accuracies and better inter-observer agreement, we need not only more experience with CE but also consensus regarding CEST terminology.",
keywords = "Capsule endoscopy analysis, Inter-observer variability, Small-bowel lesions",
author = "Jang, {Byung Ik} and Lee, {Si Hyung} and Moon, {Jeong Seop} and Cheung, {Dae Young} and Lee, {In Seok} and Kim, {Jin Oh} and Cheon, {Jae Hee} and Park, {Cheol Hee} and Byeon, {Jeong Sik} and Park, {Youn Sun} and Shim, {Ki Nam} and Kim, {Yong Sik} and Kim, {Kyung Jo} and Lee, {Kwang Jae} and Ryu, {Ji Kon} and Chang, {Dong Kyung} and Hoon-Jai Chun and Choi, {Myung Gyu}",
year = "2010",
month = "2",
day = "26",
doi = "10.3109/00365520903521574",
language = "English",
volume = "45",
pages = "370--374",
journal = "Scandinavian Journal of Gastroenterology",
issn = "0036-5521",
publisher = "Informa Healthcare",
number = "3",

}

TY - JOUR

T1 - Inter-observer agreement on the interpretation of capsule endoscopy findings based on capsule endoscopy structured terminology

T2 - A multicenter study by the Korean gut image study group

AU - Jang, Byung Ik

AU - Lee, Si Hyung

AU - Moon, Jeong Seop

AU - Cheung, Dae Young

AU - Lee, In Seok

AU - Kim, Jin Oh

AU - Cheon, Jae Hee

AU - Park, Cheol Hee

AU - Byeon, Jeong Sik

AU - Park, Youn Sun

AU - Shim, Ki Nam

AU - Kim, Yong Sik

AU - Kim, Kyung Jo

AU - Lee, Kwang Jae

AU - Ryu, Ji Kon

AU - Chang, Dong Kyung

AU - Chun, Hoon-Jai

AU - Choi, Myung Gyu

PY - 2010/2/26

Y1 - 2010/2/26

N2 - Objective. Capsule endoscopy (CE) is a novel investigation for the diagnosis of small-bowel disease but its interpretation is highly subjective. We studied the inter-observer agreement and accuracy of the interpretation of CE findings based on capsule endoscopy structured terminology (CEST). Material and methods. Fifty-six CE video clips were collected from eight university hospitals in South Korea and were independently reviewed by 13 gastroenterology experts and 10 trainees. All investigators recorded their findings based on CEST. To determine the accuracy of individual viewers, we defined the 'gold standard' as a joint review by four experts. Results. The 56 CE video clips included five normal cases, 19 cases of protruding lesions, 21 cases of depressed lesions, three cases of flat lesions, one case of abnormal mucosa, six cases with blood in the lumen, and one case of stenotic lumen. The overall mean accuracies for the experts and trainees were 74.3% ± 22.6% and 61.7% ± 25.4%, respectively. The overall accuracy for the trainee group was significantly lower than that for the expert group (P<0.001), especially in normal, tumor, venous structure, and ulcer cases. The accuracies of the two groups varied with the CE findings. The accuracies were higher in cases with more prominent intraluminal changes (e.g. active small-bowel bleeding, ulcer, tumor, stenotic lumen). In contrast, subtle mucosal lesions, such as erosion, angioectasia, and diverticulum, had lower accuracies. The mean kappa values for the experts and trainees were 0.61 (range 0.390.97) and 0.46 (range 0.170.66), respectively. Conclusions. Our results showed that there was substantial agreement between experts and moderate agreement between trainees. In order to achieve higher accuracies and better inter-observer agreement, we need not only more experience with CE but also consensus regarding CEST terminology.

AB - Objective. Capsule endoscopy (CE) is a novel investigation for the diagnosis of small-bowel disease but its interpretation is highly subjective. We studied the inter-observer agreement and accuracy of the interpretation of CE findings based on capsule endoscopy structured terminology (CEST). Material and methods. Fifty-six CE video clips were collected from eight university hospitals in South Korea and were independently reviewed by 13 gastroenterology experts and 10 trainees. All investigators recorded their findings based on CEST. To determine the accuracy of individual viewers, we defined the 'gold standard' as a joint review by four experts. Results. The 56 CE video clips included five normal cases, 19 cases of protruding lesions, 21 cases of depressed lesions, three cases of flat lesions, one case of abnormal mucosa, six cases with blood in the lumen, and one case of stenotic lumen. The overall mean accuracies for the experts and trainees were 74.3% ± 22.6% and 61.7% ± 25.4%, respectively. The overall accuracy for the trainee group was significantly lower than that for the expert group (P<0.001), especially in normal, tumor, venous structure, and ulcer cases. The accuracies of the two groups varied with the CE findings. The accuracies were higher in cases with more prominent intraluminal changes (e.g. active small-bowel bleeding, ulcer, tumor, stenotic lumen). In contrast, subtle mucosal lesions, such as erosion, angioectasia, and diverticulum, had lower accuracies. The mean kappa values for the experts and trainees were 0.61 (range 0.390.97) and 0.46 (range 0.170.66), respectively. Conclusions. Our results showed that there was substantial agreement between experts and moderate agreement between trainees. In order to achieve higher accuracies and better inter-observer agreement, we need not only more experience with CE but also consensus regarding CEST terminology.

KW - Capsule endoscopy analysis

KW - Inter-observer variability

KW - Small-bowel lesions

UR - http://www.scopus.com/inward/record.url?scp=77149135711&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77149135711&partnerID=8YFLogxK

U2 - 10.3109/00365520903521574

DO - 10.3109/00365520903521574

M3 - Article

VL - 45

SP - 370

EP - 374

JO - Scandinavian Journal of Gastroenterology

JF - Scandinavian Journal of Gastroenterology

SN - 0036-5521

IS - 3

ER -