Streaming video segmentation via short-term hierarchical segmentation and frame-by-frame Markov random field optimization

Won Dong Jang, Chang-Su Kim

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

An online video segmentation algorithm, based on shortterm hierarchical segmentation (STHS) and frame-by-frame Markov random field (MRF) optimization, is proposed in this work. We develop the STHS technique, which generates initial segments by sliding a short window of frames. In STHS, we apply spatial agglomerative clustering to each frame, and then adopt inter-frame bipartite graph matching to construct initial segments. Then, we partition each frame into final segments, by minimizing an MRF energy function composed of unary and pair wise costs. We compute the unary cost using the STHS initial segments and the segmentation result at the previous frame. We set the pair wise cost to encourage similar nodes to have the same segment label. Experimental results on a video segmentation benchmark dataset, VSB100, demonstrate that the proposed algorithm outperforms state-of-the-art online video segmentation techniques significantly.

Original languageEnglish
Title of host publicationComputer Vision - 14th European Conference, ECCV 2016, Proceedings
PublisherSpringer Verlag
Pages599-615
Number of pages17
Volume9910 LNCS
ISBN (Print)9783319464657
DOIs
Publication statusPublished - 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9910 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Fingerprint

Video Segmentation
Video streaming
Streaming
Random Field
Segmentation
Optimization
Costs
Unary
Labels
Spatial Clustering
Graph Matching
Energy Function
Bipartite Graph
Partition
Benchmark
Experimental Results
Vertex of a graph

Keywords

  • Agglomerative clustering
  • Graph matching
  • Online segmentation
  • Streaming segmentation
  • Video segmentation

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Jang, W. D., & Kim, C-S. (2016). Streaming video segmentation via short-term hierarchical segmentation and frame-by-frame Markov random field optimization. In Computer Vision - 14th European Conference, ECCV 2016, Proceedings (Vol. 9910 LNCS, pp. 599-615). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9910 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-46466-4_36

Streaming video segmentation via short-term hierarchical segmentation and frame-by-frame Markov random field optimization. / Jang, Won Dong; Kim, Chang-Su.

Computer Vision - 14th European Conference, ECCV 2016, Proceedings. Vol. 9910 LNCS Springer Verlag, 2016. p. 599-615 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9910 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Jang, WD & Kim, C-S 2016, Streaming video segmentation via short-term hierarchical segmentation and frame-by-frame Markov random field optimization. in Computer Vision - 14th European Conference, ECCV 2016, Proceedings. vol. 9910 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9910 LNCS, Springer Verlag, pp. 599-615. https://doi.org/10.1007/978-3-319-46466-4_36
Jang WD, Kim C-S. Streaming video segmentation via short-term hierarchical segmentation and frame-by-frame Markov random field optimization. In Computer Vision - 14th European Conference, ECCV 2016, Proceedings. Vol. 9910 LNCS. Springer Verlag. 2016. p. 599-615. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-46466-4_36
Jang, Won Dong ; Kim, Chang-Su. / Streaming video segmentation via short-term hierarchical segmentation and frame-by-frame Markov random field optimization. Computer Vision - 14th European Conference, ECCV 2016, Proceedings. Vol. 9910 LNCS Springer Verlag, 2016. pp. 599-615 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{597a68f8dc11411e8d4f73e1617024d2,
title = "Streaming video segmentation via short-term hierarchical segmentation and frame-by-frame Markov random field optimization",
abstract = "An online video segmentation algorithm, based on shortterm hierarchical segmentation (STHS) and frame-by-frame Markov random field (MRF) optimization, is proposed in this work. We develop the STHS technique, which generates initial segments by sliding a short window of frames. In STHS, we apply spatial agglomerative clustering to each frame, and then adopt inter-frame bipartite graph matching to construct initial segments. Then, we partition each frame into final segments, by minimizing an MRF energy function composed of unary and pair wise costs. We compute the unary cost using the STHS initial segments and the segmentation result at the previous frame. We set the pair wise cost to encourage similar nodes to have the same segment label. Experimental results on a video segmentation benchmark dataset, VSB100, demonstrate that the proposed algorithm outperforms state-of-the-art online video segmentation techniques significantly.",
keywords = "Agglomerative clustering, Graph matching, Online segmentation, Streaming segmentation, Video segmentation",
author = "Jang, {Won Dong} and Chang-Su Kim",
year = "2016",
doi = "10.1007/978-3-319-46466-4_36",
language = "English",
isbn = "9783319464657",
volume = "9910 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "599--615",
booktitle = "Computer Vision - 14th European Conference, ECCV 2016, Proceedings",

}

TY - GEN

T1 - Streaming video segmentation via short-term hierarchical segmentation and frame-by-frame Markov random field optimization

AU - Jang, Won Dong

AU - Kim, Chang-Su

PY - 2016

Y1 - 2016

N2 - An online video segmentation algorithm, based on shortterm hierarchical segmentation (STHS) and frame-by-frame Markov random field (MRF) optimization, is proposed in this work. We develop the STHS technique, which generates initial segments by sliding a short window of frames. In STHS, we apply spatial agglomerative clustering to each frame, and then adopt inter-frame bipartite graph matching to construct initial segments. Then, we partition each frame into final segments, by minimizing an MRF energy function composed of unary and pair wise costs. We compute the unary cost using the STHS initial segments and the segmentation result at the previous frame. We set the pair wise cost to encourage similar nodes to have the same segment label. Experimental results on a video segmentation benchmark dataset, VSB100, demonstrate that the proposed algorithm outperforms state-of-the-art online video segmentation techniques significantly.

AB - An online video segmentation algorithm, based on shortterm hierarchical segmentation (STHS) and frame-by-frame Markov random field (MRF) optimization, is proposed in this work. We develop the STHS technique, which generates initial segments by sliding a short window of frames. In STHS, we apply spatial agglomerative clustering to each frame, and then adopt inter-frame bipartite graph matching to construct initial segments. Then, we partition each frame into final segments, by minimizing an MRF energy function composed of unary and pair wise costs. We compute the unary cost using the STHS initial segments and the segmentation result at the previous frame. We set the pair wise cost to encourage similar nodes to have the same segment label. Experimental results on a video segmentation benchmark dataset, VSB100, demonstrate that the proposed algorithm outperforms state-of-the-art online video segmentation techniques significantly.

KW - Agglomerative clustering

KW - Graph matching

KW - Online segmentation

KW - Streaming segmentation

KW - Video segmentation

UR - http://www.scopus.com/inward/record.url?scp=84990062971&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84990062971&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-46466-4_36

DO - 10.1007/978-3-319-46466-4_36

M3 - Conference contribution

AN - SCOPUS:84990062971

SN - 9783319464657

VL - 9910 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 599

EP - 615

BT - Computer Vision - 14th European Conference, ECCV 2016, Proceedings

PB - Springer Verlag

ER -