Real-Time GPU Based Video Segmentation with Depth Information

Nilangshu Bidyanta, Ali Akoglu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In the context of video segmentation with depth sensor, prior work maps the Metropolis algorithm, a simulated annealing based key routine during segmentation, onto an Nvidia Graphics Processing Unit (GPU) and achieves real-time performance for 320×256 video sequences. However that work utilizes depth information in a very limited manner. This paper presents a new GPU-based method that expands the use of depth information during segmentation and shows the improved segmentation quality over the prior work. In particular, we discuss various ways to restructure the segmentation flow, and evaluate the impact of several design choices on throughput and quality. We introduce a scaling factor for amplifying the interaction strength between two spatially neighboring pixels and increasing the clarity of borderlines. This allows us to reduce the number of required Metropolis iterations by over 50% with the drawback of over-segmentation. We evaluate two design choices to overcome this problem. First, we incorporate depth information into the perceived color difference calculations between two pixels, and show that the interaction strengths between neighboring pixels can be more accurately modeled by incorporating depth information. Second, we pre-process the frames with Bilateral filter instead of Gaussian filter, and show its effectiveness in terms of reducing the difference between similar colors. Both approaches help improve the quality of the segmentation, and the reduction in Metropolis iterations helps improve the throughout from 29 fps to 34 fps for 320×256 video sequences.

Original languageEnglish (US)
Title of host publication2018 IEEE/ACS 15th International Conference on Computer Systems and Applications, AICCSA 2018
PublisherIEEE Computer Society
ISBN (Electronic)9781538691205
DOIs
StatePublished - Jan 14 2019
Event15th IEEE/ACS International Conference on Computer Systems and Applications, AICCSA 2018 - Aqaba, Jordan
Duration: Oct 28 2018Nov 1 2018

Publication series

NameProceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA
Volume2018-November
ISSN (Print)2161-5322
ISSN (Electronic)2161-5330

Conference

Conference15th IEEE/ACS International Conference on Computer Systems and Applications, AICCSA 2018
CountryJordan
CityAqaba
Period10/28/1811/1/18

Fingerprint

Pixels
Color
Simulated annealing
Throughput
Sensors
Graphics processing unit

Keywords

  • CUDA
  • depth
  • GPU
  • kinect
  • video segmentation

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Signal Processing
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Cite this

Bidyanta, N., & Akoglu, A. (2019). Real-Time GPU Based Video Segmentation with Depth Information. In 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications, AICCSA 2018 [8612854] (Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA; Vol. 2018-November). IEEE Computer Society. https://doi.org/10.1109/AICCSA.2018.8612854

Real-Time GPU Based Video Segmentation with Depth Information. / Bidyanta, Nilangshu; Akoglu, Ali.

2018 IEEE/ACS 15th International Conference on Computer Systems and Applications, AICCSA 2018. IEEE Computer Society, 2019. 8612854 (Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA; Vol. 2018-November).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Bidyanta, N & Akoglu, A 2019, Real-Time GPU Based Video Segmentation with Depth Information. in 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications, AICCSA 2018., 8612854, Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA, vol. 2018-November, IEEE Computer Society, 15th IEEE/ACS International Conference on Computer Systems and Applications, AICCSA 2018, Aqaba, Jordan, 10/28/18. https://doi.org/10.1109/AICCSA.2018.8612854
Bidyanta N, Akoglu A. Real-Time GPU Based Video Segmentation with Depth Information. In 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications, AICCSA 2018. IEEE Computer Society. 2019. 8612854. (Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA). https://doi.org/10.1109/AICCSA.2018.8612854
Bidyanta, Nilangshu ; Akoglu, Ali. / Real-Time GPU Based Video Segmentation with Depth Information. 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications, AICCSA 2018. IEEE Computer Society, 2019. (Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA).
@inproceedings{e9d42a5d9cf640689a712f412a0df76d,
title = "Real-Time GPU Based Video Segmentation with Depth Information",
abstract = "In the context of video segmentation with depth sensor, prior work maps the Metropolis algorithm, a simulated annealing based key routine during segmentation, onto an Nvidia Graphics Processing Unit (GPU) and achieves real-time performance for 320×256 video sequences. However that work utilizes depth information in a very limited manner. This paper presents a new GPU-based method that expands the use of depth information during segmentation and shows the improved segmentation quality over the prior work. In particular, we discuss various ways to restructure the segmentation flow, and evaluate the impact of several design choices on throughput and quality. We introduce a scaling factor for amplifying the interaction strength between two spatially neighboring pixels and increasing the clarity of borderlines. This allows us to reduce the number of required Metropolis iterations by over 50{\%} with the drawback of over-segmentation. We evaluate two design choices to overcome this problem. First, we incorporate depth information into the perceived color difference calculations between two pixels, and show that the interaction strengths between neighboring pixels can be more accurately modeled by incorporating depth information. Second, we pre-process the frames with Bilateral filter instead of Gaussian filter, and show its effectiveness in terms of reducing the difference between similar colors. Both approaches help improve the quality of the segmentation, and the reduction in Metropolis iterations helps improve the throughout from 29 fps to 34 fps for 320×256 video sequences.",
keywords = "CUDA, depth, GPU, kinect, video segmentation",
author = "Nilangshu Bidyanta and Ali Akoglu",
year = "2019",
month = "1",
day = "14",
doi = "10.1109/AICCSA.2018.8612854",
language = "English (US)",
series = "Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA",
publisher = "IEEE Computer Society",
booktitle = "2018 IEEE/ACS 15th International Conference on Computer Systems and Applications, AICCSA 2018",

}

TY - GEN

T1 - Real-Time GPU Based Video Segmentation with Depth Information

AU - Bidyanta, Nilangshu

AU - Akoglu, Ali

PY - 2019/1/14

Y1 - 2019/1/14

N2 - In the context of video segmentation with depth sensor, prior work maps the Metropolis algorithm, a simulated annealing based key routine during segmentation, onto an Nvidia Graphics Processing Unit (GPU) and achieves real-time performance for 320×256 video sequences. However that work utilizes depth information in a very limited manner. This paper presents a new GPU-based method that expands the use of depth information during segmentation and shows the improved segmentation quality over the prior work. In particular, we discuss various ways to restructure the segmentation flow, and evaluate the impact of several design choices on throughput and quality. We introduce a scaling factor for amplifying the interaction strength between two spatially neighboring pixels and increasing the clarity of borderlines. This allows us to reduce the number of required Metropolis iterations by over 50% with the drawback of over-segmentation. We evaluate two design choices to overcome this problem. First, we incorporate depth information into the perceived color difference calculations between two pixels, and show that the interaction strengths between neighboring pixels can be more accurately modeled by incorporating depth information. Second, we pre-process the frames with Bilateral filter instead of Gaussian filter, and show its effectiveness in terms of reducing the difference between similar colors. Both approaches help improve the quality of the segmentation, and the reduction in Metropolis iterations helps improve the throughout from 29 fps to 34 fps for 320×256 video sequences.

AB - In the context of video segmentation with depth sensor, prior work maps the Metropolis algorithm, a simulated annealing based key routine during segmentation, onto an Nvidia Graphics Processing Unit (GPU) and achieves real-time performance for 320×256 video sequences. However that work utilizes depth information in a very limited manner. This paper presents a new GPU-based method that expands the use of depth information during segmentation and shows the improved segmentation quality over the prior work. In particular, we discuss various ways to restructure the segmentation flow, and evaluate the impact of several design choices on throughput and quality. We introduce a scaling factor for amplifying the interaction strength between two spatially neighboring pixels and increasing the clarity of borderlines. This allows us to reduce the number of required Metropolis iterations by over 50% with the drawback of over-segmentation. We evaluate two design choices to overcome this problem. First, we incorporate depth information into the perceived color difference calculations between two pixels, and show that the interaction strengths between neighboring pixels can be more accurately modeled by incorporating depth information. Second, we pre-process the frames with Bilateral filter instead of Gaussian filter, and show its effectiveness in terms of reducing the difference between similar colors. Both approaches help improve the quality of the segmentation, and the reduction in Metropolis iterations helps improve the throughout from 29 fps to 34 fps for 320×256 video sequences.

KW - CUDA

KW - depth

KW - GPU

KW - kinect

KW - video segmentation

UR - http://www.scopus.com/inward/record.url?scp=85061901656&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061901656&partnerID=8YFLogxK

U2 - 10.1109/AICCSA.2018.8612854

DO - 10.1109/AICCSA.2018.8612854

M3 - Conference contribution

T3 - Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA

BT - 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications, AICCSA 2018

PB - IEEE Computer Society

ER -