Enhancing a laparoscopy training system with augmented reality visualization

Hao Jiang, Siqing Xu, Andrei State, Fan Feng, Henry Fuchs, Minsik Hong, Jerzy Rozenblit

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We report work in progress towards a system for laparoscopy training that is enhanced with an augmented reality display. The system can be used to perform peg-transfer training tasks using a Microsoft Hololens AR device, with full 6 degrees of freedom (DoF) shared presence with the pegboard workspace. This mode is in contrast to the conventional visualization, whose views are solely through a 2D or stereo laparoscope. In order to achieve this enhanced visualization, the system extracts the pose of the triangular prism being manipulated, and of the laparoscopic instruments with which the user manipulates the prism. These two objects are added to a CAD model of a (fixed) pegboard, and all the objects are visualized within the AR display. In the near future we expect to conduct user studies measuring accuracy and time-to-completion of this peg-transfer task using these new enhancements and compare it to conventional visualization techniques. We report our initial results of enhancing this trainer with a head-mounted AR display, with which the surgeon can visualize the laparoscopic scene and instruments from arbitrary viewpoints while naturally moving around the (internal) surgical scene. We expect that this capability, combined with the haptics feedback guidance, will provide more natural visualization, comprehension, and thus yield improved task performance.

Original languageEnglish (US)
Title of host publicationSimulation Series
PublisherThe Society for Modeling and Simulation International
Edition5
ISBN (Electronic)9781510892521, 9781510892538, 9781510892545, 9781510892552, 9781510892569
DOIs
StatePublished - Jan 1 2019
Event2019 Modeling and Simulation in Medicine, MSM 2019, Part of the 2019 Spring Simulation Multi-Conference, SpringSim 2019 - Tucson, United States
Duration: Apr 29 2019May 2 2019

Publication series

NameSimulation Series
Number5
Volume51
ISSN (Print)0735-9276

Conference

Conference2019 Modeling and Simulation in Medicine, MSM 2019, Part of the 2019 Spring Simulation Multi-Conference, SpringSim 2019
CountryUnited States
CityTucson
Period4/29/195/2/19

Fingerprint

Laparoscopy
Augmented reality
Visualization
Display devices
Prisms
Computer aided design
Feedback

Keywords

  • Augmented reality
  • Laparoscopic
  • Visualization

ASJC Scopus subject areas

  • Computer Networks and Communications

Cite this

Jiang, H., Xu, S., State, A., Feng, F., Fuchs, H., Hong, M., & Rozenblit, J. (2019). Enhancing a laparoscopy training system with augmented reality visualization. In Simulation Series (5 ed.). (Simulation Series; Vol. 51, No. 5). The Society for Modeling and Simulation International. https://doi.org/10.23919/SpringSim.2019.8732876

Enhancing a laparoscopy training system with augmented reality visualization. / Jiang, Hao; Xu, Siqing; State, Andrei; Feng, Fan; Fuchs, Henry; Hong, Minsik; Rozenblit, Jerzy.

Simulation Series. 5. ed. The Society for Modeling and Simulation International, 2019. (Simulation Series; Vol. 51, No. 5).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Jiang, H, Xu, S, State, A, Feng, F, Fuchs, H, Hong, M & Rozenblit, J 2019, Enhancing a laparoscopy training system with augmented reality visualization. in Simulation Series. 5 edn, Simulation Series, no. 5, vol. 51, The Society for Modeling and Simulation International, 2019 Modeling and Simulation in Medicine, MSM 2019, Part of the 2019 Spring Simulation Multi-Conference, SpringSim 2019, Tucson, United States, 4/29/19. https://doi.org/10.23919/SpringSim.2019.8732876
Jiang H, Xu S, State A, Feng F, Fuchs H, Hong M et al. Enhancing a laparoscopy training system with augmented reality visualization. In Simulation Series. 5 ed. The Society for Modeling and Simulation International. 2019. (Simulation Series; 5). https://doi.org/10.23919/SpringSim.2019.8732876
Jiang, Hao ; Xu, Siqing ; State, Andrei ; Feng, Fan ; Fuchs, Henry ; Hong, Minsik ; Rozenblit, Jerzy. / Enhancing a laparoscopy training system with augmented reality visualization. Simulation Series. 5. ed. The Society for Modeling and Simulation International, 2019. (Simulation Series; 5).
@inproceedings{766e2cffae5e4bc89d19d553658db101,
title = "Enhancing a laparoscopy training system with augmented reality visualization",
abstract = "We report work in progress towards a system for laparoscopy training that is enhanced with an augmented reality display. The system can be used to perform peg-transfer training tasks using a Microsoft Hololens AR device, with full 6 degrees of freedom (DoF) shared presence with the pegboard workspace. This mode is in contrast to the conventional visualization, whose views are solely through a 2D or stereo laparoscope. In order to achieve this enhanced visualization, the system extracts the pose of the triangular prism being manipulated, and of the laparoscopic instruments with which the user manipulates the prism. These two objects are added to a CAD model of a (fixed) pegboard, and all the objects are visualized within the AR display. In the near future we expect to conduct user studies measuring accuracy and time-to-completion of this peg-transfer task using these new enhancements and compare it to conventional visualization techniques. We report our initial results of enhancing this trainer with a head-mounted AR display, with which the surgeon can visualize the laparoscopic scene and instruments from arbitrary viewpoints while naturally moving around the (internal) surgical scene. We expect that this capability, combined with the haptics feedback guidance, will provide more natural visualization, comprehension, and thus yield improved task performance.",
keywords = "Augmented reality, Laparoscopic, Visualization",
author = "Hao Jiang and Siqing Xu and Andrei State and Fan Feng and Henry Fuchs and Minsik Hong and Jerzy Rozenblit",
year = "2019",
month = "1",
day = "1",
doi = "10.23919/SpringSim.2019.8732876",
language = "English (US)",
series = "Simulation Series",
publisher = "The Society for Modeling and Simulation International",
number = "5",
booktitle = "Simulation Series",
edition = "5",

}

TY - GEN

T1 - Enhancing a laparoscopy training system with augmented reality visualization

AU - Jiang, Hao

AU - Xu, Siqing

AU - State, Andrei

AU - Feng, Fan

AU - Fuchs, Henry

AU - Hong, Minsik

AU - Rozenblit, Jerzy

PY - 2019/1/1

Y1 - 2019/1/1

N2 - We report work in progress towards a system for laparoscopy training that is enhanced with an augmented reality display. The system can be used to perform peg-transfer training tasks using a Microsoft Hololens AR device, with full 6 degrees of freedom (DoF) shared presence with the pegboard workspace. This mode is in contrast to the conventional visualization, whose views are solely through a 2D or stereo laparoscope. In order to achieve this enhanced visualization, the system extracts the pose of the triangular prism being manipulated, and of the laparoscopic instruments with which the user manipulates the prism. These two objects are added to a CAD model of a (fixed) pegboard, and all the objects are visualized within the AR display. In the near future we expect to conduct user studies measuring accuracy and time-to-completion of this peg-transfer task using these new enhancements and compare it to conventional visualization techniques. We report our initial results of enhancing this trainer with a head-mounted AR display, with which the surgeon can visualize the laparoscopic scene and instruments from arbitrary viewpoints while naturally moving around the (internal) surgical scene. We expect that this capability, combined with the haptics feedback guidance, will provide more natural visualization, comprehension, and thus yield improved task performance.

AB - We report work in progress towards a system for laparoscopy training that is enhanced with an augmented reality display. The system can be used to perform peg-transfer training tasks using a Microsoft Hololens AR device, with full 6 degrees of freedom (DoF) shared presence with the pegboard workspace. This mode is in contrast to the conventional visualization, whose views are solely through a 2D or stereo laparoscope. In order to achieve this enhanced visualization, the system extracts the pose of the triangular prism being manipulated, and of the laparoscopic instruments with which the user manipulates the prism. These two objects are added to a CAD model of a (fixed) pegboard, and all the objects are visualized within the AR display. In the near future we expect to conduct user studies measuring accuracy and time-to-completion of this peg-transfer task using these new enhancements and compare it to conventional visualization techniques. We report our initial results of enhancing this trainer with a head-mounted AR display, with which the surgeon can visualize the laparoscopic scene and instruments from arbitrary viewpoints while naturally moving around the (internal) surgical scene. We expect that this capability, combined with the haptics feedback guidance, will provide more natural visualization, comprehension, and thus yield improved task performance.

KW - Augmented reality

KW - Laparoscopic

KW - Visualization

UR - http://www.scopus.com/inward/record.url?scp=85073671839&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85073671839&partnerID=8YFLogxK

U2 - 10.23919/SpringSim.2019.8732876

DO - 10.23919/SpringSim.2019.8732876

M3 - Conference contribution

AN - SCOPUS:85068609848

T3 - Simulation Series

BT - Simulation Series

PB - The Society for Modeling and Simulation International

ER -