Skip to main content

Research Repository

Advanced Search

Towards Graph Representation Learning Based Surgical Workflow Anticipation

Zhang, Xiatian; Al Moubayed, Noura; Shum, Hubert P.H.

Towards Graph Representation Learning Based Surgical Workflow Anticipation Thumbnail



Surgical workflow anticipation can give predictions on what steps to conduct or what instruments to use next, which is an essential part of the computer-assisted intervention system for surgery, e.g. workflow reasoning in robotic surgery. However, current approaches are limited to their insufficient expressive power for relationships between instruments. Hence, we propose a graph representation learning framework to comprehensively represent instrument motions in the surgical workflow anticipation problem. In our proposed graph representation, we maps the bounding box information of instruments to the graph nodes in the consecutive frames and build inter-frame/inter-instrument graph edges to represent the trajectory and interaction of the instruments over time. This design enhances the ability of our network on modeling both the spatial and temporal patterns of surgical instruments and their interactions. In addition, we design a multi-horizon learning strategy to balance the understanding of various horizons indifferent anticipation tasks, which significantly improves the model performance in anticipation with various horizons. Experiments on the Cholec80 dataset demonstrate the performance of our proposed method can exceed the state-of-the-art method based on richer backbones, especially in instrument anticipation (1.27 v.s. 1.48 for inMAE; 1.48 v.s. 2.68 for eMAE). To the best of our knowledge, we are the first to introduce a spatial-temporal graph representation into surgical workflow anticipation.


Zhang, X., Al Moubayed, N., & Shum, H. P. (2022). Towards Graph Representation Learning Based Surgical Workflow Anticipation. .

Conference Name 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI)
Conference Location Ioannina, Greece
Start Date Sep 27, 2022
End Date Sep 30, 2022
Acceptance Date Jul 19, 2022
Online Publication Date Nov 4, 2022
Publication Date 2022
Deposit Date Aug 9, 2022
Publicly Available Date Oct 1, 2022
Series ISSN 2641-3590,2641-3604


Accepted Conference Proceeding (1.3 Mb)

Copyright Statement
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

You might also like

Downloadable Citations